Training method, training system and readable medium of neural network model

Abstract

The present disclosure provides a training method of a neural network model, a training system and a readable medium. The training method comprises: using a transpose-invariant sparse weight matrix to calculate the activation of the neural network model by a sparse matrix-matrix multiplication module in forward propagation; using the transpose of the sparse weight matrix received from a weight transpose module to calculate the activation gradient of the neural network model by the sparse matrix-matrix multiplication module in backward propagation; and using the activation received from the sparse matrix-matrix multiplication module in forward propagation to calculate the weight gradient of the neural network model by a sampling dense-dense matrix multiplication module in backward propagation. The present disclosure uses a transpose-invariant sparse weight matrix to train the neural network model by non-zero calculation, which advantageously eliminates the redundant calculation of the sparse matrix-matrix multiplication module and the sampling dense-dense matrix multiplication module, and reduces the training time and memory consumption of the neural network model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of neural network, and in particular, to a neural network model training method, a training system and a readable medium. BACKGROUND

[0002] In artificial intelligence, artificial neural networks (ANN), also commonly referred to as neural networks (NN), can enable machines to learn. Through practical application of various mathematical functions, neural network models can learn various relationships from a dataset, thereby improving computer performance compared to traditional software routines. For example, a neural network can learn a mapping function from input to output by updating weights of a neural network model in response to an error produced by the neural network model on a training dataset. The updating is iteratively performed to reduce the error until the model reaches a desired level of generalization performance. Thereafter, the neural network can be used to reason an output from an input.

[0003] Reference is now made to Figure 1A , Figure 1A A neural network training method according to conventional techniques is shown. The method can include inputting a training dataset into a neural network model to produce an output in a forward pass. The output is compared to a target. A difference between the output and the target can be utilized in a backward pass to adjust weights of the neural network model. The training process of the neural network model is iteratively repeated until the output reaches a desired accuracy with respect to the target. Thereafter, as shown, the trained neural network model can be used to reason an output from an input. Figure 1B

[0004] In many applications, the size of neural network models and the time required to train neural network models is continually increasing. Accordingly, there is a continuing need for improved training systems and methods for neural network (NN) models. SUMMARY

[0005] The present disclosure can best be understood by reference to the following description and drawings, which illustrate embodiments of the training systems and methods for neural network (NN) models of the present disclosure.

[0006] ​In one embodiment, a method of training a neural network model can include computing activations in a forward pass through a sparse matrix-matrix multiplication (spMM) module using a transpose invariant sparse weight matrix. The method can also include computing activation gradients in a backward pass through the sparse matrix-matrix multiplication (spMM) module using a transpose of the sparse weight matrix received from a weight transpose module. The method can further include computing weight gradients of the neural network (NN) model in the backward pass through a sampled dense-dense matrix multiplication (SDDMM) module using activations received from the sparse matrix-matrix multiplication (spMM) module in the forward pass. The activation computation in the forward pass can further include computing, in response to an input dataset, activations of a current layer through sparse matrix-matrix multiplication (spMM) based on activations of a previous layer, sparse weight data of the sparse weight matrix of the current layer, and sparse weight indices of the sparse weight matrix of the current layer. The activation gradient computation in the backward pass can further include computing, based on a transpose of the sparse weight indices of the current layer, a transpose of the sparse weight data of the current layer, and activation gradients of the current layer, activation gradients of the previous layer through sparse matrix-matrix multiplication (spMM). The weight gradient computation in the backward pass can further include computing, based on the activations of the previous layer, the sparse weight indices of the current layer, and the activation gradients of the current layer, weight gradients of the current layer through sampled dense-dense matrix multiplication (SDDMM).

[0007] In one embodiment, a training system of a neural network (NN) model can include a multiplication module, a weight data transpose module, a weight index transpose module, and a weight update module. The multiplication module can include one or more sparse matrix-matrix multiplication (spMM) modules and one or more sampled dense-dense matrix multiplication (SDDMM) modules. The one or more sparse matrix-matrix multiplication (spMM) modules can be configured to compute, in a forward pass of a current batch dataset, an activation of a current layer based on an activation of a previous layer, sparse weight data of the current layer, and a sparse weight index of the current layer, and to compute, in a backward pass, an activation gradient of the previous layer based on transposed sparse weight data of the current layer, a transposed sparse weight index of the current layer, and the activation. The one or more sampled dense-dense matrix multiplication (SDDMM) modules can be configured to compute, in the backward pass, a weight gradient of the current layer based on the activation gradient of the current layer, the sparse weight index of the current layer, and the activation of the previous layer. The weight update module can be configured to compute new sparse weights based on the sparse weight data of the current layer and the weight gradient of the current layer.

[0008] In one embodiment, a method of training a neural network model can include computing, in a forward pass, an activation using a transpose-invariant sparse weight matrix. The method can further include computing, in a backward pass, an activation gradient and a weight gradient using the sparse weight matrix. The step of computing, in the forward pass, the activation can include computing, in response to an input dataset, the activation of a current layer by a sparse matrix-matrix multiplication (spMM) based on an activation of a previous layer, sparse weight data of a sparse weight matrix of the current layer, and a sparse weight index of the sparse weight matrix of the current layer. The step of computing, in the backward pass, the activation gradient can include computing, by a sparse matrix-matrix multiplication (spMM), an activation gradient of the previous layer based on a transpose of the sparse weight index of the current layer, a transpose of the sparse weight data of the current layer, and the activation gradient of the current layer. The step of computing, in the backward pass, the weight gradient can include computing, by a sampled dense-dense matrix multiplication (SDDMM), a weight gradient of the current layer based on the activation of the previous layer, the sparse weight index of the current layer, and the activation gradient of the current layer.

[0009] The disclosed embodiments advantageously eliminate redundant computations of sparse matrix-matrix multiplication (spMM) modules and sampled dense-dense matrix multiplication (SDDMM) modules using transpose-invariant sparse weight matrices for non-zero computations in training a neural network model, and thus can reduce training time and memory consumption of the neural network model.

[0010] In order to introduce the selection of concepts in a simplified form, the summary section is provided, which further describes the concepts in the detailed description section below. The summary section is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0011] Embodiments of the disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0012] Figure 1A A training method of a neural network according to a conventional technique is shown;

[0013] Figure 1B An inference method using a trained neural network according to a conventional technique is shown;

[0014] Figure 2 A neural network (NN) training system according to embodiments of the disclosure is shown;

[0015] Figure 3A An exemplary transpose invariant sparse weight matrix is shown;

[0016] Figure 3B An exemplary transpose invariant sparse weight matrix is shown;

[0017] Figure 4 A training system of a neural network (NN) model according to embodiments of the disclosure is shown;

[0018] Figure 5 A training method of a neural network according to embodiments of the disclosure is shown;

[0019] Figure 6 A training method of a neural network according to embodiments of the disclosure is shown;

[0020] Figure 7 A training method of a neural network according to embodiments of the disclosure is shown;

[0021] Figure 8 A block diagram of an exemplary processing unit including embodiments for implementing neural network (NN) model training according to embodiments of the disclosure is shown. DETAILED DESCRIPTION

[0022] Reference will now be made in detail to various embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. While the present disclosure will be described in conjunction with these embodiments, it is understood that they are not intended to limit the present disclosure to these embodiments. On the contrary, the present disclosure is intended to cover alternatives, modifications, and equivalents, which can be included within the scope of the present disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be recognized that the present disclosure can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present disclosure.

[0023] Embodiments of the present disclosure below are presented in terms of routines, modules, logic blocks, and other symbolic representations of operations on data within one or more electronic devices. The procedures and descriptions are the means used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. Here, a routine, module, logic block, and / or similar term is understood to be a self-consistent sequence of operations or instructions leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, compared, and otherwise manipulated in an electronic device. For convenience, and in reference to general use, these signals are referred to as data, bits, values, elements, symbols, characters, terms, numbers, strings, and / or the like in accordance with embodiments of the present disclosure.

[0024] It should be kept in mind, however, that these terms should be interpreted as referring to physical manipulations and quantities and are merely convenient labels and should not be too limiting. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the present disclosure, discussions using terms such as “receiving” and / or the like, refer to the actions and processes of an electronic device, such as an electronic computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the electronic device’s circuits, registers, memories, and / or the like.

[0025] In this application, features described individually can also mean that these features can be combined with each other. The definite article or indefinite article is not intended to indicate the cardinal number. In particular, the reference to "the" object or "one" object is also intended to mean that it can be one of a plurality of such objects. The words "include", "contain", "have" and the like are used to specify the existence of the declared elements, but not to exclude the presence or addition of one or more other elements and combinations thereof. It should also be understood that although the words "first", "second" and the like can be used herein to describe various elements, these elements should not be limited by these words. These words are used in this disclosure to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can also be referred to as a first element, without going beyond the scope of the embodiments of the disclosure. It should also be understood that when an element is described as "coupled" to another element, it can be directly or indirectly connected to the other element, or there can be an intermediate element between the two. In contrast, when an element is described as "directly connected" to another element, there is no intermediate element between the two. It should also be understood that the word "and / or" includes any and all combinations of one or more associated elements. It should also be understood that the phrases and terms used in this disclosure are intended to describe the disclosure, and should not be regarded as limiting the disclosure.

[0026] Reference Figure 2 , Figure 2 A neural network (NN) training system is shown in accordance with aspects of the disclosure. The neural network (NN) training system 200 can include a matrix multiplication module 210, a weight data transpose module 220, a weight index transpose module 230, a non-multiplication module 240, and a memory 250. The matrix multiplication module 210 can include one or more sparse matrix-matrix multiplication (spMM) modules 260 and one or more sampled dense-dense matrix multiplication (SDDMM) modules 270. In one embodiment, the matrix multiplication module 210 can be configured to compute sparse matrix-matrix multiplication (spMM) in accordance with Equation 1, which is:

[0027] C = A x B (1)

[0028] where A and B are matrices. Each row i of C can be computed in accordance with Equation 2, which is:

[0029]

[0030] In one embodiment, the matrix multiplication module 210 can be configured to compute sampled dense-dense matrix multiplication (SDDMM) in accordance with Equation 3, which is:

[0031] F(D *s ET )oS (3)

[0032] wherein, and are dense matrices, is a sample sparse matrix. These modules can be implemented in software, firmware, hardware, or any combination thereof. In one embodiment, these modules can be implemented as computing device executable instructions (e.g., software) stored in a computing device readable medium (e.g., computer memory) and executed by one or more computing devices (e.g., processing units). In one example embodiment, a sparse matrix-matrix multiplication (spMM) can be performed by computing device executable instructions that are:

[0033]

[0034] In one example embodiment, a sample dense-dense matrix multiplication (SDDMM) can be performed by computing device executable instructions that are:

[0035]

[0036] One or more sparse matrix-matrix multiplication (spMM) modules 260 can be configured to compute activations using transpose invariant sparse weight matrices in a forward pass during training of a neural network (NN) model. One or more sparse matrix-matrix multiplication (spMM) modules 260 can also be configured to compute activation gradients using sparse weight matrices in a backward pass during training of a neural network (NN) model. One or more sample dense-dense matrix multiplication (SDDMM) modules 270 can be configured to compute weight gradients using sparse weight matrices in a backward pass during training of a neural network (NN) model.

[0037] A sparse matrix is a matrix in which a large number of the elements have a value of zero. A dense matrix is generally considered to be a matrix in which a majority of the elements have a value of non-zero. The sparsity of a matrix is generally considered to be the ratio of the number of zero-valued elements to the total number of elements in the matrix. For example, if half of the elements of a matrix have a value of zero and half have a value of non-zero, then the sparsity of the matrix is 50%. For sparse matrices, the amount of memory used for storage can be reduced by storing only the non-zero element values. The compressed format of a sparse matrix can also reduce computational effort by eliminating calculations involving zero-valued matrix elements. There are many data structures for storing sparse matrices in compressed format, including but not limited to dictionary of keys (DOK), list of lists (LIL), coordinate list (COO), compressed sparse row (CSR), compressed sparse column (CSC), and others. A CSR data structure represents a sparse matrix with three arrays: a row pointer array, a column index array, and a value array. The value array includes the non-zero values. The column index array indicates the column in which a non-zero value in a given row is located. The row pointer array indicates the starting position of the non-zero values for a corresponding row in the value array. Similarly, a CSC data structure can represent a sparse matrix with a column pointer array, a row index array, and a value array. Generally, compressed format matrix data structures, such as CSR and CSC, reduce the amount of storage consumed by the matrix. Compressed format matrix data structures, such as CSR or CSC, can also reduce computational effort by eliminating calculations involving zero-valued matrix elements.

[0038] The transpose of a matrix is an operator that flips the matrix over the diagonal. When a matrix is transposed, the rows and columns are switched, which can be performed by switching the row and column indices of the matrix. In one example embodiment, the transpose of a matrix can be performed by a computing device executable instruction that is:

[0039]

[0040] Referring now to Figure 3A , Figure 3A An example transpose invariant sparse weight matrix and its transpose are shown in accordance with aspects of the present disclosure. For example, a 4x4 matrix can be a sparse weight matrix with a sparsity of 50% in which two elements in each row 305 are non-zero values (e.g., 310 and 320) and two elements in the same row are zero values (e.g., 315, 325). The 4x4 matrix can be a transpose invariant sparse matrix with a sparsity of 50% when the transposed rows 350 also include two non-zero values (e.g., 310, 330) and two zero values (e.g., 340, 345). As shown in Figure 3BAs shown, when the sparsity of one or more rows 355 of the transposed 4x4 matrix is not 50%, the sparse matrix is not transpose invariant. For example, if the original matrix is a sparse matrix with a sparsity of 50% having two non-zero elements for every four elements in the selected row 360, the transposed transform matrix one or more rows can include more or less than two non-zero elements for every four elements in the corresponding row 355.

[0041] For ease of explanation, Figure 3A and 3B A 4x4 matrix is shown, but it can be appreciated that matrices of much larger size are commonly used in neural network (NN) processing. Further, it can be appreciated that, for ease of processing, matrices of large size can be divided into windows, tiles, sections, etc. For example, multiple 4x4 element windows of a matrix can be scheduled for processing by respective threads of a software-based neural network (NN) processor, or multiple 4x4 element windows of a matrix can be scheduled to respective hardware accelerators of a neural network (NN) processor.

[0042] Referring again to Figure 2 , one or more sparse matrix-matrix multiplication modules 260 of the matrix multiplication module 210 can compute an activation of a current layer based on an activation of a previous layer, weight data of a sparse weight matrix of the current layer, and weight indices of the sparse weight matrix of the current layer. The one or more sparse matrix-matrix multiplication (spMM) modules 260 can compute the activation of the current layer in a forward pass in response to a training dataset input. The weight index transpose module 230 can be configured to transpose the sparse weight indices of the current layer. Similarly, the weight data transpose module 220 can be configured to transpose the sparse weight data of the current layer. The one or more sparse matrix-matrix multiplication (spMM) modules 260 can compute an activation gradient of the previous layer in a backward pass based on the transposed sparse weight indices of the current layer, the transposed sparse weight data of the current layer, and the activation of the current layer. The one or more sample dense-dense matrix multiplication (SDDMM) modules 270 of the matrix multiplication module 210 can compute a weight gradient of the current layer in the backward pass based on the activation of the previous layer, the sparse weight indices of the current layer, and the activation gradient of the current layer.

[0043] The sparse matrix-matrix multiplication (spMM) module 260, the sampled dense-dense matrix multiplication (SDDMM) module 270, the weight data transpose module 220, and the weight index transpose module 230 can iteratively perform the above functions for each of the plurality of training data sets. Further, the non-multiplication operation module 240 can be configured to provide non-multiplication operation support to the sparse matrix-matrix multiplication (spMM) module 230, the sampled dense-dense matrix multiplication (SDDMM) module 270, the weight data transpose module 220, and the weight index transpose module 230. In one embodiment, the non-multiplication operation module 240 can add together the weight gradient of the current layer and the sparse weight data of the current layer to generate the sparse weight data for the next iteration. In one example embodiment, the addition of the weight gradient of the current layer and the sparse weight data of the current layer can be performed by computer-executable instructions, which are:

[0044]

[0045]

[0046] Further, the memory 250 can store training data sets, activations, activation gradients, sparse weight matrices, weight indices, weight data, transposed weight matrices, transposed weight indices, transposed weight data, weight gradients, etc., for use by the sparse matrix-matrix multiplication (spMM) module 260, the sampled dense-dense matrix multiplication (SDDMM) module 270, the weight data transpose module 220, the weight index transpose module 230, and / or the non-multiplication operation module 240. Although illustrated as a single block, the memory 250 can include one or more types of memory arranged in one or more hierarchical levels. Further, although the sparse matrix-matrix multiplication (spMM) module 260 and the sampled dense-dense matrix multiplication (SDDMM) module 270 are illustrated as separate modules, it should be understood that the sparse matrix-matrix multiplication (spMM) module 260 can be a subset of the sampled dense-dense matrix multiplication (SDDMM) module 270. For example, the sampled dense-dense matrix multiplication module shares most of the functions of the sparse matrix-matrix multiplication (spMM) module 260, and thus the sampled dense-dense matrix multiplication module can be integrated therein.

[0047] It should be understood that the calculation of activations in the forward pass is typically first performed for each of the plurality of layers of the neural network (NN) model. The calculation of activation gradients and weight gradients in the backward pass can then be performed for each of the plurality of layers of the neural network (NN) model for a training data set.

[0048] The use of transpose-invariant sparse weight matrices advantageously eliminates redundant computations of the sparse matrix-matrix multiplication (spMM) module 260 and the sampled dense-dense matrix multiplication (SDDMM) module 270. Because zero-valued elements do not participate in the computations within the sparse matrix-matrix multiplication (spMM) module 260 and the sampled dense-dense matrix multiplication (SDDMM) module 270, the computations of the sparse matrix-matrix multiplication (spMM) module 260 and the sampled dense-dense matrix multiplication (SDDMM) module 270 can be completed more quickly. As a result, training time can be reduced, or a larger scale model can be trained in the same time. Sparse weight matrices also advantageously utilize less memory 250 compared to dense weight matrices. Furthermore, sparse weight matrices can also advantageously be stored in a compressed format. Moreover, the transpose of the weight indices can be performed directly on the weight indices stored in the compressed format.

[0049] Reference will now be made to Figure 4 , Figure 4 A training system for a neural network (NN) model is shown in accordance with various aspects of the present disclosure. The system 400 can include one or more sparse matrix-matrix multiplication (spMM) modules 410 configured to receive a training dataset, activations (Act L-1 ) of a previous layer, sparse weight data (W L ) of a current layer, and sparse weight indices (W IDX L ) of the current layer. The one or more sparse matrix-matrix multiplication (spMM) modules 410 can generate activations (Act L ) of the current layer in a forward pass as a function of the activations (Act L-1 ) of the previous layer, the sparse weight data W L ) of the current layer, and the sparse weight indices (W IDX L ) of the current layer.

[0050] The system 400 can also include a weight data transpose module 420 to generate transpose sparse weight data L ) of the current layer from the sparse weight data (W The system can also include a weight index transpose module 430 to generate transpose sparse weight indices L ) of the previous layer from the sparse weight indices (W IDX The one or more sparse matrix-matrix multiplication (spMM) modules 410 can generate activation gradients (ActGrad L-1 ) of the previous layer in a backward pass as a function of the activation gradients (ActGrad L ) of the current layer, the transpose sparse weight data transpose sparse weight index of the current layer .

[0051] The system 400 can also include one or more sampled dense-dense matrix multiplication (SDDMM) modules 440 configured to receive the activations of the previous layer (Act L-1 ), the activation gradient of the current layer (ActGrad L ), and the sparse weight index of the current layer (W IDX L ). The one or more sampled dense-dense matrix multiplication (SDDMM) modules 440 can generate, in the backward pass, the weight gradient of the current layer (WGrad L ) as a function of the activations of the previous layer (Act L-1 ), the activation gradient of the current layer (ActGrad L ), and the sparse weight index of the current layer (W IDX L ). The weight update module 450 of the system can generate, in the backward pass, the sparse weight data of the next iteration (W L Nxt Iter ) as a function of the weight gradient of the current layer (WGrad L ) and the sparse weight data of the current layer (W L ).

[0052] The modules can be implemented in software, firmware, hardware, or any combination thereof. In one embodiment, the modules can be implemented as computer device executable instructions (e.g., software) stored in a computer device readable medium (e.g., computer memory) and executed by one or more computer devices (e.g., processing units).

[0053] Likewise, it should be understood that the calculation of the activations in the forward pass is typically first performed for each of the plurality of layers of the neural network (NN) model. The calculation of the activation gradient and the weight gradient in the backward pass can then be performed for each of the plurality of layers of the neural network (NN) model with respect to the training data set.

[0054] Likewise, the system 400 advantageously eliminates redundant calculations of the sparse matrix-matrix multiplication (spMM) module 410 and the sampled dense-dense matrix multiplication (SDDMM) module 420. Moreover, the function of the sparse matrix-matrix multiplication (spMM) module 410 can be reused in the sampled dense-dense matrix multiplication (SDDMM) module 420. Also, the transpose of the weight index can be performed directly on the weight index stored in compressed format.

[0055] Reference is now made to Figure 5 , Figure 5A method of neural network training is shown in accordance with various methods of the present disclosure. The method can be implemented in software, firmware, hardware, or any combination thereof. In one embodiment, the method can be implemented as computer device executable instructions (e.g., software) stored in a computer device readable medium (e.g., computer memory) and executed by one or more computer devices (e.g., processing units). The method of training a neural network model can include, in step 510, computing activations using transpose invariant sparse weight matrices. In one embodiment, sparse matrix-matrix multiplication (spMM) can be performed on a batch dataset to compute activations of a current layer using transpose invariant sparse weight matrices of the current layer and activations of a previous layer in a forward pass. In one embodiment, the sparse weight matrices can include weight data in a compressed format and separate weight indices for inferring dense activations. By using transpose invariant sparse matrices, zero valued elements do not participate in the computation, thereby eliminating redundant computations. The model can be trained using non-zero computations with transpose invariant sparse weight matrices. Furthermore, memory consumption can also be reduced since non-zero weight values are only a fraction of the size of the corresponding dense weight matrix.

[0056] In step 520, activation gradients and weight gradients can be computed using sparse weight matrices. In one embodiment, sparse matrix-matrix multiplication (spMM) can be performed on transpose weight matrices of a current layer and activation gradients of the current layer in a backward pass to compute activation gradients of a previous layer. Furthermore, sample dense-dense matrix multiplication (SDDMM) can be performed on weight index matrices of the current layer, activations of the previous layer, and activation gradients of the current layer to compute weight gradients of the current layer. The weight matrices of the current layer and the weight gradients can be used to compute weight matrices for the next iteration. Activation and activation gradients can be advantageously computed using sparse matrix-matrix multiplication (spMM).

[0057] Likewise, it should be understood that computing activations in a forward pass in step 510 is typically first performed for each of the plurality of layers of the neural network (NN) model. Then, computing activation gradients and weight gradients in a backward pass in step 520 can be performed for each of the plurality of layers of the neural network (NN) model with respect to a training dataset.

[0058] Reference is now made to Figure 6 , Figure 6A method of training a neural network is shown in accordance with various aspects of the present disclosure. The method can be implemented in software, firmware, hardware, or any combination thereof. In one embodiment, the method can be implemented as computing device executable instructions (e.g., software) stored in a computing device readable medium (e.g., computer memory) and executed by one or more computing devices (e.g., processing units). The method of training a neural network model can include, in step 610, computing activations in a forward pass by a sparse matrix-matrix multiplication (spMM) module using a transpose invariant sparse weight matrix. In one embodiment, because the compressed format does not include zero values, the activations can be computed by the sparse matrix-matrix multiplication (spMM) module using the compressed sparse weight matrix to reduce the amount of computation. The sparse matrix-matrix multiplication can be performed by the sparse matrix-matrix multiplication (spMM) module as described above.

[0059] In step 620, the activation gradients of the neural network (NN) model can be computed in a backward pass by the sparse matrix-matrix multiplication (spMM) module using the transposed sparse weight matrix received from the weight transpose module. In one embodiment, because the compressed format does not include zero values, the activation gradients can also be computed by the sparse matrix-matrix multiplication (SPM) module using the compressed sparse weight matrix to reduce the amount of computation. The sparsity, compression, and transposition of the weight matrix can be performed as described above.

[0060] In step 630, the weight gradients of the neural network (NN) model can be computed in a backward pass by the sampled dense-dense matrix multiplication (SDDMM) module using the activations received from the sparse matrix-matrix multiplication (spMM) module in a forward pass. The sampled dense-dense matrix multiplication (SDDMM) can be performed by the sampled dense-dense matrix multiplication (SDDMM) module as described above.

[0061] Reference is now made to Figure 7 , Figure 7 A method of training a neural network is shown in accordance with various aspects of the present disclosure. The method can be implemented in software, firmware, hardware, or any combination thereof. In one embodiment, the method can be implemented as computing device executable instructions (e.g., software) stored in a computing device readable medium (e.g., computer memory) and executed by one or more computing devices (e.g., processing units). The method of training a neural network model can include, in step 710, computing activations of a current layer in response to an input data set by sparse matrix-matrix multiplication (spMM) on weight values of a transpose invariant sparse weight matrix of the current layer, indices of the transpose invariant sparse weight matrix of the current layer, and activations of a previous layer. The sparse matrix-matrix multiplication can be performed as described above.

[0062] In step 720, the sparse weight data of the current layer can be transposed to generate transposed sparse weight data of the current layer. In step 730, the sparse weight index of the current layer can be transposed to generate transposed sparse weight index. It can be appreciated that because the sparse weight matrix is transpose invariant, the sparse weight index is also transpose invariant. Moreover, when the sparse weight data is in compressed sparse format, the sparse weight data can be transposed. The transposition of the sparse weight data and the transposition of the sparse weight index can be transposed as described above.

[0063] In step 740, the activation gradient of the previous layer can be calculated by performing sparse matrix-matrix multiplication (spMM) on the transposed sparse weight data of the current layer, the transposed sparse weight index of the current layer, and the activation gradient of the current layer. The sparse matrix-matrix multiplication can be performed as described above. In step 750, the weight gradient of the current layer can be calculated by performing sample dense-dense matrix multiplication (SDDMM) on the activation of the previous layer, the activation gradient of the current layer, and the index of the sparse weight matrix of the current layer. The sample dense-dense matrix multiplication can be performed as described above. In step 760, the weight value of the sparse weight matrix for the next iteration can be calculated from the current weight value of the sparse weight matrix and the weight gradient of the current layer. The steps 710-760 of the neural network training method can be iteratively repeated for multiple input data sets until a desired accuracy is reached.

[0064] Likewise, it should be appreciated that for each of the plurality of layers of the neural network (NN) model, the calculating activation in forward pass in step 710 is typically performed first. Then, for each of the plurality of layers of the neural network (NN) model, the calculating activation gradient and weight gradient in backward pass in steps 720-760 can be performed for a training data set.

[0065] Reference is now made to Figure 8 , Figure 8 An exemplary processing unit including embodiments for implementing neural network (NN) model training in accordance with various aspects of the present disclosure is shown. The processing unit 805 can include one or more communication interfaces, such as a peripheral component interface (PCIe4) 810 and an inter-integrated circuit (I2C) 815. The processing unit 805 can also include one or more processors, such as a central processing unit (CPU) 820 and a graphics processing unit (GPU) 825. The processing unit 805 can also include one or more memory devices, such as a dynamic random access memory (DRAM) 830 and a read-only memory (ROM) 835. The processing unit 805 can also include one or more input / output (I / O) devices, such as a universal serial bus (USB) 840 and a display 845. 2C) interfaces 815, on-chip circuit testers, such as a joint test action group (JTAG) engine 820, a direct memory access engine (DMA) 825, a command processor (CP) 830, and one or more cores 835-850. The one or more cores 835-850 can execute one or more sets of computer device executable instructions to perform the training system and training method of a neural network model as described above. The one or more cores 835-850 may, for example, include one or more matrix multiplication modules 855, one or more weight transpose modules 860, and one or more non-multiplication operation modules 865. As described above, the one or more matrix multiplication modules 855 can be configured to compute activations of a neural network (NN) model in a forward pass using a transpose-invariant sparse weight matrix by a sparse matrix-matrix multiplication (spMM) module. As described above, the one or more matrix multiplication modules 855 can also be configured to compute gradients of activations of a neural network (NN) model in a backward pass using a transposed sparse weight matrix received from a weight transpose module by a sparse matrix-matrix multiplication (spMM) module. As described above, the one or more weight transpose modules 860 can be configured to compute a transpose of a transpose-invariant sparse weight matrix. The one or more non-multiplication operation modules 865 can be configured to provide non-multiplication operation support to the one or more weight transpose modules 860. The one or more functions can be executed on a single core 835-850, can be distributed across multiple cores 835-850, can be executed with one or more other functions on one or more cores, and the like.

[0066] The processor unit 805 can be a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a vector processor, a memory processing unit, or the like, or a combination thereof. In one embodiment, the one or more processors 805 can be implemented in a computing device such as, but not limited to, a cloud computing platform, an edge computing device, a server, a workstation, a personal computer (PC), and the like.

[0067] For a transpose invariant sparse weight matrix with a sparsity of 50%, the training kernel runtime can be improved by 1.6-1.7 times compared to training using a dense weight matrix. In addition, the runtime for an end-to-end model used to train a bidirectional encoder representations for transformer (BERT) neural network model can be improved by about 1.3 times.

[0068] A neural network (NN) model in accordance with aspects of the present disclosure enables a computing device to learn a function during training. In contrast to a traditional software routine, a neural network model can learn various relationships from a dataset through the practical application of various mathematical functions, thereby improving the performance of a computer. In contrast, a traditional computing process performs a function based on knowledge encoded in a corresponding instruction set by a programmer prior to execution by the computing device. Whereas a neural network model enables a computing device to learn and encode knowledge during a training process and apply the learned knowledge to perform a corresponding function during an inference process. Thus, a neural network model enables a computing device to improve its operations to solve real-world problems. Moreover, aspects of the present disclosure utilize transpose invariant sparsity to reduce the training time of a neural network, thereby further improving the performance of a computing device.

[0069] The foregoing description of specific implementations of the disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical application to thereby enable others skilled in the art to best utilize the disclosure and various embodiments with various modifications as are suited to the particular use contemplated. The scope of the disclosure is defined by the following claims and their equivalents.

Claims

1. A method of training a neural network model, comprising: computing activations of the neural network model in a forward pass using a transpose invariant sparse weight matrix through a sparse matrix-matrix multiplication module; computing activation gradients of the neural network model in a backward pass using a transpose of the sparse weight matrix received from a weight transpose module through the sparse matrix-matrix multiplication module; computing weight gradients of the neural network model in the backward pass using activations received from the sparse matrix-matrix multiplication module in the forward pass through a sampled dense-dense matrix multiplication module; wherein the transpose invariant sparse weight matrix is partitioned into windows, the windows are scheduled for processing by respective threads of a software-based neural network processor or the windows are scheduled to respective hardware accelerators of a neural network processor; wherein specifically comprising: in response to an input data set, computing activations of a current layer through the sparse matrix-matrix multiplication module based on activations of a previous layer, sparse weight data of the sparse weight matrix of the current layer, and sparse weight indices of the sparse weight matrix of the current layer; transposing the sparse weight indices of the current layer through the weight transpose module; transposing the sparse weight data of the current layer through the weight transpose module; computing activation gradients of the previous layer through the sparse matrix-matrix multiplication module based on the transposed sparse weight indices of the current layer, the transposed sparse weight data of the current layer, and the activation gradients of the current layer; computing weight gradients of the current layer through the sampled dense-dense matrix multiplication module based on the activations of the previous layer, the sparse weight indices of the current layer, and the activation gradients of the current layer; and computing sparse weight data of the current layer for a next iteration based on the weight gradients and current sparse weight data.

2. The training method of claim 1, wherein, the sparse weight data is transpose invariant; the sparse weight indices are transpose invariant.

3. The training method of claim 1, wherein, the sparse matrix-matrix multiplication module in the forward pass is reused to compute activation gradients in the backward pass with the transpose invariant sparse weight matrix.

4. The training method of claim 1, wherein, further comprising training the neural network model using the transpose invariant sparse weight matrix through non-zero computation.

5. The training method of claim 1, wherein, the transposed sparse weight data is transposed from the sparse weight data in a compressed format.

6. A system for training a neural network model, comprising: a multiplication module comprising one or more sparse matrix-matrix multiplication modules and one or more sampled dense-dense matrix multiplication modules; a weight data transpose module configured to transpose sparse weight data of a transpose invariant sparse weight matrix of a current layer; a weight index transpose module configured to transpose sparse weight indices of a sparse weight matrix of a current layer; the one or more sparse matrix-matrix multiplication modules are configured to: compute activations of a current layer based on activations of a previous layer, sparse weight data of the current layer, and sparse weight indices of the current layer in a forward pass of a current cycle of a batch data set; and the one or more sampled dense-dense matrix multiplication modules are configured to: compute weight gradients of the current layer based on the activations of the previous layer, the sparse weight indices of the current layer, and activation gradients of the current layer in the backward pass of the current cycle of the batch data set. In the backward pass, an activation gradient of a previous layer is computed based on transpose sparse weight data of the current layer, transpose sparse weight indices of the current layer, and an activation gradient of the current layer; The one or more sample dense-dense matrix multiplication modules are configured to: In the backward pass, a weight gradient of the current layer is computed based on the activation gradient of the current layer and an activation of the previous layer; and A weight update module is configured to compute new sparse weights based on the current weight gradient. wherein the transpose invariant sparse weight matrix is partitioned into windows, the windows are scheduled for processing by respective threads of a software-based neural network processor or the windows are scheduled to respective hardware accelerators of a neural network processor.

7. The training system of claim 6, wherein, The sparse weight data is transpose invariant; the sparse weight indices are transpose invariant.

8. The training system of claim 6, wherein, The one or more sparse matrix-matrix multiplication modules used in the forward pass are reused to compute an activation gradient of a current layer using the transpose invariant sparse weight matrix in the backward pass.

9. The training system of claim 6, wherein, The neural network model is trained using the sparse weight matrix with non-zero computation.

10. The training system of claim 6, wherein, The weight data transpose module computes transpose sparse weight data of a current layer from sparse weight data of the current layer in compressed format.

11. The training system of claim 6, wherein, The training system further includes a memory configured to: store an activation of the current layer for use as an activation of a previous layer for a next batch of processing data sets; store an activation gradient of the current layer for use as an activation gradient of the previous layer for the next batch of processing data sets; store the new sparse weight data for use as sparse weight data of the current layer for the next batch of processing data sets; and and store the sparse weight data of the current layer and the sparse weight indices of the current layer.

12. The training system of claim 11, wherein, Compared to training a neural network using a dense weight matrix, the utilization of the memory is reduced in proportion to the ratio of non-zero weight values of the sparse weight matrix.

13. One or more computing device readable media comprising instructions stored thereon that, when executed by one or more processing units, perform the method of any of claims 1-5.

Images