TRAINING DEEP NEURAL NETWORKS WITH 4-BIT INTEGER NUMBERS

Abstract

Techniques for training deep neural networks with 4-bit integers are described.According to one aspect of the present disclosure, a computer-implemented method for accelerating operations of a deep neural network comprises: for each of a plurality of matrix multiplications during forward propagation, obtaining a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; obtaining a second matrix by multiplying a weighting matrix, one transpose of which is another input to the matrix multiplication, by the block-diagonal Hadamard transform matrix; quantizing the first matrix and the second matrix into 4-bit integer matrices; obtaining a resultant matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix; and dequantizing the resultant matrix into a floating-point matrix.

Description

AREA

[0001] The aspects of this disclosure relate generally to artificial intelligence (AI) and in particular to the training of deep neural networks with 4-bit integers. BACKGROUND

[0002] Artificial neural networks (ANNs), usually simply called neural networks (NNs), are computing systems inspired by the information processing and distributed communication nodes found in biological neural network systems and have been studied for several decades. Deep neural networks (DNNs), which are artificial neural networks with multiple layers between the input and output layers, have seen tremendous development in recent years due to significant advances in both machine learning algorithms and models, as well as in the performance of computer hardware. Current state-of-the-art deep neural network models, such as Transformer, introduced in 2017 by Ashish Vaswani et al., have been widely applied in a variety of fields, including natural language processing, machine translation, speech recognition, computer vision, and more., where they achieved results with remarkable performance.

[0003] There is a constant need to accelerate the operations of deep neural networks, especially in the operating environments of mobile computing, vehicle computing, edge computing, and similar applications, which typically have relatively limited computing resources. The fundamental idea behind quantization as a method for accelerating neural networks is to replace computationally intensive, high-precision representations and calculations (e.g., 32-bit floating-point numbers (FP32), 16-bit floating-point numbers (FP16)) with more cost-effective, low-precision representations (e.g., 8-bit integers (INT8)) for use within neural networks. Quantization can effectively improve computational efficiency, reduce memory requirements and storage space, and consequently lower power consumption.

[0004] Training neural networks is computationally intensive. Quantizing activations, weights, and gradients to extremely low precision promises to accelerate neural network training. Current research in this area aims to reduce numerical precision without significantly compromising model accuracy. Although several proposals have been made, significant optimization challenges remain when training neural networks at the extremely low 4-bit level. SUMMARY

[0005] This summary serves to present a selection of concepts in simplified form, which are explained in more detail below. This summary is neither intended to identify principal features or essential characteristics of the claimed subject matter, nor is it meant to serve as an aid in determining the scope of protection of the claimed subject matter.

[0006] According to one embodiment of the present disclosure, a computer-implemented method for accelerating operations of a deep neural network is provided. The method comprises: for each of a plurality of matrix multiplications during forward propagation, obtaining a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; obtaining a second matrix by multiplying a weighting matrix, one transpose of which is another input to the matrix multiplication, by the block-diagonal Hadamard transform matrix; quantizing the first matrix and the second matrix into 4-bit integer matrices; obtaining a resultant matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix; and dequantizing the resultant matrix into a floating-point matrix.

[0007] According to a further embodiment of the present disclosure, a computing device is provided. The computing device comprises a memory for storing instructions and one or more processing unit(s) coupled to the memory, wherein the instructions, when executed on the one or more processing unit(s), cause the one or more processing unit(s) to: for each of a plurality of matrix multiplications during forward propagation, obtain a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; obtain a second matrix by multiplying a weighting matrix, a transpose of which is another input to the matrix multiplication, by the block-diagonal Hadamard transform matrix;Quantize the first matrix and the second matrix into 4-bit integer matrices; obtain a resulting matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix, and dequantize the resulting matrix into a floating-point matrix.

[0008] According to a further embodiment of the present disclosure, a non-volatile, computer-readable storage medium is provided on which instructions are stored which, when executed by one or more processing units, cause the one or more processing units to: for each of a plurality of matrix multiplications during forward propagation, obtain a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; obtain a second matrix by multiplying a weighting matrix, one transpose of which is another input to the matrix multiplication, by the block-diagonal Hadamard transform matrix; quantize the first matrix and the second matrix into 4-bit integer matrices;Obtaining a resulting matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix, and dequantizing the resulting matrix into a matrix with floating-point representation.

[0009] According to yet another embodiment of the present disclosure, a computer program product is provided which includes instructions which, when executed by one or more processing units, cause the one or more processing units to: for each of a plurality of matrix multiplications during forward propagation, obtain a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; obtain a second matrix by multiplying a weighting matrix, one transpose of which is another input to the matrix multiplication, by the block-diagonal Hadamard transform matrix; quantize the first matrix and the second matrix into 4-bit integer matrices;Obtaining a resulting matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix, and dequantizing the resulting matrix into a matrix with floating-point representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the present disclosure are illustrated by way of example and without limitation in the figures of the accompanying drawings, wherein identical reference numerals refer to identical or similar elements and in which: Fig. 1 illustrates a typical architecture of an exemplary neural network; Fig. 2A is a histogram illustrating the original activation distribution of an exemplary transformer model; Fig. 2B is a histogram illustrating the activation distribution of an exemplary transformer model after a Hadamard transformation according to some embodiments of the present disclosure; Fig. 3 is a diagram illustrating performance variations associated with different group sizes according to some embodiments of the present disclosure; Fig. 4 is a flowchart of an exemplary computer-implemented method according to some embodiments of the present disclosure; Fig. 5 is a flowchart of an exemplary matrix multiplication method according to some embodiments of the present disclosure; Fig. 6 is a flowchart of an exemplary matrix multiplication method according to some embodiments of the present disclosure and Fig. 7 is a block diagram of an exemplary computing device according to some embodiments of the disclosure. DETAILED DESCRIPTION

[0011] The following description presents numerous specific details for clarification. However, it is understood that embodiments of the present disclosure can also be implemented without these specific details. In other cases, known circuits, structures, and techniques have not been shown in detail so as not to complicate the understanding of the disclosure.

[0012] References to “an embodiment”, “a form”, “an example”, “some embodiments”, “various embodiments”, or the like throughout the description indicate that the embodiment(s) so described in this disclosure may include certain features, structures, or properties, but not every embodiment necessarily includes these certain features, structures, or properties. Furthermore, some embodiments may have some, all, or none of the features described for other embodiments.

[0013] In the following description and claims, the terms "coupled" and "connected," as well as their derivatives, may be used. It is understood that these terms are not intended to be synonymous. Rather, in certain embodiments, "connected" is used to indicate that two or more elements are in direct physical or electrical contact with each other, while "coupled" is used to indicate that two or more elements work together or interact with each other, but need not be in direct physical or electrical contact.

[0014] Fig. Figure 1 illustrates a typical architecture of an exemplary neural network, which has an input layer, an output layer and two hidden layers.

[0015] In this example neural network, there are four neurons (or "nodes") in the input layer, three neurons in the first hidden layer, two neurons in the second hidden layer, and one neuron in the output layer. As shown here, a neuron or node in one of the layers of the example neural network receives input signals from the neurons in the previous layer, which are passed along connections with different weights. An activation function (denoted as "f") associated with the neuron is then applied to the sum of the weights (denoted as "Σ") of input signals received by the neuron to produce the neuron's output, called the "activation," which serves as input for a neuron(s) in the next layer. It is understood that for the neurons in the input layers, the corresponding activations correspond to the respective input values.

[0016] A deep neural network (DNN) is a neural network with multiple layers (i.e., very deep) between the input and output layers. Transformer, as a state-of-the-art model of a deep neural network, is increasingly becoming the model of choice for various tasks, such as natural language processing (NLP). For NLP applications, the deep neural network uses vector representations of words in some typical scenarios, generated based on word embedding.A typical transformer uses an encoder-decoder architecture employing an attention mechanism, wherein the encoder consists of coding layers to generate coding information about which parts of the inputs are relevant to each other, which is then passed as input to the next coding layer, and wherein the decoder consists of decoding layers that take all the coding information and use their contained context information to generate an output sequence. Although transformers are referred to in the following discussion as exemplary models of deep neural networks, it should be noted that the concept proposed in the present disclosure according to the invention can also be applied to other types of deep neural networks.

[0017] The training process of a deep neural network, of which the Transformer is a typical example, generally consists of two phases: forward propagation and backpropagation. Forward Propagation

[0018] The forward propagation of a neural network can be formulated as a combination of linear operations (matrix multiplications) and nonlinear operations (GeLU, normalization, softmax, etc.). According to some embodiments of the present disclosure, all linear operations, i.e., matrix multiplications, are accelerated using 4-bit integer arithmetic (INT4 arithmetic). For these less computationally intensive nonlinear operations, the original floating-point (FP) format, such as the 16-bit floating-point format (FP16), can be retained. It should be noted that in the following discussion, FP16 is generally referred to as an example of a floating-point representation, and that other floating-point formats with higher or lower precision, such as FP32, may also be used in implementations of the present disclosure.

[0019] All linear operations in a deep neural network, for which Transformer is a typical and the most popular instantiation, can be written in the form of a matrix multiplication (MM), which, according to some embodiments of the present disclosure, is to be accelerated. For the sake of simplicity, the matrix multiplication can be expressed as follows: Y=XWT where Y ∈ ℝ N×C , X ∈ ℝ N×D and W ∈ ℝ C×D The predominant use case of such matrix multiplication is the fully connected layer. A transformer with an input form of (batch size S, sequence length T, dimensionality D) is assumed. A fully connected layer can be written as equation (1), where X is the activation matrix for N=ST tokens and W is the weighting matrix.

[0020] Given an FP matrix X, a typical quantization method called Learned Step Size Quantization (LSQ) can quantize X into an integer using: intsX(X):=clamp(XsX,−QN,QP), in which s X a learnable floating-point scalar parameter called the step size; the clamp function restricts its input to the range [-Q N , Q P ]; the operator ⋅ represents a rounding operation to the nearest integer and X / s X The calculation should be element-wise. The resulting matrix takes values ​​from {-Q N , -Q N + 1,..., Q P Since the calculation is to be performed using 4-bit integers (INT4), according to some embodiments of the present disclosure it may be preferable that Q N = Q P = 7, especially considering symmetric quantization.

[0021] The integer matrix can be dequantized back into floating-point representation: float(intsX(X))=sXintsX(X)≈X,

[0022] Similarly, the FP weighting matrix W can be quantized as: intsX(W):=clamp(WsX,−QN,QP), in which s W a learnable floating-point scalar parameter for the weighting matrix.

[0023] Therefore, the above equation (1) can be approximated as follows: Y=XWT≈sXsWintsX(X)intsW(W)T,

[0024] Here, the INT4 matrix multiplication int sX (X) int sX (W) T efficiently implemented on hardware such as Tensor Core CPUs.

[0025] However, it has been found that directly using the above equations, such as equation (2), to accelerate integer arithmetic training can lead to a significant deterioration in accuracy. In particular, the outlier problem exists with the quantization method mentioned above. As in Fig. Figure 2A, a histogram illustrating the original activation distribution of an example transformer model, shows that activations exhibit some outlier entries that are much larger in magnitude than other entries. In this case, the step size s represents X represents a compromise between quantization granularity and representable numerical range. If s X If the sample size is large, outliers can be clearly displayed, at the expense of a very coarse representation of most other entries. On the other hand, if the sample size is large... X If the range is small, the entries outside the range [-Q] must be included. N sX , Q P s X ] must be truncated. Transformers tend to store information in these outliers, and such a truncation would seriously impair accuracy. The outlier problem is particularly significant when the training task is to optimize a pre-trained model for some new subsequent tasks.

[0026] The outliers in an activation matrix X form a feature-wise structure. They are typically concentrated in a few dimensions, i.e., only a few columns of X are significantly larger than others. According to some embodiments of the present disclosure, the Hadamard transform, which is a linear transform, is introduced hereto to solve the outlier problem.

[0027] In particular, the Hadamard transformation H k a 2 k × 2 k -matrix, in which H0=1,Hk=12[Hk−1Hk−1Hk−1−Hk−1],

[0028] Hadamard matrices are orthogonal and symmetrical: Hk=HkT=Hk−1, so H k H k = I, ∀k ≥ 0. According to some embodiments of the present disclosure, a block-diagonal matrix H ∈ ℝ D×D used, where D is a multiple of 2 k is: H=BlockDiag(Hk,…,Hk),

[0029] To suppress outliers, a transformed version of the activation matrix X can be quantized as follows: X=(XH)HT≈sXintsX(XH)HT,

[0030] The practical impact of the Hadamard transformation on outlier suppression in a quantization-friendly manner is discussed in Fig. Figure 2B shows that the Hadamard transform distributes the information contained in outliers to the nearby entries of the matrix, thus reducing the numerical range of the outliers. Quantization using the Hadamard transform, also referred to here as Hadamard quantization (HQ), successfully quantizes the matrix into a further linear space that contains fewer outliers.

[0031] For Hadamard quantization, there is a group size hyperparameter M = 2. k , as defined in equation (7). The group size controls a trade-off between accuracy and time complexity. With a larger group size, outliers can be amortized over more entries, but at the cost of higher time complexity. Fig. 3 is a diagram illustrating performance variations associated with different group sizes according to some embodiments of the present disclosure; As in Fig. As shown in Figure 3, a larger group size steadily improves the accuracy, proving the effectiveness of Hadamard quantization, and the improvement begins at a group size M = 2. 5 = 32 to saturate. Therefore, according to some embodiments of the present disclosure, the value of k is chosen to be k = 5.

[0032] Similarly, the weighting matrix W can be transformed using another block-diagonal Hadamard transformation matrix H ∈ ℝ D×D to be quantized: W=(WH)HT≈sWintsX(WH)HT,

[0033] By combining the quantized matrices, the following can be obtained: Y=XWT≈sXsWintsX(XH)HTHintsW(WH)T=sXsWintsX(XH)intsW(HTWT),

[0034] Therefore, according to some embodiments of the present disclosure, a computer-implemented method for accelerating operations of a deep neural network is disclosed, which can be used during a training process of the deep neural network. Fig. Figure 4 is a flowchart of an exemplary process 400 according to some embodiments of the present disclosure.

[0035] For each of the many matrix multiplications during forward propagation, the exemplary procedure 400 begins with step 410, obtaining a first matrix (XH) by multiplying an activation matrix (X), which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix (H). It is understood that the first matrix obtained in this step is still in a floating-point format, such as FP16.

[0036] Similarly, procedure 400, step 420, obtains a second matrix (WH) by multiplying a weighting matrix (W), one of whose transposes is another input to the matrix multiplication, with the block-diagonal Hadamard transform matrix (H). Following the preceding example, the obtained second matrix is ​​also in FP16 form.

[0037] Procedure 400 then proceeds to step 430, quantizing the first matrix and the second matrix into 4-bit integer matrices. This quantization can be implemented using the quantization method discussed above, namely learned step-size quantization. In some embodiments of the present disclosure, the quantization may involve applying a clamp function to the first matrix and the second matrix, respectively, to restrict each element in the first and second matrix to a range representable by 4-bit integers. Here, a first step size for the activation matrix and a second step size for the weighting matrix, each a parameter of the clamp function, are floating-point scalars that can be learned during a training process of the deep neural network. In this case, the quantized first matrix can be determined by reference to the equations above. sX(XH) in equation (8) correspond, while the quantized second matrix int sW (WH) can correspond to equation (9).

[0038] Procedure 400 continues with step 440, obtaining a resulting matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix. It is evident that in this step, the matrix multiplication is performed on two 4-bit integer matrices (INT4 matrices) instead of the original computationally intensive FP16 matrix multiplication.

[0039] Step 450 then follows, dequantizing the resulting matrix into a matrix with floating-point representation. In some embodiments of the present disclosure, the dequantization with reference to equation (10) may include multiplying the resulting matrix, which as described above is an integer matrix, by the first step size and the second step size, which are each floating-point scalars.

[0040] It should be noted that the sequence discussed above, from step 410 to step 450, in no way implies that the exemplary procedure 400 can only be carried out in this particular sequential order. Instead, it is readily apparent that some of the operations can be performed simultaneously, in parallel, or in a different order. For example, steps 410 and 420 can be performed concurrently. Likewise, other procedure steps and processes disclosed in this disclosure need not necessarily be carried out in the described sequential order.

[0041] For time complexity, steps 410 and 420 require O(2 kN(D + C)) FP16-Multiply-Accumulates (MACs); steps 430 and 450 together require O(N(D + C)) FP16-MACs, and step 440 requires O(NDC) INT-MACs. Compared to directly using the Learned Step Size Quantization (LSQ) discussed above with reference to equations (2) to (5), the number of FP16-MACs increases 2 k -times, from O(N(D + C)) to O(2 k N(D + C)). However, the Hadamard quantization proposed herein is still much more favorable than FP16 matrix multiplication when 2 k << D, 2 k << C.

[0042] It should be noted that although the preceding discussions take place in the context of the deep neural network training process, the technique discussed herein could also be applied to speed up the inference process that uses a trained neural network to make predictions about new data, which is also a type of forward propagation. Backpropagation

[0043] The backpropagation phase of the deep neural network training process refers to a procedure that minimizes the loss (or cost) incurred using a loss function. L (e.g., cross-entropy) is calculated and fed back through the layers of the neural network to adjust parameters (e.g., weights) within the neural network using a type of gradient descent. The loss function serves this purpose. L The purpose of backpropagation is to measure the difference between the predicted output of the deep neural network and the expected output (or the basic truth), and backpropagation aims to minimize the loss function.

[0044] To accelerate the backpropagation of the training process, the linear operation of matrix multiplication, defined in equation (10) and having four inputs, is now considered: activation X, weight W and step sizes s. X , s W . Given the gradient ∇YL (for the sake of simplicity, hereafter referred to as ∇) Y (designated) the loss function L With respect to the output Y of the matrix multiplication, the gradients for all four inputs are to be calculated.

[0045] Firstly, the Straight-Through Estimator can be used x'=1 and the chain rule, the weighting gradient, and the activation gradient are calculated as follows: ∇W=sX(∇YTX^∘IW)HT∇X=sWIX∘∇YW^HT, where X̂ = int sX (XH), Ŵ = int sW (WH), IX=I(−QN≤X / sX≤QP) and IW−I(−QN≤W / sW≤QP), where IX and IW 0 / 1 matrices are and the operator ◦ represents an element size multiplication.

[0046] According to some embodiments of the present disclosure, among other things, the matrix multiplication of the FP16 gradient matrix ∇ is performed. Y with an INT4 matrix X̂ or Ŵ, i.e., two matrix multiplications ∇YTX^ and ∇ Y Ŵ in the preceding equation, by quantizing ∇ Y accelerated to INT4.

[0047] It was found that the gradient matrix ∇ Y It tends to be very sparsely populated during the training process, and the sparse population exhibits a structure: few rows (i.e., tokens) of ∇ YThey exhibit large entries, while most other rows are close to a vector consisting only of zeros. Such structural sparsity arises from the strong overparameterization of modern neural networks, and it has been observed that in pre-training tasks, such structural sparsity quickly appears after only a few training epochs, while in fine-tuning tasks, the gradient remains sparse throughout the entire training process.

[0048] As described above, many lines of the gradient are so small that they have little impact on the parameter gradient, yet they waste a significant amount of computing power. On the other hand, the large lines may not be accurately represented using 4-bit integers (INT4).

[0049] In view of the aforementioned problems for calculations related to the FP16 gradient matrix ∇ YThis matrix must first be divided into higher 4 bits and lower 4 bits: ∇Y≈s↑∇Y↑+s↓∇Y↓ in which s ↑ , s ↓ two floating-point scalars are and ∇Y↑,∇Y↓ INT4 matrices are the higher and lower 4 bits of ∇, respectively. Y represent.

[0050] According to some embodiments of the present disclosure, this bit splitting (BS) can be implemented by first splitting the FP16 gradient matrix ∇ Y into the first (higher) INT4 gradient matrix ∇Y↑ is quantized as ∇Y≈s↑∇Y↑; and then the residue, i.e., the difference between ∇ Y and s↑∇Y↑, into the second (lower) INT4 gradient matrix ∇Y↓ is quantized as ∇−s↑∇Y↑≈s↓∇Y↓.

[0051] Bit splitting can be represented as an INT8 representation of the original gradient matrix ∇.Y to be considered, whereby ∇Y↑ or ∇Y↓ These are the higher and lower 4 bits of the INT8 representation, respectively.

[0052] Next, we will discuss the calculation using the weighting gradient and the activation gradient. weighting gradient

[0053] As previously discussed, the weighting gradient involves matrix multiplication. ∇YTX^, wherein ∇ Y ∈ ℝ N×C and X̂ is an N × D-INT4 matrix. This matrix multiplication can be approximated as follows: ∇YTX^≈(s↑∇Y↑T+s↓∇Y↓T)X^=∇Y↕X↕, wherein ∇Y↕=[s↑∇Y↑;s↓∇Y↓]∈ℝ2N×C and X ↕ = [X̂ ↕ ; X̂ ↕ ], which is supposed to be a 2N × D-INT4 matrix.

[0054] Equation (13) shows that the matrix multiplication of an FP16 matrix is ​​the transpose of the gradient matrix ∇ Yand the INT4 matrix X̂ by two INT4 matrix multiplications ∇Y↑TX^ and ∇Y↓TX^ can be approximated. More precisely, matrix multiplication can be approximated. ∇YTX^, which is discussed herein, can be implemented according to some embodiments of the present disclosure by the following steps: Obtaining a matrix ∇Y↑TX^ by multiplying the transposes of the first (higher) INT4 gradient matrix ∇Y↑ and the INT4 matrix X̂; Obtaining a matrix ∇Y↓TX^ by multiplying the transposes of the second (lower) INT4 gradient matrix ∇Y↓ and the INT4 matrix X̂; dequantizing the two obtained matrices into matrices with floating-point representation (e.g., FP16); and subsequently obtaining a resulting matrix by summing the two dequantized matrices.

[0055] Such a matrix multiplication in equation (13) is exact, since the gradient matrix ∇ Y represented with 8 bits, effectively avoiding situations where, if an element in the gradient matrix is ​​quantized to only a 4-bit integer, the inaccurate gradients could adversely slow down the training process and even cause the training to become unstable or diverge.

[0056] Compared to a naive quantization of ∇ Y In INT4, the bit splitting described above can further double the number of INT4 operations for matrix multiplication. According to some embodiments of the present disclosure, Leverage Score Sampling (LSS) is proposed herein to halve the number of operations of equation (13) to the same amount as naive matrix multiplication. s↑∇Y↑TX^ to reduce. This means that by exploiting the structural sparsity, some small rows of the gradient matrix can be deleted, and the saved computing power can therefore be used for a more accurate representation of large rows.

[0057] The matrix multiplication of equation (13) can be written as the sum of 2N matrix matrices: ∇Y↕X↕=∑i=12N∇Y:,i↕TXi↕=∑i2N∇Wi, wherein ∇Wl=∇Y:,i↕Xi↕. Due to the sparse population of ∇ Y The matrices differ ∇ Wi in size, and small matrices can be discarded without significantly affecting the result.

[0058] According to some embodiments of the present disclosure, when obtaining the matrix ∇Y↑TX^ and the matrix ∇Y↓TX^, As discussed above, rows in the first (higher) INT4 gradient matrix and the second (lower) INT4 gradient matrix are sampled, with random masks per row (m i ), which correspond to the Bernoulli distribution with corresponding probabilities per row (p i ) follow. More precisely, each ∇ Wi a probability p i ∈ [0,1], i = 1, ...,2N assigned, the ∑i=12Npi=N fulfilled. Random masks m i are defined such that they correspond to the Bernoulli distribution with parameter p i follow, referred to as m i ~ Bern(p i ), and a diagonal matrix M̃ is defined as M˜=diag(m1p1,…,m2Np2N) defined. So it can ∇Y↕TX↕ can be approximated without bias as follows: ∇Y↕TX↕≈∇Y↕TM˜X↕=∑i=12N∇Y↕Tmipi∇Y:,i↕TXi↕

[0059] As expected, there are only N m inon-zero. Therefore, leverage score sampling reduces the cost of matrix multiplication by half. One coefficient per row, here called the leverage score for the weighting gradient, is ci:=‖∇Yi,:↕T‖‖Xi,:↕‖, which can be easily calculated with low time complexity. If p i ∝ c i , reaches the variance for ∑i=12Nmipi∇Y:,i↕TXi↕ their minimum.

[0060] Here is M ↑ defined as the upper left N × N submatrix of M, and M ↓ is defined as the lower right N × N submatrix of M. Therefore, the following holds: ∇Y↕TM˜X↕=s↑∇Y↑TM˜↑X^+s↓∇Y↓TM˜↓X^, which can be implemented by two INT4 matrix multiplications with sampled rows / columns.

[0061] Therefore, using the above equations, a matrix multiplication method 500 for calculating the weighting gradient can be as follows, as shown in Fig. 5 shown: ➢ Step 510: Quantizing ∇ Y with bit splitting to ∇Y↑ and ∇Y↓ to obtain in INT4; ➢ Step 520: Calculating the Leverage Score ci:=‖∇Yi,:↕‖‖Xi,:↕‖ in FP16; ➢ Step 530: Scanning the masks m i , which corresponds to the Bernoulli distribution with parameter p i follow, in which p i ∝ c i ; ➢ Step 540: Calculating INT4 matrix multiplications ∇Y↑TM˜↑X^ and ∇Y↓TM˜↓X^; ➢ Step 550: Dequantizing the resulting matrices into FP16 matrices and ➢ Step 560: Summing the FP16 matrices to obtain the FP16 result ∇Y↕TM˜X↕ to obtain.

[0062] The two INT4 matrix multiplications in step 540 require approximately 2NCD INT4 MACs, as expected. The total complexity of all other steps is O(NC + ND). Activation gradient

[0063] Just as in the discussion of the weighting gradient, matrix multiplication ∇ γ The Ŵ in the calculation of the activation gradient should be written as follows: ∇YW^≈(s↑∇Y↑+s↓∇Y↓)W^=s↑∇Y↑W^+s↓∇Y↓W^=(I^↕∇Y↕)W^, wherein ∇Y↕=[s↑∇Y↑;s↓∇Y↓]∈ℝ2N×C, as described above, and Î ↕ = [II], which is an N × 2N-INT4 matrix, where I is an N × N identity matrix.

[0064] Equation (17) shows that the matrix multiplication of the FP16 gradient matrix ∇ Y and the INT4 matrix Ŵ also by two INT4 matrix multiplications ∇Y↑W^ and ∇Y↓W^ They can be approximated. Matrix multiplication ∇ can be used more precisely. Y The Ŵ discussed here can be implemented according to some embodiments of the present disclosure by the following steps: Obtaining a matrix ∇Y↑W^ by multiplying the first (higher) INT4 integer gradient matrix ∇Y↑ and the INT4 matrix; Obtaining a matrix ∇Y↓W^ by multiplying the second (lower) INT4 gradient matrix ∇Y↓ and the INT4 matrix Ŵ; Dequantizing the two obtained matrices into matrices with floating-point representation (e.g., FP16); and subsequently obtaining a resulting matrix by summing the two dequantized matrices.

[0065] Unlike the weighting gradient, the focus is now on I^↕∇Y↕ in Eq. (17), and the leverage score sampling is performed by this matrix multiplication. This matrix multiplication can also be written as the sum of 2N smaller multiplications: I^↕∇Y↕=∑i=12NI^:,i↕∇Yi↕=∑i=12N∇Yi, wherein ∇Yi=I^:,i↕∇Yi↕.

[0066] According to some embodiments of the present disclosure, when obtaining the matrix ∇Y↑W^ and the matrix ∇Y↓W^, As discussed above, rows in the first (higher) INT4 gradient matrix and the second (lower) INT4 gradient matrix are sampled, with random masks per row m. i , which correspond to the Bernoulli distribution with corresponding probabilities per row p i follow. More precisely, each ∇ Yi a probability p i ∈ [0,1], i = 1, ...,2N assigned, the ∑i=12Npi=N fulfilled. Random masks m i are defined such that they correspond to the Bernoulli distribution with parameter p i follow, referred to as m i ~ Bern(p i ), and a diagonal matrix M̃ is defined as M˜=diag(m1p1,…,m2Np2N) So can I^↕∇Y↕ can be approximated without bias as follows: I^↕∇Y↕≈I^↕M˜∇Y↕=∑i=12NmipiI^:,i↕∇Yi↕

[0067] Here would be the leverage score for the activation gradient. ci:=‖∇Yi↕‖, and the variance for ∑i=12NmipiI^:,i↕∇Yi↕ It reaches its minimum when p i ∝ c i .

[0068] M is back again ↑ defined as the upper left N × N submatrix of M, and M ↓ is defined as the lower right N × N submatrix of M, and the matrix multiplication equation (17) can be decomposed into two INT4 matrix multiplications: (I^↕M˜∇Y↕)W^=s↑IM˜↑∇Y↑W^+s↓IM˜↓∇Y↓W^

[0069] In this case, point IM˜↑∇Y↑ and IM˜↓∇Y↓ Both only have non-zero parts of their rows, and the remaining rows are zeros because they are discarded. Subsequently, when multiplying by Ŵ, half of the rows are zeros in . IM˜↑∇Y↑W^ and IM˜↓∇Y↓W^, Therefore, it is not necessary to calculate them, and in this case, half of the calculation is omitted.

[0070] Therefore, using the above equations, a matrix multiplication method 600 for calculating the activation gradient can be formulated as follows, as shown in Fig. 6 shown: ➢ Step 610: Quantizing ∇ Y with bit splitting to ∇Y↑ and ∇Y↓ to obtain in INT4; ➢ Step 620: Calculating the Leverage Score ci:=‖∇Yi↕‖ in FP16; ➢ Step 630: Scanning the masks m i , which corresponds to the Bernoulli distribution with parameter p i follow, in which p i ∝ c i ; ➢ Step 640: Calculating of IM˜↑∇Y↑ and IM˜↓∇Y↓, by discarding some of their lines; ➢ Step 650: Calculating INT4 matrix multiplications IM˜↑∇Y↑W^ and IM˜↓∇Y↓W^; ➢ Step 660: Dequantizing the resulting matrices into FP16 matrices and ➢ Step 670: Summing the FP16 matrices to obtain the FP16 result (I^↕∇Y↕)W^ to obtain.

[0071] The two matrix multiplications in step 640 require approximately 2NCD INT4 MACs, as expected. Learn quantization parameters

[0072] The following section discusses how the quantization step sizes s X , s W They can be learned or optimized. According to the chain rule, the gradients for the step sizes s can be determined. X , s W to be written as follows: ∇sW=g(sW)∇YTX^∘δW(sW)∇sX=g(sX)∇YW^∘δX(sX), wherein g(sW)=1 / QpNW,g(sX)=1 / QpNX,NW and N X the number of weighting and activation elements are, δW(sW)=intsW(W)−IW∘(W / sW) and δX(sW)=intsX(X)−IX∘(X / sX).

[0073] For the calculation of ∇ sW and ∇ sX are the most complex matrix multiplications ∇YTX^ and ∇ Y Ŵ, which, however, have already been calculated in previous calculations, so no additional calculation is required. The element-wise multiplication by δ W (s W ) and δ X (s X ) requires only minor calculations.

[0074] Furthermore, according to some embodiments of the present disclosure, the determination of whether or not to begin optimizing each of the first and second step sizes is made when the rate of change of the respective step size over one or more iterations in the training process is below a predetermined threshold, or when a predetermined number of epochs have been completed in the training process. This effectively avoids the cold start problem that occurs when the neural network is trained from scratch.

[0075] The mechanism proposed in this disclosure combines quantization techniques for forward and backpropagation and uses INT4 matrix multiplications for all linear operations in deep neural networks, such as transformers. The mechanism described herein is evaluated for training transformers for a wide variety of tasks, including natural language understanding, question answering, machine translation, image classification, etc., and achieves competitive or superior accuracy compared to existing work.Training deep neural networks at an extremely low 4-bit level using the mechanism disclosed herein can effectively reduce the need for computer hardware resources, including but not limited to processing units, registers, memory, and storage, compared to solutions with costly high-precision representation and computation, thereby significantly improving computational efficiency and reducing power consumption. Furthermore, the mechanism disclosed herein is compatible with modern hardware, such as GPUs, as it does not require the custom numerical formats needed for existing applications. Moreover, a prototype implementation of the mechanism disclosed herein exhibits a higher speed, being 1.7 to 2.9 times faster than the baseline of FP16 matrix multiplication.

[0076] Now, with reference to Fig.Figure 7 illustrates a block diagram of an exemplary computing device 700 according to some embodiments of the present disclosure. As illustrated herein, the computing device 700 may comprise one or more processing units 710 and a memory 720. The one or more processing units 710 may include any type of general-purpose processing unit / core (for example, without being limited to CPU, GPU) or application-specific processing units, cores, circuits, controllers, or the like. The memory 720 coupled to the one or more processing units 710 may include any type of medium that can be used to store data.Memory 720 is configured to store instructions which, when executed, cause one or more processing units 710 to perform operations of a procedure described herein, including, but not limited to, CPU, Exemplary Procedure 400, Exemplary Process 500 and Exemplary Process 600.

[0077] Various embodiments described herein may include or operate on a number of components, elements, units, modules, instances, or mechanisms that may be implemented using hardware, software, firmware, or any combination thereof. Examples of hardware may include, but are not limited to, devices, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and the like), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), memory units, logic gates, registers, semiconductor devices, chips, microchips, chipsets, and the like.Examples of software may include, but are not limited to, software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, software modules, routines, subroutines, functions, procedures, processes, software interfaces, application programming interfaces (APIs), instruction sets, computer code, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware, software, and / or firmware may vary according to a number of factors, such as desired computing speed, performance levels, thermal tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints, as required for a particular embodiment.

[0078] Some embodiments described herein may comprise a product. A product may comprise a storage medium. Examples of a storage medium may include volatile and non-volatile, removable and non-removable media implemented in a method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.The storage medium may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technologies, compact discs (CDs), digital versatile discs (DVDs) or other optical storage media, magnetic cassettes, magnetic tapes, magnetic disk storage media or other magnetic storage devices, and any other medium that can be used to store information. In some embodiments, a product may store executable computer program instructions which, when executed by one or more processing units, cause the processing units to perform the operations described herein.The executable computer program instructions can include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. The executable computer program instructions can be implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0079] Below are some examples of the present revelation described herein.

[0080] Example 1 may include a computer-implemented procedure for accelerating operations of a deep neural network, comprising: for each of a plurality of matrix multiplications during forward propagation, obtaining a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; obtaining a second matrix by multiplying a weighting matrix, one transpose of which is another input to the matrix multiplication, by the block-diagonal Hadamard transform matrix; quantizing the first matrix and the second matrix into 4-bit integer matrices; obtaining a resultant matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix, and dequantizing the resultant matrix into a floating-point matrix.

[0081] Example 2 can include the subject of Example 1, with the procedure being carried out during a training process of the deep neural network.

[0082] Example 3 can include the subject of Example 1, where the block-diagonal Hadamard transform matrix is ​​a block-diagonal matrix consisting of several Hadamard transform matrices H k consists of, where k = 5.

[0083] Example 4 can include the subject of Example 1, where quantization includes: applying a clamp function to the first matrix and the second matrix, respectively, to restrict each of the elements in the first matrix and the second matrix to a range representable by 4-bit integers, where a first step size for the activation matrix and a second step size for the weighting matrix, each a parameter of the clamp function, are floating-point scalars that can be learned during a training process of the deep neural network.

[0084] Example 5 can include the subject of Example 4, where dequantization includes: multiplying the resulting matrix by the first step size and the second step size.

[0085] Example 6 can include the subject of Example 4, where determining whether to start optimizing the first step size and the second step size is done when the rate of change of the corresponding step size over one or more iterations in the training process is below a predetermined threshold or when a predetermined number of epochs have been trained in the training process.

[0086] Example 7 may include the subject of Example 1, wherein the procedure during backpropagation in a training process of the deep neural network further comprises: for computations relating to a gradient matrix representing a gradient of a loss function with respect to the output of the matrix multiplication, splitting the gradient matrix into a first 4-bit integer gradient matrix and a second 4-bit integer gradient matrix, wherein the first 4-bit integer gradient matrix and the second 4-bit integer gradient matrix represent higher 4 bits and lower 4 bits of the gradient matrix, respectively.

[0087] Example 8 can include the subject of Example 7, wherein the splitting includes: quantizing the gradient matrix into the first 4-bit integer gradient matrix associated with a first floating-point scalar; and quantizing the difference between the gradient matrix and a product of the first 4-bit integer gradient matrix and the first floating-point scalar into the second 4-bit integer gradient matrix associated with a second floating-point scalar.

[0088] Example 9 may include the subject of Example 8, wherein the procedure for calculating a weighting gradient, which involves a second matrix multiplication of a transpose of the gradient matrix with the quantized first matrix, further comprises: obtaining a third matrix by multiplying the transpose of the first 4-bit integer gradient matrix and the quantized first matrix; obtaining a fourth matrix by multiplying the transpose of the second 4-bit integer gradient matrix and the quantized first matrix; dequantizing the third matrix and the fourth matrix into matrices with floating-point representation; and obtaining a resulting fifth matrix by summing the dequantized third matrix and the dequantized fourth matrix.

[0089] Example 10 can include the subject of Example 9, wherein, when obtaining the third matrix and when obtaining the fourth matrix, rows in the first 4-bit integer gradient matrix and the second 4-bit integer gradient matrix are sampled, with random masks per row following the Bernoulli distribution with corresponding probabilities per row.

[0090] Example 11 can include the subject of Example 10, where each of the probabilities per row is proportional to a corresponding coefficient per row equal to a product of a first norm associated with the quantized gradient matrix and a second norm associated with the quantized first matrix.

[0091] Example 12 may include the subject of Example 7, wherein the procedure for calculating an activation gradient, which involves a third matrix multiplication of the gradient matrix with the quantized second matrix, further comprises: obtaining a sixth matrix by multiplying the first 4-bit integer gradient matrix and the quantized second matrix; obtaining a seventh matrix by multiplying the second 4-bit integer gradient matrix and the quantized second matrix; dequantizing the sixth matrix and the seventh matrix into matrices with floating-point representation; and obtaining a resulting eighth matrix by summing the dequantized sixth matrix and the dequantized seventh matrix.

[0092] Example 13 can include the subject of Example 12, wherein, when obtaining the sixth matrix and when obtaining the seventh matrix, rows in the first 4-bit integer gradient matrix and the second 4-bit integer gradient matrix are sampled, with random masks per row following the Bernoulli distribution with corresponding probabilities per row.

[0093] Example 14 can include the subject of Example 13, where each of the probabilities per row is proportional to a corresponding coefficient per row corresponding to a third norm associated with the quantized gradient matrix.

[0094] Example 15 can include the subject of Example 4, wherein during backpropagation in the training process of the deep neural network, the gradient of a loss function with respect to the first step size is determined based on the maximum of the range and the number of elements of the activation matrix, and wherein the gradient of the loss function with respect to the second step size is determined based on the maximum of the range and the number of elements of the weighting matrix.

[0095] Example 16 can include the subject of Example 1, where the deep neural network includes a transformer.

[0096] Example 17 can include the subject of Example 1, where the deep neural network is to be used for natural language processing, with the deep neural network taking as input vector representations of words generated based on word embedding.

[0097] Example 18 may include a computing device comprising: memory for storing instructions and one or more processing unit(s) coupled to the memory, wherein the instructions, when executed by the one or more processing unit(s), cause the one or more processing unit(s) to perform one of the methods disclosed herein.

[0098] Example 19 may include a non-volatile, computer-readable storage medium on which instructions are stored which, when executed by one or more processing units, cause one or more processing units to carry out one of the methods disclosed herein.

[0099] Example 20 may include a computer program product comprising instructions which, when executed by one or more processing units, cause the one or more processing units to perform one of the methods disclosed herein.

[0100] The foregoing includes examples of the disclosed architecture. It is naturally not possible to describe every conceivable combination of components and / or methods, but those skilled in the art can recognize that many further combinations and permutations are possible. Accordingly, the novel architecture is intended to include all changes, modifications, and variations that fall within the scope of the invention and the scope of protection of the accompanying claims.

Claims

[1] Computer-implemented method for accelerating operations of a deep neural network, comprising: For each of a multitude of matrix multiplications during forward propagation, obtaining a first matrix by multiplying an activation matrix, which is an input to the matrix multiplication, by a block-diagonal Hadamard transform matrix; Obtaining a second matrix by multiplying a weighting matrix, one transpose of which is another input to the matrix multiplication, with the block-diagonal Hadamard transformation matrix; Quantize the first matrix and the second matrix into 4-bit integer matrices; obtain a resulting matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix and Dequantize the resulting matrix into a matrix with floating-point representation. [2] Computer-implemented method according to claim 1, wherein the method is to be carried out during a training process of the deep neural network. [3] Computer-implemented method according to claim 1, wherein the block-diagonal Hadamard transform matrix is ​​a block-diagonal matrix composed of several Hadamard transform matrices H k consists of, where k = 5. [4] Computer-implemented method according to claim 1, wherein quantizing includes: Applying a clamp function to the first matrix and the second matrix, respectively, to restrict each of the elements in the first matrix and the second matrix to a range representable by 4-bit integers, where a first step size for the activation matrix and a second step size for the weighting matrix, each a parameter of the clamp function, are floating-point scalars that can be learned during a training process of the deep neural network. [5] Computer-implemented method according to claim 4, wherein dequantizing includes: multiplying the resulting matrix by the first step size and the second step size. [6] Computer-implemented method according to claim 4, wherein the determination of whether to start optimizing the first step size and the second step size is made when the rate of change of the corresponding step size over one or more iterations in the training process is below a predetermined threshold or when a predetermined number of epochs have been trained in the training process. [7] Computer-implemented method according to claim 1, further comprising during backpropagation in a training process of the deep neural network: for calculations related to a gradient matrix representing the gradient of a loss function with respect to the output of matrix multiplication, Splitting the gradient matrix into a first 4-bit integer gradient matrix and a second 4-bit integer gradient matrix, where the first 4-bit integer gradient matrix and the second 4-bit integer gradient matrix represent higher 4 bits and lower 4 bits of the gradient matrix, respectively. [8] Computer-implemented method according to claim 7, wherein the splitting includes: Quantizing the gradient matrix into the first 4-bit integer gradient matrix that is mapped to a first floating-point scalar; and Quantizing the difference between the gradient matrix and a product of the first 4-bit integer gradient matrix and the first floating-point scalar into the second 4-bit integer gradient matrix, which is mapped to a second floating-point scalar. [9] Computer-implemented method according to claim 8, further comprising for calculating a weighting gradient which includes a second matrix multiplication of a transpose of the gradient matrix with the quantized first matrix: Obtaining a third matrix by multiplying the transpose of the first 4-bit integer gradient matrix and the quantized first matrix; Obtaining a fourth matrix by multiplying the transpose of the second 4-bit integer gradient matrix and the quantized first matrix; Dequantizing the third matrix and the fourth matrix into matrices with floating-point representation and Obtaining a resulting fifth matrix by summing the dequantized third matrix and the dequantized fourth matrix. [10] Computer-implemented method according to claim 9, wherein, when obtaining the third matrix and when obtaining the fourth matrix, rows in the first 4-bit integer gradient matrix and the second 4-bit integer gradient matrix are sampled, with random masks per row following the Bernoulli distribution with corresponding probabilities per row. [11] Computer-implemented method according to claim 10, wherein each of the probabilities per row is proportional to a corresponding coefficient per row, which is equal to a product of a first norm associated with the quantized gradient matrix and a second norm associated with the quantized first matrix. [12] Computer-implemented method according to claim 7, further comprising for calculating an activation gradient which includes a third matrix multiplication of the gradient matrix with the quantized second matrix, Obtaining a sixth matrix by multiplying the first 4-bit integer gradient matrix and the quantized second matrix; Obtaining a seventh matrix by multiplying the second 4-bit integer gradient matrix and the quantized second matrix; Dequantizing the sixth and seventh matrices into matrices with floating-point representation and Obtaining a resulting eighth matrix by summing the dequantized sixth matrix and the dequantized seventh matrix. [13] Computer-implemented method according to claim 12, wherein, when obtaining the sixth matrix and when obtaining the seventh matrix, rows in the first 4-bit integer gradient matrix and the second 4-bit integer gradient matrix are sampled, with random masks per row following the Bernoulli distribution with corresponding probabilities per row. [14] Computer-implemented method according to claim 13, wherein each of the probabilities per row is proportional to a corresponding coefficient per row corresponding to a third norm associated with the quantized gradient matrix. [15] Computer-implemented method according to claim 4, wherein during backpropagation in the training process of the deep neural network the gradient of a loss function with respect to the first step size is determined based on the maximum of the range and the number of elements of the activation matrix and wherein the gradient of the loss function with respect to the second step size is determined based on the maximum of the range and the number of elements of the weighting matrix. [16] Computer-implemented method according to claim 1, wherein the deep neural network includes a transformer. [17] Computer-implemented method according to claim 1, wherein the deep neural network is to be used for natural language processing, wherein the deep neural network uses as input vector representations for words that were generated based on word embedding. [18] Computing device comprising: Memory for storing instructions and one or more processing unit(s) coupled with the memory, wherein the instructions, when executed by the one or more processing unit(s), cause the one or more processing unit(s) to carry out the method according to any one of claims 1 to 17. [19] Non-volatile, computer-readable storage medium on which instructions are stored which, when executed by one or more processing units, cause the one or more processing units to carry out the method according to any one of claims 1 to 17. [20] Computer program product comprising instructions which, when executed by one or more processing units, cause the one or more processing units to carry out the method according to any one of claims 1 to 17.

Images