Sparse training methods for pre-trained language models and deep language computing systems

Abstract

A sparse training method and deep language computing system for pre-trained language models are disclosed. The method includes: feeding language training samples from the current batch of the target task into the model and calculating the gradient of the weight matrix W, where the weight matrix W has a dimension of n×k; based on the weight matrix W and the gradient, calculating a low-rank importance row matrix A and a low-rank importance column matrix B, where the low-rank importance row matrix A has a dimension of n×r1, the low-rank importance column matrix B has a dimension of r1×k, and the product of A and B is equivalent to a weight importance matrix S of dimension n×k; and constructing a sparse mask M based on the product of A and B to reset the weights at corresponding positions in the weight matrix W to zero. This invention utilizes the low-rank property of the weight matrix and gradient matrix, using a set of low-rank matrices to characterize the importance of each weight in the weight matrix based on the training data, thereby significantly reducing the number of parameters involved in sparse training and reducing the resource overhead of sparse training.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and in particular to a sparse training method for a pre-trained language model and a deep language computing system. Background Technology

[0002] In recent years, an increasing number of large-scale models have been proposed, with parameters ranging from hundreds of millions to tens of billions, and even reaching trillions or hundreds of billions. These models require a large amount of hardware resources for training and deployment, making them difficult to apply in real-world scenarios. Therefore, model sparsification, which removes redundant parameters and retains only those most relevant to the inference results, has become a research direction for reducing the size of model deployments.

[0003] However, sparse training of a model directly consumes more training resources than conventional models due to the size of the parameters to be sparsed and the additional parameters introduced by sparse training. Therefore, how to perform sparse training on models, especially large models, with fewer hardware resources has become a key problem to be solved in this field.

[0004] Therefore, an improved method for sparse model training is needed. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a parameter-efficient sparse training method. By utilizing the low-rank property of the weight matrix and gradient matrix, a set of low-rank matrices is used to characterize the importance of each weight in the weight matrix, thereby greatly reducing the number of parameters involved in sparse training and reducing the resource overhead of sparse training.

[0006] According to a first aspect of this disclosure, a sparse training method for a pre-trained language model is provided, comprising: feeding language training samples of the current batch for a target task into the model and calculating the gradient of the weight matrix W, wherein the dimension of the weight matrix W is... Based on the weight matrix W and the gradient, the low-rank row matrix A and the low-rank column matrix B of importance are obtained, wherein the dimension of the low-rank row matrix A is... The dimension of the importance low-rank column matrix B is... And the product of A and B is equivalent to a product of dimension . The weight importance matrix S; and the sparse mask M constructed based on the product of A and B to reset the weights at the corresponding positions of the weight matrix W to zero.

[0007] Optionally, the method further includes: calculating the importance structure row matrix R and the importance structure column matrix C based on the weight matrix W and the gradient, wherein the dimension of the importance structure row matrix R is... The dimension of the importance structure column matrix C is... And used to represent the importance of a single row and a single column of the weight matrix W, wherein the product of A and B and the sum of R and C are equivalent to the weight importance matrix S, and the sparse mask M is constructed based on the product of A and B and the sum of R and C.

[0008] Optionally, the method further includes: constructing the sparse mask M based on the product of A and B accumulated in each batch of the target task and the sum of R and C.

[0009] Optionally, the method further includes: constructing the sparse mask M based at least in part on the weight values ​​at each position of the weight matrix W.

[0010] Optionally, the method further includes: an initial weight matrix W based on the pre-trained model. (0) Given the gradient, calculate the weighted low-rank row matrix U and the weighted low-rank column matrix V, where the dimension of the weighted low-rank row matrix U is... The dimension of the weighted low-rank column matrix V is And the product of U and V with W (0) The sum is equivalent to the weight matrix W.

[0011] Optionally, the method further includes: performing sparsity training for the target task based on the following loss function:

[0012]

[0013] Where v refers to the number of weights to be retained, and p refers to the target sparsity. The target task is represented by the language training sample set, β, α1 and α2 are hyperparameters, and the function f(S, v) represents the function used to find the largest v values ​​in S to generate a binary mask M for sparsification.

[0014] According to a second aspect of this disclosure, a deep learning computing system is provided for performing a sparse training method for a pre-trained language model according to the first aspect for a target task.

[0015] Optionally, during the sparse training of the pre-trained language model for the target task, a corresponding... The memory space for each parameter is used to store the trainable parameters.

[0016] Optionally, after the sparse training, inference is performed for the target task using the following formula: ,in, It is sparse, and .

[0017] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0018] According to a fourth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0019] Therefore, the parameter-efficient sparse training method of the present invention calculates the importance of each weight by using a set of low-rank matrices and a set of structured matrices, which not only reduces the amount of computation and parameters, but also achieves efficient acceleration of sparse neural networks. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0021] Figure 1A -B shows an example of sparse training using different weight importances.

[0022] Figure 2 A schematic flowchart of a sparse training method for a pre-trained language model according to an embodiment of the present invention is shown.

[0023] Figure 3 An example of sparse training using a set of low-rank matrices to represent the importance of weights is shown.

[0024] Figure 4 The distribution trends of parameters in binary mask M under various network structures are shown.

[0025] Figure 5 An example of sparse training using a set of low-rank matrices and a set of structure matrices to represent the importance of weights is shown.

[0026] Figure 6 An example of sparse training is shown, which uses the values ​​of the weight parameters and a set of low-rank and structural matrices to represent the importance of the weights.

[0027] Figure 7 A schematic diagram of a computing device is shown, which can be used to implement the sparse training method of the pre-trained language model described above, according to an embodiment of the present invention.

[0028] Figure 8 An application example of efficient sparse training of parameters according to the present invention is shown. Detailed Implementation

[0029] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0030] Many applications in natural language processing follow this paradigm: first, a large language model is pre-trained, and then it is fine-tuned for multiple downstream tasks. Despite great success, large language models with millions or even billions of parameters require significant memory and computing power (e.g., GPU memory and computing power when using heterogeneous processors such as GPUs) for fine-tuning downstream datasets or during inference, which greatly limits the application of language models in many tasks.

[0031] It's important to understand that deep learning models require training before they can perform correct reasoning. The training process doesn't change the structure of the neural network model, but it does update the parameters of each layer. That is, by importing a large amount of training data, the parameters of each layer of the neural network model, such as weights and biases, are determined.

[0032] Training the model means learning (determining) ideal values ​​for all weights and biases using labeled samples. These determined weights and biases then enable high-accuracy inference on the input feature values ​​during the neural network deployment phase.

[0033] In supervised learning, machine learning algorithms learn parameters by examining multiple samples and trying to find a model that minimizes loss; this process is called empirical risk minimization.

[0034] Loss is a penalty for poor predictions. That is, loss can be a numerical value representing how accurately the model predicts for a single sample. If the model's predictions are perfectly accurate, the loss is zero; otherwise, the loss will be large. The goal of training a model is to find a set of weights and biases with a "small" average loss from all samples.

[0035] During the training of a neural network, a loss function needs to be defined to quantify whether the current weights and biases can allow the network input to fit all network inputs. Therefore, the goal of training the network can be transformed into minimizing the loss function of the weights and biases. Typically, gradient descent (and backpropagation in multi-layer neural network training) is used to achieve this minimization.

[0036] The backpropagation algorithm involves repeated iterative processes of forward and backward propagation. Forward propagation involves neurons in different layers being connected through weight matrices, allowing stimuli (feature values) to be continuously transmitted from one layer to the next via the activation function of each layer. In backpropagation, the error of the current layer is derived from the error of the next layer. Through these iterative processes of forward and backward propagation, the weights and biases are continuously adjusted until the loss function gradually approaches its minimum, thus completing the training of the neural network.

[0037] To alleviate the computational and memory burden in language model inference, parameters can be pruned to remove unimportant weights, channels, and layers, thereby reducing computational and memory overhead. Common sparsity methods can be divided into structured sparsity and unstructured sparsity. Structured sparsity prunes convolutions and matrix multiplications along a specific dimension (feature channels, convolution kernels, etc.) to generate a smaller model structure, allowing reuse of existing convolution and matrix multiplication calculations without the need for special inference operators. Unstructured sparsity sparsifies each parameter as a unit, without changing the shape of the parameter matrix, but transforming it into a sparse matrix containing a large number of zero values. In terms of effectiveness, structured sparsity can provide some speedup at lower sparsity levels, but often introduces significant accuracy loss at high sparsity levels. Unstructured sparsity, on the other hand, can achieve several times the inference speedup at high sparsity levels with minimal accuracy loss. Therefore, this invention focuses on optimizing unstructured sparsity, which performs well in both inference acceleration and accuracy preservation.

[0038] During the training process of a neural network, the weight matrix can be defined as follows: When sparse training of a network is required, an importance parameter needs to be introduced. Used to determine the weight matrix Which weights need to be removed? Based on the importance parameter. It can be used for calculation Generate a binary mask, where It is the output of the current layer. The input of the current layer. and These are all activation values ​​passed from the input data in the model. The Hadamard product, also known as the basic product, is the operation of multiplying corresponding terms of two matrices of the same order to obtain another matrix of the same order. A common sparsity reduction method is based on the importance parameter. Find the first v parameters (the value of v determines the sparsity) and retain them, while setting the other parameters to zero. Therefore, a function f(S,v) can be defined to find the largest v values ​​in S to generate a binary mask M for sparsification:

[0039] (1)

[0040] To mitigate the impact of sparse training on model accuracy, iterative sparse training is required, that is, gradually removing unimportant parameters and updating important parameters. (Here, "stepwise" can refer to the process of removing unimportant parameters and updating important parameters layer by layer in each batch of computation within each epoch, based on the backpropagation algorithm.) ).

[0041] Therefore, the standard optimization formula for sparse training of language models is as follows:

[0042] (2)

[0043] Where W is the model weight, S is the weight importance, v is the number of weights to retain, n and k represent the number of rows and columns of the model weights, respectively, and p is the objective sparsity. Represents the training dataset. This represents the loss function used in network training.

[0044] Sparse training involves optimizing the weights W and their importance S. Under the constraint of the target sparsity, the v most important weights (i.e., top-v weights) are determined to be retained based on their importance. In other words, the remaining parameters are set to zero using a binary mask M.

[0045] Various methods can be used to calculate the weight importance S, and sparse training can be performed accordingly. Figure 1A -B illustrates an example of sparse training using different weight importance. Figure 1A and 1B In the example, the white-boxed matrix represents activation values, that is, activation values ​​that flow between different layers of the neural network, such as those input to a specific layer. and the output after calculation by this layer The gray boxes represent trainable parameters. Figure 1A In the example, the model weights W are trainable parameters; while Figure 1B In the example, besides the model weights W, the weight importance S is also a trainable parameter. The black and white boxes correspond to the mask matrix M, which includes 0s and 1s.

[0046] Specifically, Figure 1AAn example of model training based on the weight importance determined by the properties of the model weights themselves is shown. Figure 1A In this example, the determination of weight importance does not take into account the specific training dataset used for model training. It is data-free. In this case, there is no need to construct a separate weight importance matrix S. Instead, we can directly calculate the absolute value of each parameter in the current layer model weight matrix W, select the top v parameters with the largest absolute values ​​(i.e., the top-v weights), and obtain the binary mask M (i.e., the positions corresponding to the top-v weights are set to 1, and the rest are set to 0). Corresponding to the sparsified model weights, This corresponds to the layer output obtained by sparsening the model weights of the current layer input X.

[0047] Figure 1B This illustrates an example where a separate weight importance matrix S needs to be constructed for model training. Here, the weight importance S cannot be derived from the values ​​of the weights themselves, but is determined based on the dataset used for training. For example, based on different datasets, the parameters in the weight matrix that retain their influence after calculation across multiple layers are not the same. In this case, a data-driven weight importance matrix S needs to be constructed in addition to the weight matrix W. As shown in the figure, the dimension of the weight importance matrix S is the same as that of the weight matrix W, and each parameter can be used to represent the importance of the weight parameter at the corresponding position. After obtaining the weight importance matrix S of the current layer in the current batch of training based on different rules, the top v parameters with the largest absolute values ​​(i.e., the top-v weights) can be selected, thus obtaining the binary mask M (i.e., the positions corresponding to the top-v weights are set to 1, and the rest are set to 0). Corresponding to the sparsified model weights, This corresponds to the layer output obtained by sparsening the model weights of the current layer input X.

[0048] When training a pre-trained language model for downstream tasks, the parameters of importance in the weight matrix W differ depending on the type of data samples. Therefore, a dedicated data-driven weight importance matrix S needs to be constructed in sparse training to represent the importance of each parameter in the weight matrix W, thereby preserving the model performance as much as possible after sparse training. Thus, the weight importance parameter matrix S can be defined as:

[0049] (3)

[0050] in, This represents the weight importance driven by the data at step t. Here, step t can refer to a specific batch of data in an iteration. If the sparse training for the current task includes 10 iterations, with 8 data batches in each iteration, then completing the sparse training requires 80 steps, where t ranges from 1 to 80. In one embodiment, The value can be That is, the importance score of a parameter is represented by the product of a parameter and its gradient, and a dedicated memory space (the same as the space occupied by W) is required to store the weight importance matrix S, which is used to accumulate the importance scores of each parameter in each step.

[0051] As mentioned earlier, with the improvement of language model performance, the number of model parameters can reach hundreds of millions or even tens of billions. Although a well-trained sparse model can greatly reduce the number of parameters, sparse training itself consumes more training resources than conventional models due to the scale of the parameters to be sparsed and the additional parameters introduced by sparse training. Maintaining a weight importance matrix S (with the same dimension as the weight matrix W) as shown in Equation (3) will greatly increase the training overhead in the sparse model training process. Therefore, how to perform sparse training on models, especially large models, with less hardware resources has become a key problem that needs to be solved in this field.

[0052] To address this, the present invention proposes a parameter-efficient sparse training method. This method utilizes the low-rank property of the weight matrix and gradient matrix, and uses a set of low-rank matrices to characterize the importance of each weight in the weight matrix, thereby greatly reducing the number of parameters involved in sparse training and reducing the resource overhead of sparse training.

[0053] Figure 2 A schematic flowchart of a sparse training method for a pre-trained language model according to an embodiment of the present invention is shown.

[0054] In step S210, the language training samples of the current batch for the target task are fed into the model, and the gradient of the weight matrix W is calculated. Here, the dimension of the weight matrix W can be... When performing sparse training on large pre-trained language models, The value of can be as high as millions or even hundreds of millions, thus occupying a large amount of memory (e.g., video memory). The gradient obtained can be expressed as... The gradient described above can be used to represent the transformation direction of each weight parameter in the weight matrix W, therefore the dimension is also... .

[0055] In step S220, based on the weight matrix W and the gradient, the low-rank row matrix A and the low-rank column matrix B of importance are obtained. The dimension of the low-rank row matrix A is... The dimension of the importance low-rank column matrix B is... And the product of A and B is equivalent to a product of dimension . The weight importance matrix S. Here, low rank is the low rank relative to the dimension of the weight importance matrix S.

[0056] In other words, in step S220, a set of low-rank matrices is used to replace the weight importance matrix S. As mentioned earlier, And in one embodiment, The value can be That is, the importance score of a parameter is represented by the product of a parameter and its gradient. In other words, if we are currently at step t of sparse training, we can accumulate the sum of the fundamental products of the weights and gradients of each step from step 1 to the current step t to obtain the weight importance matrix S of the current step (i.e., the current batch in the current iteration of the target task). In the prior art, dedicated memory space (the same as the space occupied by W) is required to store the weight importance matrix S, which is used to accumulate the importance scores of each parameter in each step. The sparse training method of the present invention can utilize the low-rank property of the matrix to use low-rank matrices A and B instead of storing matrix S.

[0057] Given W and The rank mean is very small; in other words, W and The amount of information contained in each is actually unnecessary. This is represented by several parameters. Furthermore, based on background knowledge of the matrix rank, we know that... That is, multiplying a term that is zero by its non-zero term results in 0, thus further reducing the information content of the fundamental product matrix of two matrices of the same order, each containing a large number of zeros. Therefore, the sparsification scheme of this invention can use a set of low-rank matrices to fit the original importance index based on the data part, that is, it can introduce... Here, r1 is a hyperparameter used to control the size of this set of low-rank matrices. As mentioned earlier, the low rank of A and B is relative to the dimension of the weight importance matrix S; in other words, the range of values ​​for the variable parameter r1 is much smaller than the range of values ​​for n and k. For example, in one embodiment, r1 can be 8. To ensure that the data-based importance score for each weight is the same at the beginning, Gaussian initialization can be used to initialize A, zero initialization to initialize B, and optimization can be performed directly based on stochastic gradient descent (SGD). In one embodiment, In this G A Let λ represent the gradient of A calculated based on the network loss function, and λ represent the learning rate. A similar method can be used to calculate B.

[0058] After obtaining a set of low-rank matrices A and B that are equivalent to the weight importance matrix S, in step S230, a sparse mask M can be constructed based on the product of A and B to reset the weights at the corresponding positions of the weight matrix W to zero.

[0059] Specifically, according to the previous equation (1), the product of A and B equals S, therefore, the constructed value in the product of A and B can be obtained. In the matrix, the v largest data points are selected, and a sparse mask M is constructed from them. This binary sparse mask M can then be multiplied by the weight matrix W (i.e., ...). Implement the sparse operation in step t of this current operation.

[0060] In specific sparse training, the above steps can be performed batch by batch. After training a predetermined batch of data, the next iteration can be performed, and training can be completed after a predetermined number of iterations (or after the model accuracy reaches a predetermined requirement). In each of the above steps, the results can be obtained by accumulating the values. (at this time, The value of v is gradually reduced so that the predetermined sparsity is achieved after a predetermined number of iterations (which also corresponds to a predetermined number of steps).

[0061] Therefore, the sparse training method of this invention utilizes the low rank of the weight matrix and gradient matrix, and utilizes dimensions respectively... and The low-rank matrices A and B replace the dimension as The weight importance matrix S is used to construct the sparse mask M and set the corresponding weights to zero, thereby reducing the number of weight importance parameters that need to be maintained in memory during sparse training and reducing the resource overhead of sparse training.

[0062] Figure 3 An example of sparse training using a set of low-rank matrices to represent weight importance is shown. Figure 1A and Figure 1B similar, Figure 3 The white-box matrix still represents activation values, gray-boxes represent trainable parameters, and black-and-white boxes correspond to a mask matrix M containing 0s and 1s. Similar to... Figure 1B , Figure 3 It also calculates the importance of each weight parameter based on the data by constructing an additional matrix. But compared to Figure 1B The difference is, Figure 3 Use two low-rank matrices A and B to replace the original matrix of dimension 1. We can obtain the weight importance matrix S and construct a sparse binary mask M with approximate accuracy. Thus, by introducing a set of low-rank matrices, we can significantly reduce the memory space required to compute data-based weight importance.

[0063] Furthermore, the inventors of this application have discovered that although unstructured sparse training can remove weight parameters at any position in the weight matrix without restriction, i.e., the distribution of the sparse result corresponding to the binary mask M is uncontrolled, however, the data-driven weight importance matrix S (e.g., based on...) The binary mask M generated by the weights (obtained) has a clear structural distribution. Figure 4 The distribution trends of parameters in the binary mask M under various network structures are shown. The left figures in sections (a)-(d) (corresponding to the attention query layer, attention output layer, FFN input layer, and FFN output layer, respectively) show a visualization of the binary mask M of BERT on the SST-2 dataset with a sparsity of 90%. FFN stands for Feed-Forward Network. As shown, each left figure illustrates a binary mask M exhibiting a row or column distribution pattern. To quantify this phenomenon, the inventors calculated the sparsity of each row / column in the binary mask M and obtained the histograms of row (bottom right) and column (top) sparsity distribution shown on the right side of sections (a)-(d) by dividing the sparsity into multiple intervals and analyzing the number of columns and rows falling into the corresponding intervals. Here, the x-axis represents the sparsity, and the y-axis represents the number of rows or columns falling into the corresponding sparsity interval. For example, the left chart in section (a) shows many rows with very few remaining weights, while the right chart shows that all weights in approximately 250 rows have been removed. In contrast, section (b) shows that most columns have extremely high sparsity.

[0064] Based on the above observations, the inventors discovered that the row and column distributions of the weights in the weight importance matrix S exhibit significant differences. Therefore, in a preferred embodiment, the sparse training method of the present invention may further include: calculating the importance structure row matrix R and the importance structure column matrix C based on the weight matrix W and the gradient, wherein the dimension of the importance structure row matrix R is... The dimension of the importance structure column matrix C is... And it is used to represent the importance of a certain row and a certain column in the weight matrix W.

[0065] In other words, the sparsification scheme of this invention can introduce an additional set of structured matrices, namely... This is used to characterize the structure within the importance metrics. In one embodiment, this set of structured matrices can be obtained as follows:

[0066] (4)

[0067] At this point, the product of A and B, summed with R and C, is equivalent to the weight importance matrix S, and the sparse mask M is constructed based on the product of A and B, summed with R and C. In other words, the data-driven importance score can be... In a preferred embodiment, the importance score can be modified as follows:

[0068] (5)

[0069] Here, α1 and α2 are hyperparameters used to trade off between low rank and structural importance score. In one embodiment, α1 and α2 can each be equal to 1.

[0070] Figure 5 An example of sparse training using a set of low-rank matrices and a set of structure matrices to represent weight importance is shown. Figure 1A and Figure 1B and Figure 3 similar, Figure 5 The white-box matrix still represents activation values, gray-boxes represent trainable parameters, and black-and-white boxes correspond to a mask matrix M containing 0s and 1s. Similar to... Figure 3 , Figure 5 It also includes a set of low-rank matrices A and B. However, compared to... Figure 3 The difference is, Figure 5 An additional set of structured matrices R and C is included, thereby representing the importance of the weights from the structural distribution of rows and columns. In this case, two low-rank matrices A and B, along with two structured matrices R and C, are used to replace the original dimensionless matrix. The system can calculate the weight importance matrix S and construct a sparse binary mask M with more accurate approximation. This significantly reduces the memory space required to compute data-based weight importance while maintaining accuracy.

[0071] Here, step t can refer to a data batch in a certain iteration. If the sparse training for the current task includes 10 iterations, and each iteration includes 8 data batches, then 80 steps are required to complete the sparse training, and the value of t is 1 to 80. Therefore, as shown in equation (5) above, the sparse mask M can be constructed based on the product of A and B accumulated in each batch of the target task and the sum of R and C (for example, according to equation (1)).

[0072] In the example above, That is, the importance of a weight parameter is determined solely by its data-based importance. However, in practice, the value of the weight parameter itself can also affect the importance of the weight. Therefore, in a preferred embodiment of the present invention, a new formula for calculating the importance score can be proposed: , where represent the data-independent weight importance and the data-based (i.e., data-driven) weight importance at step t, respectively. In this case, the sparse training method of the present invention may further include: constructing the sparse mask M based at least partially on the weight values ​​at each position of the weight matrix W.

[0073] In one embodiment, the definition of the weight importance parameter matrix S can be modified based on equation (3) as follows:

[0074] (6)

[0075] Here, the importance of the weights is decomposed into a weight-based component |W (t) | and data-based components . Specifically, |W (t) | represents the L1 norm value of the weight, corresponding to the weight-based part (i.e., the absolute value of each weight parameter). This represents the cumulative result of multiplying the gradients of the weights in the previous t steps by their respective weights, corresponding to the data-based part. (t) represents the importance of the weights at step t of model training. Furthermore, a hyperparameter α can be introduced to compromise between data-independent and data-based weight importance scores. In one embodiment, α can be equal to 1. Due to the data-independent weight importance score |W (t) | can be directly derived from the weight matrix without additional parameters, thus avoiding the need to construct extra parameters and being resource-efficient. For the data-based part Originally requiring a matrix of the same size as the weights for storage, it is optimized using the sparsification scheme of the present invention to the form shown in equation (5) above. .

[0076] Figure 6 This illustrates an example of sparse training using weight parameter values ​​and a set of low-rank and structure matrices to represent weight importance. Figure 1A-1B , Figure 3 and Figure 5 similar, Figure 6 The white-box matrix still represents activation values, gray-boxes represent trainable parameters, and black-and-white boxes correspond to a mask matrix M containing 0s and 1s. Similar to... Figure 5 , Figure 6 It also includes a set of low-rank matrices A and B and a set of structured matrices R and C, thereby representing the importance of weights from the structural distribution of rows and columns. Furthermore, Figure 6 It also determines the importance of the weights at each position based on weight importance scores (i.e., the absolute value of each weight parameter itself), which are independent of the training data. Since |W (t) The | Calculation does not require the construction of an additional matrix, thus significantly reducing the memory space required to calculate the importance of data-based weights while maintaining greater accuracy.

[0077] Meanwhile, to further reduce the parameters of sparse training of the model, in one embodiment, the sparsification scheme of the present invention can also use a set of low-rank matrices for updating the weights. Therefore, the sparse training method of the present invention may include: based on the initial weight matrix W of the pre-trained model. (0) Given the gradient, calculate the weighted low-rank row matrix U and the weighted low-rank column matrix V, where the dimension of the weighted low-rank row matrix U is... The dimension of the weighted low-rank column matrix V is And the product of U and V with W (0) The sum is equivalent to the weight matrix W.

[0078] Here, the initial weight matrix W (0) This can be the initial weights of a pre-trained language model before performing the target task. In subsequent operations, it can be used through W... (t) = W (0) +βU (t) V (t) The weight matrix W is decomposed. Similarly, here, β is a hyperparameter used to compromise between the initial weights and the weight update. In one embodiment, β can be equal to 1. As mentioned earlier, since the gradient matrix is ​​low-rank, the update of the weight matrix can be performed using a set of low-rank matrices. In other words, the low rank of U and V is relative to the dimension of the weight matrix W; that is, the range of values ​​for the variable parameter r2 is much smaller than the range of values ​​for n and k. For example, in one embodiment, r2 can also be 8.

[0079] Therefore, the weight importance score obtained based on equation (6) can ultimately be replaced by:

[0080] (7)

[0081] In this way, sparse training can optimize the six small matrices using stochastic gradient descent. This achieves parameter efficiency and reduces training overhead.

[0082] Therefore, sparsity training for the target task can be performed based on the following loss function:

[0083]

[0084] Where v refers to the number of weights to be retained, and p refers to the target sparsity. The target task is represented by the language training sample set, β, α1 and α2 are hyperparameters, and the function f(S, v) represents the function used to find the largest v values ​​in S to generate a binary mask M for sparsification.

[0085] Accordingly, the calculation for each layer based on equation (7) becomes:

[0086] (9)

[0087] In other words, after optimization, all weights are fixed, and the reasoning process for the target task can be... ,in, It is sparse, and Therefore, the reasoning process will also be efficient in terms of parameters and resources.

[0088] In one embodiment, the present invention can also be implemented as a deep learning computing system for performing a sparse training method based on a pre-trained language model as described above for a target task. During the sparse training of the pre-trained language model for the target task, assignments are made corresponding to... The memory space for each parameter is used to store trainable parameters. By reducing the number of parameters involved in training, the sparsity scheme of this invention can effectively reduce the training overhead of sparse training. On mainstream natural language processing models, this scheme only requires 1.5% of the original training parameters to achieve a sparse model with comparable accuracy to the original sparse training.

[0089] Following the sparse training, the system can also perform inference for the target task using the following formula:

[0090] ,in, It is sparse, and .

[0091] This enables efficient subsequent inference with minimal parameters and resources.

[0092] Figure 7 A schematic diagram of a computing device is shown, which can be used to implement the sparse training method of the pre-trained language model described above, according to an embodiment of the present invention.

[0093] See Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0094] Processor 720 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 720 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0095] Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0096] The memory 710 stores executable code, which, when processed by the processor 720, enables the processor 720 to execute the sparse training method of the pre-trained language model described above.

[0097] Figure 8 An application example of efficient sparse training of parameters according to the present invention is shown. Similarly, Figure 8The white-box matrix still represents activation values, gray boxes represent trainable parameters, and black and white boxes correspond to a mask matrix M containing 0s and 1s. The difference is that... Figure 8 It also includes a fixed parameter matrix represented by a slash matrix, i.e., W shown in the figure, which can correspond to the initial weight matrix W. (0) In the sparse target training scenario of the pre-trained language model of this invention, the fixed parameter matrix can be the pre-trained model BERT. Here, the pre-trained language model BERT includes massive weight parameters, for example, millions or even hundreds of millions of parameters, and contains a large amount of implicit linguistic knowledge. Then, for a target task (e.g., understanding the intent of a user's interaction with a chatbot on a shopping website), a target task training dataset (e.g., manually labeled user communication corpus) needs to be prepared, and the parameters of the language model based on this invention are efficiently and sparsely trained based on the training dataset. During this training process, the language model BERT can learn specialized knowledge for the target task and sparsify at a certain rate.

[0098] Specifically, in each step (corresponding to the training of each batch of data), the gradient with respect to the existing weights can be calculated, and a set of low-rank matrices U and V can be constructed to represent the gradients compared to the initial weight matrix W. (0) The increment, and based on W (t) = W (0) +βU (t) V (t) The weight matrix is ​​represented by this. Then, it can be determined according to... This is used to replace the calculation of the smaller matrix. For example, to calculate |W (0) +βU (t) V (t) |As a data-independent weighted importance score, calculate α1 A (t) B (t) + α2(R (t) +C (t) The weights are used as the weight importance scores based on the data, and the parameters of the top-v are selected to construct the binary mask M, thereby completing the training and sparsification of each layer.

[0099] The sparse training method for the pre-trained language model and the deep language computing system according to the present invention have been described in detail above with reference to the accompanying drawings. Previous sparse training methods required a matrix of the same size as the weights to calculate the importance of the weights. In contrast, the sparsification scheme of the present invention proposes a parameter-efficient sparse training method that calculates the importance of each weight using a set of low-rank matrices and a set of structured matrices, thereby significantly reducing the number of parameters involved in sparse training and reducing the resource overhead of sparse training.

[0100] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0101] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0102] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0104] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A sparse training method for a pre-trained language model, comprising: The current batch of language training samples for the target task is fed into the model, and the gradient of the weight matrix W is calculated, where the dimension of the weight matrix W is... The language training samples are from the SST-2 dataset. Based on the weight matrix W and the gradient, the low-rank row matrix A and the low-rank column matrix B of importance are obtained, wherein the dimension of the low-rank row matrix A is... The dimension of the importance low-rank column matrix B is... , The value range of is less than the value range of n and k; the importance low-rank row matrix A and the importance low-rank column matrix B are stored in memory; based on the weight matrix W and the gradient, the importance structure row matrix R and the importance structure column matrix C are obtained, wherein the dimension of the importance structure row matrix R is . The dimension of the importance structure column matrix C is... And it is used to represent the importance of a single row and a single column of the weight matrix W, where the product of A and B and the sum of R and C are equivalent to a dimension of The weight importance matrix S; A sparse mask M is constructed based on the product of A and B and the sum of R and C to reset the weights at the corresponding positions of the weight matrix W to zero.

2. The method according to claim 1, further comprising: The sparse mask M is constructed based on the product of A and B accumulated in each batch of the target task and the sum of R and C.

3. The method according to claim 2, further comprising: The sparse mask M is constructed based at least in part on the weight values ​​at each position of the weight matrix W.

4. The method according to claim 3, further comprising: The initial weight matrix W based on the pre-trained model (0) Given the gradient, calculate the weighted low-rank row matrix U and the weighted low-rank column matrix V, where the dimension of the weighted low-rank row matrix U is... The dimension of the weighted low-rank column matrix V is And the product of U and V with W (0) The sum is equivalent to the weight matrix W.

5. The method according to claim 1, further comprising: Based on the following loss function, sparsity training is performed for the target task: Where v refers to the number of weights to be retained, and p refers to the target sparsity. Let S represent the language training sample set for the target task, where β, α1, and α2 are hyperparameters, and the function f(S, v) represents the function used to find the largest v values ​​in S to generate a binary mask M for sparsification. It represents the Hadamardi (or Hadama) stack.

6. A deep learning computing system for performing a sparse training method for a pre-trained language model according to any one of claims 1-5 for a target task.

7. The system of claim 6, wherein, During the sparse training of the pre-trained language model for the target task, the corresponding... The memory space for each parameter is used to store the trainable parameters.

8. The system of claim 7, wherein, Following the sparse training, inference is performed for the target task using the following formula: ,in, It is sparse, and ,in, It represents the Hadamardi (or Hadama) stack.

9. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-5.

10. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-5.

Images