System and method for fine tuning large language models (LLMS) using sparse embedded adapters via hierarchical approximate second-order information

Abstract

A system and method are provided for fine-tuning large language models (LLMs). The method provides high system performance during fine-tuning and attains state-of-the-art model quality on downstream applications. The method identifies the most important LLM weights via second-order information in a pre-processing step, and significantly reduces the computation and storage costs of the pre-processing step via i) a hierarchical approximation of second-order information, and ii) an online projection and rediagonalization algorithm. The method may train only the sparse important weights and embeds these sparse weights into the pre-trained LLM during fine-tuning to provide high system performance.

Description

SYSTEM AND METHOD FOR FINE TUNING LARGE LANGUAGE MODELS (LLMS) USING SPARSE EMBEDDED ADAPTERS VIA HIERARCHICAL APPROXIMATE SECOND-ORDER INFORMATION CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 655,134 filed on June 3, 2024, the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The following generally relates to fine tuning LLMs and, in particular, to fine tuning LLMs while maintaining performance and quality on downstream applications, for example, using sparse embedded adapters via hierarchical approximate second-order information. BACKGROUND

[0003] Large language models (LLMs) are deployed on various applications, e.g., machine translation [4, 54], code generation [6, 27 ], and text-to-image generation [2, 45]. Training an LLM from scratch for each application is typically prohibitively expensive since LLMs typically have billions of parameters [4, 53]. LLM training requires multiple accelerators with large memory capacities (e.g., NVIDIA A10080 GB) and ultra-high bandwidth interconnects (e.g., InfiniBand EDR 100Gb / s

[0035] ).

[0004] A more cost-efficient approach is fine-tuning for each downstream task after obtaining a pre-trained LLM, i.e., a general-purpose LLM trained on large corpora. With fine- tuning, the learned knowledge of the pre-trained LLM is transferred to downstream tasks by continuing to train the LLM on the task-specific datasets [41, 58].

[0005] The naive full fine-tuning approach (see FIG.1, option (a)) trains all weights of the pre-trained LLM and thus requires the same hardware capacities as pre-training, which are typically in high demand and expensive or difficult to access [16, 48]. Memory optimizations such as Zero Redundancy Optimizer (ZeRO) [42, 43] are proposed to reduce graphics processing unit (GPU) memory usage for full fine-tuning.

[0006] To make LLM fine-tuning more affordable, two directions have been explored: adapters [8, 10, 19, 20, 31, 46, 59] and pruners [12, 13, 18, 23, 24, 26, 28, 49]. On the one hand, adapters (see FIG.1, options (b) and (c)) keep the pre-trained LLM parameters as frozen weights and inject a small amount of trainable weights during fine-tuning. 1 CPST Doc: 1379-4645-4294.1

[0007] Low-Rank Adaptation (LoRA)

[0020] , the state-of-the-art adapter approach, inserts trainable weights as low-rank decompositions during fine-tuning.

[0008] Adapters attain state-of-the-art model quality by combining the frozen pre-trained weights and injected trainable weights. However, they introduce computation overheads during fine-tuning due to the injected trainable weights as discussed further below. For example, LoRA requires forward-backward passes on both the frozen weights and the injected trainable weights.

[0009] On the other hand, pruners (see FIG.1, option (d)) introduce a pre-processing step that identifies and removes the less important weights of the pre-trained LLM. A few pioneering works [18, 26] propose to identify weight importance leveraging second-order information, specifically the Hessian [15, 29, 50]. However, the computation and storage cost of the Hessian is considered prohibitive for modern deep neural networks. Kronecker- Factorized Approximate Curvature (K-FAC) [17, 34] improves on them by using a block- diagonal approximation to the Fisher Information Matrix (FIM) and further approximates each block by the Kronecker products of two much smaller matrices. Indicatively, K-FAC pruner

[0056] is practical for convolutional neural networks, but it still requires excessive memory in the pre-processing step for multi-billion parameter LLMs (e.g., LLaMA 3B model incurs out- of-memory errors during pre-processing on 24GB GPUs).

[0010] Moreover, although pruners provide high system performance in the fine-tuning process by leveraging weight sparsity, they typically degrade the model quality since they remove a large subset of the pre-trained LLM weights and incur high computation / storage costs in the pre-processing. SUMMARY

[0011] A method is proposed herein, which is illustrated in FIG.1, option (e) (labeled as “Sylva” in that diagram), which combines the strengths of adapters and pruners while mitigating their weaknesses. The method can optimize two goals, namely: i) system performance during fine-tuning and ii) model quality on downstream tasks. For the former goal, the method can identify a small number of important weights in the pre-trained LLMs leveraging second-order information and only trains these sparse weights during fine-tuning (similar to pruners). For the latter goal, the method can keep the pre-trained LLM weights frozen and employs these frozen weights in prediction (similar to adapters).

[0012] The method can also address the following two shortcomings of prior solutions. First, it can significantly reduce the computation and storage costs of the pre-processing step in pruners via (i) a hierarchical approximation of the second-order information and (ii) an 2 CPST Doc: 1379-4645-4294.1online projection and rediagonalization algorithm (below). Second, it can eliminate the computation overheads in adapters via embedding the sparse trainable weights into the frozen pre-trained LLM weights throughout the fine-tuning procedure.

[0013] In one aspect, there is provided a method of fine-tuning a large language model (LLM), comprising: identifying a set of important LLM weights via second-order information in a pre-processing step; and reducing computation and storage of the pre-processing step by: applying a hierarchical approximation of the second-order information, and executing an online projection and rediagonalization process.

[0014] In an implementation, only sparse important weights are trained, and the method further comprises embedding the sparse weights into the pre-trained LLM during fine-tuning.

[0015] In an implementation, the sparse weights and frozen pre-trained weights are kept together to reduce computation requirements during the fine-tuning.

[0016] In an implementation, the method includes determining weight importance by minimizing the impact of pruning on LLM quality.

[0017] In an implementation, the hierarchical approximation of the second-order information comprises a hierarchical approximation of Kronecker factors.

[0018] In an implementation, the method includes recursively halving the Kronecker factors and approximating off-diagonal blocks.

[0019] In an implementation, the off-diagonal blocks are approximated using singular value decomposition (SVD).

[0020] In an implementation, the method includes using an online algorithm to directly update SVD factors without materializing approximated blocks.

[0021] In an implementation, for each layer, both inputs and output gradients are saved.

[0022] In an implementation, the method includes updating corresponding ones of the SVD factors in the hierarchical approximation using the inputs and the output gradients.

[0023] In an implementation, the hierarchical approximation of the second-order information, comprises: projecting incoming data onto an existing basis; rediagonalizing to obtain rotation matrices; and rotating an extended subspace to obtain updated SVD factors.

[0024] In an implementation, the method includes applying a fine-tuning step by: (i) computing a forward pass; and (ii) computing a backward pass by computing gradients of loss with respect to the inputs and computing the gradients of loss with response to the sparse weights. 3 CPST Doc: 1379-4645-4294.1

[0025] In an implementation, a dataset being processed by the method is divided into a plurality of batches and each batch is processed one by one.

[0026] In another aspect, there is provided a computer readable medium storing processor-executable instructions for fine-tuning a large language model (LLM), comprising instructions that, when executed by a processor, cause a computer system to perform the method(s) above.

[0027] In another aspect, there is provided a computer system comprising at least one processor and at least one memory, the memory storing processor-executable instructions for fine-tuning a large language model (LLM), comprising instructions that, when executed by the at least one processor, cause the computer system to perform the method(s) above. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Embodiments will now be described with reference to the appended drawings wherein:

[0029] FIG.1 illustrates fine-tuning and inference execution flow of various approaches.

[0030] FIG.2a illustrates a K-FAC approximation of the FIM.

[0031] FIG.2b illustrates a naïve approach.

[0032] FIG.3a illustrates the present method using a hierarchical approximation of Kronecker factors.

[0033] FIG.3b illustrates a comparison between the brute force approach and the proposed online algorithm (projection and rediagonalization) to update the SVD factors in the hierarchical approximations.

[0034] FIGS.4a and 4b illustrate a comparison between naively injecting sparse trainable weights and embedded sparse trainable weights.

[0035] FIG.5 illustrates an algorithm for a two-stage fine-tuning procedure.

[0036] FIG.6 illustrates an algorithm for a hierarchical approximate curvature (HAC).

[0037] FIG.7 illustrates an algorithm for online projection and diagonalization (OPD).

[0038] FIGS.8a, 8b, 8c, 8d, and 8e illustrate fine-tuning time per iteration (Y-axis) on GPT2-Large, LLaMA-3B, and LLaMA-7B, as the number of GPUs (X-axis) increases from 4 to 16, wherein the fine-tuning time is broken down into forward pass time (in orange), backward pass time (in purple) and optimization step time (in green). 4 CPST Doc: 1379-4645-4294.1

[0039] FIGS.9a, 9b, and 9c illustrate latency of forward-backward propagation and optimization step for a single layer at various layer dimensions and sequence length.

[0040] FIG.10 illustrates an example of a computing environment in which the present method may be deployed and executed.

[0041] FIG.11 is an example of one of one or more computing devices operable to communicate in the computing environment. DETAILED DESCRIPTION

[0042] Fine-tuning is considered the gateway to transferring learned knowledge in a pre-trained LLM on many downstream applications. To make LLM fine-tuning more affordable, prior works follow two paths: i) adapters freeze the pre-trained LLM weights and inject a small number of trainable weights during fine-tuning, and ii) pruners remove the less important weights in pre-trained LLMs and train the remaining sparse weights during fine- tuning.

[0043] It has been found that the former introduces computation overheads due to the injected trainable parameters, while the latter introduces an expensive pre-processing step to identify the important weights and degrades model quality. To get the best of both worlds, the method proposed herein provides a novel LLM fine-tuning procedure that provides high system performance during fine-tuning and attains state-of-the-art model quality on downstream applications. The method identifies the most important LLM weights via second- order information in a pre-processing step, and significantly reduces the computation and storage costs of the pre-processing step via i) a hierarchical approximation of second-order information, and ii) an online projection and rediagonalization algorithm. The method may train only the sparse important weights and embeds these sparse weights into the pre- trained LLM during fine-tuning to provide high system performance. 1.0 Introduction

[0044] It is shown herein that, on average, the end-to-end fine-tuning with the proposed method can be up to 5.1 times faster than full fine-tuning with ZeRO optimizer and 1.2 times faster than LoRA, the state-of-the-art adapter approach. The following also studies the sensitivity of the method’s performance on various layer dimensions and sequence lengths. For example, the method’s forward and backward pass time can be up to 2.3 times faster on fully connected layers. Additionally, the method can reduce GPU memory usage by up to 35% compared to LoRA during fine-tuning. In the pre-processing step, the method’s 5 CPST Doc: 1379-4645-4294.1hierarchical approximation can reduce the peak GPU memory by 2.3× compared to K-FAC, the most widely-used approximation to second-order information.

[0045] The present disclosure therefore can provide the following contributions:

[0046] - Comprehensively analyze prior LLM fine-tuning approaches and propose an effective fine-tuning method for LLMs.

[0047] - Design a hierarchical approximation of the second-order information along with an online projection and rediagonalization algorithm to significantly reduce the computation and storage costs, when identifying the most important weights of a pretrained LLM. The following examples include embedding the most important weights as trainable parameters into the pre-trained LLM to eliminate the extra computation costs that would have been introduced by adapters (e.g., LoRA) injecting trainable weights.

[0048] - Evaluate the method by fine-tuning state-of-the-art LLMs on multi-node systems and showing that the method can outperform prior attempts in performance and memory efficiency. The method can also provide high model quality and low pre-processing costs. 2.0 Background 2.1 Overview of Prior Solutions

[0049] FIG.1 summarizes various approaches that accelerate fine-tuning. The naive fine-tuning approach, known as full fine-tuning (FIG.1, option (a)), keeps all weights of the pre-trained large language model (LLM) and continues training them on task-specific datasets. Zero Redundancy Optimizer (ZeRO) [42, 43] can improve the per-device memory footprint of full fine-tuning via sharding and offloading. However, ZeRO can also introduce communication overheads because they require gathering sharded weights at every iteration. Another line of prior works [21, 37] improves fine-tuning performance using pipeline or tensor parallelism. Existing implementations of these works are only tailored for a few LLM families and thus lack generality. For example, Megatron-LM

[0037] optimizes the GPT family [4] of models, and supporting the LLaMA family

[0053] can require significant engineering efforts and tuning to provide high system performance.

[0050] A group of prior works [19, 20, 31, 46], named “adapters”, can keep all parameters of the pre-trained LLM as frozen weights and injects a small number of new parameters as trainable weights. Only the latter are trained during fine-tuning, and thus performance is improved. Houlsby et al.,

[0019] propose the serial adapter (FIG.1, option (b)) that injects trainable weights as new layers into the pre-trained LLM. Serial adapter achieves state-of-the-art model quality; however, adding additional layers in an LLM is found to 6 CPST Doc: 1379-4645-4294.1worsen inference performance compared to full fine-tuning. Some works [31, 46] improve inference performance via reducing the trainable weights in the serial adapter, however, at the cost of degrading model quality. Low-Rank Adaptation (LoRA)

[0020] (FIG.1, option (c)) injects trainable weights as low-rank decompositions to each existing fully connected (FC) layer of the pre-trained LLM. After fine-tuning, the injected trainable weights are merged with the frozen weights via summation. This way, LoRA does not introduce additional inference latency while attaining high model quality.

[0051] Another group of prior works [12, 13, 18, 23, 26, 49], named as pruners, identifies and removes the less important weights in pre-trained LLM in a pre-processing step (FIG.1, option (d)). In fine-tuning, only the non-zero elements of the induced sparse weight matrices are trained and updated using optimized sparse linear algebra libraries. LeCun et al.

[0026] and Hassibi et al.

[0018] propose to determine the importance of the pre- trained weights via second order information, specifically the Hessian [15, 29]. Singh et al.

[0049] and Kurtic et al.

[0023] improve on computation and storage costs by using a block- diagonal approximation of the Fisher Information Matrix (FIM) [1, 22, 33] in place of the Hessian. Kurtic et al.

[0023] enables to scale up to medium-sized language models (e.g. BERT [9]). However, it faces a trade-off between computation / storage costs and the accuracy of the approximation. Specifically, the GPU memory capacity limits the block size to be small. However, the smaller the block size, the more off-diagonal information is disregarded. Frantar et al. [12, 13] enable to prune GPT-family models [4] using second-order derivatives; however, at the price of using a layer-wise reconstruction loss as the pruning objective instead of the model’s prediction loss, and thus losing theoretical groundings of prior works [18, 26]. Kronecker-Factorized Approximate Curvature (K-FAC) [17, 34] is the state-of-the- art approximation to second order information since it significantly reduces computation and storage costs compared to using the Hessian [18, 26].

[0052] As shown in FIG.2a, (i), K-FAC employs a layer-wise approximation to the FIM and further approximates each block (corresponding to a layer in the neural network) using the Kronecker product of two much smaller matrices (see FIG.3a). Although K-FAC is practical for medium sized LLMs (e.g., BERT), the memory footprints of the Kronecker factors are excessive for multi-billion parameter LLMs, making it prohibitive to prune large LLMs on most data-center GPUs. 2.2 Comparison of Prior Solutions

[0053] Table 1, shown below, qualitatively compares the aforementioned methods. 7 CPST Doc: 1379-4645-4294.1

[0054] Memory Footprint. In ZeRO full fine-tuning, although the gradients and optimizer states are sharded and offloaded, each GPU still needs to temporarily store the gradients for all LLM weights in the backward pass. Serial and LoRA adapters only train the injected weights, thus reducing the peak memory footprints compared to naive full fine-tuning. K-FAC pruner sparsifies the LLM weights. Thus, the memory footprints of weights, gradients and optimizer states are all reduced compared to ZeRO and adapter approaches. However, K- FAC pruner introduces excessive memory footprints during the pre-processing step for the second-order information, which might exceed the GPU memory capacity. For example, KFAC’s pre-processing step requires at least 40GB memory for the LLaMA-3B, thus causing out-of-memory errors on 24GB GPUs. The computations during the K-FAC’s pre-processing step to approximate second-order information are also expensive.

[0055] Fine-Tuning Time. On multi-GPU systems, the fine-tuning time aggregates computation and communication costs. ZeRO requires computing the gradients of all LLM weights and has large communication overheads due to gathering the sharded weights at each iteration. Serial adapter and LoRA eliminate the need to compute gradients for the frozen weights; however, they introduce extra computation costs because they attach the trainable weights as either extra layers (serial adapter) or extra computation paths at each FC layer of the LLM (LoRA). In LoRA, the trainable weights and pre-trained weights are processed separately in the forward and backward passes. Serial and LoRA adapters reduce the communication time compared to full fine-tuning, since they All-Reduce only the gradients of the small amount of injected trainable weights. When setting K-FAC to have the same number of trainable parameters with adapters (regardless of model quality degradation led by the pruned weights), K-FAC pruner provides much higher computation efficiency than adapters since it does not introduce extra computation paths and has a comparable communication cost.

[0056] Inference Time. Serial adapter introduces extra inference latency compared to full fine-tuning due to extra LLM layers added during fine-tuning, while LoRA merges the trainable weights into the frozen weights after fine-tuning. Thus, the inference time of LLM 8 CPST Doc: 1379-4645-4294.1fine-tuned using LoRA is the same as that using full fine-tuning. Pruners remove weights from the pre-trained LLM and use sparse weight matrices, thus reducing the inference time compared to ZeRO and adapters.

[0057] Model Quality. ZeRO’s memory optimizations do not affect the quality of LLMs, thus, ZeRO provides high model quality, since it trains all LLM weights with the task-specific dataset. Serial adapter and LoRA achieve comparable model quality with full fine-tuning, since they combine the learned knowledge of the pre-trained frozen weights with the knowledge of the additional weights trained with the task-specific dataset. Instead, K-FAC pruner trains a small subset of pre-trained LLM weights with the task-specific dataset; thus, it typically degrades model quality.

[0058] Checkpoint Saving. ZeRO does not allow sharing the checkpoints of fine-tuned LLMs since all weights are specialized to a particular downstream task. Instead, adapters save storage resources for LLM checkpoints by enabling sharing the frozen weights across multiple downstream tasks, while the injected task-specialized trainable weights are stored separately. K-FAC pruner also reduces the LLM checkpoints storage costs: this benefit is obtained via weight pruning, i.e., creating sparsified fine-tuned LLMs, instead of sharing the weight values across downstream tasks as adapters do.

[0059] This analysis shows that adapters achieve high model quality by exploiting all the pre-trained LLM weights during fine-tuning, but they introduce additional computation costs for to the injected trainable weights. Instead, pruners leverage sparse trainable weights to reduce computation costs during fine-tuning significantly. However, they downgrade model quality and introduce an expensive pre-processing step. To get the best of both approaches, the method proposed herein provides a combination of adapters and pruners, as described in the next section. Table 1 above shows that the method can provide the most effective solution in pre-processing memory and performance in both fine-tuning and inference and achieves state-of-the-art model quality. The method can also reduce the storage costs for checkpoints by storing only the sparse trainable weights for each downstream task. 3. Method Overview

[0060] The present method interpolates between the adapter and pruner approach to overcome their shortcomings and acquire their strengths. FIG.1, option (e) presents an overview of the method.

[0061] 1) The method adopts the pruner’s notion of identifying a small number of important weights of the pretrained LLM (named sparse trainable weights) via a pre- 9 CPST Doc: 1379-4645-4294.1processing step, and only training these sparse weights during fine-tuning to achieve low computation costs.

[0062] 2) The method adopts the adapter’s notion of freezing all the pre-trained LLM weights in the finetuning step and leveraging them in prediction to provide high model quality. In this design, the method is configured to address two key challenges: i) how to minimize the computation and storage costs required by the second-order information in the pre-processing step (Challenge 1), and ii) how to eliminate the computation overheads introduced by injecting trainable weights into the pre-trained LLM during finetuning (Challenge 2).

[0063] To address these challenges, the following proposes three techniques.

[0064] 1) Hierarchical Approximate Kronecker Factors. During preprocessing, the weight importance is determined by minimizing the impact of pruning on the model’s prediction loss, formally, a quadratic model of the loss landscape [18, 26]. Solving this minimization problem by Lagrange multipliers yields the weight importance that involves the Hessian H. In K-FAC pruner, the Hessian H is approximated by the Kronecker products A ⊗ G, where A and G are the covariances of input and output gradients. Henceforth, A and G are called the Kronecker factors. The (^^^^, ^^^^)-th element in the Kronecker factor A stands for the correlation (i.e., the degree to which two variables are related) between the ^^^^-th and ^^^^-th neuron in the LLM layer’s inputs. A similar representation holds for the Kronecker factor G.

[0065] To reduce the memory overheads of the pre-processing step of multi-billion parameter LLMs (Challenge 1), the method proposes a hierarchical approximation of the Kronecker factors. It may be observed that the magnitude of entries in the Kronecker factors tends to decrease as their distances to the diagonal increase. This is because the neurons closer to each other in the LLMs have a higher correlation. If the ^^^^-th and ^^^^-th dimensions are close in the LLM layer’s inputs, then the (^^^^, ^^^^)-th entry in the Kronecker factor A is close to the diagonal. Based on this observation, the method recursively halves the Kronecker factors and approximates the off-diagonal blocks using Singular Value Decomposition (SVD). Let ^^^^ be the rank in the lowrank decomposition provided by SVD and ^^^^ be the size of a block in the partition.

[0066] As shown in FIG.3a, each off-diagonal block B ∈ R^^^^×^^^^(in purple for A and ingreen for G) is approximated using SVD: B ≈ UΛV⊤, where each of U ∈ R^^^^×^^^^and V ∈ R^^^^×^^^^contains a set of ^^^^ orthonormal bases that span the approximating subspace, Λ contains a vector of ^^^^ singular values that stretches the bases. As a result, at each partition, the method 10 CPST Doc: 1379-4645-4294.1only stores (a) two tall and skinny matrices (U and V matrices in blue) and a vector (Λ in blue) for each off-diagonal block (instead of the entire block B), and (b) the exact diagonal blocks of the finest partition (shown in red in FIG.3a). Denote the dimension of A and G by ^^^^ and ^^^^, respectively. Let ^^^^ be the total number of recursive partitions in the hierarchical approximation. The original Kronecker factor A takes ^^^^(^^^^2) storage, while that of the method’s hierarchical approximation of A is much smaller: the approximated off-diagonal blocks take ^^^^(^^^^^^^^ (2^^^^− 1)) storage, and the exact diagonal blocks of the finest partition take ^^^^(^^^^2 / 2^^^^). The storage complexity of G is analog to that of A except that it is on the output dimension ^^^^.

[0067] 2) Online Projection and Rediagonalization. In the method, one can divide the dataset into many mini-batches of data and then process the mini-batches one by one because fitting the entire dataset onto the GPU is prohibitive, in the pre-processing step. Thus, one needs to aggregate second-order information from a mini-batch of data with the previous ones, i.e., update an existing hierarchical approximation using incoming data. To further reduce the pre-processing step’s storage cost and minimize the computation costs of hierarchically approximating the Kronecker factors (Challenge 1), the method proposes an online algorithm that directly updates the SVD factors without materializing the approximated blocks. For each layer, the inputs I are saved and output gradients G. One can use z to uniformly denote the input I and output gradients G since the process is similar. Then, the method updates the hierarchical approximation of A and G using I and G, respectively. Naively, as shown in FIG.2b, the method would need to (1) materialize the approximated block using the SVD factors, (2) add the blocks and (3) do another SVD on the aggregated block. Materializing the approximated blocks leads to high memory footprints, and the computation of SVD is expensive. To address these issues, the method proposes online projection and rediagonalization (see FIG.3b).

[0068] There are three main steps in the algorithm: (1) projecting the incoming dataonto the existing basis (2) rediagonalization to get the rotation matrices ^�^^^ , Λ�and V�, and (3)rotating the extended subspace (concatenation of existing basis (U, V) and the orthogonal vectors (u, v) to obtain the updated SVD factors (U′, V′). The updated singular values Λ′ aresimplyΛ�. Put another way, the algorithm directly operates on the SVD factors (U, Λ, V)without materializing the corresponding approximated block B, which has a much larger number of elements than that of the SVD factors. Thus, both computation and storage costs are reduced. Specifically, in the naive approach, (1) takes ^^^^(^^^^2^^^^ +^^^^^^^^) computation to retrieve the approximated block B from the SVD factors, (2) takes ^^^^(^^^^2) computation to add 11 CPST Doc: 1379-4645-4294.1the retrieved block and new block (outer product of z and itself), and (3) takes ^^^^(^^^^3) to recompute the SVD of the aggregated block. Throughout the entire process, ^^^^(^^^^2) storage is required. With online projection and rediagonalization, (1) obtains the new basis (u and v) orthogonal to existing subspace (spanned by U and V) takes ^^^^(^^^^^^^^) computation and storage, (2) takes ^^^^((^^^^ +1)3) computation and ^^^^((^^^^ + 1)2) storage, since the method computes SVD on the lower dimension ^^^^ instead of the higher dimension ^^^^, and (3) to obtain the new SVD factors via rotating the extended subspace takes ^^^^(^^^^(^^^^ + 1)2) computation and ^^^^(^^^^(^^^^ + 1)) storage. The proposed algorithm has much lower computation and storage costs than the naive approach since ^^^^ ≪ ^^^^.

[0069] 3) Embedded Sparse Trainable Weights. To eliminate extra computation introduced by injected trainable weights (Challenge 2), the method proposes to embed the sparse trainable weights into the pretrained weights during fine-tuning. Specifically, one may keep a single copy of weights as shown in FIG.4b, instead of two: one for the frozen weights and one for the trainable weights as the naïve approach shown in FIG.4a. This technique can eliminate the extra computation (red rectangles in FIG.4a) due to injecting trainable weights. Furthermore, during the backward pass, the method leverages an optimized library that provides fast Sampled Dense-Dense Matrix Multiplication (SDDMM) operation to compute the sparse gradients. The sparse gradients are stored in block sparse matrix format. Specifically, the non-zero blocks are stored contiguously, and there is a mask indicating whether each block is non-zero or not. The method scatters and adds the sparse gradients to the dense weights in each optimization step. One can use a single copy of weights, i.e., the frozen weights and trainable weights are stored in the same dense matrix, while the frozen weights are not updated throughout the fine-tuning. The method fuses the optimization steps of layers into one CUDA kernel launch instead of performing a sequence of small updates. Denote the mini-batch size by ^^^^ and the sequence length by ^^^^. Let the sparsity (i.e., percentage of zeros in the sparse weight matrix) be ^^^^. For the naive approach, the extra computations (red rectangles in FIG.4a) take ^^^^(^^^^^^^^^^^^2^^^^^^^^ + ^^^^^^^^^^^^) in the forward pass and ^^^^(^^^^^^^^^^^^^^^^2^^^^ + ^^^^^^^^^^^^) in the backward pass, which are completely eliminated thanks to the embedding of sparse trainable weights into the pre-trained weights. Instead, the method requires ^^^^(^^^^^^^^^^^^2^^^^) in the forward pass. In the backward pass, the method requires ^^^^(^^^^^^^^^^^^^^^^2) computation for the loss with respect to the inputs and ^^^^(^^^^^^^^^^^^^^^^^^^^) computation for the loss with respect to the output gradients. 4. Method Design Details 4.1 A Two-Stage Fine-Tuning Procedure 12 CPST Doc: 1379-4645-4294.1

[0070] Based on the ideas that were presented above in Section 3, the method proposes a two-stage fine-tuning procedure as shown in FIG.5, referred to as “Algorithm 1”.

[0071] Pre-processing. The method samples a subset from the fine-tuning dataset to approximate second-order information. The number of preprocessing samples is a configurable hyperparameter in the method (see Section 5.5). The method performs a forward-backward pass and stores per-sample activation I and output gradients G (lines 5 - 7). At the first iteration, one can initialize hierarchical approximate of A and G (lines 8 - 11) using Algorithm 2 (see FIG.6). For the following iterations, one may update the hierarchical approximations (lines 13 - 15) using online projection and rediagonalization (see Algorithm 3 in FIG.7). After one round over the sampled data, the method computes the inversion (line 19) by recursively applying Eqn. (5) - (8). The method computes the importance of each input dimension using Eqn. (19) and that of each output dimension using Eqn. (20) (line 20). To obtain the element-wise importance of the weights, the method computes an outer product of the importance of input and output dimensions. The method sorts the weights based on their importance to obtain the weight mask and keep the top candidates (line 21). The selected weights are set as trainable and the other weights as frozen, i.e., one computes only the gradients and update the sparse weights (line 21).

[0072] Fine-Tuning. During fine-tuning, the method computes the forward pass as in Eqn. (21) (line 26). The backward pass includes two parts: (i) computing the gradients of loss with respect to the inputs, as in Eqn. (22), and (ii) compute the gradients of loss with respect to the sparse weights, as in Eqn. (16) (line 27). The sparse gradients are all-reduced in data-parallel training (line 28). In every optimization step, the method scatter-adds the sparse gradients to the dense weight matrix (line 29). The optimization step takes a list of dense weight matrices, the index of non-zero elements and the sparse gradients stored contiguously. 4.2 Hierarchical Approximate Curvature

[0073] Partition of Kronecker Factors. To reduce the storage required by the Kronecker factors (A and G), note that the off-diagonal elements of A and G stand for the correlation between two dimensions in input or output. Intuitively, the neurons closer to each other in terms of their positions have a higher correlation The number of partitions in the hierarchical approximation is a configurable hyperparameter in the method (see Section 5.5 below). Let ^^^^ denote the current level of partition and B^^^^ denote a block at level ^^^^. So trivially, A = B0. The same holds for G. Leveraging this favorable structural property, one can get the hierarchical 13 CPST Doc: 1379-4645-4294.1approximation for each of the Kronecker factors A and G using the following algorithm: (1) Partition a matrix B^^^^ into four equally sized blocks:

[0074] 2) One can approximate the off-diagonal blocks ^^^^^^^^+1 ^^^^+112and ^^^^21using Singular Value Decomposition (SVD):

[0075] where U, V are orthonormal rotation matrices and Λ are diagonal matrices of the top singular values. (3) the method can recursively partition each of the diagonal blocks ^^^^^^^^+112and ^^^^^^^^+121into smaller blocks by repeating steps (1) and (2), until the dimension of the finest partition is smaller than a user-defined threshold.

[0076] A hierarchical approximation is (^^^^, ^^^^) if the off-diagonal blocks are approximated using rank-^^^^ SVD and the matrix is partitioned ^^^^ times.

[0077] One can denote a (^^^^, ^^^^) hierarchical. A block at the ^^^^-th level partition is:

[0078] for ^^^^ = 0, ..., ^^^^. The dimension of B^^^^is 1 / 2^^^^of the original matrix. The blocks at the finest partition can be kept as exact.

[0079] Recursive Block Inversion. Given a hierarchical approximation, one can compute the matrix inversion in a bottom-up manner by recursively applying the block inversion formula given the inversion (or SVD) of submatrices. Consider a block B in the hierarchical 14 CPST Doc: 1379-4645-4294.1approximation. The base case is the inversion of the diagonal blocks. Consider block at the ^^^^-th level partition as in Equation 4. Given the inverses of the diagonal blocks ^^^^1−11= [^^^^^^^^+1]-1and ^^^^−1= [^^^^+1 -111 22^^^^12] , one can compute the kernel matrix:

[0080] The dimension of K is ^^^^ × ^^^^, where ^^^^ is the rank in the hierarchical approximation. Denote the inversion of diagonal blocks by:

[0081] and the rotation matrices by:

[0082] As a result, the inverse of B is: 4.3 Online Projection and Rediagonalization

[0083] To efficiently aggregate second-order information over many mini-batches of training data into the hierarchical approximation of Kronecker factors, the method proposes an online algorithm that does not need to materialize the approximated blocks. Brand [3] proposes a numerical method for fast additive low-rank modifications for tracking singular values and subspaces. Based on it, one can design an online algorithm that directly updates the SVD factors in the hierarchical approximation without recomputing the approximated blocks and maintains the best rank-^^^^ approximation greedily. Consider a block B in the 15 CPST Doc: 1379-4645-4294.1hierarchical approximation. Denote the block after ^^^^ iterations of update by B^^^^. In each iteration, given the per-sample input or output z of the mini-batch, the method wants to update the singular value decomposition B^^^^= U^^^^Λ^^^^V^^^^to incorporate the rank-1 perturbation zz⊤resulting from an incoming data. Firstly, one can find the additional basis u^^^^ and v^^^^ that are orthogonal to the rotation matrices U^^^^and V^^^^:

[0084] Similarly, one can have:

[0085] Next, the method computes the rediagonalization:

[0086] where (1) is a diagonal matrix and (2) is a rank-1 matrix (outer product of two vectors). One can obtain the rotation of extended subspaces by diagonalizing Ψ:

[0087] The updated SVD is then:16 CPST Doc: 1379-4645-4294.1

[0088] One can greedily select the top-^^^^ singular values and their corresponding eigenbasis and discard the rest to maintain a limited memory usage, regardless of the number of incoming data. By the Eckart-Young- Mirsky theorem [11, 36], the rank-^^^^ SVD provides the optimal approximation for a given ^^^^ . By the end, the method obtains the best rank-^^^^ approximation for each block in the hierarchy. 4.4 Embedded Sparse Trainable Weights

[0089] To optimize the memory footprint of the model during fine-tuning, the method proposes to construct sparse adapter modules, which provide fine-grained control over thenumber of trainable parameters. Denote the weights trained to convergence by ^�^^^ . Let theweight mask be M. The injected trainable parameters are denoted as W. The method uses ⊙ to denote the element-wise product. The adapter W and its gradient ^^^^L / ^^^^W are sparse. Wis initialized as ^�^^^ ⊙ M. The rest of the pre-trained weights ^�^^^ ⊙￢M are kept frozenthroughout the fine-tuning. In the forward pass, the method computes:

[0090] In the backward pass, the method computes the gradient of loss with respect to the input:

[0091] and the gradient of loss with respect to the weight:

[0092] In the optimization step, the method updates the sparse trainable weights with their gradients:17 CPST Doc: 1379-4645-4294.1

[0093] FIM-Induced Sparsity. The method can leverage second-order information to obtain the weight mask M with fined-grained sparsity. The rank deficiency of Hessian provides source and reasoning of parameter redundancy in trained networks

[0050] . In other words, the Hessian induces the effective dimension of trained overparameterized networks. Taking advantage of this observation, since the method wants to maximize the information in sparsified gradients on the effective dimensions of the network, one can project the gradients onto Hessian’s range, i.e., the space spanned by rows / columns of the Hessian. Denote the gradient of loss with respect to the parameters in the ^^^^-th layer by Γ = ∇WL ∈ R^^^^×^^^^, where ^^^^ and ^^^^ are the input and output dimensions of a layer in the neural network. The method uses ΔΓ to represent the difference in the gradient and the targeting sparse gradients. The vectorized gradients and change in gradients are ^^^^ and Δ^^^^, respectively. One can use ^^^^^^^^to denote the gradients corresponding to ^^^^-th input or output dimension. The method computes the importance of input and output dimensions separately. For each of them, one can solve the following constrained optimization problem:

[0094] where e^^^^is a canonical basis, all elements are zeros except that the ^^^^-the element is one. Using the Kronecker factorization (FIG.2a), the method can rewrite the inner objective as Tr(Δ^^^^⊤AΔ^^^^G). Solving problem (18) by the Lagrange multiplier, the method has the importance metrics for input and output dimensions:

[0095] and. 18 CPST Doc: 1379-4645-4294.1

[0096] The outer product of Eqn. (19) and Eqn. (20) yields an importance score for each of the parameters. The method sorts the parameters in each layer by descending importance and select the top candidates as trainable weights in fine-tuning.

[0097] Sparse Embedded Adapters. To reduce the computation overhead caused by the additional forward-backward propagation through the adapters, we observe that some of the matrix operations can be merged as one, leading to an implicit parameterization and eliminating the redundant computation. Computing Eqn. (14) as is introduces additional computation and storage compared to a standard forward pass, as shown in FIG.4. Specifically, it requires another Sparse Matrix Multiply (SpMM) and activation memory for the intermediate tensors W ⊙￢MX and WX. Instead of storing the frozen and trainable weights separately, we keep a unified dense weight matrix while only computing and storing the gradients for the sparse trainable weights. As a result, the forward pass takes the same computation and activation memory as the standard one:

[0098] Similarly, Eqn. (15) becomes a General Matrix Multiply:

[0099] instead of an addition of two, as in Eqn. (15). The computation sparse gradient in Eqn. (16) is a Sampled Dense Dense Matrix Multiply (SDDMM). Only the rows of ^^^^L / ^^^^Y and columns of X that correspond to the non-zero elements in W are loaded to the memory and used to compute the gradients. The method may use block sparsity to demonstrate the performance benefit in Section 5. The block size and sparsity in adapters are configurable hyperparameters in the method. 5. Evaluation 5.1 Methodology

[0100] Model and Datasets. One may use four set of benchmarks: (i) RoBERTaLarge (355M parameters)

[0032] with the GLUE benchmark

[0055] . (ii) GPT2-Large (812M parameters)

[0040] with the E2E NLG challenge

[0039] . (iii) T5-Large (738M parameters)

[0041] with the Super- Natural Instructions

[0057] dataset. (iv) LLaMA-3B and 7B

[0053] on the OpenAssistant Conversations Dataset (OASST1)

[0025] . 19 CPST Doc: 1379-4645-4294.1

[0101] Baselines. One can compare with (1) full fine-tuning, (2) LoRA

[0020] as the adapter baseline, and (3) K-FAC pruner

[0056] as the pruner baseline. For the performance evaluation of full fine-tuning baseline, one may use ZeRO optimizer [42, 43] for GPT2-Large and LLaMA, and gradient checkpointing [7] for T5-Large. For RoBERTa-Large, there is no memory optimization needed, and we simply do full fine-tuning in PyTorch

[0038] . One can choose the memory optimization that provides better fine-tuning time and does not result in out-of-memory (OOM) errors. The choice of memory optimization does not affect model quality.

[0102] Hardware. One can use g2-standard-48 instances on Google Cloud Platform (GCP). Each node has 4 NVIDIA L4 GPUs (24 GB) and an Intel Cascade Lake CPU. Evaluate all methods on up to 16 GPUs. All instances are located in the same regions and connected via a network bandwidth of around 15 Gbps.

[0103] Software. One can use PyTorch 2.1.0

[0038] for all methods and Deep-Speed 0.10.0

[0044] for the ZeRO optimizer. To showcase the performance benefit of leveraging sparsity, one can use Triton block sparse module [47, 52], which provides CUDA kernels for block sparse SpMM and SDDMM operations. Note that although the methodology uses block sparsity to demonstrate the method’s performance benefit, the method is not limited to block sparse patterns and can benefit from other CUDA kernels or hardware-enabled acceleration. 5.2 Fine-Tuning Time

[0104] One can compare the end-to-end fine-tuning time per iteration using up to 16 GPUs. The present method can be, on average, 5.1 times faster than ZeRO optimizer on GPT2-Large and LLaMA. As shown in FIG.8, ZeRO is significantly slower than LoRA and the present method at a typical data center network bandwidth (10 ∼ 20 Gbps) without ultra- high bandwidth interconnects (e.g., InfiniBand EDR 100Gb / s

[0035] ). This is because ZeRO trades off latency for larger memory efficiency. ZeRO stage 1 partitions optimizer states across the GPUs, and stage 2 additionally shards reduced gradients. Both need additional gathering for every optimization step. Besides, the communication overhead is exacerbated 20 CPST Doc: 1379-4645-4294.1as we increase the number of GPU nodes. This is because ZeRo adds communication overheads in both backward and optimization steps, thus increasing execution time the method can be 1.2 times faster than full fine-tuning with gradient checkpointing enabled on T5-Large.

[0105] The forward pass time in K-FAC Pruner may be the best on GPT2-Large and LLaMA because it uses sparse weight matrices, significantly reducing computation costs. The forward and backward pass time of the method can be 1.2 ∼ 1.4 times faster than LoRA on GPT2-Large and LLaMA. This is because the method eliminates extra computation led by the injected trainable parameters in LoRA. On T5-Large, LoRA and the method perform similarly. On RoBERTa-Large and T5-Large, K-FAC Pruner performs worse than LoRA and the method, because the layer dimensions are relatively small. Also, there is a trade-off between the performance gain from reduced computation FLOPs and the overhead of performing irregular memory access in the memory hierarchy. On RoBERTa-Large, LoRA may perform slightly better than the method, specifically on the backward pass, because the method uses sparse instead of dense matrices, which causes irregular memory access. The optimization step time is relatively small compared to forward / backward pass time in all methods except for ZeRO. 5.3 Model Quality

[0106] The present method can achieve comparable or slightly worse model quality compared to full fine-tuning and LoRA at a high sparsity in the range of 95% ∼ 99% (see Table 3 below). As shown in Section 5.5, the method may perform better using a moderate sparsity in the range of 86% ∼ 96%, while LoRA obtains its best scores with a small rank of 4. KFAC Pruner degrades model quality because it uses sparse weights, which do not fully utilize learned knowledge in pre-trained weights. 21 CPST Doc: 1379-4645-4294.15.4 Memory Footprint

[0107] One may compare the peak GPU memory usage on various rank ^^^^ (for LoRA) and sparsity ^^^^ (for K-FAC Pruner and the present method). As shown in Table 2, the method requires less GPU memory than LoRA on all models. The advantage of the method grows as sparsity decreases, i.e., if one injects a larger number of trainable weights into LLMs. For example, LoRA causes OOM errors on LLaMA-7B with a rank of 256, while the present method can fit onto a 24 GB GPU in an equivalent configuration (^^^^ = 90%). This is because LoRA requires storing intermediate results provided by both the frozen pre-trained weights and injected trainable weights and then taking the sum of them. In contrast, the method only requires one pass on the embedded weights and thus needs to store fewer intermediate results. K-FAC Pruner requires the least GPU memory during fine-tuning. However, their preprocessing step requires excessive GPU memory and might lead to OOM errors on the GPUs, as shown later (See Section 5.6 Pre-Processing Costs). 5.5 Sensitivity Study

[0108] Model Quality. One can compare the hyperparameter sensitivity of LoRA and the prsent method on GPT2-Large model and the E2E NLG Challenge. Control the method’s sparsity and LoRA’s rank together so that the number of trainable parameters is the same during fine-tuning. One can also study the method’s sensitivity to block size, the number of 22 CPST Doc: 1379-4645-4294.1partitions in the hierarchical approximation, and the number of samples in the pre-processing step.

[0109] (i) Sparsity: the method obtains better model quality than LoRA with sparsity in the range of 86.7% ∼ 96.7% (see Table 4 below).

[0110] At a sparsity of 99.5%, the method obtains a worse model quality than LoRA (with a rank of 4).

[0111] (ii) Block size: As shown in Table 5, the quality slightly decreases as the block size increases.

[0112] A moderate block size such as 64 balances the model quality and performance tradeoff. Specifically, it obtains high model quality while being 10% faster than a block size of 16 and only 4% slower compared to a block size of 128. With a high sparsity of over 90%, the end-to-end latency of different block sizes is comparable.

[0113] (iii) Number of Partitions: The model quality decreases as we increase the number

[0114] of partitions (see Table 6 below). 23 CPST Doc: 1379-4645-4294.1

[0115] One may prioritize a smaller number of partitions, because it yields a more accurate approximation and has less preprocessing time as long as it is able to fit onto the GPU memory.

[0116] (iv) Number of Samples in Pre-Processing: On average, the model quality is the highest with 1024 samples (see Table 7).

[0117] It was noticed that the method’s highest score for MET and ROUGE-L is obtained with 128 samples, although the average score with 128 samples is lower than that obtained with 1024 samples. This indicates that more pre-processing samples provide richer information on the dataset, thus yielding a higher average score.

[0118] Performance. Besides comparing the end-to-end fine-tuning time, one may benchmark the forward / backward pass computation time for a single Linear module (common in attention or FC modules) on various layer dimensions and sequence lengths. As shown in FIG.9, the advantage of the present method over full fine-tuning and LoRA enlarges as the layer dimensions increase and the maximum sequence length increases. The method can be up to 2.3 times faster than LoRA and 1.3 times faster than full fine- tuning. This is because the computation complexity for activations increases linearly to the layer dimensions and sequence lengths. K-FAC Pruner has the lowest latency because it uses sparse weight matrices. Thus, computation in forward / backward passes is sparse matrix operations (SpMM or SDDMM) instead of dense. However, K-FAC Pruner sacrifices model quality, as shown in Section 5.3. Note that the discrepancy between FIGS.8 and 9 is 24 CPST Doc: 1379-4645-4294.1because there are layers that LoRA, K-FAC Pruner and the method do not optimize (e.g., embedding, normalization), which results in lower end-to-end speedup compared to that on a single layer. 5.6 Pre-Processing Costs

[0119] The method can reduce the peak GPU memory by up to 2.3 times compared to KFAC Pruner (see Table 2). For LLaMA-7B, the pre-trained model weights and Kronecker factors alone take 48 GB of GPU memory, without considering the activation memory and gradients allocated in the pre-processing step. The hierarchical approximation and algorithm reduce peak GPU memory usage to 20.1 GB during computing second-order information. Since the pre-trained weights take about 13 GB, the storage is reduced by at least 5.0 times compared to naively computing the Kronecker factors. To approximate the second-order information in the pre-processing step, one may sample about 10% points of the dataset. For example, the pre-processing step of GPT2-Large takes 17.2 minutes on 4 GPUs, while fine- tuning it for 5 epochs takes about 3 hours. Thus, the method’s pre-processing does not add significant overhead overall. 5.7 Inference Performance

[0120] Since the adapters are merged back to the pre-trained weights after fine-tuning for both LoRA and the present method, LoRA and the present method have a similar inference performance to full fine-tuning. The K-FAC pruner removes a subset of the pre- trained weights in the preprocessing step, yielding a sparse model after fine-tuning. Thus, K- FAC pruner obtains 1.4× speedup during inference compared to the other methods on LLaMA-7B. 6. Related Work

[0121] The present method can significantly reduce (i) computation costs in LLM fine- tuning by embedding sparse trainable weights into the pre-trained LLM, and (ii) storage costs to approximate second-order information via a hierarchical approximation, and online projection and rediagonalization algorithm, while also providing high model quality, in contrast to prior solutions.

[0122] Pruners using Second-Order Information. Previously [18, 26] solutions may employ Hessian to prune weight matrices. Hessian is shown to be closely related to the source of rank deficiency in pre-trained deep neural networks

[0050] . However, using Hessian as a second order information method results in excessive memory footprints in the pre- processing step, making it prohibitive to be used in common data-center GPUs. K-FAC [17, 25 CPST Doc: 1379-4645-4294.134, 56] uses the Fisher Information Matrix (FIM) to approximate the Hessian and significantly reduces the memory footprints needed during the pre-processing step. The FIM is shown to be equivalent to the Hessian for commonly used loss (e.g. cross-entropy, squared-error)

[0033] . In the present evaluation, it was shown that the present method can reduce storage costs compared to K-FAC. Frantal et al. [12, 13] use the layer-wise reconstruction loss to identify redundancy between elements of weight matrices. Still, there is not known to be theoretical result that shows optimizing this reconstruction loss is as good as that proposed in prior works [18, 26]. A recent work

[0028] approximates pre-trained weights via a combination of low-rank and sparse matrix. Finally,

[0024] proposes a mask search, rearrangement and tuning technique for fast post-training pruning. Although their method is guided by FIM, it uses a crude diagonal approximation and disregards correlations between weight elements.

[0123] Adapters for Parameter-Efficient Fine-Tuning. Houlsby et al.

[0019] proposes parameter-efficient transfer learning via injecting new layers after every submodule consisting of a few layers. This significantly reduces the number of trainable parameters and thus accelerates fine-tuning compared to full fine-tuning. However, this adds computation overheads to the fine-tuned model since the number of parameters increases. A few prior works [31, 46] attempt to improve on Houlsby et al. by reducing the adapter size, however, at the price of compromising model quality. LoRA

[0020] is the state-of-the-art adapter approach, that reduces GPU memory usage, attains high model quality and is easy to use. Recent works [10, 59] improve upon LoRA by dynamically controlling rank in adapters throughout fine-tuning. Dettmers et al. [8] further reduces GPU memory usage by quantizing the frozen pre-trained weights. One may compare with LoRA in the evaluations (Section 5).

[0124] Other LLM Fine-Tuning Approaches. Sung et al.

[0051] use Fisher diagonal to extract a fixed sparse mask on the pre-trained LLM, and identify a set of sparse masked weights (i.e., a subset of the pretrained LLM weights) to be re-trained in fine-tuning. This method has been shown to be effective only in BERTLARGE model with 345 million parameters. Furthermore, there is no efficient implementation to take advantage of the sparsity in the computation.

[0125] Efficient Sparse Linear AlgebraKernels for GPUs. Prior works [5,14, 30] have proposed software optimizations using CUDA for sparse linear algebra (e.g., Sparse Matrix Multiply) for GPU architectures. The present method can work synergistically with these prior works, i.e., integrating optimized sparse computation kernels to improve performance on GPUs further. 26 CPST Doc: 1379-4645-4294.17. Conclusion

[0126] Herein, the present method provides a novel approach that accelerates the ubiquitous fine-tuning process that adapts pre-trained models to downstream tasks. One can design a hierarchical approximation of the second-order information, and an online projection and rediagonalization algorithm to significantly reduce the pre-processing costs to identify important weights compared to K-FAC, the most widely used approximation to the Fisher Information Matrix. This effectively embeds the sparse trainable weights into the pre- trained LLM weights to eliminate computation overheads introduced in adapters. The method’s end-to-end fine-tuning can be, on average, 5.1 times faster than ZeRO, a memory efficient optimizer for full fine-tuning, and on average 1.2 times faster than LoRA, the state- of-the-arts adapter approach for fine-tuning LLMs. In comparison to K-FAC Pruner, the method’s hierarchical approximation can reduce the peak GPU memory by 2.3 times in the pre-processing step and provides better model quality after fine-tuning. It may be concluded that the present method is a highly efficient finetuning method for LLMs. 8. Computing Environment

[0127] Referring now to FIG.10a, an example of a computing environment 10 is shown in which the present method may be deployed. The computing environment 10 includes an LLM fine-tuning system 12, which takes an LLM 14 as an input and applies the present method, e.g., as described above, and fine tunes the LLM 14 for one or more downstream applications 16. In FIG.10, the present method is referred to as a fine-tuning method 18. The system 12 also includes or otherwise has access to fine-tuning data 20. The data 20 may include information, constraints, parameters, etc., as discussed above (e.g., as shown in FIG.1, option (e)) and may also cache or store other data such as weights and biases 15 that are part of an LLM 14, etc.

[0128] It can be appreciated that the system 12 may be running on one or more computing devices 40 (e.g., see FIG.11). Such computing devices 40 (or computing systems) may include, but are not limited to, a personal (e.g., desktop) computer, a server computer or other computing system that is equally or more powerful or otherwise suitably adapted to fine-tune LLMs and other large amounts of data. Such computing devices 40 may also, where applicable, include a mobile phone, a laptop computer, a tablet computer, a notebook computer, a hand-held computer, a personal digital assistant, a portable navigation device, a wearable device, a gaming device, an embedded device, edge device, a virtual reality device, an augmented reality device, etc. 27 CPST Doc: 1379-4645-4294.1

[0129] The system 12 may be hosted or otherwise run on the one or more computing devices 40 or may be accessed by the computing device(s) 40 over a communication network (not shown). Such communication network(s) may include a the Internet, accessed via, for example, a telephone network, cellular, and / or data communication network to connect different types of client- and / or server-type devices. For example, the communication network may include a private or public switched telephone network (PSTN), mobile network (e.g., code division multiple access (CDMA) network, global system for mobile communications (GSM) network, and / or any 3G, 4G, or 5G wireless carrier network, etc.), WiFi or other similar wireless network, and a private and / or public wide area network (e.g., the Internet). The system 12 may be embodied in an application, which may take the form of a mobile-type application (also referred to as an “app”), a desktop-type application, an embedded application in customized computing systems, or an instance or page contained and provided within a web / Internet browser, to name a few.

[0130] The LLM 14 may be provided by a separate one or more computing devices 40 or computing system, by a separate entity or may be integrated with the system 12 within the same computing device(s) 40 or computing system. As such, the configuration shown in FIG.10 is illustrative and other computing device / system configurations are possible. For example, the computing environment 10 shown in FIG.10 may represent a single device such as a portable electronic device or the integration / cooperation of multiple electronic devices such as a client device and server device or a client device and a remote or offsite storage or processing entity or service or multiple client or server devices working together in a fine-tuning job collectively (e.g., data / tensor / pipeline / expert-parallel distributed training) using more than one computing device (e.g., utilizing GPUs). That is, the computing environment 10 may be implemented using any one or more electronic devices including standalone devices and those connected to offsite storage and processing operations (e.g., via cloud-based computing storage and processing facilities).

[0131] FIG.10b illustrates an example workflow that may be executed in the computing environment shown in FIG.10a. In this example, the fine-tuning system 12 and fine-tuning method 18 receive, as inputs, the LLM 14 and its weights and biases 15 and a downstream task or application 16 and its dataset 24. The fine-tuning method 18 may then be applied as described herein to generate a fine-tuned LLM 26, with fine-tunes weights and biases 28.

[0132] FIG.11 shows an example of one such computing device 40, e.g. from a set of one or more computing devices 40, which may be utilized by any one or more of the entities shown in FIG.10, for example, a personal electronic device or server used to provide the system 12 or other computing device 40 used to communicate with the LLM 14 via the 28 CPST Doc: 1379-4645-4294.1system 12. The computing device 40 in FIG.11 may, additionally or alternatively, provide an example of a device on which the LLM 14 may be deployed or accessed. Similarly, the computing device 40 in FIG.11 may provide an example of a device on which the downstream application(s) 16 may be deployed.

[0133] In this example, the computing device 40 includes one or more processors 42 (e.g., a microprocessor, microcontroller, embedded processor, digital signal processor (DSP), central processing unit (CPU), media processor, graphics processing unit (GPU) or other hardware-based processing units) and one or more network interfaces 44 (e.g., a wired or wireless transceiver device connectable to a network via a communication connection).

[0134] Examples of such communication connections can include wired connections such as twisted pair, coaxial, Ethernet, fiber optic, etc. and / or wireless connections such as LAN, WAN, PAN and / or via short-range communications protocols such as Bluetooth, WiFi, NFC, IR, etc.

[0135] The computing device(s) 40 may also include the system 12 (or other application(s)), a data store 52, and client application data 54. The data store 52 may represent a database or library or other computer-readable medium configured to store data and permit retrieval of data by the computing device 40. The data store 52 may be read-only or may permit modifications to the data. The data store 52 may also store both read-only and write accessible data in the same memory allocation. In this example, the data store 52 stores the application data 54 for the system 12 (and / or an application) that is configured to be executed by the computing device 40 for a particular role or purpose.

[0136] While not delineated in FIG.11, the computing device(s) 40 include(s) at least one memory or memory device that can include a tangible and non-transitory computer- readable medium having stored therein computer programs, sets of instructions, code, or data to be executed by processor(s) 42. The processor(s) 42 and network interface(s) 44 are connected to each other via a data bus or other communication backbone to enable components of the computing device 40 to operate together as described herein. FIG.11 illustrates examples of modules and applications stored in memory on the computing device 40 and executed by the processor(s) 42.

[0137] It can be appreciated that any of the modules and applications shown in FIG.11 may be hosted externally and may be available to the computing device 40, e.g., via a network interface 44. The data store 52 in this example stores, among other things, the application data 54 that can be accessed and utilized by an application (and / or the system 29 CPST Doc: 1379-4645-4294.112). The data store 52 may additionally store one or more software functions or routines in a cache or in other types of memory.

[0138] As shown in FIG.11, the computing device(s) 40 may, optionally (e.g., when configured as a personal electronic device such as a smartphone or tablet), include a display 46 and one or more input device(s) 48 that may be utilized via an input / output (I / O) module 50. That is, such components may be omitted when the computing device 40 does not interact with a user.

[0139] While examples referred to herein may refer to a single display 46 for ease of illustration, the principles discussed herein may also be applied to multiple displays 46, e.g., to view portions of UIs rendered by or with an application on separate side-by-side screens. That is, any reference to a display 46 may include any one or more displays 46 or screens providing similar visual functions. The application receives one or more inputs from one or more input devices 48, which may include or incorporate inputs made via the display 46 as well as any other available input to the computing environment 10 (e.g., via the I / O module 50), such as haptic or touch gestures, voice commands, eye tracking, biometrics, keyboard or button presses, etc. Such inputs may be applied by a user interacting with the computing environment 10, e.g., by operating the computing device 40 as illustrated in FIG.10.

[0140] For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the examples described herein. However, it will be understood by those of ordinary skill in the art that the examples described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the examples described herein. Also, the description is not to be considered as limiting the scope of the examples described herein.

[0141] It will be appreciated that the examples and corresponding diagrams used herein are for illustrative purposes only. Different configurations and terminology can be used without departing from the principles expressed herein. For instance, components and modules can be added, deleted, modified, or arranged with differing connections without departing from these principles.

[0142] It will also be appreciated that any module or component exemplified herein that executes instructions may include or otherwise have access to computer readable media such as transitory or non-transitory storage media, computer storage media, or data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical 30 CPST Doc: 1379-4645-4294.1disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory computer readable medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the computing environment 10, any component of or related thereto, etc., or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable / executable instructions that may be stored or otherwise held by such computer readable media.

[0143] The steps or operations in the flow charts and diagrams described herein are provided by way of example. There may be many variations to these steps or operations without departing from the principles discussed above. For instance, the steps may be performed in a differing order, or steps may be added, deleted, or modified.

[0144] Although the above principles have been described with reference to certain specific examples, various modifications thereof will be apparent to those skilled in the art as having regard to the appended claims in view of the specification as a whole. 31 CPST Doc: 1379-4645-4294.1References

[0145] [1] Shun-ichi Amari.1998. Natural Gradient Works Efficiently in Learning. Neural Computation 10, 2 (021998), 251–276. https: / / doi.org / 10.1162 / 089976698300017746

[0146] [2] Fengxiang Bie, Yibo Yang, Zhongzhu Zhou, Adam Ghanem, Minjia Zhang, Zhewei Yao, XiaoxiaWu, Connor Holmes, Pareesa Golnari, David A. Clifton, Yuxiong He, Dacheng Tao, and Shuaiwen Leon Song.2023. RenAIssance: A Survey into AI Text-to- Image Generation in the Era of Large Model. arXiv:2309.00810 [cs.CV].

[0147] [3] Matthew Brand.2006. Fast low-rank modifications of the thin singular value decomposition. Linear Algebra Appl.415, 1 (2006), 20–30. https: / / doi.org / 10.1016 / j.laa.2005.07.021 Special Issue on Large Scale Linear and Nonlinear Eigenvalue Problems.

[0148] [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol.33. Curran Associates, Inc., Vancouver, Canada, 1877–1901.

[0149] [5] Roberto L. Castro, Andrei Ivanov, Diego Andrade, Tal Ben-Nun, Basilio B. Fraguela, and Torsten Hoefler.2023. VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores. arXiv:2310.02065 [cs.DC].

[0150] [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, PeterWelinder, Bob McGrew, Dario Amodei, Sam 32 CPST Doc: 1379-4645-4294.1McCandlish, Ilya Sutskever, and Wojciech Zaremba.2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]

[0151] [7] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.2016. Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174 [cs.LG]

[0152] [8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems, A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol.36. Curran Associates, Inc., New Orleans, USA, 10088–10115.

[0153] [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https: / / doi.org / 10.18653 / v1 / N19-1423.

[0154]

[0010] Ning Ding, Xingtai Lv, QiaosenWang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun.2023. Sparse Low-rank Adaptation of Pre-trained Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 4133–4145. https: / / doi.org / 10.18653 / v1 / 2023.emnlp-main.252.

[0155]

[0011] Carl Eckart and G. Marion Young.1936. The Approximation of One Matrix by Another Of Lower Rank. Psychometrika 1 (1936), 211–218.

[0156]

[0012] Elias Frantar and Dan Alistarh.2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol.35. Curran Associates, Inc., New Orleans, USA, 4475–4488.

[0157]

[0013] Elias Frantar and Dan Alistarh.2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, Honolulu Hawaii USA, 10323–10337.

[0158]

[0014] Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen.2020. Sparse GPU Kernels for Deep Learning. In Proceedings of the International Conference for High 33 CPST Doc: 1379-4645-4294.1Performance Computing, Networking, Storage and Analysis (SC ’20). IEEE Press, Atlanta, Georgia, Article 17, 14 pages.

[0159]

[0015] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.2019. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, USA, 2232–2241.

[0160]

[0016] Erin Griffith.2023. The Desperate Hunt for the A.I. Boom’s Most Indispensable Prize. The New York Times. https: / / www.nytimes.com / 2023 / 08 / 16 / technology / ai-gpuchips-shortage.html.

[0161]

[0017] Roger Grosse and James Martens.2016. A Kronecker-factored approximate Fisher matrix for convolution layers. arXiv:1602.01407 [stat.ML]

[0162]

[0018] Babak Hassibi, David Stork, and Gregory Wolff.1993. Optimal Brain Surgeon: Extensions and performance comparisons. In Advances in Neural Information Processing Systems, J. Cowan, G. Tesauro, and J. Alspector (Eds.), Vol.6. Morgan- Kaufmann, Denver, USA.

[0163]

[0019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.2019. Parameter- Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, USA, 2790–2799.

[0164]

[0020] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL]

[0165]

[0021] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol.32. Curran Associates, Inc., Vancouver, Canada. https: / / proceedings.neurips.cc / paper_files / paper / 2019 / file / 093f65e080a295f8076b1c5722a4 6aa2-Paper.pdf 34 CPST Doc: 1379-4645-4294.1

[0166]

[0022] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari.2019. Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol.89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, Naha, Okinawa, Japan, 1032–1041.

[0167]

[0023] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh.2022. The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 4163– 4181.https: / / doi.org / 10.18653 / v1 / 2022.emnlp-main.279

[0168]

[0024] Woosuk Kwon, Sehoon Kim, MichaelWMahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami.2022. A Fast Post-Training Pruning Framework for Transformers. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol.35. Curran Associates, Inc., New Orleans, 24101– 24116.https: / / proceedings.neurips.cc / paper_files / paper / 2022 / file / 987bed997ab668f91c822a 09bce3ea12-Paper-Conference.pdf

[0169]

[0025] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.2023. OpenAssistant Conversations – Democratizing Large Language Model Alignment. arXiv:2304.07327 [cs.CL]

[0170]

[0026] Yann Le Cun, John S. Denker, and Sara A. Solla.1989. Optimal Brain Damage. In Proceedings of the 2nd International Conference on Neural Information Processing Systems (NIPS’89). MIT Press, Cambridge, MA, USA, 598–605.

[0171]

[0027] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.2022. Competition-level code generation with AlphaCode. Science 378, 6624 (Dec.2022), 1092–1097.https: / / doi.org / 10.1126 / science.abq1158 35 CPST Doc: 1379-4645-4294.1

[0172]

[0028] Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He,Weizhu Chen, and Tuo Zhao.2023. LoSparse: structured compression of large language models based on low-rank and sparse approximation. In Proceedings of the 40th International Conference on Machine Learning (ICML’23). JMLR.org, Honolulu, Hawaii, USA ,Article 839, 15 pages.

[0173]

[0029] Zhenyu Liao and Michael W Mahoney.2021. Hessian Eigenspectra of More Realistic Nonlinear Models. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol.34. Curran Associates, Inc., Virtual, 20104–20117.

[0174]

[0030] Bin Lin, Ningxin Zheng, Lei Wang, Shijie Cao, Lingxiao Ma, Quanlu Zhang, Yi Zhu, Ting Cao, Jilong Xue, Yuqing Yang, and Fan Yang.2023. Efficient GPU Kernels for N:M-SPARSE Weights in Deep Learning. In Sixth Conference on Machine Learning and Systems (MLSys’23). Machine Learning and Systems, Miami, USA.

[0175]

[0031] Zhaojiang Lin, Andrea Madotto, and Pascale Fung.2020. Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 441– 459.https: / / doi.org / 10.18653 / v1 / 2020.findings-emnlp.41

[0176]

[0032] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.2020. Ro{BERT}a: A Robustly Optimized {BERT} Pretraining Approach.

[0177]

[0033] James Martens.2020. New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research 21, 146 (2020), 1–76.

[0178]

[0034] James Martens and Roger Grosse.2015. Optimizing Neural Networks with Kronecker-factored Approximate Curvature. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.37), Francis Bach and David Blei (Eds.). PMLR, Lille, France, 2408–2417.

[0179]

[0035] Nvidia Mellanox.2014. Introducing EDR 100Gb / s - Enabling the Use of Data. https: / / network.nvidia.com / pdf / whitepapers / wp_introducing_edr_100gb_enabling_use_data. pdf

[0180]

[0036] L. MIRSKY.1960. Symmetric Gauge Functions And Unitarily Invariant Norms. The Quarterly Journal of Mathematics 11, 1 (Jan 1960), 50–59. https: / / doi.org / 10.1093 / qmath / 11.1.50 36 CPST Doc: 1379-4645-4294.1

[0181]

[0037] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia.2021.Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 58, 15 pages. https: / / doi.org / 10.1145 / 3458817.3476209

[0182]

[0038] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 721, 12 pages.

[0183]

[0039] Yevgeniy Puzikov and Iryna Gurevych.2018. E2E NLG Challenge: Neural Models vs. Templates. In Proceedings of the 11th International Conference on Natural Language Generation, Emiel Krahmer, Albert Gatt, and Martijn Goudbeek (Eds.). Association for Computational Linguistics, Tilburg University, The Netherlands, 463–471. https: / / doi.org / 10.18653 / v1 / W18-6557

[0184]

[0040] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.2019. Language Models are Unsupervised Multitask Learners. Open AI.

[0185]

[0041] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67.

[0186]

[0042] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. ArXiv.

[0187]

[0043] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He.2021. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 59, 14 pages. https: / / doi.org / 10.1145 / 3458817.3476205 37 CPST Doc: 1379-4645-4294.1

[0188]

[0044] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.2020. Deep-Speed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, NY, USA, 3505–3506. https: / / doi.org / 10.1145 / 3394486.3406703

[0189]

[0045] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.2022. High-Resolution Image Synthesis with Latent Diffusion Models.arXiv:2112.10752 [cs.CV]

[0190]

[0046] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych.2021. AdapterDrop: On the Efficiency of Adapters in Transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7930–7946. https: / / doi.org / 10.18653 / v1 / 2021.emnlp-main.626.

[0191]

[0047] Alec Radford Scott Gray and Diederik P. Kingma.2017. GPU Kernels for Block-Sparse Weights. https: / / cdn.openai.com / blocksparse / blocksparsepaper.pdf

[0192]

[0048] Anton Shilov.2023. TSMC: Shortage of Nvidia’s AI GPUs to Persist for 1.5 Years. Tom’s Hardware. https: / / www.tomshardware.com / news / tsmc-shortage-ofnvidias-ai- gpus-to-persist-for-15-years.

[0193]

[0049] Sidak Pal Singh and Dan Alistarh.2020. WoodFisher: Efficient Second-Order Approximation for Neural Network Compression. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol.33. Curran Associates, Inc., Virtual, 18098–18109.

[0194]

[0050] Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann.2021. Analytic Insights into Structure and Rank of Neural Network Hessian Maps. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J.Wortman Vaughan (Eds.), Vol.34. Curran Associates, Inc., Virtual, 23914–23927.

[0195]

[0051] Yi-Lin Sung, Varun Nair, and Colin A Raffel.2021. Training Neural Networks with Fixed Sparse Masks. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol.34. Curran Associates, Inc., Virtual, 24193–24205. 38 CPST Doc: 1379-4645-4294.1

[0196]

[0052] Philippe Tillet, H. T. Kung, and David Cox.2019. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (Phoenix, AZ, USA) (MAPL 2019). Association for Computing Machinery, New York, NY, USA, 10–19. https: / / doi.org / 10.1145 / 3315508.3329973.

[0197]

[0053] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.2023. LLaMa: Open and Efficient Foundation Language Models.arXiv:2302.13971 [cs.CL]

[0198]

[0054] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H.Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol.30. Curran Associates, Inc., Long Beach, USA.

[0199]

[0055] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Tal Linzen, Grzegorz Chrupała, and Afra Alishahi (Eds.). Association for Computational Linguistics, Brussels, Belgium, 353–355. ttps: / / doi.org / 10.18653 / v1 / W18-5446

[0200]

[0056] Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang.2019. EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.).PMLR, Long Beach, USA, 6566–6575.

[0201]

[0057] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language 39 CPST Doc: 1379-4645-4294.1Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 5085–5109. https: / / doi.org / 10.18653 / v1 / 2022.emnlp-main.340

[0202]

[0058] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu.2023. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. CoRR abs / 2304.13712 (2023).

[0203]

[0059] Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao.2024. APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference. arXiv:2401.12200 [cs.CL] 40 CPST Doc: 1379-4645-4294.1

Claims

Claims:

1. A method of fine-tuning a large language model (LLM), comprising: identifying a set of important LLM weights via second-order information in a pre- processing step; and reducing computation and storage of the pre-processing step by: applying a hierarchical approximation of the second-order information, and executing an online projection and rediagonalization process.

2. The method of claim 1, wherein only sparse important weights are trained, and the method further comprises embedding the sparse weights into the pre-trained LLM during fine-tuning.

3. The method of claim 2, wherein the sparse weights and frozen pre-trained weights are kept together to reduce computation requirements during the fine-tuning.

4. The method of claim 1, further comprising determining weight importance by minimizing the impact of pruning on LLM quality.

5. The method of claim 1, wherein the hierarchical approximation of the second-order information comprises a hierarchical approximation of Kronecker factors.

6. The method of claim 5, comprising recursively halving the Kronecker factors and approximating off-diagonal blocks.

7. The method of claim 6, wherein the off-diagonal blocks are approximated using singular value decomposition (SVD).

8. The method of claim 7, comprising using an online algorithm to directly update SVD factors without materializing approximated blocks.

9. The method of claim 8, wherein, for each layer, both inputs and output gradients are saved. 41 CPST Doc: 1379-4645-4294.

110. The method of claim 9, comprising updating corresponding ones of the SVD factors in the hierarchical approximation using the inputs and the output gradients.

11. The method of any one of claims 7 to 10, wherein the hierarchical approximation of the second-order information, comprises: projecting incoming data onto an existing basis; rediagonalizing to obtain rotation matrices; and rotating an extended subspace to obtain updated SVD factors.

12. The method of any one of claims 1 to 11, further comprising: applying a fine-tuning step by: (i) computing a forward pass; and (ii) computing a backward pass by computing gradients of loss with respect to the inputs and computing the gradients of loss with response to the sparse weights.

13. The method of any one of claims 1 to 12, wherein a dataset being processed by the method is divided into a plurality of batches and each batch is processed one by one.

14. A computer readable medium storing processor-executable instructions for fine- tuning a large language model (LLM), comprising instructions that, when executed by a processor, cause a computer system to perform the method of any one of claims 1 to 13.

15. A computer system comprising at least one processor and at least one memory, the memory storing processor-executable instructions for fine-tuning a large language model (LLM), comprising instructions that, when executed by the at least one processor, cause the computer system to perform the method of any one of claims 1 to 13. 42 CPST Doc: 1379-4645-4294.1

Images