Parameter-efficient large-language fine-tuning federated learning framework

Abstract

Provided in the present invention is a parameter-efficient large-language fine-tuning federated learning framework, comprising the following steps: performing modeling on LoRA adapters of different edge clouds; since different weights exhibit different average performances on the LoRA adapters, using singular values to quantify the importance of the weights, and therefore, before each round of independent training of the LoRA adapters using N edge clouds, using a matrix singular value to decompose a BA matrix in the LoRA adapter for each trainable weight; configuring heterogeneous LoRA adapters on the basis of the importance of the weights; and using different numbers of quantization bits to quantize a pre-trained model, and performing high-precision inverse quantization on the pre-trained model only when matrix multiplication is executed, wherein the pre-trained model is quantized to the maximum number of quantization bits on the basis of the memory budget of the edge clouds. The present invention has the following beneficial effects: the present invention determines the optimal fine-tuning model structure, thereby improving the performance of LLM fine-tuning, and adapts to heterogeneous and resource-constrained edge clouds.

Description

A parameter-efficient large language fine-tuning federated learning framework Technical Field

[0001] This invention relates to the field of large-scale language model fine-tuning technology, and in particular to a parameter-efficient federated learning framework for large-scale language fine-tuning. Background Technology

[0002] 1.1 Parameter-efficient fine-tuning: PEFT (Parameter-efficient fine-tuning) is a recently emerging efficient LLM fine-tuning method that achieves a balance between the number of trainable parameters and training accuracy in NLP tasks. Among various PEFT methods, Adapter and LoRA are the most attractive techniques, which can significantly reduce the number of trainable parameters without compromising LLM fine-tuning performance. Adapter inserts small trainable modules into each Transformer block while keeping the parameters of the pre-trained model frozen to improve training efficiency. Compactor is a variant of Adapter, which achieves lower parameter complexity based on shared information between adapters and the low-rank subspace of the model. However, the Adapter method introduces additional computational overhead at the adapter layer, increasing the latency of LLM fine-tuning, especially at small batch sizes. In contrast, LoRA

[0018] optimizes the changes in dense layers through low-rank decomposition matrices while keeping the parameters of the pre-trained model frozen. Unlike the serial insertion method of Adapter, the low-rank matrix is ​​inserted into the fine-tuning module in parallel, alleviating any additional training latency and thus improving the efficiency of LLM fine-tuning.

[0003] 1.2 FL Paradigm for LLM: PEFT requires fine-tuning of the LLM based on domain-specific data for downstream tasks. However, due to data privacy considerations, especially for privacy-sensitive data such as medical images, clients may be unwilling to share the original data with the server. A promising solution to this problem is the FL paradigm, which allows multiple users to train the model without sharing the original data. Recently, some research has combined LLM fine-tuning with the FL paradigm, and this is the first work to combine FL with LLM fine-tuning, comprehensively validating the effectiveness of the FedNLP framework in four common NLP tasks: text classification, sequence labeling, question answering, and sequence generation. FedAdapter addresses the high training cost problem of FedNLP by progressively upgrading adapter configurations and continuously optimizing future adapter configurations, thus accelerating model convergence. However, existing work only focuses on traditional language models, such as BART or BERT, while neglecting the most common LLM models (such as GPT-2) with a large number of parameters.

[0004] 1.3 Model Compression and Quantization: The number of parameters in LLM is unprecedented, making computation and storage a major bottleneck for LLM fine-tuning. Model compression and quantization have become promising techniques that can reduce the storage size of the model and accelerate the inference process by reducing bit precision. GPTQ uses second-order information for error compensation, which significantly improves the accuracy and training efficiency of the model. An edge clipping quantization method called AWQ is proposed, which quantizes the weights into low-order integers, thereby reducing hardware requirements and accelerating label generation. Unlike previous quantization work on LLM inference, the SwitchBack layer studies backpropagation on quantized weights with 1 billion parameters, while QLoRA quantizes the weights into 4-bit NormalFloat and backpropagates the gradients to the low-rank structure of the pre-trained LLM.

[0005] 1.4 Resource Heterogeneity: Resource heterogeneity poses a significant challenge to FL frameworks deploying LLM. Clients with limited computing or memory resources require longer times to complete local model updates, hindering the FL training process or even excluding them from model training. Currently, there is no existing work addressing resource heterogeneity in FL frameworks for LLM fine-tuning.

[0006] In practice, deploying FL on an LLM system encounters several problems, including the following:

[0007] 2.1 Introducing PEFT into Federated Learning of LLM: Due to the limited computing resources in edge clouds (e.g., commercial GPUs, such as RTX 3090) and the large number of parameters in LLMs, full parameter fine-tuning in each edge cloud (i.e., initializing the model with pre-trained weights, updating all parameters, and generating separate instances for various tasks) leads to unacceptable computational and network latency in model training and aggregation. Figure 16 shows the number and size of parameters in several popular LLMs. Clearly, the smallest model, GPT-2, has 774 million parameters, more than 7 times larger than the 11 million parameters of traditional language models (such as BERT). To address the aforementioned impractical obstacles in LLMFL, PEFT, used for downstream task training, is a potential solution. Unlike full parameter fine-tuning, PEFT only fine-tunes a subset of neurons to reduce the number of trainable parameters in the model, thereby reducing computational costs. Recently, there have been two main types of PEFT: ADAPTER and LoRA, as shown in Figure 2. ADAPTER inserts several layers into each Transformer block and trains only these layers from the pre-trained model. Clearly, ADAPTER introduces additional latency during the inference phase due to its extra layers; however, LoRA, a low-rank adaptive adapter, does not introduce additional inference latency. It injects a trainable rank factorization matrix into each transformer layer but freezes all weights of the pre-trained model. Many studies have verified that LoRA outperforms ADAPTER, so we will focus on LoRA in the following sections. To better understand the computational and communication bottlenecks in LLMFL, we set up two experiments using one of the most common LLMs, GPT-2, Figures 3a and 3b (where r is the rank size of the LoRA adapter) to show the batch data training time and parameter exchange time of full parameter fine-tuning and PEFT. Compared to full parameter fine-tuning, PEFT reduces the training time and parameter exchange time by more than 1.4 times and 1000 times, respectively. Clearly, PEFT can significantly reduce computational and communication overhead compared to fine-tuning the entire model. However, three main challenges were encountered in integrating PEFT into LLMFL, and the term "training" will be used to refer to fine-tuning in the following sections.

[0008] 2.2 Severe Lag Issues in LLMFL: State-of-the-art LLMFL frameworks typically assume that each edge cloud has sufficient local resources (e.g., computing power, communication capabilities, and memory size) to perform local model training. Therefore, the central cloud can easily aggregate these local models to achieve good performance. However, in practice, the available resources of each edge cloud vary significantly, and the allocation of resources for training may change depending on runtime needs and priorities. More importantly, due to the large number of parameters in LLM, the resource heterogeneity of the edge clouds leads to significant differences in training time and results in severe lag issues in model aggregation. To better understand the impact of heterogeneous resources on lag issues in LLMFL, a standard model and an LLM... Two experiments were conducted, integrating LoRA into FedAvg, one of the most commonly used fine-tuning (FL) frameworks, to evaluate its performance on the widely accepted LLM / PT-2 and the standard model BERT (for heterogeneous and homogeneous scenarios). Two PC servers (i.e., edge clouds) were first set up. Different GPU memory configurations (22GB and 24GB) were used, with the same 24GB memory configured for both heterogeneous and homogeneous scenarios to simulate varying computational capabilities. The results are shown in Figure 4. The undertraining rate refers to the ratio of undertrained edge clouds to the total number of edge clouds, and the average aggregation time refers to the time per round from the start of local model training on a batch of data to the end of model aggregation. It is clear that in heterogeneous scenarios, both the standard model and LLM exhibit lag issues. However, even with a 10 TFLOPS difference in computational power between the two edge clouds, LLM performs significantly worse than the standard model in terms of both undertraining rate and aggregation time. Therefore, a new FL fine-tuning framework needs to be developed for LLM to adapt to heterogeneous resources on different edge clouds.

[0009] 2.3 Weight Level Configuration: Although LoRA reduces a significant amount of computational overhead in LLM (see Section 2.1), training all weights in each layer using LoRA still incurs a considerable workload due to the limited computational power of edge clouds. Multiple LoRA adapter versions are trained by increasing the number of trainable weights in each layer, as shown in Figure 5a. Perplexity (PPL) is used as a performance metric to evaluate the model's learning of the distribution of its training data; a lower PPL indicates better performance. Clearly, the more weights a LoRA adapter has, the higher the computational overhead, but the better the desired and actual performance. A simple solution is to configure the number of trainable weights according to the computational budget. However, determining the prior weights is a non-trivial task. Another measurement experiment was conducted to investigate the impact of weight indexing in LoRA training. First, four trainable weights (i.e., Wq, Wk, Wv, Wo, as shown in Figure 2) were selected from the Transformer block to train the LoRA adapter, as shown in Figure 5b. The experiment demonstrates that different weight indices have different performance characteristics. Therefore, it is necessary to investigate how to automatically identify and prioritize important trainable weights within the computational budget of each edge cloud.

[0010] 2.4 Budget-Aware Model Parameter Alignment: Although the training layers are determined, several parameters still affect LoRA training, especially batch size, decomposition rank, dropout, and alpha. Evaluating all these parameters reveals that batch size and decomposition rank are the two parameters with the greatest impact on training performance. To investigate the impact of batch size and decomposition rank, they were increased to train a pre-trained GPT-2 model, and the results are illustrated in Figure 6. It is clear that a larger batch size or decomposition rank always yields better performance and greater memory usage. However, given the computational budget of the edge cloud, the largest batch size and rank cannot always be chosen. The optimal combination of batch size and rank needs to be found within a parameter-efficient, large-language fine-tuning federated learning framework to adapt to the computational budget of each edge cloud.

[0011] Summary of the Invention

[0012] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a parameter-efficient federated learning framework for large language fine-tuning, which solves the problem that existing fine-tuning model structures have poor fine-tuning performance for large language models and are not suitable for heterogeneous and resource-constrained edge clouds.

[0013] To achieve the above and other related objectives, the present invention provides the following technical solution:

[0014] A parameter-efficient large language fine-tuning federated learning framework includes the following steps: modeling LoRA adapters for different edge clouds, wherein N edge clouds independently train LoRA adapters, and after training, a central cloud aggregates all LoRA adapters from the edge clouds; since different weights have different average performance on the LoRA adapters, singular values ​​are used to quantify the importance of the weights, therefore, before each round of training LoRA adapters independently using N edge clouds, the BA matrix in each LoRA adapter with trainable weights is decomposed using matrix singular values;

[0015] The heterogeneous LoRA adapter is configured according to the importance of the weights. The purpose of the heterogeneous LoRA adapter configuration is to select the batch size and rank of each edge cloud, and to perform rank adjustment for each weight iteration. The optimal LoRA adapter for each edge cloud is dynamically reconfigured during each training round. The pre-trained model is quantized using different quantization bits, and the pre-trained model is dequantized with high precision only when performing matrix multiplication. The pre-trained model is quantized to the maximum quantization bit according to the memory budget of the edge cloud, but the quantization bit of the LoRA adapter remains unchanged.

[0016] In one embodiment of the present invention, modeling the LoRA adapter for different edge clouds includes: overparameterizing the model to have a substantially low intrinsic dimension, thus reducing the number of trainable parameters; and LoRA enhancing the parameters by combining the parameters of the frozen pre-trained model with additional factorization projections.

[0017] In one embodiment of the present invention, before training the LoRA adapter independently using N edge clouds in each round, the BA matrix in the LoRA adapter of each trainable weight is decomposed using matrix singular values. This includes: since larger singular values ​​represent richer data information, the average singular value of each trainable weight is used to characterize the importance of the weight. Therefore, in the t-th round of training, the importance metric for correctly measuring the parameter contribution of the m-th trainable weight (LoRA adapter) of the i-th edge cloud can be calculated as follows: in, This represents the j-th singular value of the m-th trainable weight (LoRA adapter) of the i-th edge cloud. This represents the m-th trainable weight (LoRA adapter) of the i-th edge cloud, where d1 is the total number of singular values, and φ (t) (·) is a sensitivity-based importance scoring function used to identify the importance weights of a model.

[0018] In one embodiment of the present invention, configuring the heterogeneous LoRA adapter according to the importance of weights includes: taking into account the data scale C = [C1,max C 2,max ,…,C N,max ] Calculate the proportion of all values ​​in C, expressed as U r =[u1:u2:…:u N Furthermore, the batch size of the edge cloud is determined by the formula B = U. r ×b max It is confirmed that, note that any value in B less than b... min The value of b i All will be replaced with b min After determining the batch size B, for the i-th edge cloud, according to formula b i ·f c (W i R i, )≤C i,,max Then through Obtain rank-related computational resource constraints.

[0019] In one embodiment of the present invention, configuring the heterogeneous LoRA adapter according to the importance of the weights further includes: sorting the weights according to their importance, starting with the most important weights and then trying Q-factors in descending order. m The rank value in the equation is determined if the computational power corresponding to the highest rank is lower than the maximum computational constraint. If the condition is not met, the next highest rank is used for evaluation. The above steps are repeated for other weights until the combination of weights and ranks meets the computational constraints. Among them, Q m It is a matrix used to store rank values ​​from highest to lowest.

[0020] In one embodiment of the present invention, quantizing the pre-trained model using different quantization bits and performing high-precision dequantization of the pre-trained model only during matrix multiplication includes: since the weights of the pre-trained model typically follow a zero-mean normal distribution, first converting all weights to a zero-mean normal distribution in the range [-1, 1] using the standard deviation, and then estimating 2 k +1 quantile to obtain a k-bit quantized data type, and normalize the values ​​of this data type to the range [-1, 1]. Finally, quantize the input weight tensor to the range [-1, 1] using absolute maximum rescaling. Once the weight range and data type range match, estimate the 2^k values ​​of the data type as follows: k The value q i : Where Q(·) is the quantile function of the standard normal distribution.

[0021] In one embodiment of the present invention, the method further includes: aggregating and deploying the configured heterogeneous LoRA adapters, wherein the incremental matrices of the edge clouds have a consistent dimension and are used for incremental matrices with the same trainable weights. Therefore, instead of directly aggregating the rank decomposition matrix, incremental matrices with the same trainable weights are aggregated from the edge clouds, and the incremental matrices are also aggregated for use in the pre-trained model for model updates.

[0022] In one embodiment of the present invention, the aggregation and deployment of the configured heterogeneous LoRA adapters includes: after receiving the updated rank decomposition matrices B and A that participate in the edge cloud transmission, the central cloud multiplies these decomposition matrices to obtain the corresponding incremental matrices; then the central cloud aggregates the preprocessed incremental matrices from different edge devices; and finally, the local pre-trained model is updated according to the incremental matrices.

[0023] As described above, the parameter-efficient large language fine-tuning federated learning framework of the present invention has the following beneficial effects:

[0024] This invention aims to facilitate fine-tuning of LLMs in edge clouds with heterogeneous computing resources. The problem is modeled as a MILP (Multi-Input Multi-Output) model to guide the design of this invention. To address this issue, significant weights are first selected, and then appropriate parameters are configured under the computational budget of the LoRA adapter in the edge cloud. Heterogeneous memory and LLM quantization based on the edge cloud budget are also considered, further reducing memory resource usage. Extensive evaluations were conducted by comparing with state-of-the-art benchmarks, showing that FedPiL achieves excellent performance in LLM FL (Fluid-Free Optimization). Therefore, this invention identifies the optimal fine-tuning model structure to improve the performance of LLM fine-tuning and adapt to heterogeneous and resource-constrained edge clouds. Attached Figure Description

[0025] Figure 1 shows a schematic diagram of the joint framework of federated learning and local model update in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention, and the scenario of edge cloud.

[0026] Figure 2 shows an example of the LoRA method in the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention;

[0027] Figure 3 shows a schematic diagram of the training time and parameter swapping of LoRA and fully parameter-tuned GPT-2 in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0028] Figure 4 shows a schematic diagram of the latency issues in GPT-2 and BERT in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0029] Figure 5 shows a schematic diagram of the training performance of GPT-2 under different trainable weights in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0030] Figure 6 shows the impact of the LoRA adapter in the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention on the performance and memory usage of GPT-2 under different training batch sizes and ranks.

[0031] Figure 7 shows a schematic diagram of the system overview of FedPiL in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0032] Figure 8 shows a schematic diagram of a scenario for training LoRA in LLMFL within the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention.

[0033] Figure 9 shows a schematic diagram of calculating the average normalized importance index and perplexity (PPL) for different GPT-2 models in 20 training rounds in the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention.

[0034] Figure 10 shows a schematic diagram of the automatic LoRA adapter configuration solution in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0035] Figure 11 shows a schematic diagram of adapter aggregation in the FedPiL framework of the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0036] Figure 12 shows the convergence accuracy of the Bert and GPT-2 models under heterogeneous and ideal settings in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0037] Figure 13 shows a schematic diagram of the number of trainable parameters of the BERT and GPT-2 models in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0038] Figure 14 shows a schematic diagram of the convergence rate of the Bert and GPT-2 models under heterogeneous and ideal settings in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0039] Figure 15 shows a schematic diagram of the elimination evaluation of the GPT2-L model in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiment of the present invention.

[0040] Figure 16 shows a schematic diagram of the well-known large model trainable parameters and size statistics table in the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0041] Figure 17 shows a schematic diagram of the statistical tables of GPT-2 medium (M) and large (L) models using different adaptation methods in the E2ENLG challenge within the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention.

[0042] Figure 18 shows a schematic diagram of the statistical table of the GPT-2 large model using different combinations of trainable weights in the E2ENLG challenge within the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention.

[0043] Figure 19 shows a schematic diagram of the statistical table of a large GPT-2 model with different heterogeneous configurations in the E2ENLG challenge within the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention.

[0044] Figure 20 shows a schematic diagram of the statistical table of the FedPiL framework with different quantization bits on the GPT-2 medium-sized model in the E2ENLG challenge of the parameter-efficient large language fine-tuning federated learning framework disclosed in the embodiments of the present invention.

[0045] Figure 21 shows a schematic diagram of the overall configuration process of the heterogeneous LoRA adapter in the parameter-efficient large language fine-tuning federated learning framework disclosed in this embodiment of the invention. Detailed Implementation

[0046] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0047] This invention relates to a parameter-efficient, large-language fine-tuning federated learning framework, as detailed below:

[0048] 3.1 Overview of a Parametrically Efficient Large Language Fine-Tuning Federated Learning Framework: A parametrically efficient large language fine-tuning federated learning framework (federated pipeline: FedPiL) is used for fine-tuning the FL (Flexible Interchange) of LLMs. Unlike existing methods, FedPiL's goal is not to introduce new PEFT techniques, but rather to serve as a foundational framework to enhance any existing PEFT methods. This goal is achieved by systematically decoupling, abstracting, and automating the operations and parameters of these methods. As a result, application and algorithm developers can efficiently outsource the implementation details of LLM FL fine-tuning, allowing them to focus on their main service or model design. First, a multi-objective integer linear programming problem (MILP) is theoretically established to simulate LLM. To address the lag issues in FL under heterogeneous and limited computing resources, and to guide the design of the parameter-efficient large language fine-tuning federated learning framework (FedPiL), a two-level solution is proposed: an important weight identification mechanism and heterogeneous adapter configuration. Heterogeneous memory and edge cloud budget-based quantization of LLM are also considered. Finally, an effective method is designed to aggregate all local models with different quantization LoRA adapters. After completion, the next round of training continues. See Figure 1 for details.

[0049] 3.2 The specific steps for modeling heterogeneous LoRA adapters are as follows: Model the LoRA adapters for different edge clouds, considering heterogeneous computing resources, as shown in Figure 8. N edge clouds independently train LoRA adapters, and each edge cloud has its own computing budget C. i,max (representing the computational budget of the i-th edge cloud), after training, the central cloud aggregates all LoRA adapters of the edge clouds; LoRA is constructed based on the following view: overparameterized models have essentially low intrinsic dimensionality, thus reducing trainable parameters. LoRA enhances parameters by combining the parameters of the frozen pre-trained model with additional factorization projections, which express the weight updates as a low-rank decomposition W = W0 + ΔW, where ΔW = BA. Rank r << min(d) i ,d o (For example, when d) i =d o When r = 1024, r = 8), the dimensions of the neural network input and output are d and d, respectively. i and d o Then you can use f p (r)=r(d i +d o )Calculate the total number of trainable parameters for LoRA;

[0050] The LoRa adapter for the i-th edge cloud is determined by its weights and biases W.i This indicates that its local training dataset is... Where, x i,j and y i,j Let represent the j-th input data and its corresponding label, respectively; therefore, the local loss function of the i-th edge cloud can be expressed as: Among them, L i,j (x i，j ,y i,j W i ) represents the dataset The sample-by-sample loss function for the j-th data sample in the dataset, expressed as follows: The global loss function can be expressed as a weighted average of the local loss functions. This is to find the optimal model parameters W. * The learning objective of the global model can be constructed as Equation (1): in, Referring back to Section 2.3, the direct impact of batch size and rank on LoRA training performance, and rewriting equation (1) in the form of formula (2a):

[0051] in, Let st represent the set of trainable weights and their corresponding rank, and let equations (2b) to (2e) be as follows: min ≤b i ≤b max , b i ·f c (W i R i )≤C i,max Among them, constraint equation (2b) relates to the range of batch sizes for model training, b i It is the batch size of the i-th edge cloud; constraint equation (2c) is used to select the important weights in the transformer layer (see Section 2.3); equation (2c) ensures that the m-th weights are in Q m The selected value is m, which ranges from 1 to M=4, representing different weights (i.e., W). q W k W v W o Equation (2e) represents the computational budget for the i-th edge cloud; where f c (·) Mapping the trainable weight structure to floating-point operations per second (FLOPS) is an optimization problem that is a MILP problem and is generally NP-hard. Therefore, in Sections 3.3 to 3.5, a parameter-efficient large language fine-tuning federated learning framework is proposed to solve this problem.

[0052] 3.3 The specific steps for important weight identification are as follows: Referring back to Figure 5 (see Section 2.3), after the entire training process is completed, it is observed that different weights have different average performance on the LoRA adapter. In fact, the situation is more complicated because different weight combinations will also lead to different training performance. More importantly, under the computational budget, the i-th edge cloud needs to identify and predict the weight combination with the LoRA adapter for the next round of training.

[0053] To identify a set of important weights for each edge cloud, FedPiL introduces an important weight metric to quantify the importance of parameters and their sensitivity to training loss. The well-known principal component analysis (PCA) algorithm uses singular value decomposition (SVD) to identify information-rich dimensions with large singular values. Inspired by this, singular values ​​are first used to quantify the importance of weights. Therefore, before each training round, the BA (see Section 3.2) of each trainable weight is decomposed by SVD. Since larger singular values ​​represent richer data information, the average singular value of each trainable weight is used to characterize the importance of the weight. In addition, a sensitivity-based importance scoring function is designed. Therefore, in the t-th training round, the importance metric for correctly measuring the parameter contribution of the m-th trainable weight (LoRA adapter) of the i-th edge cloud can be calculated as shown in Equation (3):

[0054] In formula (3), This represents the j-th singular value of the m-th trainable weight (LoRA adapter) of the i-th edge cloud. Let d1 represent the m-th trainable weight (LoRA adapter) of the i-th edge cloud, where d1 is the total number of singular values, and φ is the sum of the sums ... t (·) is a sensitivity-based importance scoring function used to identify the importance weights of the model. To explain φ... t For the (·) function, first define the magnitude of the gradient-weight product as shown in formula (4):

[0055] In formula (4), w j and Let d1 represent the j-th trainable parameter of model parameter W and its corresponding gradient, and d2 be the total number of model parameters W. Equation (4) basically approximates the average change of each parameter of the loss function when a parameter is set to zero [25,31]. However, in equation (4), sensitivity is not a reliable indicator of the importance of weight W because it is estimated based on a small batch of data samples. Under random sampling and complex training dynamics, using equation (4) to estimate sensitivity will lead to high variability and significant uncertainty. Therefore, this problem is solved by designing sensitivity smoothing and uncertainty quantification, as shown in equations (5) and (6):

[0056] In formulas (5) and (6), 0 < λ1, λ2 < 1, This indicates the sensitivity of smoothing using an exponential moving average. Representing the uncertainty term, subsequently, an importance scoring function φ based on sensitivity is used. t (·) is defined as and The product is shown in formula (7): As shown in Figure 9, trainable weights with higher importance metrics exhibit better fine-tuning performance, i.e., lower PPL values. This further demonstrates the effectiveness of the proposed importance metric, based on the importance of the weights. The i-th edge cloud can be configured with a set of LoRA adapters in the next section.

[0057] 3.4 The specific steps for configuring the heterogeneous LoRA adapter are as follows: An automatic configuration solution based on the proposed important weight index is proposed to select the batch size and rank of each edge cloud; due to the huge search space, it is impossible to perform an exhaustive search to determine all possible batch sizes, weights, and ranks; FedPiL uses a two-stage search algorithm to solve this problem. The first stage is to determine the batch size based on data diversity, and the second stage is to find a suitable rank within the computational budget; in order to synchronize model aggregation among various edge clouds (to align the completion time of local model updates from different edge clouds), the selection of the batch size is related to the computational power of each edge cloud. For example, two edge clouds with computational powers of 20 TFLOPs and 40 TFLOPs respectively contribute differently to the LoRA adapter. By studying the impact of data diversity on the batch size (as shown in Figure 6), it is found that a larger data scale contributes better to the LoRA adapter. Therefore, considering the data scale of N edge clouds C = [C 1,max C 2,max ,…,C N,max FedPiL calculates the proportion of all values ​​in C, denoted as U. r= [u1:u2:…:u N ], and the batch size of the edge cloud is determined by B=U r ×b max It is confirmed that, note that any value in B less than b... min The value of b i All will be replaced with b min Please refer to Figure 10 for details.

[0058] After determining the batch size B, for the i-th edge cloud, according to formula 2e, FedPiL can be used... Given the computational resource constraints related to rank, in order to select the rank and weights, since the importance of the weights has already been ordered (see Section 3.3), FedPiL first starts with the most important weights and then tries Q in descending order of importance. m The rank value in the equation is determined if the computational power corresponding to the highest rank is lower than the maximum computational constraint. If the condition is not met, the next highest rank is used for evaluation. This process is repeated for other weights until the combination of weights and ranks meets the computational constraints. Finally, for each weight, the rank is adjusted iteratively, and the optimal LoRA adapter for each edge cloud is dynamically reconfigured during each training round. The entire process is summarized as shown in Algorithm 1. For details of the process of Algorithm 1, please refer to Figure 21.

[0059] 3.5 The specific steps of quantization for adapter training are as follows: Besides the LoRA adapter configuration, the limited and heterogeneous GPU memory resources of edge clouds may lead to an increase in the undertraining rate, thus posing a significant challenge to LLM fine-tuning. Although model quantization has become a promising method to address memory constraints, existing methods do not consider the different quantization bit depths between edge clouds; considering the heterogeneous GPU memory budget (M... i,max For each i = 1, 2, ..., N, the pre-trained model is quantized using different quantization bits, and high-precision dequantization is only performed when matrix multiplication is executed. The pre-trained model consumes a large amount of memory and requires a high compression ratio to save memory (usually 4 bits or 8 bits). In contrast, the LoRA adapter requires less memory and is used for updates, so a lower compression ratio (e.g., 16-bit floating-point (FP16)) is used to maintain accuracy. Therefore, in FedPiL, the pre-trained model is quantized to the maximum quantization bit according to the memory budget of the edge cloud, but the LoRA adapter remains unchanged. The quantization process will be described in detail below.

[0060] In this invention, a NormalFloat (NF) quantization method based on Quantile Quantization is adopted. This method has significant advantages in model compression, reduced computational complexity, accuracy preservation, and hardware compatibility. The NF data type ensures that each quantization interval of the input tensor has an equal number of values. It estimates quantiles through an empirical cumulative distribution function, thereby improving quantization accuracy. Since the weights of the pre-trained model usually follow a zero-mean normal distribution, all weights are first converted to a zero-mean normal distribution in the range [-1, 1] using the standard deviation, and then the 2... k We increment the quantile by +1 to obtain a k-bit quantized data type, normalize the values ​​of this data type to the range [-1, 1], and finally quantize the input weight tensor to the range [-1, 1] using absolute maximum rescaling. Once the weight range and data type range match, we estimate the 2^k values ​​of the data type as follows: k The value q i : Here, Q(·) is the quantile function of the standard normal distribution, and in this way, the quantization process of the pre-trained model is completed.

[0061] 3.6 The specific steps for the aggregation and deployment of LoRA adapters are as follows: The current FL model aggregation aggregates the entire model, but for LLM, transmitting them to the central cloud would incur significant communication overhead (see Section 2.2). Therefore, in the FedPiL design, only the LoRA adapters of the edge cloud are transmitted, instead of the entire model. However, due to the heterogeneous configuration approach in Section 3.4, FedPiL needs to merge various LoRA adapters from different edge clouds.

[0062] A novel model aggregation method is proposed to merge the heterogeneous LoRA adapter structures obtained in Section 3.4. These different LoRA adapter structures are realized by aggregating rank factorization matrices. Fortunately, the increment matrices of the edge clouds have a consistent dimension, which provides a basis for the design of the model aggregation scheme. Therefore, instead of directly aggregating rank factorization matrices, increment matrices with the same trainable weights are aggregated from the edge clouds. This design transforms the heterogeneous LoRA adapters into a unified dimension, achieving efficient aggregation.

[0063] Furthermore, since the rank decomposition matrix cannot be accurately recovered from the aggregated increment matrix, directly updating the adapter may lead to a deterioration rather than an improvement in training performance. Therefore, the increment matrix is ​​aggregated into a pre-trained model for model updating, thereby avoiding the overhead and performance degradation caused by the inaccurate decomposition of the increment matrix. As shown in Figure 11, the customized heterogeneous model aggregation scheme includes the following three stages:

[0064] a) Model preprocessing: After receiving the updated rank decomposition matrices B and A that participate in the edge cloud transmission, the central cloud multiplies these decomposition matrices to obtain their corresponding increment matrices. In the t-th round of training, the increment matrix of the m-th trainable weight (LoRA adapter) of the i-th edge cloud is represented as shown in formula (9): In formula (9), and Let represent the rank decomposition matrix of the m-th trainable weight of the i-th edge cloud.

[0065] b) Incremental matrix aggregation: In this stage, the central cloud aggregates preprocessed incremental matrices from different edge devices, as shown in formula (10):

[0066] c) Local model update: due to the rank decomposition matrix and Unable to obtain the aggregated incremental matrix ΔW m For accurate recovery, directly updating the adapter may worsen rather than improve training performance, as indicated by the incremental matrix ΔW. m This can be viewed as a model update of a pre-trained model running in parallel with the LoRA adapter, therefore the increment matrix ΔW... m Absorbed into the pre-trained model In the process, the LoRA adapter for the (t+1)th round of training is reinitialized. and The expressions are as follows, as shown in formulas (11) and (12): In formulas (11) and (12), and This is the reinitialized rank factorization matrix. Note that different quantization bits were used to alleviate the storage resource constraints of the edge cloud; therefore, the increment matrix ΔW... m Merging into pre-trained model parameters Previously, quantification was crucial, and based on the aforementioned stages, heterogeneous LoRA adapter structures were effectively aggregated.

[0067] 4. Implementation and Experimental Setup: 4.1 Models: Two well-known LLM models, BERT and GPT-2, were deployed. The BERT model is widely used for various natural language understanding tasks, including text classification, question answering, and named entity recognition. GPT-2 is known for its ability to generate coherent and relevant content, making it well-suited for text generation tasks such as text completion and language translation. The BERT model has 12 layers and 11 million parameters, while the medium and large GPT-2 models have 24 layers and 35.5 million parameters, and 36 layers and 77.4 million parameters, respectively.

[0068] 4.2 Datasets and Tasks: The training performance of FedPiL on the Text Classification (TC) task is evaluated on the 20NEWS dataset, and on the Natural Language Generation (NLG) task on the E2E dataset. The 20NEWS dataset contains approximately 20,000 newsgroup files, while the E2E dataset contains approximately 42,000 training samples, 4,600 validation samples, and 4,600 test samples from the restaurant domain. The 20NEWS dataset is used to train the BERT model for the TC task, which outputs labels selected from predefined sets (such as politics, sports, and entertainment). In addition, the E2E dataset is used to train the GPT-2 model for the NLG task, which aims to transform various forms of input into readable natural language.

[0069] 4.3 Benchmarking: FedPiL is compared with three classic benchmarks: (1) Traditional Fine-Tuning (FT): FT fine-tunes all parameters of the LLM on each edge cloud, which is the default fine-tuning technique in most natural language processing literature; (2) Low-Rank Adaptation (LoRA): LoRA introduces low-rank trainable modules in parallel with the weight matrix of the LLM for fine-tuning. Only the low-rank trainable modules are trained, while the remaining model parameters are frozen, which can significantly reduce the number of trainable parameters without increasing additional inference latency; (3) FedAdapter: FedAdapter is an edge-end federated learning framework customized for efficient fine-tuning of LLM. It determines the optimal adapter configuration during training to minimize the training cost of fine-tuning the LLM while protecting privacy. For fair comparison, all benchmarks use the same model aggregation algorithm (FedAvg) and random client sampling scheme, consistent with the default settings in previous federated learning literature.

[0070] 4.4 Hardware and Hyperparameters: Experiments were conducted using an NVIDIA GeForce RTX 3090 GPU. To ensure fair comparison, consistent settings were maintained between FedPiL and benchmark tests across different tasks. For the TC task on the 20NEWS dataset, the mini-batch size, learning rate, and maximum sequence length were set to 4, 0.1, and 256, respectively. Similarly, for the NLG task on the E2E dataset using a medium-sized GPT-2 model, the mini-batch size, learning rate, and maximum sequence length were set to 8, 0.0002, and 512, respectively. Considering the limited 24GB of memory on the NVIDIA GeForce RTX 3090, the mini-batch size was reduced to 4 when training a large GPT-2 model, while keeping other hyperparameters unchanged. By default, the selected number of participating edge clouds was 5, and the rank was set to 4. All edge clouds were run in synchronous mode.

[0071] 5. Evaluation: This section provides numerical results to evaluate the training performance of the FedPiL framework and the effectiveness of each carefully designed component. 5.1 Overall Performance Evaluation:

[0072] Convergence Accuracy: Figure 12 shows the comparison of the convergence accuracy of FedPiL with three other benchmarks on the BERT and GPT-2 models under heterogeneous and ideal environments. It is clear that the proposed FedPiL framework outperforms the other benchmarks in both settings. Under homogeneous settings, the FedPiL framework achieves comparable accuracy on the BERT model and outperforms the FT, FedAdapter and LoRA benchmarks by 0.22, 0.31 and 0.06 respectively in terms of PPL. This advantage is attributed to the carefully designed critical weight identification, which prioritizes the fine-tuning of key weights, thereby improving the efficiency of LLM fine-tuning and achieving higher convergence accuracy. In addition, due to the lack of adaptive rank adjustment when computing heterogeneity, all benchmarks show significantly poor convergence accuracy in ideal environments. In contrast, the FedPiL framework automatically adjusts the rank size according to heterogeneous computing resources and only slightly increases the PPL by about 0.01 in heterogeneous rather than ideal settings. Furthermore, the performance gap between FedPiL and other benchmarks is much larger in heterogeneous settings than in ideal settings, further demonstrating FedPiL's superior adaptability to heterogeneous scenarios.

[0073] Trainable Parameters: Figure 13 shows the number of trainable parameters on BERT and GPT-2 models when deploying FedPiL and other benchmarks in heterogeneous and ideal environments. In the ideal environment, the FedPiL framework and the LoRA benchmark with a fixed rank size are observed to have the same number of trainable parameters. This is because FedPiL can fine-tune all four trainable weights and select the maximum rank of each weight without resource constraints. In the heterogeneous setting, FedPiL has the fewest trainable parameters, nearly 20% lower than in the ideal setting. Deploying FedPiL on BERT and GPT-2 models yields approximately 0.22M and 0.29M trainable parameters, respectively. In contrast, the number of trainable parameters for the FT and FedAdapter frameworks exceeds that of FedPiL on the BERT model by 480 times and on the GPT-2 model by 1200 times, respectively, and is 30 times greater than that of FedPiL on the GPT-2 model. The reduction in trainable parameters significantly reduces the computational and communication overhead of fine-tuning LLMs, facilitating LLM deployment. It is worth noting that FedPiL does not require iteratively searching for the optimal depth and width of the adapter layer, thus reducing the time overhead of the training process; in contrast, FedAdapter selects an adapter from three different configuration groups, while FedPiL only adapts to one initialized adapter configuration, thus saving storage resources significantly compared to FedAdapter.

[0074] Convergence Speed: Figure 14 compares the convergence speed of FedPiL with three other benchmarks under heterogeneous and ideal settings. In both ideal and heterogeneous settings, FedPiL exhibits the fastest convergence speed, outperforming FT, FedAdapter, and LoRA by 6x, 2.7x, and 2x respectively, achieving convergence speeds of 3.6 PPL and 24, 18, and 2, as well as 4.0 PPL and 2 for the Bert model. Furthermore, in the ideal setting, FedPiL's accuracy is comparable to FT, only 0.23% lower on the Bert model, and even outperforms FT by 0.22% on the PPL of the GPT-2 model. This is mainly due to... Our designed key layer identification strategy prioritizes fine-tuning of critical layers to reduce undertraining and thus accelerate model training. Conversely, in heterogeneous settings, FedPiL adaptively adjusts the rank of LoRA to suit the heterogeneous computing resources of the client, thus outperforming other benchmarks in convergence speed and accuracy. Due to the lack of heterogeneous LoRA adapter configurations, the convergence speed and accuracy of FT, FedAdapter, and LoRA rapidly decline in heterogeneous settings. For the GPT-2 model, we also conducted a comprehensive comparison of the accuracy of various metrics in homogeneous and heterogeneous settings, which is summarized in Figure 17.

[0075] 5.2 Ablation Assessment:

[0076] Importance Weight Identification (IWI): Figure 15a shows the impact of trainable weight selection on the GPT2-L model in a heterogeneous setting. Considering that the average number of trainable weights during training is about 3, the IWI method is fairly compared with three randomly combined trainable weights. The proposed IWI method shows a lower PPL than the three random weight combinations, which proves the effectiveness of the IWI method. This is because large eigenvalues ​​usually correspond to the main direction of model weight changes and reflect the location with high main information density. Therefore, prioritizing fine-tuning of trainable weights with larger eigenvalues ​​is efficient. In addition, Figure 18 summarizes the training performance on other metrics.

[0077] Heterogeneous Configuration (HC): Figure 15b illustrates the impact of rank size selection on model training. Comparing the HC method of this invention with traditional LoRA schemes with different fixed rank sizes, it is observed that the PPL of the traditional LoRA framework increases rapidly with increasing rank, indicating that heterogeneous computing resources among clients have an increasingly severe impact on model training. This is because a larger rank means a heavier computational workload, resulting in a lower undertraining rate. Conversely, the PPL of the HC method increases slightly with increasing rank, which is attributed to the design of adaptively adjusting the rank according to heterogeneous computing capabilities. Furthermore, the performance gap between the HC method and LoRA with a fixed rank size widens significantly with increasing rank, further revealing the effectiveness of the HC method. Similarly, the training performance on other metrics is shown in Figure 19.

[0078] Quantization: Figure 15c shows the GPU memory size and training accuracy at different quantization bit depths. The GPU memory size of FP32 is significantly larger than other quantization levels, at 20.5GB, of which the model occupies 3.14GB, which is approximately 1.9 times, 3.2 times, and 4.9 times that of FP16, NF8, and NF4, respectively. The number of quantization bits will trade off between the memory size and training accuracy of LLM. It was observed that as the number of quantization bits decreases, the training accuracy of the model only decreases slightly. FP32 achieves a slightly higher training accuracy, i.e., PPL of 3.224, while the training accuracy of FP16, NF8, and NF4 is lower, with PPL of approximately 3.228. The proposed quantization scheme further reduces the memory requirements of the FedPiL framework, thereby improving its scalability and facilitating its promotion and use. For a more comprehensive comparison, see Figure 20.

[0079] This invention aims to facilitate fine-tuning of LLMs in edge clouds with heterogeneous computing resources. The problem is modeled as a MILP (Multi-Input Multi-Output) model to guide the design of this invention. To address this issue, significant weights are first selected, and then appropriate parameters are configured under the computational budget of the LoRA adapter in the edge cloud. Heterogeneous memory and LLM quantization based on the edge cloud budget are also considered, further reducing memory resource usage. Extensive evaluations were conducted by comparing with state-of-the-art benchmarks, showing that FedPiL achieves excellent performance in LLM FL (Fluid-Free Optimization). Therefore, this invention identifies the optimal fine-tuning model structure to improve the performance of LLM fine-tuning and adapt to heterogeneous and resource-constrained edge clouds.

[0080] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. All equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this invention should still be covered by the claims of this invention.

Claims

1. A parameter-efficient large language fine-tuning federated learning framework, characterized in that, Includes the following steps: Modeling LoRA adapters for different edge clouds, where N edge clouds train LoRA adapters independently, and after training, the central cloud aggregates all LoRA adapters from the edge clouds; Since different weights have different average performance on the LoRA adapter, singular values ​​are used to quantify the importance of the weights. Therefore, before training the LoRA adapter independently with N edge clouds in each round, the BA matrix in the LoRA adapter with trainable weights is decomposed using matrix singular values. The heterogeneous LoRA adapter is configured according to the importance of the weights. The purpose of the heterogeneous LoRA adapter configuration is to select the batch size and rank of each edge cloud, and to adjust the rank for each weight iteration, and to dynamically reconfigure the optimal LoRA adapter for each edge cloud during each round of training. The pre-trained model is quantized using different quantization bits, and high-precision dequantization is performed only when matrix multiplication is performed. The pre-trained model is quantized to the maximum quantization bits according to the memory budget of the edge cloud, but the quantization bits of the LoRA adapter remain unchanged.

2. The parameter-efficient large language fine-tuning federated learning framework according to claim 1, characterized in that: The modeling of LoRA adapters for different edge clouds includes: Overparameterized models generally have lower intrinsic dimensionality, thus reducing the number of trainable parameters. LoRA enhances the parameters by combining the parameters of the frozen pre-trained model with additional factorization projections.

3. The parameter-efficient large language fine-tuning federated learning framework according to claim 1, characterized in that: Before training the LoRA adapter independently using N edge clouds in each round, the BA matrix in each trainable weight LoRA adapter is decomposed using matrix singular value decomposition, including: Since larger singular values ​​represent richer data information, the average singular value of each trainable weight is used to characterize the importance of the weight. Therefore, in the t-th training round, the importance metric for correctly measuring the parameter contribution of the m-th trainable weight (LoRA adapter) of the i-th edge cloud can be calculated as follows: in, This represents the j-th singular value of the m-th trainable weight (LoRA adapter) of the i-th edge cloud. This represents the m-th trainable weight (LoRA adapter) of the i-th edge cloud, where d1 is the total number of singular values, and φ (t) (·) is a sensitivity-based importance scoring function used to identify the importance weights of a model.

4. The parameter-efficient large language fine-tuning federated learning framework according to claim 1, characterized in that: The configuration of heterogeneous LoRA adapters based on the importance of weights includes: Considering the data size C = [C] of N edge clouds 1,max C 2,max ,…,C N,max ] Calculate the proportion of all values ​​in C, expressed as U r =[u1:u2:…:u N Furthermore, the batch size of the edge cloud is determined by the formula B = U. r ×b max Confirmed, things to note Yes, any element in B that is less than b min The value of b i All will be replaced with b min After determining the batch size B, for the i-th edge cloud, according to formula b i ·f c (W i R i， )≤C i,,max Then through Obtain rank-related computational resource constraints.

5. The parameter-efficient large language fine-tuning federated learning framework according to claim 4, characterized in that: The configuration of heterogeneous LoRA adapters based on the importance of weights also includes: Sort by weight importance, starting with the most important weight, then try Q in descending order of importance. m The rank value in the equation is determined if the computational power corresponding to the highest rank is lower than the maximum computational constraint. If the condition is not met, the next highest rank is used for evaluation. The above steps are repeated for other weights until the combination of weights and ranks meets the computational constraints. Among them, Q m It is a matrix used to store rank values ​​from highest to lowest.

6. The parameter-efficient large language fine-tuning federated learning framework according to claim 1, characterized in that: The method of quantizing the pre-trained model using different quantization bits, and performing high-precision dequantization on the pre-trained model only during matrix multiplication, includes: Since the weights of the pre-trained model typically follow a zero-mean normal distribution, we first transform all weights to a zero-mean normal distribution in the range [-1, 1] using the standard deviation, and then estimate 2. k +1 quantile to obtain a k-bit quantized data type, and normalize the values ​​of this data type to the range [-1, 1]. Finally, quantize the input weight tensor to the range [-1, 1] using absolute maximum rescaling. Once the weight range and data type range match, estimate the 2^k values ​​of the data type as follows: k The value q i : Where Q(·) is the quantile function of the standard normal distribution.

7. The parameter-efficient large language fine-tuning federated learning framework according to claim 1, characterized in that: Also includes: The configured heterogeneous LoRA adapters are aggregated and deployed. The incremental matrices of the edge clouds have a consistent dimension and are used for incremental matrices with the same trainable weights. Therefore, instead of directly aggregating the rank decomposition matrix, incremental matrices with the same trainable weights are aggregated from the edge clouds. The aggregated incremental matrices are also used for the pre-trained model for model updates.

8. The parameter-efficient large language fine-tuning federated learning framework according to claim 7, characterized in that: The aggregation and deployment of the configured heterogeneous LoRA adapters includes: After receiving the updated rank decomposition matrices B and A that participate in the edge cloud transmission, the central cloud multiplies these decomposition matrices to obtain the corresponding increment matrices; then the central cloud aggregates the preprocessed increment matrices from different edge devices; finally, the local pre-trained model is updated based on the increment matrices.

Images