Large Language Model Training Method Based on Parameter-Efficient Fine-Tuning

Abstract

The present invention discloses a large language model training method based on parameter-efficient fine-tuning. The large language model training method includes: dividing the large language model for the training object into two parts, namely a client pre-trained model and a server pre-trained model; jointly performing parallel collaborative training on the client pre-trained model and the server pre-trained model, and during the training process, using the parameter-efficient fine-tuning technology to perform parallel training on the client pre-trained model and the server pre-trained model; after the training is completed, using the model aggregation method to merge the client pre-trained model and the server pre-trained model; repeating the above steps until the large language model as the training object converges. The large language model training method of the present invention is based on the split federated learning framework and combines the parallel training advantages of FL and the model splitting advantages of SL, greatly improving the training efficiency.

Description

Technical Field

[0001] The present invention relates to a machine learning and natural language processing technology, and particularly to a method for training a large language model based on parameter-efficient fine-tuning. Background Art

[0002] Currently, the application of large language models (LLMs) in processing high-complexity models and large-scale datasets has achieved remarkable success. With the growing demand for more training data, the exhaustion of high-quality public datasets has become a worrying reality. To address this issue, the federated learning (FL) LLM fine-tuning paradigm has been proposed, which allows multiple data owners to collaborate in fine-tuning a shared LLM without sharing the original data.

[0003] The main problems at present are as follows:

[0004] 1) Heavy computational and communication burdens.

[0005] Specifically, the huge model size of LLMs brings heavy computational and communication burdens to edge devices / servers, which is a computational overload for resource-constrained computing entities and hinders the popularization of distributed learning paradigms such as FL and SL in LLM fine-tuning.

[0006] In addition, there are some other problems as follows:

[0007] 2) Data privacy issues: In the FL LLM fine-tuning paradigm, although the original data is not shared, deep learning models, especially large ones, have the ability to memorize training data, which may raise privacy concerns. In LLM fine-tuning, this risk is particularly serious because of their powerful capabilities, which may inadvertently memorize and potentially expose more detailed information.

[0008] 3) Model partitioning and heterogeneous configuration issues: In the SL LLM fine-tuning framework, choosing the appropriate cut layer is crucial for controlling the amount of data transmitted to the central server and the distribution of computational workloads between the client server and the central server. In addition, resource heterogeneity between different client servers / devices may lead to serious straggler problems in model aggregation.

[0009] 4) Efficiency issues: When actually deploying large language model systems, it is usually necessary to utilize private data on edge devices, which have fewer computational and storage resources than data centers. Training LLMs with billions of parameters on these edge devices faces significant obstacles due to limited storage and computational capabilities. Summary of the Invention

[0010] The object of the present invention is to provide a training method for large language models based on efficient parameter fine-tuning. This large language model training method is based on a split federated learning framework and combines the parallel training advantages of FL and the model splitting advantages of SL, thus greatly improving the training efficiency.

[0011] To achieve the above technical object, the present invention adopts the following technical solutions:

[0012] A training method for large language models based on efficient parameter fine-tuning, the large language model training method comprising:

[0013] S1. For the large language model to be trained, divide the large language model into two parts: a client pre-trained model and a server pre-trained model;

[0014] S2. Jointly perform parallel collaborative training on the client pre-trained model and the server pre-trained model. During the training process, use the efficient parameter fine-tuning technology to perform parallel training on the client pre-trained model and the server pre-trained model;

[0015] S3. After the training is completed, use the model aggregation method to merge the client pre-trained model and the server pre-trained model;

[0016] S4. Repeat the above steps S2 to S3 until the large language model to be trained converges.

[0017] Further, the step S1 specifically includes:

[0018] S11. Select the splitting point:

[0019] Determine the network structure of the large language model and select an appropriate layer as the splitting point to divide the large language model into two parts: a client pre-trained model and a server pre-trained model;

[0020] The factors to be considered in the selection of the splitting point include: the complexity of the network structure, the computing capabilities of the client and the server, and the data transmission efficiency;

[0021] S12. Initialize the large language model parameters:

[0022] According to the selected splitting point, initialize the parameters of the large language model into the client pre-trained model and the server pre-trained model respectively to ensure the consistency of the parameters and the effectiveness of the initialization for effective cooperation during the subsequent training process;

[0023] S13. Client model deployment:

[0024] Deploying the client pre-trained model on the client device requires considering the computing resources of the client device to ensure the efficient operation of the client pre-trained model;

[0025] S14, Server-side model deployment:

[0026] Deploy the server-side pre-trained model on the server side, deploy the more complex model parts, and be responsible for centralized parameter updates and model optimization work;

[0027] S15, Define the communication protocol:

[0028] Establish a data communication protocol between the client and the server for transmitting large language model parameters, gradient information, and necessary data;

[0029] S16, Model synchronization mechanism:

[0030] Design and implement a synchronization mechanism for the model parameters of the client and the server, including the aggregation and update of gradients, and the exchange of parameters; The synchronization mechanism needs to consider network bandwidth limitations and communication delays to ensure the efficiency and stability of the training process;

[0031] S17, Federated learning strategy:

[0032] Formulate a federated learning strategy between the client and the server to optimize the federated training process of the model, specifically including:

[0033] Client local training: The client device uses local data for local updates of the model;

[0034] Model upload: The client regularly uploads the updated model parameters or gradients to the server;

[0035] Model aggregation: The server aggregates the model parameters or gradients from multiple clients and updates the global model;

[0036] Global model update: Distribute the updated global model parameters to each client.

[0037] Furthermore, the specific steps of step S2 include:

[0038] S21, Initialize the parallel collaborative training environment:

[0039] 1) Client model initialization: Load the initial pre-trained model on all client devices, and the pre-trained model is a copy of a general pre-trained large language model;

[0040] 2) Server model initialization: Load the corresponding copy of the pre-trained model on the central server side to ensure that the model version is consistent with the client;

[0041] 3) Prepare the parameter efficient fine-tuning technology: Select the required fine-tuning method and initialize its corresponding parameter settings;

[0042] S22, Data allocation and preprocessing:

[0043] 1) Client data preprocessing: Each client device reads the local dataset and performs data cleaning, tokenization, and encoding operations to ensure unified data format;

[0044] 2) Sharding and synchronization: Shard the preprocessed data and prepare for batch processing, and at the same time synchronize with the server for the preliminary data distribution plan;

[0045] S23, Preliminary activation value calculation:

[0046] 1) Local training: Each client performs preliminary forward propagation on the local data shard using the current pre-trained model to calculate the activation value;

[0047] 2) Activation value caching: Temporarily store the calculated activation values locally and prepare to synchronize with the server;

[0048] S24, Gradient aggregation and synchronization:

[0049] 1) Upload local activation values: Each client uploads the calculated activation values to the server;

[0050] 2) Server-side gradient update: The server receives the activation values uploaded by all clients, performs forward propagation and backward propagation, and calculates the global gradient update direction;

[0051] 3) Parameter update: Update the parameters of the pre-trained model on the server once;

[0052] 4) Distribute gradient values: The server distributes the gradient values back to each client;

[0053] 5) Client gradient update: The client receives the gradient downloaded from the server, performs backward propagation, and updates the client model parameters;

[0054] S25, Synchronize model parameters back to the client:

[0055] 1) Distribute updated model parameters: The server distributes the model parameters after global gradient update back to each client;

[0056] 2) Local update: Each client receives the new parameters, updates its local model, and prepares for the next round of training;

[0057] S26, Loop and enhanced training:

[0058] 1) Multi-round collaborative training: Repeatedly execute steps S23 to S26 to perform multi-round gradient calculation, update, and efficient fine-tuning, gradually improving the performance of the model;

[0059] 2) Periodic evaluation: After a specific iteration frequency, evaluate the performance of the model to determine the training effect and the optimization degree of the fine-tuning strategy;

[0060] S27, Dynamic tuning and model convergence:

[0061] Specifically, this step includes:

[0062] 1) Dynamic parameter adjustment: Dynamically adjust the hyperparameters of the efficient fine-tuning technique according to the performance of the model during training;

[0063] 2) Convergence judgment: Judge whether the model reaches the convergence state by monitoring the performance metrics of the model on the validation set.

[0064] Furthermore, step S3 specifically includes:

[0065] S31, Aggregation preparation:

[0066] Specifically, this step includes:

[0067] 1) Model version confirmation: Ensure that all client and server model versions are consistent, that is, all models have undergone the same number of rounds of training and fine-tuning;

[0068] 2) Synchronization check: Verify whether each client has completed all training tasks and confirm that all necessary data has been uploaded to the server;

[0069] S32, Model parameter collection:

[0070] Specifically, this step includes:

[0071] 1) Client parameter upload: Each client uploads its locally optimized model parameters to the server;

[0072] 2) Parameter compression and packaging: Before uploading, the parameters can be compressed and packaged as necessary to reduce transmission overhead and improve efficiency;

[0073] S33, Global parameter calculation:

[0074] 1) Parameter decompression and merging: The server decompresses and preliminarily merges the parameters received from each client to prepare for global parameter calculation;

[0075] 2) Aggregation strategy selection: Determine which model aggregation method to use, such as average aggregation, weight aggregation, gradient-based aggregation, etc.;

[0076] S34, Preliminary aggregation:

[0077] 1) Local aggregation: If there are multiple server nodes, local aggregation can be performed on each server node first to reduce the single-point computing burden;

[0078] 2) Global aggregation: Upload the local aggregation results to the main server for the final global aggregation calculation;

[0079] S35, Parameter weight adjustment:

[0080] 1) Weight allocation: Reasonably allocate weights according to the importance or training volume of each client's data, especially in the case of uneven training data;

[0081] 2) Weighted average: If the weighted average strategy is selected, the parameters are merged by weighted average according to the allocated weights;

[0082] S36, Consistency correction:

[0083] 1) Conflict detection: Check whether there are conflicts or inconsistencies in the merged model parameters and make necessary corrections;

[0084] 2) Parameter pruning: Prune redundant or overfitting parameters to optimize the overall performance of the model;

[0085] S37, Model fusion verification:

[0086] 1) Preliminary verification: Use the merged model on the server side for preliminary verification to check the model performance and consistency;

[0087] 2) Performance testing: Conduct forward testing on a small batch of test data to ensure that the model still maintains high performance after global aggregation.

[0088] The beneficial effects of the large language model training method of the present invention compared with the prior art are as follows:

[0089] The large language model is divided into a client side and a server side, where the client side is responsible for forward propagation and only generates and transmits activation values, while the server side is responsible for subsequent fine-tuning and backpropagation of gradients; this method significantly reduces the size of the transmitted data because it avoids the need to transmit the entire model; in addition, the large language model training method also utilizes the advantages of Split Federated Learning (SFL) by performing the main training workload on a powerful central server, thereby reducing the computing pressure on edge devices. The large language model training method of the present invention is based on the Split Federated Learning (SFL) framework and combines the parallel training advantages of FL and the model splitting advantages of SL, thereby greatly improving the training efficiency and providing the first open-source benchmark for SL LLM fine-tuning. Description of the Drawings

[0090] Figure 1 Overview diagram of the SplitLoRA framework constructed according to the inventive concept;

[0091] Figure 2 Schematic diagram of the convergence accuracy of the GPT2-S and GPT2-M models constructed according to the inventive concept. In the figure, the perplexity (PPL) is an index used to measure the quality of the prediction of the language model. The lower the PPL, the better the prediction performance;

[0092] Figure 3 Schematic diagram of the training performance of GPT2-S and GPT2-M in the E2E NLG challenge constructed according to the inventive concept;

[0093] Figure 4 Comparison table of the convergence accuracy of various indicators of GPT2-S for the E2E NLG challenge constructed according to the inventive concept;

[0094] Figure 5 Comparison table of the convergence accuracy of various indicators of GPT2-M for the E2E NLG challenge constructed according to the inventive concept;

[0095] Figure 6 Comparison table of the number of trainable parameters on GPT2-S and GPT2-M constructed according to the inventive concept. Detailed implementation manners

[0096] The present invention will be further described below with specific embodiments:

[0097] This embodiment provides a large language model training method based on parameter-efficient fine-tuning. By implementing this large language model training method, it is mainly possible to reduce the computational and communication burdens.

[0098] See Figure 1 , the large language model training method of this embodiment specifically includes the following steps S1 to S4.

[0099] S1, for the large language model to be trained, divide the large language model into two parts: a client pre-trained model and a server pre-trained model.

[0100] S2, jointly perform parallel collaborative training on the client pre-trained model and the server pre-trained model. During the training process, existing parameter-efficient fine-tuning techniques (such as Adapter Tuning, Prompt Tuning, Low-Rank Adaptation (LoRA), etc.) are used to perform parallel training on the client pre-trained model and the server pre-trained model.

[0101] S3. After the training is completed, use the existing model aggregation method to merge the client pre-trained model and the server pre-trained model again to achieve the training effect without splitting.

[0102] S4. Repeat the above steps S2 to S3 until the "large language model as the training object" converges.

[0103] The above steps S1 to S4 are the most basic concepts of this embodiment. Based on such a concept, the large language model training method of this embodiment can mainly achieve the good technical effect of "reducing the computational and communication burdens".

[0104] The following introduces the specific details of each step:

[0105] The step S1 specifically includes the following steps S11 to S17.

[0106] S11. Select a splitting point:

[0107] Specifically,

[0108] First, determine the network structure of the large language model and select an appropriate layer as the splitting point to divide the large language model into two parts: the client pre-trained model and the server pre-trained model. The selection of the splitting point needs to consider the following factors:

[0109] 1) The complexity of the network structure;

[0110] 2) The computing capabilities of the client and the server;

[0111] 3) The data transmission efficiency.

[0112] For example, the first few layers of the large language model can be used as the client pre-trained model, and the rest can be used as the server pre-trained model.

[0113] S12. Initialize the large language model parameters:

[0114] Specifically,

[0115] According to the previously selected splitting point, initialize the parameters of the large language model into the client pre-trained model and the server pre-trained model respectively. Ensure the consistency of the parameters and the effectiveness of the initialization for effective cooperation in the subsequent training process.

[0116] S13. Deploy the client model:

[0117] Specifically,

[0118] Deploy the client pre-trained model on the client device. This step needs to consider the computing resources of the client device, such as CPU / GPU performance, memory capacity, etc., to ensure the efficient operation of the client pre-trained model.

[0119] S14, Server-side model deployment:

[0120] Specifically,

[0121] Deploy the server-side pre-trained model on the server. Servers usually have more powerful computing and storage capabilities, so more complex model parts can be deployed, and centralized parameter updates and model optimization tasks are responsible.

[0122] S15, Define the communication protocol:

[0123] Specifically,

[0124] Establish a data communication protocol between the client and the server for transmitting large language model parameters, gradient information, and other necessary data. This requires ensuring the efficiency and security of data transmission, and compression algorithms and encryption technologies may be used.

[0125] S16, Model synchronization mechanism:

[0126] Specifically,

[0127] Design and implement a synchronization mechanism for the model parameters between the client and the server. This may include the aggregation and update of gradients, the exchange of parameters, etc. The synchronization mechanism needs to consider network bandwidth limitations and communication delays to ensure the efficiency and stability of the training process.

[0128] S17, Federated learning strategy:

[0129] Specifically,

[0130] Formulate a federated learning strategy between the client and the server to optimize the federated training process of the model. Specifically, it includes:

[0131] 1) Client-side local training: The client device uses local data to perform local updates of the model;

[0132] 2) Model upload: The client periodically uploads the updated model parameters or gradients to the server;

[0133] 3) Model aggregation: The server aggregates the model parameters or gradients from multiple clients and updates the global model;

[0134] 4) Global model update: The updated global model parameters are sent to each client.

[0135] Step S2 specifically includes the following steps S21 to S27

[0136] S21, Initialize the parallel collaborative training environment:

[0137] Specifically, this step includes:

[0138] 1) Initialize the client model: Load the initial pre-trained model on all client devices. It can be a copy of a general pre-trained large language model.

[0139] 2) Initialize the server model: Load the corresponding copy of the pre-trained model on the central server side to ensure that the model version is consistent with the clients.

[0140] 3) Prepare the parameter-efficient fine-tuning technique: Select the required fine-tuning method (such as Adapter Tuning, Prompt Tuning, LoRA) and initialize its corresponding parameter settings.

[0141] S22, Data allocation and preprocessing:

[0142] Specifically, this step includes:

[0143] 1) Client data preprocessing: Each client device reads the local dataset and performs data cleaning, tokenization, and encoding operations to ensure unified data format;

[0144] 2) Sharding and synchronization: Shard the preprocessed data and prepare for batch processing, and at the same time synchronize the preliminary data distribution plan with the server side.

[0145] S23, Calculate the preliminary activation values:

[0146] Specifically, this step includes:

[0147] 1) Local training: Each client performs a preliminary forward pass on the local data shard using the current pre-trained model to calculate the activation values.

[0148] 2) Cache the activation values: Temporarily store the calculated activation values locally and prepare to synchronize with the server side.

[0149] S24, Gradient aggregation and synchronization:

[0150] Specifically, this step includes:

[0151] 1) Upload the local activation values: Each client uploads the calculated activation values to the server side;

[0152] 2) Server-side gradient update: The server side receives the activation values uploaded by all clients, performs forward and backward passes, and calculates the global gradient update direction;

[0153] 3) Parameter update: Update the parameters of the server-side pre-trained model once on the server side;

[0154] 4) Distribute gradient values: The server distributes the gradient values back to each client;

[0155] 5) Client gradient update: The client receives the gradient downloaded from the server, performs backpropagation, and updates the client model parameters.

[0156] S25, Synchronize model parameters back to the client:

[0157] Specifically, this step includes:

[0158] 1) Distribute updated model parameters: The server distributes the model parameters after global gradient update back to each client.

[0159] 2) Local update: Each client receives the new parameters, updates its local model, and prepares for the next round of training.

[0160] S26, Loop and enhanced training:

[0161] Specifically, this step includes:

[0162] 1) Multiple rounds of collaborative training: Repeatedly execute steps S23 to S26 for multiple rounds of gradient calculation, update, and efficient fine-tuning to gradually improve the performance of the model.

[0163] 2) Periodic evaluation: After a specific iteration frequency, evaluate the performance of the model to determine the training effect and the optimization degree of the fine-tuning strategy.

[0164] S27, Dynamic tuning and model convergence:

[0165] Specifically, this step includes:

[0166] 1) Dynamic parameter adjustment: Dynamically adjust the hyperparameters (such as learning rate, adapter size, etc.) of the efficient fine-tuning technique according to the performance of the model during training.

[0167] 2) Convergence judgment: Judge whether the model reaches the convergence state by monitoring the performance metrics of the model on the validation set, as Figure 3 shown.

[0168] The above-mentioned step S3 specifically includes the following steps S31 to S37.

[0169] S31, Aggregation preparation:

[0170] Specifically, this step includes:

[0171] 1) Model version confirmation: Ensure that the model versions on all clients and the server are consistent, i.e., all models have undergone the same number of training and fine-tuning rounds.

[0172] 2) Synchronization check: Verify whether each client has completed all training tasks and confirm that all necessary data has been uploaded to the server.

[0173] S32, Model parameter collection:

[0174] Specifically, this step includes:

[0175] 1) Client parameter upload: Each client uploads its locally optimized model parameters to the server.

[0176] 2) Parameter compression and packaging: Before uploading, necessary compression and packaging can be performed on the parameters to reduce transmission overhead and improve efficiency.

[0177] S33, Global parameter calculation:

[0178] Specifically, this step includes:

[0179] 1) Parameter decompression and merging: The server decompresses and preliminarily merges the parameters received from each client to prepare for global parameter calculation.

[0180] 2) Aggregation strategy selection: Determine which model aggregation method to use, such as average aggregation, weighted aggregation, gradient-based aggregation, etc.

[0181] S34, Preliminary aggregation:

[0182] Specifically, this step includes:

[0183] 1) Local aggregation: If there are multiple server nodes, local aggregation can be performed on each server node first to reduce the single-point calculation burden.

[0184] 2) Global aggregation: Upload the local aggregation results to the main server for final global aggregation calculation.

[0185] S35, Parameter weight adjustment:

[0186] Specifically, this step includes:

[0187] 1) Weight assignment: Reasonably assign weights according to the importance or training volume of each client's data, especially in the case of uneven training data.

[0188] 2) Weighted average: If the weighted average strategy is selected, the parameters are merged by weighted average according to the assigned weights.

[0189] S36, Consistency correction:

[0190] Specifically, this step includes:

[0191] 1) Conflict detection: Check whether there are conflicts or inconsistencies (such as gradient explosion) in the merged model parameters and make necessary corrections.

[0192] 2) Parameter pruning: Prune redundant or overfitting parameters to optimize the overall performance of the model.

[0193] S37, Model fusion verification:

[0194] Specifically, this step includes:

[0195] 1) Preliminary verification: Conduct preliminary verification using the merged model on the server side to check the model performance and consistency. As Figure 2 shown.

[0196] 2) Performance testing: Conduct forward testing on a small batch of test data to ensure that the model still maintains high performance after global aggregation. Refer to Figure 4 , 5 , and Figure 6 as shown.

[0197] Next, taking a specific large language model running scenario as an example, the implementation process of the large language model training method of this embodiment will be specifically introduced.

[0198] In this scenario, a central server, edge servers, and a federated server are set up.

[0199] In this scenario,

[0200] The role of the central server is:

[0201] 1) Global model storage and management: Maintain the central storage of the global pre-trained model to ensure the unity and consistency of the model version.

[0202] 2) Global gradient aggregation and update: During the federated learning process, aggregate the gradients uploaded by all clients or edge servers and perform global parameter updates.

[0203] 3) Task allocation and coordination: During the training process, coordinate the edge and federated servers, and allocate training tasks and data slices.

[0204] The role of the edge server is:

[0205] 1) Real-time data processing: Collect and preprocess data from local devices and use the processed data for model training.

[0206] 2) Local model update: Perform local optimization and update of the model locally, perform forward and backward propagation through local data, and calculate the model gradients.

[0207] 3) Preliminary gradient aggregation: Receive gradients from multiple local devices and perform preliminary aggregation to reduce the amount of data transmission.

[0208] The functions of the federated server are as follows:

[0209] 1) Regional gradient aggregation: Collect local aggregated gradients from edge servers and perform further aggregation within the partition.

[0210] 2) Intermediate result synchronization: Synchronize the aggregated intermediate gradient results to the central server, or receive the global update parameters from the central server for distribution.

[0211] 3) Regional model distribution: Distribute the updated model parameters to edge servers according to actual needs to ensure the real-time nature and continuous improvement of edge training.

[0212] In the process of this implementation, a software system was constructed based on the large language model training method of this implementation manner. For the convenience of description, the constructed software system is named "SplitLoRA".

[0213] (Corresponding to step S1) Before the large language model training starts, the central server initializes the large language model and divides it into a client pre-training model and a server-side pre-training model.

[0214] (Corresponding to step S2) Subsequently, perform model fine-tuning for I consecutive rounds of training on the large language model.

[0215] (Corresponding to step S3) Then perform the aggregation of the client LoRA adapters.

[0216] (Corresponding to step S4) This process is repeated until the model converges.

[0217] The training process of SplitLoRA for the large language model includes two main stages: split fine-tuning (corresponding to step S2) and client LoRA adapter aggregation (corresponding to step S3). Split fine-tuning is performed in each round of training, while client LoRA adapter aggregation is performed once every I rounds of training.

[0218] See Figure 1 , for one training round The training process of SplitLoRA is as follows in detail.

[0219] (Corresponding to step S2)

[0220] The split fine-tuning stage: The split fine-tuning stage includes the client and server-side fine-tuning of the edge server and the central server in each training round, specifically including the following five steps.

[0221] 1) Forward Propagation (FP) of the Client Pre-trained Model:

[0222] Specifically, all participating edge servers perform the forward propagation of the client pre-trained model in parallel.

[0223] More specifically, each edge server i randomly selects a mini-batch from its local dataset. This mini-batch contains b data samples and is used to fine-tune the client pre-trained model.

[0224] In training round t, the input data and corresponding labels of the mini-batch are respectively denoted as and denotes the set of trainable LoRA adapters of the client pre-trained model on edge server i at the t-th round of training. After inputting a set of mini-batch data into the client pre-trained model, activation values are generated at the cut layer. The activation values of edge server i are denoted as

[0225]

[0226] where represents the mapping relationship between the input data x and its predicted value given the model parameters w and the set of trainable LoRA adapters r.

[0227] 2) Activation Value Transmission:

[0228] After completing the forward propagation of the client pre-trained model, all participating edge servers transmit their respective activation values and corresponding labels to the central server (usually via a wireless channel). Then, in the central server, the received activation values from the participating edge servers are used to drive the fine-tuning of the server-side pre-trained model.

[0229] 3) Forward Propagation and Backward Propagation of the Server-side Pre-trained Model:

[0230] In the central server, after receiving the activation values from the participating edge servers, the central server inputs these activation values into the server-side pre-trained model to perform the forward propagation (FP) on the server side.

[0231] represents the set of trainable LoRA adapters of the server-side pre-trained model in the t-th round of training.

[0232] The connected activation matrix S t is denoted as Therefore, the predicted value is denoted as

[0233]

[0234] The predicted values and labels are used to calculate the loss function value and further derive the gradients of the server-side LoRA adapters. Therefore, the m-th server-side LoRA adapter can be updated as follows:

[0235]

[0236] where and represent the gradients of the decomposed matrices A and B of the m-th LoRA adapter of the server-side pre-trained model, and γ s is the learning rate on the server side.

[0237] 4) Activate gradient transmission:

[0238] After the server-side backpropagation (BP) is completed, the central server transmits the activated gradients to the corresponding participating edge servers.

[0239] 5) Client model backpropagation:

[0240] In this step, each edge server fine-tunes the client pre-trained model according to the received activated gradients.

[0241] For edge server i, the j-th client LoRA adapter is updated as follows

[0242]

[0243] where and represent the gradients of the decomposed matrices A and B of the n-th LoRA adapter of the server-side pre-trained model, and γ c is the learning rate on the client side.

[0244] (corresponding to step S3)

[0245] Client LoRA adapter aggregation phase: The main focus is on aggregating the client LoRA adapters on the federated server, which is performed once every I rounds of training. This phase consists of the following three steps.

[0246] 1) Client LoRA adapter upload:

[0247] In this step, each participating edge server sends its client LoRA adapter to the federated server via a wireless / wired link.

[0248] 2) Client LoRA adapter aggregation:

[0249] The federated server aggregates the received client LoRA adapters into an aggregated LoRA adapter. The decomposed matrices A and B of the n-th client LoRA adapter are aggregated as follows respectively

[0250]

[0251]

[0252] 3) Downlink transmission of the client LoRA adapter:

[0253] After the aggregation of the client LoRA adapter is completed, the federated server sends the aggregated client LoRA adapter to the participating edge servers. Then, each edge server uses the received aggregated LoRA adapter as the initial LoRA configuration for the next round of training.

[0254] The following provides a performance evaluation of implementing the large language model training method in the above specific scenario:

[0255] This evaluation includes convergence accuracy, convergence speed, and resource efficiency.

[0256] 1) Experimental setup

[0257] Hardware: We implemented SplitLoRA using Python 3.7 and PyTorch 1.7.1, and used an NVIDIA GeForce RTX 3090 GPU for training.

[0258] Dataset and task: We evaluated the training performance of SplitLoRA on the E2E dataset

[29] for the natural language generation (NLG) task. The E2E dataset consists of approximately 42,000 training, 4,600 validation, and 4,600 test examples in the restaurant domain.

[0259] Model: We adopted the well-known GPT2 small (GPT-S) and GPT2 medium (GPT2-M) models. GPT2-S is the smallest version in the GPT2 series, with 124 million parameters and 12 layers of Transformer encoders, while GPT2-M is the medium version of GPT2, with 355 million parameters and 24 layers of Transformer encoders.

[0260] Experimental Setup: In the simulation, we default to deploying N = 3 edge servers unless otherwise stated. The client model of each edge server has the first three Transformer layers of the GPT-2 model, while the server-side model of the central server retains the remaining 9 Transformer layers (GPT2-S) or 21 layers (GPT2-M). The computing power of each edge server is 35.6 TFLOTS (peak performance of NVIDIA RTX 3090), while the computing power of the central server is set to 284.8 TFLOTS. The communication rate between the edge server and the federated server is set to 300 Mbps, and the communication rate between the edge server and the central server is set to 600 Mbps. We set the mini-batch size, learning rate, and maximum sequence length of the GPT2-S model to 8, 0.0002, and 512, while those of the GPT2-M model are 4, 0.0002, and 512, respectively. The rank of the LoRA adapter is set to r = {1, 2, 4, 8}.

[0261] Benchmarks: To study the advantages of SplitLoRA, we compare it with three typical benchmarks: (1) Centralized LoRA (CenLoRA): The edge server collects the original data from other participating servers for full model fine-tuning using the LoRA adapter. (2) Federated LoRA (FedLoRA): Each participating edge server locally fine-tunes the full model and then transmits the updated LoRA adapter to the central server for adapter aggregation.

[0262] 2) Performance Evaluation

[0263] Convergence Accuracy: Figure 2The training performance of SplitLoRA and other baselines on the GPT2-S and GPT2-M models in the E2E NLG challenge was compared, with PPL as the performance metric. The convergence accuracy of FedLoRA is lower than that of SplitLoRA and CenLoRA, and its PPL is about 0.08 / 0.11 (GPT2-S / GPT2-M) and 0.73 / 0.09 (GPT2-S / GPT2-M) higher than that of SplitLoRA and CenLoRA respectively. The poor training performance of FedLoRA stems from its fully distributed training mode, where the entire model is updated through model aggregation, making it vulnerable to model biases caused by data heterogeneity in edge servers. The convergence accuracy achieved by SplitLoRA is comparable to that of CenLoRA, especially on GPT2-M, with the accuracy difference being less than 0.04. There are two reasons for this: First, the server submodels in SplitLoRA are trained in a centralized manner, making them more resistant to data heterogeneity; Second, SplitLoRA offloads the main workload to the central server, and only a small part of the LLM is affected by client data heterogeneity. It can be seen that as the rank r increases, the PPL of SplitLoRA and CenLoRA gets closer and closer. This is because a higher r means more trainable parameters, thus enhancing the fitting ability of the server-side submodels. This may partially offset the impact of data heterogeneity, making the performance of SplitLoRA closer to that of CenLoRA. For the comparison of training performance on other metrics, please refer to Figure 4 and Figure 5 。

[0264] Convergence speed: Figure 3 (a) and Figure 3 (b) show the training performance of SplitLoRA and other baselines on GPT2-S and GPT-M in the E2E NLG challenge. Obviously, SplitLoRA is significantly superior to FedLoRA and CenLoRA in terms of convergence speed. The training latency required for FedLoRA and CenLoRA to achieve model convergence is approximately 1.7 times and 4.7 times, as well as 2.1 times and 4.8 times that of SplitLoRA on GPT-S and GPT-M. Compared with CenLoRA, FedLoRA speeds up model convergence due to the parallel training mode of multiple edge servers. Based on FedLoRA, by offloading the main computational workload to a more powerful central server, SplitLoRA further shortens the time required to achieve model convergence while supporting parallel training of multiple edge servers.

[0265] Trainable Parameters: Table III shows the number of trainable parameters for GPT2-S and GPT2-M. SplitLoRA alleviates the computational workload on the edge server by partitioning the model and keeping fewer Transformer layers locally. In our experimental setup, for the GPT2-S and GPT2-M models, the parameters used for fine-tuning on the edge server in SplitLoRA only account for one-fourth and one-eighth of the entire model, respectively. This significant reduction in local computational requirements makes SplitLoRA efficient and suitable for deployment in resource-constrained environments. In contrast, CenLoRA and FedLoRA require the edge server to process the entire GPT-2 model for fine-tuning, which poses significant requirements for computational power and memory resources. In particular, in 5G and future edge computing systems, small base stations and access points with limited capabilities act as edge repositories. Therefore, SplitLoRA not only improves computational efficiency but also enhances the system's flexibility and adaptability to better handle various computationally and resource-constrained scenarios.

[0266] The large language model training method of this embodiment has the advantages that

[0267] 1) Reduce the computational and communication burdens through model partitioning and parallel training (addressing Problem 1 raised in the background art). The model is divided into a client side and a server side, where the client is responsible for forward propagation and only generates and transmits activation values, while the server side is responsible for subsequent fine-tuning and the backpropagation of gradients. This method significantly reduces the size of the transmitted data because it avoids the need to transmit the entire model. In addition, the large language model training method also leverages the advantages of Split Federated Learning (SFL) by performing the main training workload on a powerful central server, thereby alleviating the computational pressure on edge devices.

[0268] 2) Protect privacy by aggregating the LoRA adapters of clients on the federated server instead of directly aggregating model parameters (addressing Problem 2 raised in the background art). This method ensures that client data does not need to leave the local device because only the adapters of the model (which are a small part of the model parameters) are used to update the global model. In addition, the large language model training method employs differential privacy techniques by adding controlled noise to the model gradients or updates to further protect the privacy of the training data.

[0269] 3) Allow for the selection of different split layers and the rank of LoRA adapters based on the resource heterogeneity of the client-server / devices (addressing problem 3 raised in the background art). By carefully selecting the split layers and the rank of the adapters, the computational workload between the client and server sides can be balanced, and the amount of data that needs to be transmitted by the client device can be reduced. This approach enables resource-constrained devices to also participate in the model training while ensuring the efficiency and effectiveness of the model training.

[0270] 4) Use model compression techniques such as knowledge distillation, pruning, and quantization techniques to reduce the size and computational complexity of the model without significantly affecting the model performance (addressing problem 4 raised in the background art). These techniques enable the deployment of smaller and more efficient versions of the LLM in resource-constrained environments. Additionally, the large language model training method improves efficiency by reducing the number of parameters for client-side fine-tuning (only a small fraction of the entire model), enabling fine-tuning of the LLM even on edge devices with low computational power. In this way, different computational and resource-constrained scenarios can be accommodated while maintaining the model performance.

[0271] The above are only the preferred embodiments of the present invention and are not used to limit the protection scope of the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A training method for large language models based on parameter - efficient fine - tuning, characterized in that: The large language model training method includes: S1. For the large language model to be trained, divide the large language model into two parts: a client-side pre-training model and a server-side pre-training model; S2. Jointly perform parallel collaborative training on the client-side pre-training model and the server-side pre-training model. During the training process, use parameter-efficient fine-tuning technology to perform parallel training on the client-side pre-training model and the server-side pre-training model; S3. After the training is completed, use the model aggregation method to merge the client-side pre-training model and the server-side pre-training model; S4. Repeat the above steps S2 to S3 until the large language model to be trained converges; The specific steps of S2 include: S21. Initialize the parallel collaborative training environment: 1) Client model initialization: Load the initial pre-training model on all client devices. The pre-training model is a copy of a general pre-trained large language model; 2) Server model initialization: Load the corresponding pre-training model copy on the central server side to ensure that the model version is consistent with the client; 3) Prepare parameter-efficient fine-tuning technology: Select the required fine-tuning method and initialize its corresponding parameter settings; S22. Data allocation and preprocessing: 1) Client data preprocessing: Each client device reads the local dataset and performs data cleaning, tokenization, and encoding operations to ensure that the data format is unified; 2) Sharding and synchronization: Shard the preprocessed data and prepare for batch processing, and at the same time synchronize the preliminary data distribution plan with the server side; S23. Preliminary activation value calculation: 1) Local training: Each client uses the current pre-training model on the local data slice to perform preliminary forward propagation and calculate the activation value; 2) Activation value caching: Temporarily store the calculated activation value locally and prepare to synchronize it with the server side; S24. Gradient aggregation and synchronization: 1) Upload local activation values: Each client uploads the calculated activation values to the server side; 2) Server-side gradient update: The server side receives the activation values uploaded by all clients, performs forward propagation and backward propagation, and calculates the global gradient update direction; 3) Parameter update: Update the parameters of the server-side pre-training model once on the server side; 4) Distribute gradient values: The server side distributes the gradient values back to each client; 5) Client gradient update: The client receives the gradient downloaded from the server side, performs backward propagation, and updates the client model parameters; S25. Synchronize the model parameters back to the client: 1) Distribute updated model parameters: The server side distributes the model parameters after the global gradient update back to each client; 2) Local update: Each client receives the new parameters, updates its local model, and prepares for the next round of training; S26. Loop and enhanced training: 1) Multi-round collaborative training: Repeatedly execute steps S23 to S26 to perform multiple rounds of gradient calculation, update, and efficient fine-tuning to gradually improve the performance of the model; 2) Periodic evaluation: After a specific iteration frequency, evaluate the performance of the model to determine the training effect and the optimization degree of the fine-tuning strategy; S27. Dynamic tuning and model convergence: Specifically, this step includes: 1) Dynamic parameter adjustment: Dynamically adjust the hyperparameters of the efficient fine-tuning technique according to the performance of the model during training. 2) Convergence judgment: Determine whether the model has reached the convergence state by monitoring the performance metrics of the model on the validation set.

2. The method for training a large language model based on efficient fine-tuning of parameters according to claim 1, wherein: The specific steps of step S1 include: S11, Select the splitting point: Determine the network structure of the large language model and select an appropriate layer as the splitting point to divide the large language model into two parts: the client pre-trained model and the server pre-trained model. The factors to be considered in the selection of the splitting point include: the complexity of the network structure, the computing capabilities of the client and the server, and the data transmission efficiency. S12, Initialize the large language model parameters: According to the selected splitting point, initialize the parameters of the large language model into the client pre-trained model and the server pre-trained model respectively, ensuring the consistency of the parameters and the effectiveness of the initialization for effective cooperation in the subsequent training process. S13, Client model deployment: Deploy the client pre-trained model on the client device, considering the computing resources of the client device to ensure the efficient operation of the client pre-trained model. S14, Server-side model deployment: Deploy the server pre-trained model on the server side, deploy the more complex model part, and be responsible for centralized parameter update and model optimization work. S15, Define the communication protocol: Establish a data communication protocol between the client and the server for transmitting large language model parameters, gradient information, and data. S16, Model synchronization mechanism: Design and implement a synchronization mechanism for the model parameters of the client and the server, including the aggregation and update of gradients and the exchange of parameters; the synchronization mechanism needs to consider network bandwidth limitations and communication delays to ensure the efficiency and stability of the training process. S17, Federated learning strategy: Formulate a federated learning strategy between the client and the server to optimize the federated training process of the model, specifically including: Client local training: The client device uses local data to perform local updates of the model. Model upload: The client regularly uploads the updated model parameters or gradients to the server. Model aggregation: The server aggregates the model parameters or gradients from multiple clients and updates the global model. Global model update: Distribute the updated global model parameters to each client.

3. The method for training a large language model based on efficient parameter fine-tuning according to claim 1, wherein: The specific steps of step S3 include: S31, Aggregation preparation: Specifically, this step includes: 1) Model version confirmation: Ensure that the model versions of all clients and the server are consistent, that is, all models have undergone the same number of training and fine-tuning rounds. 2) Synchronization check: Verify whether each client has completed all training tasks and confirm that all data has been uploaded to the server side. S32, Model parameter collection: Specifically, this step includes: 1) Client parameter upload: Each client uploads its locally optimized model parameters to the server side. 2) Parameter compression and packaging: Before uploading, the parameters can be compressed and packaged to reduce transmission overhead and improve efficiency. S33, Global parameter calculation: 1) Parameter decompression and merging: The server decompresses and preliminarily merges the parameters received from each client to prepare for global parameter calculation; 2) Aggregation strategy selection: Determine which model aggregation method to use, including average aggregation, weighted aggregation, and gradient-based aggregation; S34, Preliminary aggregation: 1) Local aggregation: If there are multiple server nodes, local aggregation can be performed on each server node first to reduce the single-point calculation burden; 2) Global aggregation: Upload the local aggregation results to the main server for the final global aggregation calculation; S35, Parameter weight adjustment: 1) Weight assignment: Reasonably assign weights according to the importance or training volume of each client's data in the case of uneven training data; 2) Weighted average: If the weighted average strategy is selected, the parameters are merged by weighted average according to the assigned weights; S36, Consistency correction: 1) Conflict detection: Check whether there are conflicts or inconsistencies in the merged model parameters and make corrections; 2) Parameter pruning: Prune redundant or overfitting parameters to optimize the overall performance of the model; S37, Model fusion verification: 1) Preliminary verification: Use the merged model on the server side for preliminary verification to check the model performance and consistency; 2) Performance test: Conduct a forward test on a small batch of test data to ensure that the model still maintains high performance after global aggregation.

Images