Large language model training method based on efficient fine tuning of parameters

Abstract

The invention discloses a large language model training method based on efficient fine tuning of parameters. The large-scale language model training method comprises the following steps: aiming at a large-scale language model of a training object, dividing the large-scale language model into a client pre-training model and a server pre-training model; the client pre-training model and the server pre-training model are combined for parallel cooperative training, and in the training process, the client pre-training model and the server pre-training model are subjected to parallel training by adopting an efficient parameter fine tuning technology; after training is completed, combining the client pre-training model and the server pre-training model by adopting a model aggregation method; and repeating the steps until the large language model serving as the training object is converged. The large language model training method is based on the segmentation federal learning framework, and combines the parallel training advantage of the FL and the model segmentation advantage of the SL, so that the training efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to a machine learning and natural language processing technology, and in particular to a large language model training method based on efficient parameter fine-tuning. Background Art

[0002] Currently, large language models (LLMs) have achieved remarkable success in handling high-complexity models and large-scale datasets. With the growing demand for more training data, the exhaustion of high-quality public datasets has become a worrying reality. To address this issue, a federated learning (FL) LLM fine-tuning paradigm is proposed, which allows multiple data owners to collaboratively fine-tune a shared LLM without sharing the original data.

[0003] The main problem at present is that

[0004] 1) Heavy computation and communication burden.

[0005] Specifically, the huge model size of LLM imposes heavy computational and communication burdens on edge devices / servers, which is a computational overload for resource-constrained computing entities and hinders the popularization of distributed learning paradigms such as FL and SL in LLM fine-tuning.

[0006] In addition, there are some other problems, as follows:

[0007] 2) Data privacy issues: In the FL LLM fine-tuning paradigm, although the original data is not shared, deep learning models, especially large ones, have the ability to memorize training data, which may raise privacy concerns. This risk is particularly serious in LLM fine-tuning because of their powerful ability to inadvertently memorize and potentially expose more detailed information.

[0008] 3) Model splitting and heterogeneous configuration issues: In the SL LLM fine-tuning framework, choosing an appropriate cutlayer is crucial to control the amount of data transferred to the central server and the computational workload distribution between client servers and the central server. In addition, the resource heterogeneity between different client servers / devices may lead to serious stragglers in model aggregation.

[0009] 4) Efficiency Issue: When deploying large language model systems in practice, it is usually necessary to utilize private data on edge devices, which have less computing and storage resources than data centers. Training LLMs with billions of parameters on these edge devices faces significant obstacles due to limited storage and computing power. Summary of the invention

[0010] The purpose of the present invention is to provide a large-scale language model training method based on efficient parameter fine-tuning, which is based on a segmented federated learning framework and combines the parallel training advantages of FL and the model segmentation advantages of SL, thereby greatly improving the training efficiency.

[0011] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0012] A large language model training method based on efficient parameter fine-tuning, the large language model training method comprising:

[0013] S1, for a large language model as a training object, dividing the large language model into a client-side pre-training model and a server-side pre-training model;

[0014] S2, jointly conducts parallel collaborative training on the client-side pre-trained model and the server-side pre-trained model. During the training process, efficient parameter fine-tuning technology is used to train the client-side pre-trained model and the server-side pre-trained model in parallel;

[0015] S3, after the training is completed, the client-side pre-trained model and the server-side pre-trained model are merged using the model aggregation method;

[0016] S4, repeating the above steps S2 to S3 until the large language model used as the training object converges.

[0017] Furthermore, the step S1 specifically includes:

[0018] S11, select the split point:

[0019] Determine the network structure of the large language model and select an appropriate layer as the split point to divide the large language model into two parts: the client-side pre-training model and the server-side pre-training model.

[0020] The selection of the split point needs to consider factors including: the complexity of the network structure, the computing power of the client and server, and the data transmission efficiency;

[0021] S12, initialize large language model parameters:

[0022] Based on the selected split point, the parameters of the large language model are initialized into the client-side pre-trained model and the server-side pre-trained model respectively to ensure the consistency of the parameters and the effectiveness of the initialization, so as to facilitate effective collaboration in the subsequent training process;

[0023] S13, client model deployment:

[0024] When deploying a client-side pre-trained model on a client device, you need to consider the computing resources of the client device to ensure that the client-side pre-trained model can run efficiently.

[0025] S14, server-side model deployment:

[0026] Deploy server-side pre-trained models on the server side, deploy more complex model parts, and be responsible for centralized parameter updates and model optimization;

[0027] S15, define the communication protocol:

[0028] Establish a data communication protocol between the client and server to transmit large language model parameters, gradient information and necessary data;

[0029] S16, model synchronization mechanism:

[0030] Design and implement a synchronization mechanism for client and server model parameters, including gradient aggregation and update, and parameter exchange. The synchronization mechanism needs to take into account network bandwidth limitations and communication delays to ensure the efficiency and stability of the training process.

[0031] S17, Federated Learning Strategy:

[0032] Formulate a federated learning strategy between the client and server to optimize the federated training process of the model, including:

[0033] Client-side local training: The client device uses local data to perform local updates on the model;

[0034] Model upload: The client periodically uploads updated model parameters or gradients to the server;

[0035] Model aggregation: The server aggregates model parameters or gradients from multiple clients and updates the global model;

[0036] Global model update: Send the updated global model parameters to each client.

[0037] Furthermore, the step S2 specifically includes:

[0038] S21, initialize the parallel collaborative training environment:

[0039] 1) Client model initialization: Load the initial pre-trained model on all client devices. The pre-trained model is a copy of a universal pre-trained large language model.

[0040] 2) Server model initialization: Load the corresponding pre-trained model copy on the central server to ensure that the model version is consistent with the client;

[0041] 3) Prepare parameter-efficient fine-tuning techniques: select the required fine-tuning method and initialize its corresponding parameter settings;

[0042] S22, data allocation and preprocessing:

[0043] 1) Client data preprocessing: Each client device reads the local data set, performs data cleaning, word segmentation and encoding operations to ensure the uniformity of data format;

[0044] 2) Sharding and synchronization: Shard the preprocessed data and prepare batch processing, and synchronize the preliminary data distribution plan with the server;

[0045] S23, preliminary activation value calculation:

[0046] 1) Local training: Each client uses the current pre-trained model to perform preliminary forward propagation on the local data slice and calculate the activation value;

[0047] 2) Activation value cache: temporarily store the calculated activation value locally and prepare to synchronize with the server;

[0048] S24, gradient aggregation and synchronization:

[0049] 1) Upload local activation value: Each client uploads the calculated activation value to the server;

[0050] 2) Server-side gradient update: The server receives the activation values ​​uploaded by all clients, performs forward propagation and backward propagation, and calculates the global gradient update direction;

[0051] 3) Parameter update: Update the parameters of the server-side pre-trained model on the server side;

[0052] 4) Distribute gradient values: The server distributes the gradient values ​​back to each client;

[0053] 5) Client gradient update: The client receives the gradients transmitted from the server, performs back propagation, and updates the client model parameters;

[0054] S25, model parameters are synchronized back to the client:

[0055] 1) Distribute and update model parameters: The server distributes the model parameters after global gradient update back to each client;

[0056] 2) Local update: Each client receives the new parameters, updates its local model, and prepares for the next round of training;

[0057] S26, Circuit and Enhancement Training:

[0058] 1) Multi-round collaborative training: Repeat steps S23 to S26 to perform multiple rounds of gradient calculation, update, and efficient fine-tuning to gradually improve the performance of the model;

[0059] 2) Periodic evaluation: After a specific iteration frequency, the model performance is evaluated to determine the training effect and the degree of optimization of the fine-tuning strategy;

[0060] S27, dynamic tuning and model convergence:

[0061] Specifically, this step includes:

[0062] 1) Dynamic parameter adjustment: Dynamically adjust the hyperparameters of efficient fine-tuning techniques based on the performance of the model during training;

[0063] 2) Convergence judgment: Determine whether the model has reached a convergence state by monitoring the performance indicators of the model on the validation set.

[0064] Furthermore, the step S3 specifically includes:

[0065] S31, polymerization preparation:

[0066] Specifically, this step includes:

[0067] 1) Model version confirmation: Ensure that all client and server model versions are consistent, that is, all models have undergone the same rounds of training and fine-tuning;

[0068] 2) Synchronization check: verify whether each client has completed all training tasks and confirm that all necessary data has been uploaded to the server;

[0069] S32, model parameter collection:

[0070] Specifically, this step includes:

[0071] 1) Client parameter upload: Each client uploads its locally optimized model parameters to the server;

[0072] 2) Parameter compression and packaging: Before uploading, the parameters can be compressed and packaged as necessary to reduce transmission overhead and improve efficiency;

[0073] S33, global parameter calculation:

[0074] 1) Parameter decompression and merging: The server decompresses and preliminarily merges the parameters received from each client to prepare for global parameter calculation;

[0075] 2) Aggregation strategy selection: determine which model aggregation method to use, such as average aggregation, weighted aggregation, gradient-based aggregation, etc.

[0076] S34, initial polymerization:

[0077] 1) Local aggregation: If there are multiple server nodes, local aggregation can be performed on each server node to reduce the computing burden at a single point;

[0078] 2) Global aggregation: upload the local aggregation results to the main server for final global aggregation calculation;

[0079] S35, parameter weight adjustment:

[0080] 1) Weight allocation: Reasonably allocate weights based on the importance or training volume of each client’s data, especially when the training data is uneven;

[0081] 2) Weighted average: If the weighted average strategy is selected, the parameters are weighted averaged and merged according to the assigned weights;

[0082] S36, consistency correction:

[0083] 1) Conflict detection: Check whether there are conflicts or inconsistencies in the merged model parameters and make necessary corrections;

[0084] 2) Parameter pruning: pruning redundant or overfitting parameters to optimize the overall performance of the model;

[0085] S37, model fusion verification:

[0086] 1) Preliminary verification: Use the merged model to perform preliminary verification on the server side to check model performance and consistency;

[0087] 2) Performance testing: Forward testing is performed on a small batch of test data to ensure that the model still maintains high performance after global aggregation.

[0088] Compared with the prior art, the large language model training method of the present invention has the following beneficial effects:

[0089] The large language model is divided into the client and the server, where the client is responsible for forward propagation and only generates and transmits activation values, while the server is responsible for subsequent fine-tuning and back-propagation of gradients; this method significantly reduces the size of transmitted data because it avoids the need to transmit the entire model; in addition, the large language model training method also takes advantage of the segmented federated learning (SFL) to reduce the computing pressure of edge devices by performing the main training workload on a powerful central server. The large language model training method of the present invention is based on the segmented federated learning (SFL) framework and combines the parallel training advantages of FL and the model segmentation advantages of SL, thereby greatly improving the training efficiency and providing the first open source benchmark for SL LLM fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 An overview diagram of the SplitLoRA framework constructed according to the present invention;

[0091] Figure 2 Schematic diagram of convergence accuracy of GPT2-S and GPT2-M models constructed according to the present invention, where perplexity level (PPL) is an indicator used to measure the quality of a language model predicting a sample, and the lower the PPL, the better the prediction performance;

[0092] Figure 3 Schematic diagram of the training performance of GPT2-S and GPT2-M constructed according to the present invention in the E2E NLG challenge;

[0093] Figure 4 A comparison table of the convergence accuracy of various indicators of GPT2-S for the E2E NLG challenge constructed according to the present invention;

[0094] Figure 5 A comparison table of the convergence accuracy of various indicators of GPT2-M for the E2E NLG challenge constructed according to the present invention;

[0095] Figure 6 Comparison table of the number of trainable parameters on GPT2-S and GPT2-M constructed according to the present inventive concept. DETAILED DESCRIPTION

[0096] The present invention will be further described below with specific embodiments:

[0097] This embodiment provides a large-scale language model training method based on efficient parameter fine-tuning. By implementing this large-scale language model training method, it is mainly possible to reduce the computational and communication burdens.

[0098] See also Figure 1 The large-scale language model training method of this embodiment specifically includes the following steps S1 to S4.

[0099] S1, for a large language model as a training object, the large language model is divided into two parts: a client-side pre-training model and a server-side pre-training model.

[0100] S2, jointly conducts parallel collaborative training with the client-side pre-trained model and the server-side pre-trained model. During the training process, the existing efficient parameter fine-tuning techniques (such as Adapter Tuning, Prompt Tuning, Low Rank Adaptation (LoRA), etc.) are used to train the client-side pre-trained model and the server-side pre-trained model in parallel.

[0101] S3, after the training is completed, the existing model aggregation method is used to merge the client-side pre-trained model and the server-side pre-trained model again to achieve the training effect without splitting.

[0102] S4, repeating the above steps S2 to S3 until the “large language model as the training object” converges.

[0103] The above steps S1 to S4 are the most basic concept of this embodiment. Based on this concept, the large-scale language model training method of this embodiment can mainly achieve the good technical effect of "reducing the computing and communication burden".

[0104] The following are the details of each step:

[0105] The step S1 specifically includes the following steps S11 to S17.

[0106] S11, select the split point:

[0107] Specifically,

[0108] First, determine the network structure of the large language model and select an appropriate layer as the split point to divide the large language model into two parts: the client-side pre-trained model and the server-side pre-trained model. The selection of the split point needs to consider the following factors:

[0109] 1) Complexity of network structure;

[0110] 2) The computing power of the client and server;

[0111] 3) Data transmission efficiency.

[0112] For example, the first few layers of a large language model can be used as a client-side pre-trained model, and the rest as a server-side pre-trained model.

[0113] S12, initialize large language model parameters:

[0114] Specifically,

[0115] Based on the previously selected split points, the parameters of the large language model are initialized into the client-side pre-trained model and the server-side pre-trained model respectively. Ensure the consistency of parameters and the effectiveness of initialization for effective collaboration in the subsequent training process.

[0116] S13, client model deployment:

[0117] Specifically,

[0118] Deploy the client-side pre-trained model on the client device. This step needs to consider the computing resources of the client device, such as CPU / GPU performance, memory capacity, etc., to ensure that the client-side pre-trained model can run efficiently.

[0119] S14, server-side model deployment:

[0120] Specifically,

[0121] Deploy the server-side pre-trained model on the server side. The server usually has more powerful computing and storage capabilities, so it can deploy more complex model parts and is responsible for centralized parameter updates and model optimization.

[0122] S15, define the communication protocol:

[0123] Specifically,

[0124] Establish a data communication protocol between the client and the server to transmit large language model parameters, gradient information, and other necessary data. This requires ensuring the efficiency and security of data transmission, which may require the use of compression algorithms and encryption technologies.

[0125] S16, model synchronization mechanism:

[0126] Specifically,

[0127] Design and implement a synchronization mechanism for client and server model parameters. This may include gradient aggregation and updating, parameter exchange, etc. The synchronization mechanism needs to take into account network bandwidth limitations and communication delays to ensure the efficiency and stability of the training process.

[0128] S17, Federated Learning Strategy:

[0129] Specifically,

[0130] Formulate a federated learning strategy between the client and server to optimize the federated training process of the model. Specifically:

[0131] 1) Client-side local training: The client device uses local data to perform local updates on the model;

[0132] 2) Model upload: The client periodically uploads the updated model parameters or gradients to the server;

[0133] 3) Model aggregation: The server aggregates model parameters or gradients from multiple clients and updates the global model;

[0134] 4) Global model update: send the updated global model parameters to each client.

[0135] The step S2 specifically includes the following steps S21 to S27

[0136] S21, initialize the parallel collaborative training environment:

[0137] Specifically, this step includes:

[0138] 1) Client model initialization: Load the initial pre-trained model on all client devices. This can be a copy of a common pre-trained large language model.

[0139] 2) Server model initialization: Load the corresponding pre-trained model copy on the central server to ensure that the model version is consistent with the client.

[0140] 3) Prepare efficient parameter tuning techniques: Select the required tuning method (such as Adapter Tuning, Prompt Tuning, LoRA) and initialize its corresponding parameter settings.

[0141] S22, data allocation and preprocessing:

[0142] Specifically, this step includes:

[0143] 1) Client data preprocessing: Each client device reads the local data set, performs data cleaning, word segmentation and encoding operations to ensure the uniformity of data format;

[0144] 2) Sharding and synchronization: Shard the preprocessed data and prepare batch processing, and synchronize the preliminary data distribution plan with the server.

[0145] S23, preliminary activation value calculation:

[0146] Specifically, this step includes:

[0147] 1) Local training: Each client uses the current pre-trained model to perform preliminary forward propagation on the local data slice and calculate the activation value.

[0148] 2) Activation value cache: The calculated activation value is temporarily stored locally, ready to be synchronized with the server.

[0149] S24, gradient aggregation and synchronization:

[0150] Specifically, this step includes:

[0151] 1) Upload local activation value: Each client uploads the calculated activation value to the server;

[0152] 2) Server-side gradient update: The server receives the activation values ​​uploaded by all clients, performs forward propagation and backward propagation, and calculates the global gradient update direction;

[0153] 3) Parameter update: Update the parameters of the server-side pre-trained model on the server side;

[0154] 4) Distribute gradient values: The server distributes the gradient values ​​back to each client;

[0155] 5) Client gradient update: The client receives the gradients transmitted from the server, performs back propagation, and updates the client model parameters.

[0156] S25, model parameters are synchronized back to the client:

[0157] Specifically, this step includes:

[0158] 1) Distribute and update model parameters: The server distributes the model parameters after global gradient update back to each client.

[0159] 2) Local update: Each client receives the new parameters, updates its local model, and prepares for the next round of training.

[0160] S26, Circuit and Enhancement Training:

[0161] Specifically, this step includes:

[0162] 1) Multi-round collaborative training: Repeat steps S23 to S26 to perform multiple rounds of gradient calculation, update, and efficient fine-tuning to gradually improve the performance of the model.

[0163] 2) Periodic evaluation: After a specific iteration frequency, the model performance is evaluated to determine the optimization degree of the training effect and fine-tuning strategy.

[0164] S27, dynamic tuning and model convergence:

[0165] Specifically, this step includes:

[0166] 1) Dynamic parameter adjustment: Dynamically adjust the hyperparameters (such as learning rate, adapter size, etc.) of efficient fine-tuning techniques based on the performance of the model during training.

[0167] 2) Convergence judgment: By monitoring the performance indicators of the model on the validation set, it is judged whether the model has reached a convergence state, such as Figure 3 shown.

[0168] The step S3 specifically includes the following steps S31 to S37.

[0169] S31, polymerization preparation:

[0170] Specifically, this step includes:

[0171] 1) Model version confirmation: Ensure that all client and server model versions are consistent, that is, all models have undergone the same rounds of training and fine-tuning.

[0172] 2) Synchronization check: Verify whether each client has completed all training tasks and confirm that all necessary data has been uploaded to the server.

[0173] S32, model parameter collection:

[0174] Specifically, this step includes:

[0175] 1) Client parameter upload: Each client uploads its locally optimized model parameters to the server.

[0176] 2) Parameter compression and packaging: Before uploading, the parameters can be compressed and packaged as necessary to reduce transmission overhead and improve efficiency.

[0177] S33, global parameter calculation:

[0178] Specifically, this step includes:

[0179] 1) Parameter decompression and merging: The server decompresses and preliminarily merges the parameters received from each client to prepare for global parameter calculation.

[0180] 2) Aggregation strategy selection: Determine which model aggregation method to use, such as average aggregation, weight aggregation, gradient-based aggregation, etc.

[0181] S34, initial polymerization:

[0182] Specifically, this step includes:

[0183] 1) Local aggregation: If there are multiple server nodes, local aggregation can be performed on each server node first to reduce the computing burden at a single point.

[0184] 2) Global aggregation: upload local aggregation results to the main server for final global aggregation calculation

[0185] S35, parameter weight adjustment:

[0186] Specifically, this step includes:

[0187] 1) Weight allocation: Reasonably allocate weights based on the importance of each client's data or the amount of training, especially when the training data is uneven.

[0188] 2) Weighted average: If the weighted average strategy is selected, the parameters are combined by weighted average according to the assigned weights.

[0189] S36, consistency correction:

[0190] Specifically, this step includes:

[0191] 1) Conflict detection: Check whether there are conflicts or inconsistencies in the merged model parameters (such as gradient explosion) and make necessary corrections.

[0192] 2) Parameter pruning: Prune redundant or overfitting parameters to optimize the overall performance of the model.

[0193] S37, model fusion verification:

[0194] Specifically, this step includes:

[0195] 1) Preliminary verification: Use the merged model to perform preliminary verification on the server side to check the model performance and consistency. Figure 2 shown.

[0196] 2) Performance testing: Perform forward testing on a small batch of test data to ensure that the model still maintains high performance after global aggregation. Figure 4 , 5 , as shown in Figure 6.

[0197] Next, taking a specific large-scale language model operation scenario as an example, the implementation process of the large-scale language model training method of this embodiment is specifically introduced.

[0198] In this scenario, a central server, an edge server, and a federated server are set up.

[0199] In this scenario,

[0200] The central server's role is to:

[0201] 1) Global model storage and management: Maintain the central storage of global pre-trained models to ensure the uniformity and consistency of model versions.

[0202] 2) Global gradient aggregation and update: In the federated learning process, the gradients uploaded by all clients or edge servers are aggregated to perform global parameter updates.

[0203] 3) Task allocation and coordination: During the training process, coordinate the edge and federated servers to allocate training tasks and data slices.

[0204] The role of edge servers is to:

[0205] 1) Real-time data processing: Collect and preprocess data from local devices and use the processed data for model training.

[0206] 2) Local model update: Perform local optimization and update of the model locally, perform forward and backward propagation through local data, and calculate the model gradient.

[0207] 3) Preliminary gradient aggregation: Receive gradients from multiple local devices and perform preliminary aggregation to reduce the amount of data transmission.

[0208] The role of the federation server is to:

[0209] 1) Regional gradient aggregation: Collect local aggregated gradients from edge servers for further aggregation within the partition.

[0210] 2) Intermediate result synchronization: synchronize the aggregated intermediate gradient results to the central server, or receive the global update parameters from the central server for distribution.

[0211] 3) Regional model distribution: Distribute updated model parameters to edge servers based on actual needs to ensure real-time and continuous improvement of edge training.

[0212] During this implementation, a software system was constructed based on the large-scale language model training method of this implementation. For the convenience of description, the constructed software system was named "SplitLoRA".

[0213] (Corresponding to step S1) Before the large language model training begins, the central server initializes the large language model and divides it into a client-side pre-training model and a server-side pre-training model.

[0214] (Corresponding to step S2) Then, the large language model is fine-tuned through I rounds of continuous training.

[0215] (Corresponding to step S3) Then the client LoRA adapter is aggregated.

[0216] (Corresponding to step S4) This process is repeated until the model converges.

[0217] The training process of SplitLoRA for large language models consists of two main stages: segmentation fine-tuning (corresponding to step S2) and client LoRA adapter aggregation (corresponding to step S3). Segmentation fine-tuning is performed in each round of training, while client LoRA adapter aggregation is performed once every I round of training.

[0218] See also Figure 1 , for a training round The training process of SplitLoRA is detailed as follows.

[0219] (Corresponding to step S2)

[0220] Split fine-tuning phase: The split fine-tuning phase includes client-side and server-side fine-tuning of the participating edge servers and central servers in each training round, which specifically includes the following five steps.

[0221] 1) Client pre-trained model forward propagation (FP):

[0222] Specifically, all participating edge servers perform the forward propagation of the client-side pre-trained model in parallel.

[0223] More specifically, each edge server i randomly selects a micro-batch from its local dataset This micro-batch contains b data samples and is used to fine-tune the client-side pre-trained model.

[0224] In training round t, the input data and corresponding labels of the micro-batch are represented as and represents the set of trainable LoRA adapters of the client pre-trained model of edge server i in the tth round of training. After a set of micro-batch data is input into the client pre-trained model, the activation value is generated at the slice layer. The activation value of edge server i is expressed as

[0225]

[0226] in, Represents the mapping between input data x and its predicted value given the model parameters w and the set of trainable LoRA adapters r.

[0227] 2) Activation value transmission:

[0228] After completing the forward propagation of the client-side pre-trained model, all participating edge servers transmit their respective activation values ​​and corresponding labels to the central server (usually through a wireless channel). Then, in the central server, the activation values ​​received from the participating edge servers are used to drive the fine-tuning of the server-side pre-trained model.

[0229] 3) Server-side pre-trained model forward propagation and back propagation:

[0230] In the central server, after receiving the activation values ​​from the participating edge servers, the central server inputs these activation values ​​into the server-side pre-trained model to perform server-side forward propagation (FP).

[0231] A collection of trainable LoRA adapters representing the server-side pre-trained model in the tth round of training.

[0232] The activation matrix S of the connection t Expressed as Therefore, the predicted value is expressed as

[0233]

[0234] The predicted values ​​and labels are used to calculate the loss function value and further derive the gradient of the server-side LoRA adapter. Therefore, the mth server-side LoRA adapter can be updated as follows:

[0235]

[0236] in, and represents the gradients of the decomposition matrices A and B of the mth LoRA adapter of the server-side pre-trained model, γ s is the learning rate on the server side.

[0237] 4) Activate gradient transfer:

[0238] After the server-side back-propagation (BP) is completed, the central server transmits the activated gradients to the corresponding participating edge servers.

[0239] 5) Client model back propagation:

[0240] In this step, each edge server fine-tunes the client-side pre-trained model based on the received activation gradients.

[0241] For edge server i, the jth client LoRA adapter is updated via

[0242]

[0243] in, and represents the gradient of the decomposition matrices A and B of the nth LoRA adapter of the server-side pre-trained model, γ c is the client’s learning rate.

[0244] (Corresponding to step S3)

[0245] Client LoRA adapter aggregation phase: mainly focuses on aggregating client LoRA adapters on the federated server, which is performed once every I rounds of training. This phase includes the following three steps.

[0246] 1) Client LoRA adapter upload:

[0247] In this step, each participating edge server sends its client LoRA adapter to the federation server over wireless / wired link.

[0248] 2) Client LoRA adapter aggregation:

[0249] The federated server aggregates the received client LoRA adapters into an aggregated LoRA adapter. The decomposition matrices A and B of the nth client LoRA adapter are aggregated as follows:

[0250]

[0251]

[0252] 3) Client LoRA adapter downlink transmission:

[0253] After completing the client LoRA adapter aggregation, the federation server sends the aggregated client LoRA adapter to the participating edge servers. Each edge server then uses the received aggregated LoRA adapter as the initial LoRA configuration for the next round of training.

[0254] The following provides a performance evaluation of the large language model training method implemented in the above specific scenarios:

[0255] This evaluation includes convergence accuracy, convergence speed and resource efficiency.

[0256] 1) Experimental setup

[0257] Hardware: We implemented SplitLoRA using Python 3.7 and PyTorch 1.7.1 and trained it on an NVIDIA GeForce RTX 3090 GPU.

[0258] Datasets and Tasks: We evaluate the training performance of SplitLoRA on the E2E dataset

[29] for the natural language generation (NLG) task. The E2E dataset consists of approximately 42,000 training, 4,600 validation, and 4,600 test examples in the restaurant domain.

[0259] Models: We use the well-known GPT2 Small (GPT-S) and GPT2 Medium (GPT2-M) models. GPT2-S is the smallest version in the GPT2 family with 124 million parameters and a 12-layer Transformer encoder, while GPT2-M is the medium version of GPT2 with 355 million parameters and a 24-layer Transformer encoder.

[0260] Experimental setup: In simulations, we deploy N = 3 edge servers by default unless otherwise specified. The client-side model of each edge server has the first three Transformer layers of the GPT-2 model, while the server-side model of the central server retains the remaining 9 Transformer layers (GPT2-S) or 21 layers (GPT2-M). The computing power of each edge server is 35.6TFLOTS (the peak performance of NVIDIA RTX 3090), while the computing power of the central server is set to 284.8TFLOTS. The communication rate between the edge server and the federated server is set to 300Mbps, and the communication rate between the edge server and the central server is set to 600Mbps. We set the mini-batch size, learning rate, and maximum sequence length of the GPT2-S model to 8, 0.0002, and 512, and those of the GPT2-M model to 4, 0.0002, and 512, respectively. The rank of the LoRA adapter is set to r = {1, 2, 4, 8}.

[0261] Benchmarks: To study the merits of SplitLoRA, we compare it with three typical baselines: (1) Centralized LoRA (CenLoRA): The edge server collects raw data from other participating servers for full model fine-tuning, using LoRA adapters. (2) Federated LoRA (FedLoRA): Each participating edge server fine-tunes the full model locally and then transmits the updated LoRA adapter to the central server for adapter aggregation.

[0262] 2) Performance evaluation

[0263] Convergence accuracy: Figure 2The training performance of SplitLoRA is compared with other baselines on the GPT2-S and GPT2-M models of the E2E NLG challenge, using PPL as the performance metric. FedLoRA achieves lower convergence accuracy than SplitLoRA and CenLoRA, and its PPL is about 0.08 / 0.11 (GPT2-S / GPT2-M) and 0.73 / 0.09 (GPT2-S / GPT2-M) higher than SplitLoRA and CenLoRA, respectively. The poor training performance of FedLoRA stems from its fully distributed training mode, where the entire model is updated through model aggregation, and is therefore susceptible to model bias caused by heterogeneous edge server data. SplitLoRA achieves comparable convergence accuracy to CenLoRA, especially on GPT2-M, where the accuracy difference is less than 0.04. There are two reasons for this: first, the server sub-model in SplitLoRA is trained in a centralized manner, making it more resistant to data heterogeneity; second, SplitLoRA offloads the main workload to the central server, and only a small number of LLMs are affected by client data heterogeneity. It can be seen that as the level r increases, the PPL of SplitLoRA and CenLoRA become closer. This is because a higher r means more trainable parameters, which enhances the fitting ability of the server sub-model. This may partially offset the impact of data heterogeneity and make the performance of SplitLoRA closer to CenLoRA. For a comparison of training performance of other indicators, see Figure 4 and Figure 5 .

[0264] Convergence speed: Figure 3 (a) and Figure 3 (b) shows the training performance of SplitLoRA and other baselines on GPT2-S and GPT-M of the E2E NLG challenge. Obviously, SplitLoRA significantly outperforms FedLoRA and CenLoRA in convergence speed. The training latency of FedLoRA and CenLoRA required to reach model convergence is approximately 1.7 times and 4.7 times that of SplitLoRA on GPT-S and GPT-M and 2.1 times and 4.8 times that of SplitLoRA on GPT-S and GPT-M. Compared with CenLoRA, FedLoRA speeds up model convergence due to the parallel training mode of multiple edge servers. By offloading the main computing workload to a more powerful central server, SplitLoRA is based on FedLoRA and further shortens the time required to reach model convergence while supporting multiple edge servers for parallel training.

[0265] Trainable Parameters: Table III shows the number of trainable parameters for GPT2-S and GPT2-M. SplitLoRA reduces the computational workload on edge servers by partitioning the model and keeping fewer Transformer layers locally. In our experimental setup, for GPT2-S and GPT2-M models, the parameters in SplitLoRA used for fine-tuning on edge servers account for only one-quarter and one-eighth of the entire model. This huge reduction in local computational requirements makes SplitLoRA efficient and suitable for deployment in resource-constrained environments. In contrast, CenLoRA and FedLoRA require edge servers to process the entire GPT-2 model for fine-tuning, which places significant demands on computing power and memory resources. In particular, in edge computing systems of 5G and beyond, small base stations and access points with limited capabilities act as edge libraries. Therefore, SplitLoRA not only improves computational efficiency, but also enhances the flexibility and adaptability of the system to better cope with various computationally and resource-constrained scenarios.

[0266] The large language model training method of this embodiment has the advantages that:

[0267] 1) Reduce the computational and communication burdens through model segmentation and parallel training (for problem 1 raised in the background technology). The model is divided into the client and server, where the client is responsible for forward propagation and only generates and transmits activation values, while the server is responsible for subsequent fine-tuning and backpropagation of gradients. This approach significantly reduces the size of transmitted data because it avoids the need to transmit the entire model. In addition, the large language model training method also takes advantage of segmented federated learning (SFL) to reduce the computational pressure on edge devices by performing the main training workload on a powerful central server.

[0268] 2) Privacy is protected by aggregating the client's LoRA adapters on a federated server instead of directly aggregating model parameters (addressing problem 2 raised in the background technology). This approach ensures that client data does not need to leave the local device because only the model's adapters (which are a small part of the model parameters) are used to update the global model. In addition, the large language model training method adopts differential privacy technology to further protect the privacy of training data by adding controlled noise to the model gradient or update.

[0269] 3) Allows different split layers and LoRA adapter ranks to be selected based on the resource heterogeneity of the client server / device (for problem 3 in the background technology). By carefully selecting the split layer and the adapter rank, the computational workload between the client and server can be balanced and the amount of data that the client device needs to transmit can be reduced. This approach allows resource-constrained devices to participate in model training while ensuring the efficiency and effectiveness of model training.

[0270] 4) Reduce the size and computational complexity of the model through model compression techniques such as knowledge distillation, pruning, and quantization without significantly affecting model performance (for question 4 raised in the background technology). These techniques make it possible to deploy smaller and more efficient versions of LLM in resource-constrained environments. In addition, large language model training methods improve efficiency by reducing the number of parameters fine-tuned on the client (only a small part of the entire model), so that LLM can be fine-tuned even on edge devices with lower computing power. In this way, it is possible to adapt to different computing and resource-constrained scenarios while maintaining model performance.

[0271] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A large language model training method based on efficient parameter fine-tuning, characterized by: The large language model training method comprises: S1, for a large language model as a training object, dividing the large language model into a client-side pre-training model and a server-side pre-training model; S2, jointly conducts parallel collaborative training on the client-side pre-trained model and the server-side pre-trained model. During the training process, efficient parameter fine-tuning technology is used to train the client-side pre-trained model and the server-side pre-trained model in parallel; S3, after the training is completed, the client-side pre-trained model and the server-side pre-trained model are merged using the model aggregation method; S4, repeating the above steps S2 to S3 until the large language model used as the training object converges.

2. The large-scale language model training method based on efficient parameter fine-tuning according to claim 1, characterized in that: The step S1 specifically includes: S11, select the split point: Determine the network structure of the large language model and select an appropriate layer as the split point to divide the large language model into two parts: the client-side pre-training model and the server-side pre-training model. The selection of the split point needs to consider factors including: the complexity of the network structure, the computing power of the client and server, and the data transmission efficiency; S12, initialize large language model parameters: Based on the selected split point, the parameters of the large language model are initialized into the client-side pre-trained model and the server-side pre-trained model respectively to ensure the consistency of the parameters and the effectiveness of the initialization, so as to facilitate effective collaboration in the subsequent training process; S13, client model deployment: When deploying a client-side pre-trained model on a client device, you need to consider the computing resources of the client device to ensure that the client-side pre-trained model can run efficiently. S14, server-side model deployment: Deploy server-side pre-trained models on the server side, deploy more complex model parts, and be responsible for centralized parameter updates and model optimization; S15, define the communication protocol: Establish a data communication protocol between the client and server to transmit large language model parameters, gradient information and necessary data; S16, model synchronization mechanism: Design and implement a synchronization mechanism for client and server model parameters, including gradient aggregation and update, and parameter exchange. The synchronization mechanism needs to take into account network bandwidth limitations and communication delays to ensure the efficiency and stability of the training process. S17, Federated Learning Strategy: Formulate a federated learning strategy between the client and server to optimize the federated training process of the model, including: Client-side local training: The client device uses local data to perform local updates on the model; Model upload: The client periodically uploads updated model parameters or gradients to the server; Model aggregation: The server aggregates model parameters or gradients from multiple clients and updates the global model; Global model update: Send the updated global model parameters to each client.

3. The large-scale language model training method based on efficient parameter fine-tuning according to claim 1, characterized in that: The step S2 specifically includes: S21, initialize the parallel collaborative training environment: 1) Client model initialization: Load the initial pre-trained model on all client devices. The pre-trained model is a copy of a universal pre-trained large language model. 2) Server model initialization: Load the corresponding pre-trained model copy on the central server to ensure that the model version is consistent with the client; 3) Prepare parameter-efficient fine-tuning techniques: select the required fine-tuning method and initialize its corresponding parameter settings; S22, data allocation and preprocessing: 1) Client data preprocessing: Each client device reads the local data set, performs data cleaning, word segmentation and encoding operations to ensure the uniformity of data format; 2) Sharding and synchronization: Shard the preprocessed data and prepare batch processing, and synchronize the preliminary data distribution plan with the server; S23, preliminary activation value calculation: 1) Local training: Each client uses the current pre-trained model to perform preliminary forward propagation on the local data slice and calculate the activation value; 2) Activation value cache: temporarily store the calculated activation value locally and prepare to synchronize with the server; S24, gradient aggregation and synchronization: 1) Upload local activation value: Each client uploads the calculated activation value to the server; 2) Server-side gradient update: The server receives the activation values ​​uploaded by all clients, performs forward propagation and backward propagation, and calculates the global gradient update direction; 3) Parameter update: Update the parameters of the server-side pre-trained model on the server side; 4) Distribute gradient values: The server distributes the gradient values ​​back to each client; 5) Client gradient update: The client receives the gradients transmitted from the server, performs back propagation, and updates the client model parameters; S25, model parameters are synchronized back to the client: 1) Distribute and update model parameters: The server distributes the model parameters after global gradient update back to each client; 2) Local update: Each client receives the new parameters, updates its local model, and prepares for the next round of training; S26, Circuit and Enhancement Training: 1) Multi-round collaborative training: Repeat steps S23 to S26 to perform multiple rounds of gradient calculation, update, and efficient fine-tuning to gradually improve the performance of the model; 2) Periodic evaluation: After a specific iteration frequency, the model performance is evaluated to determine the training effect and the degree of optimization of the fine-tuning strategy; S27, dynamic tuning and model convergence: Specifically, this step includes: 1) Dynamic parameter adjustment: Dynamically adjust the hyperparameters of efficient fine-tuning techniques based on the performance of the model during training; 2) Convergence judgment: Determine whether the model has reached a convergence state by monitoring the performance indicators of the model on the validation set.

4. The large-scale language model training method based on efficient parameter fine-tuning according to claim 1, characterized in that: The step S3 specifically includes: S31, polymerization preparation: Specifically, this step includes: 1) Model version confirmation: Ensure that all client and server model versions are consistent, that is, all models have undergone the same rounds of training and fine-tuning; 2) Synchronization check: Verify whether each client has completed all training tasks and confirm that all necessary data has been uploaded to the server; S32, model parameter collection: Specifically, this step includes: 1) Client parameter upload: Each client uploads its locally optimized model parameters to the server; 2) Parameter compression and packaging: Before uploading, the parameters can be compressed and packaged as necessary to reduce transmission overhead and improve efficiency; S33, global parameter calculation: 1) Parameter decompression and merging: The server decompresses and preliminarily merges the parameters received from each client to prepare for global parameter calculation; 2) Aggregation strategy selection: determine which model aggregation method to use, such as average aggregation, weighted aggregation, gradient-based aggregation, etc. S34, initial polymerization: 1) Local aggregation: If there are multiple server nodes, local aggregation can be performed on each server node to reduce the computing burden at a single point; 2) Global aggregation: upload local aggregation results to the main server for final global aggregation calculation; S35, parameter weight adjustment: 1) Weight allocation: Reasonably allocate weights according to the importance or training amount of each client data, especially when the training data is uneven; 2) Weighted average: If the weighted average strategy is selected, the parameters are weighted averaged and merged according to the assigned weights; S36, consistency correction: 1) Conflict detection: Check whether there are conflicts or inconsistencies in the merged model parameters and make necessary corrections; 2) Parameter pruning: pruning redundant or overfitting parameters to optimize the overall performance of the model; S37, model fusion verification: 1) Preliminary verification: Use the merged model to perform preliminary verification on the server side to check the model performance and consistency; 2) Performance testing: Forward testing is performed on a small batch of test data to ensure that the model still maintains high performance after global aggregation.

Images