Large language model fine-tuning federal learning excitation method based on benchmark test

Abstract

The invention discloses a benchmark test-based large language model fine-tuning federal learning excitation method. The method comprises the following steps: when fine tuning is performed on a large language model, performing local training on the model by a node; after local training is completed, training parameters are uploaded to a center node, the center node evaluates the training quality of the nodes in the round of training by using a benchmark test, and the benchmark test can accurately measure the parameter quality of local training of the nodes and judge node attacks; updating the reputation value of the node according to the benchmark test result; and then the central node performs global model aggregation on the nodes by using a federated learning aggregation method according to the evaluation result and the reputation value. According to the method, a feasible incentive mechanism is designed for federated learning of large language model fine tuning, the problem that an incentive mechanism is lacked in a large language model fine tuning federated learning scene is solved, reliable nodes can be attracted to submit high-quality data, low-quality node contribution can be eliminated, and federated learning application can be further promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning and large language models, and specifically, relates to a large language model fine-tuning federated learning incentive method based on benchmark testing. Background Art

[0002] With the continuous development of neural networks, machine learning and artificial intelligence technologies, generative pre-trained models based on transformers have been widely used. Due to its large number of parameters, complex model network structure, large amount of data required for training models and natural language interaction, it is also called a large language model. Large language models have achieved excellent results in applications such as natural language processing and pattern recognition.

[0003] Training a large language model often requires two steps: pre-training the model and fine-tuning the model. Although the pre-trained model has a lot of "knowledge", the trained model is only for the prediction of the next word. Based on pre-training, the model is fine-tuned to enable the model to understand and respond to human language.

[0004] Whether pre-training or fine-tuning a large language model, a large amount of natural language data needs to be collected as a training dataset. However, most of the existing large language model training datasets are collected from the Internet. These data collected from the Internet may contain sensitive information that users are unwilling to disclose, such as mobile phone numbers, ID numbers, and financial information. As large language models are increasingly used, improper use of private data has caused users to worry. In addition to concerns about privacy, high-quality public datasets will be exhausted in a few years. More and more high-quality datasets will be hidden in data silos.

[0005] In order to solve the problem of data silos, the common approach is to use federated learning. Federated learning is a distributed machine learning method that allows participants in federated learning training to train models without leaving the local data. This approach can reduce data transmission and is more conducive to protecting user privacy.

[0006] However, existing federated learning for fine-tuning large language models has the following two problems:

[0007] 1. Existing federated learning for large language model fine-tuning assumes that all participating nodes will participate in the federated learning training without any rules. This positive assumption is inconsistent with reality. Since the existing federated learning for large language model fine-tuning lacks an incentive mechanism, nodes with high-quality data are likely to be unwilling to participate in the federated learning for large language model fine-tuning. In addition, the uneven data quality among the nodes participating in the federated learning for large language model fine-tuning will also affect the enthusiasm of the nodes to participate in federated learning;

[0008] 2. The central server always unconditionally believes the training results uploaded by the nodes, which gives malicious nodes an opportunity to attack the federated learning network. If a malicious node uploads offensive training results during federated training, the global training of federated learning will also be affected. Summary of the invention

[0009] In view of the problems in the above-mentioned prior art, the present invention proposes a benchmark-based large language model fine-tuning federated learning incentive method, and designs an incentive mechanism for the scenario of large language model fine-tuning federated learning, which accurately measures the quality of local training of federated learning nodes and updates the reputation value of nodes. Different reputation values ​​are used to determine whether local training is aggregated into global nodes, and incentives are distributed fairly, thereby solving the problem of node enthusiasm; through the node training quality evaluation method, the attack parameters of malicious nodes are accurately screened out, and the anti-attack capability of federated learning is improved.

[0010] In order to solve the above technical defects, the present invention proposes a large language model fine-tuning federated learning incentive method based on benchmark testing. It mainly includes the following steps:

[0011] Step 1: The central server of federated learning contacts the nodes to form a federated learning network, the federated learning training parameters are initialized, the central server prepares the pre-trained model, and the nodes prepare the fine-tuning dataset;

[0012] Step 2: The central server pre-selects nodes according to the reputation value rules and sends training parameters or pre-trained models to the selected nodes;

[0013] Step 3: The node performs low-rank adaptation parameter initialization training based on local training data. After completing the low-rank adaptation parameter initialization training, the initial parameter matrix is ​​uploaded to the central server, and the central server performs the first pass of node screening based on the parameters;

[0014] Step 4: Perform local training of low-rank adaptation through the nodes screened in the first pass described in step 3 and record the training cost at the same time. After completing the local training, upload the training results and training cost records to the central server;

[0015] Step 5: The central server performs a second screening based on the parameters uploaded by the node. The second screening evaluates the node contribution through the benchmark test method and calculates the node contribution value. If the node passes the second screening, the node will participate in this round of global aggregation and increase its reputation value. On the contrary, if the node fails the second screening, the node will not participate in this round of global aggregation and decrease its reputation value. Finally, according to the node contribution value and the training cost recorded in step 4, node incentives are issued according to the principle of distribution according to work.

[0016] Step 6: Repeat Steps 2 to 5 until the preset end condition for the fine-tuning federated learning of the large language model is reached.

[0017] Furthermore, as a preferred method of the present invention, in Step 2, the method should further include the following steps:

[0018] Step 2-1: When the training batch t of the federated learning is 1, the central server randomly selects k nodes from the set N = {1, 2,... u} of all nodes according to the preset parameter node number k, and k ≤ u; when the training batch t of the federated learning is greater than 1, if there are nodes in the set of nodes that have never participated in the federated learning training, then randomly select k × v nodes from them to enter the node pre-selection, where the node update rate v satisfies 0 < v < 1; when the number of nodes that have never participated in the federated learning training is less than k × v, select the top k × (1 - v) nodes in descending order of reputation value, and finally randomly select from the remaining nodes until the number of pre-selected nodes is k;

[0019] Step 2-2: When the training batch t of the federated learning is 1, the central server will send the pre-trained model to the nodes, and at the same time record the initial reputation value rep of the node with node number i i as the preset initial value; when the training batch t of the federated learning is not 1, the central server will send the low-rank adaptation global aggregation parameters aggregated previously to the nodes;

[0020] Furthermore, as a preferred method of the present invention, in Step 3, the low-rank adaptation parameter initialization training and the first-pass screening method should further include the following steps:

[0021] Step 3-1: The node receives the pre-trained matrix W 0 and the global aggregation parameter B·A. Among them, W 0 is an m-row n-column matrix, B is an m-row r-column matrix, and A is an r-row m-column matrix; m and n will vary according to the different pre-trained matrices selected, and r is the rank of the low-rank adaptation fine-tuning, which is configured at the beginning of the federated learning. r is required to be less than the minimum of m and n.

[0022] Step 3-2: Calculate W′ = W 0 + B·A as the initial value of the full-scale parameters for this round of local training according to the pre-trained matrix W 0 and the global aggregation parameter B·A, and then perform full-scale fine-tuning on W′ to calculate the initial gradient W init . Perform singular value decomposition on the initial gradient W init W init = U·S·V T , let A init be the first r columns of the matrix U, and B initis the [r+1:2r] row of the matrix V;

[0023] Step 3-3: Node A init With B init Upload to the central server, and record the parameters uploaded by node i in batch t as (B init· A init ) i，t , through the formula diff(A, B) = ||AB|| F Calculate the difference between matrices A and B, where ||·|| F represents the Frobenius norm. For each matrix (B init ·A init ) i，t , calculate the sum of its differences with other matrices in the batch D i =∑ j≠i diff((B init ·A init ) i，t , (B init ·A init ) j，t ), and calculate the average of the differences

[0024] Step 3-4: Calculate the standard deviation For each node in the batch, compare its difference D i Difference from average If satisfied If the node uploads malicious parameters in this batch, it is considered that the node has attacked. The central node will refuse to aggregate the parameters in this round of global aggregation and reduce the node's reputation value to rep. i -h rep . Where K is the threshold defined at the beginning of federated learning.

[0025] Further, as a preferred method of the present invention, in step 5, the second screening method should also include the following steps:

[0026] Step 5-1: The central server receives the training results uploaded by node i after completing local training in batch t using the method described in step 4 (B final ·A final ) i，t and the training cost C i，t ;

[0027] Step 5-2: Based on the results of the t batch local training of node i (B final ·A final ) i，t , according to the preset questions in the node evaluation data set, generate the corresponding answers ans i，t ;

[0028] Step 5-3: Answer ans i，t The referee model will score the node, with the lowest score set to L and the highest score set to H. When the referee model makes a judgment, it will conduct two rounds of evaluation on the node's answer, and finally get the node score s i，t ;

[0029] Step 5-4: Benchmark and score all node parameters involved in the training to get U t ={s i，t , |i is the node number participating in round t training}, and the parameters of the last round of global aggregation are benchmarked and scored to obtain {s last，t-1}. Calculate the set U t ∪{s last，t-1 The standard deviation sd t and the average score avg t ;

[0030] Step 5-5: Calculate the Z score of each node in batch t, If the node Z score is greater than 2, it is considered that the node has made a positive contribution in this round of training, and the node reputation value is adjusted upward to rep i +h rep , and accept the training results of the node in this round of global aggregation; if the Z score is less than -2, it is considered that the node may have attack behavior, and the reputation value of the node is lowered to rep i -h rep , and refuse to aggregate the training results of the node in this round of global aggregation. If the node is selected to participate in federated learning in subsequent rounds and can pass the first and second rounds of node screening, the training results of the node in the new round will also be accepted by the global aggregation of the new round. After the screening is completed, the training results of the accepted nodes are globally aggregated through the federated averaging algorithm FedAvg.

[0031] In the present invention, in step 6, the preset termination condition of the federated learning for fine-tuning the large language model includes adopting a maximum number of iterations, and terminating the training when the number of training reaches the maximum number of iterations.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. In the federated learning for fine-tuning large language models, an incentive mechanism is designed. The designed incentive mechanism takes into account the fact that not all nodes are willing to participate in federated learning in reality. Compared with the unincentivized federated learning for fine-tuning large language models, the federated learning designed with an incentive mechanism can attract more nodes to participate in the federated learning training. The designed incentive mechanism will also allocate more incentives to nodes with higher training quality, prompting nodes to make more high-quality contributions. High-quality contributions are continuously involved in global aggregation, ultimately improving the training quality of the federated learning for fine-tuning large language models;

[0034] 2. The present invention designs a node initial parameter screening method in the pre-selection stage, and uses a node screening method for benchmark evaluation after the node completes training and uploads parameters to the central server. Both methods can accurately and effectively screen out malicious nodes, and then defend against malicious nodes' attacks on large language fine-tuning federated learning by reducing reputation values ​​and refusing global merging. These two methods enable federated learning of large language model fine-tuning to have the ability to detect and resist malicious node attacks, allowing federated learning to proceed more robustly. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 A flow chart of the benchmark-based large language model fine-tuning federated learning incentive method provided by the present invention.

[0037] Figure 2 This is an example diagram of the anti-attack experiment results in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0040] refer to Figure 1 The large language model fine-tuning federated learning incentive method based on benchmark testing provided by the present invention comprises the following steps:

[0041] S1: The central server of federated learning contacts the nodes to form a federated learning network. The training parameters of federated learning are initialized. The central server prepares the pre-trained model and the nodes prepare the fine-tuning dataset.

[0042] S2: The central server pre-selects nodes according to the reputation value rules and sends training parameters or pre-trained models to the selected nodes;

[0043] S3: The node performs low-rank adaptation parameter initialization training based on local training data. After completing the low-rank adaptation parameter initialization training, the initial parameter matrix is ​​uploaded to the central server, and the central server screens the parameters;

[0044] S4: Perform local training of low-rank adaptation on the selected nodes, and upload the training results to the central server after completing the local training;

[0045] S5: The central server benchmarks the quality of the parameters uploaded by the nodes. Based on the results of the benchmark tests, it selects appropriate parameters to aggregate the nodes into the global model, updates the node reputation value based on the quality of the benchmark tests, and issues node incentives based on the standard of distribution according to work.

[0046] S6: Determine whether the preset end condition of the large language model fine-tuning federated learning has been reached. If so, stop training and save the training results to the output path. If not, start the next batch of federated learning training from step S2.

[0047] Furthermore, in S1, the method should also include the following steps:

[0048] S11: Configure the large language model to fine-tune the training parameters of federated learning, including the path of the pre-trained model, the output path of the training results, the end batch of training, and the initial reputation value rep of the node i , Reputation value adjustment step h rep, the total number of nodes u in the federated learning network, the number of nodes k participating in training in each round, the rank r of low-rank adaptation fine-tuning, the standard deviation judgment threshold K, the node update rate v, the lowest value L and the highest value H of the benchmark evaluation score;

[0049] S12: When the central server prepares the pre-trained model, preferably, the prepared pre-trained model is implemented in the Huggingface format;

[0050] S13: When the node prepares the fine-tuning dataset, preferably, the prepared dataset is implemented in the Huggingface format;

[0051] Furthermore, in S2, the method should also include the following steps:

[0052] S21: When the training batch t of federated learning is 1, the central server randomly selects k nodes from the set N = {1, 2,... u} of all nodes according to the parameter node number k set in S11, and k ≤ u. When the training batch t of federated learning is greater than 1, if there are nodes in the set of nodes that have never participated in federated learning training, then randomly select k × v nodes from them to enter the node pre-selection, where the node update rate v satisfies 0 < v < 1; when the number of nodes that have never participated in federated learning training is less than k × v, select the first k × (1 - v) nodes in descending order of reputation value, and finally randomly select from the remaining nodes until the number of pre-selected nodes is k;

[0053] S22: When the training batch t of federated learning is 1, the central server will send the pre-trained model to the nodes, and at the same time record the initial reputation value rep of the node with node number i as the preset initial value; when the training batch t of federated learning is not 1, the central server will send the previously globally aggregated low-rank adaptation global aggregation parameters to the nodes; i For the preset initial value; when the training batch t of federated learning is not 1, the central server will send the previously globally aggregated low-rank adaptation global aggregation parameters to the nodes;

[0054] Furthermore, in S3, the method should also include the following steps:

[0055] S31: The node receives the pre-trained matrix W 0 And the global aggregation parameter B·A. Among them, W 0 Is an m-row n-column matrix, B is an m-row r-column matrix, and A is an r-row m-column matrix; m and n will change according to the different pre-trained matrices selected;

[0056] S32: Calculate W′ = W 0 According to the pre-trained matrix W 0 And the global aggregation parameter B·A as the initial value of the full-scale parameters for this round of local training, and then perform full-scale fine-tuning on W′ to calculate the initial gradient W init. For the initial gradient W init Perform singular value decomposition W init =U·S·V T , let A init is the first r columns of matrix U, B init is the [r+1:2r] row of the matrix V;

[0057] S33: Node A init With B init Upload to the central server, and record the parameters uploaded by node i in batch t as (B init ·A init ) i，t , through the formula diff(A, B) = ||AB|| F Calculate the difference between matrices A and B, where ||·|| F represents the Frobenius norm. For each matrix (B init ·A init ) i，t , calculate the sum of its differences with other matrices in the batch D i =∑ j≠ i diff((B init ·A init ) i，t , (B init ·A init ) j,t ), and calculate the average of the differences

[0058] S34: Calculate standard deviation For each node in the batch, compare its difference D i Difference from average If satisfied If the node uploads malicious parameters in this batch, it is considered that the node has attacked. The central node will refuse to aggregate the parameters in this round of global aggregation and reduce the node's reputation value to rep. i -h rep .

[0059] Furthermore, in S4, the method should also include the following steps:

[0060] S41: Set the initial parameter of low-rank adaptation to B obtained in S32 through the selected node init With A init , local training based on local datasets;

[0061] S42: During the training process, record the training cost C of the batch i，j =F flops ×U GPU ×Ti，j , where F flops Indicates the peak floating-point operations per second of the GPU used by the node, U GPU It represents the average GPU utilization of the node in this batch, T i，j It indicates the time taken by the node to complete local training under this batch;

[0062] S43: After completing the local training, the training result B final With A final and the training cost C i，j Upload to the central server;

[0063] Furthermore, in S5, the method should also include the following steps:

[0064] S51: In batch t, ​​the central server receives the training results uploaded by node i through the method described in S43 (B final ·A final ) i，t and the training cost C i，t ;

[0065] S52: The central server generates corresponding answers according to the preset questions in the node evaluation data set. i，t . will answer ans i，t The referee model will score the node. Preferably, the GPT4 model is used as the referee. The lowest score L is 1 and the highest score H is 10. When the referee model makes a judgment, the node answer will be evaluated twice, and the node score s will be obtained in the end. i，t ;

[0066] S53: Benchmark and score all the node parameters involved in the training, and get U t ={s i，t , |i is the node number participating in round t training}, and the parameters of the last round of global aggregation are benchmarked and scored to obtain {s last，t-1}. Calculate the set U t ∪{s last，t-1 The standard deviation sd t and the average score avg t ;

[0067] S54: Calculate the Z score of each node in batch t, If the node Z score is greater than 2, it is considered that the node has made a positive contribution in this round of training, and the node reputation value is adjusted upward to rep i +h rep If the Z score is less than -2, the node is considered to have attack behavior and the reputation value is adjusted down to rep i -h rep, and refuse to aggregate the node in the global aggregation. For the nodes that are not rejected, preferably, the training results of the nodes are globally aggregated by the federated averaging algorithm (FedAvg);

[0068] S55: Distribute incentives to nodes by distributing according to work. For nodes that have completed local training and are accepted into the global aggregation, calculate the distribution that node i deserves in batch t training. in, represents the total incentive proposed by the central node in batch t, ​​S t represents the sum of the contribution values ​​of all nodes participating in the global aggregation in batch t, ​​con i，t represents the contribution value of node i in t batch training. Preferably, con i，t satisfy rank is the ranking of the benchmark scores of the nodes participating in the global aggregation in step S53 from high to low. satisfy C i，t is the training cost in step S43, is the sum of the training costs of the nodes participating in the global aggregation.

[0069] As mentioned above, in the present invention, the reputation mechanism mainly affects the selection of nodes, while the allocation of incentives is determined based on the contribution of nodes. Nodes that make high-quality contributions will have their reputation values ​​increased, and will also receive more incentives in this round of incentive allocation. This means that if a node has a higher reputation value, it will have made more high-quality contributions in past training rounds, and if a node has made more high-quality contributions, it will have received more incentives. These two designs ensure that high-quality nodes will receive more incentives, which in turn makes nodes willing to contribute better training results, and ultimately improves the training quality of federated learning for fine-tuning large language models.

[0070] according to Figure 1 The implementation method proposed in the present invention is further described in conjunction with specific embodiments.

[0071] In this embodiment, a computing platform with RTX 4090 (24GB) GPU and 12vCPU Intel Xeon Platinum 8352V CPU is used to construct the model federated learning experimental environment. Further preferably, the end batch of the training is configured to be 100, the initial reputation value of the node is 10, and the reputation value adjustment step length is h. repThe total number of nodes u in the federated learning network is 12, the number of nodes participating in each round of training k is 8, the rank r of low-rank adaptive fine-tuning is 32, the standard deviation judgment threshold K is 2, the node update rate v is 0.8, the minimum value L of the benchmark evaluation score is 1, and the maximum value H is 10. The selected pre-trained model is the llama2-7b-hf pre-trained model, the data set is split from the alpaca-gpt4 data set, and gpt-4-1106-preview is selected as the referee large model.

[0072] In order to prove the effectiveness of the incentive mechanism proposed in the present invention, this embodiment designs a set of malicious node attack experiments. The malicious nodes are designed to attack by poisoning, and the experiments are carried out when the number of malicious nodes is 4, 6, and 8 nodes, respectively. The training is carried out with and without the incentive mechanism designed by the present invention enabled. Each set of configured experiments is run 6 times, and the parameters of the completed training are benchmarked to evaluate the training results. The training results are averaged to reduce errors. The score is the comprehensive quality of the model, and the score value is between 1 and 10. The running results of this embodiment are shown in Figure 2. Figure 2 Place.

[0073] It should be noted here that when the incentive mechanism is not enabled, the central server will not evaluate the training results uploaded by the nodes, and the nodes do not have reputation value attributes. Before each round of training, the central node randomly selects k = 8 nodes from all nodes for local training. After the local training is completed, all training results from the nodes will participate in the global aggregation. The situation where the incentive mechanism is not enabled is the same as the current common scenario of large language model fine-tuning federated learning. The contribution of malicious nodes will be added to the global aggregation, which will eventually lead to a decline in the training quality of the entire large language model fine-tuning federated learning.

[0074] like Figure 2 As shown in the figure, the horizontal axis of the chart represents the number of attack nodes. The larger the value of the horizontal axis, the more malicious nodes there are. The vertical axis represents the score of the node. The value of the vertical axis represents the quality of model training. The larger the value of the vertical axis, the better the quality of model training.

[0075] according to Figure 2It can be concluded from the running results that the participation of malicious nodes in training has significantly affected the benchmark score of the large language model fine-tuning federated learning. The benchmark score has dropped significantly. As the number of malicious nodes increases, the score has a downward trend, which indicates that the quality of the trained model will decrease due to the participation of malicious nodes in training. After enabling the incentive mechanism, the scores of the training results are maintained at a higher level compared to the case where the incentive mechanism is not enabled. This embodiment shows that the use of the incentive mechanism proposed in the present invention can effectively maintain the training quality of the large language model fine-tuning with the participation of malicious nodes, and the incentive mechanism proposed in the present invention is effective.

[0076] The above is only a preferred embodiment of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A benchmark-based large language model fine-tuning federated learning incentive method, characterized in that: The method includes the following steps: Step 1: The central server of federated learning contacts the nodes to form a federated learning network. Initialize the training parameters of federated learning. The central server prepares a pre-trained model, and the nodes prepare fine-tuning datasets. Step 2: The central server pre-selects nodes according to the reputation value rule and distributes the training parameters or the pre-trained model to the selected nodes. Step 3: The nodes perform low-rank adaptation parameter initialization training based on the local training data. After completing the low-rank adaptation parameter initialization training, upload the initial parameter matrix to the central server, and the central server screens the nodes for the first time according to the parameters. Step 4: The nodes that pass the first screening in Step 3 perform local training for low-rank adaptation and record the training cost at the same time. After completing the local training, upload the training results and the training cost record to the central server. Step 5: The central server performs a second screening according to the parameters uploaded by the nodes. The second screening evaluates the node contributions through a benchmark test method and calculates the node contribution values at the same time. If a node passes the second screening, the node participates in the current round of global aggregation and increases its reputation value; conversely, if a node fails the second screening, the node does not participate in the current round of global aggregation and decreases its reputation value. Finally, according to the node contribution values and the training costs recorded in Step 4, distribute node incentives according to the principle of distribution according to work. Step 6: Loop Steps 2 to 5 until the preset end condition of the federated learning for large language model fine-tuning is reached.

2. The benchmark-based large language model fine-tuning federated learning incentive method according to claim 1, characterized in that: Step 2 is specifically as follows: Step 2-1: When the training batch t of federated learning is 1, the central server randomly selects k nodes from the set N = {1, 2,... u} of all nodes, and k ≤ u is satisfied; when the training batch t of federated learning is greater than 1, if there are nodes in the set of nodes that have never participated in federated learning training, then randomly select k × v nodes from them to enter the node pre-selection, where the node update rate v satisfies 0 < v < 1; when the number of nodes that have never participated in federated learning training is less than k × v, select the top k × (1 - v) nodes in descending order of reputation value, and finally randomly select from the remaining nodes until the number of pre-selected nodes is k. Step 2-2: When the training batch t of federated learning is 1, the central server will send the pre-trained model to the node and record the initial reputation value rep of the node numbered i. i is the preset initial value; when the training batch t of federated learning is not 1, the central server will send the previously globally aggregated low-rank adaptation global aggregation parameters to the node.

3. The method for fine-tuning federated learning based on benchmarking of a large language model according to claim 1, characterized in that: In Step 3, the low-rank adaptation parameter initialization training and the first screening method are specifically as follows: Step 3-1: The node receives the pre-trained matrix W0 and the global aggregation parameter B·A; where, W0 is a matrix with m rows and n columns, B is a matrix with m rows and r columns, and A is a matrix with r rows and m columns; m and n vary according to the different pre-trained matrices selected, and r is the rank for low-rank adaptation fine-tuning, which is configured at the beginning of federated learning, and r is less than the minimum of m and n. Step 3-2: Calculate W′ based on the pre-training matrix W0 and the global aggregation parameter B·A, W′=W0+B·A, and use W′ as the initial value of all parameters for this round of local training. Then fine-tune W′ and calculate the initial gradient W init ; For the initial gradient W init Perform singular value decomposition W init =U·S·V T , let A init is the first r columns of matrix U, B init is the [r+1:2r] row of the matrix V; Step 3-3: Node A init With B init Upload to the central server, and record the parameters uploaded by node i in batch t as (B init ·A init ) i，t , through the formula diff(A, B) = ||AB|| F Calculate the difference between matrices A and B, where ||·|| F represents the Frobenius norm; for each matrix (B init ·A init ) i，t , calculate the sum of its differences with other matrices in the batch D i =∑ j≠ i diff((B init ·A init ) i，t , (B init ·A init ) j，t ), and calculate the average of the differences Step 3-4: Calculate the standard deviation For each node in the batch, compare its difference D i Difference from average If satisfied If the node uploads malicious parameters in this batch, it is considered that the node has attacked. The central node will refuse to aggregate the parameters in this round of global aggregation and reduce the node's reputation value to rep. i -h rep ; where K is the threshold defined at the beginning of federated learning, h rep Represents the adjustment step size of the reputation value defined at the beginning of federated learning.

4. The method for fine-tuning federated learning based on benchmarking of a large language model according to claim 1, characterized in that: In Step 5, the second screening method includes the following steps: Step 5-1: The central server receives the training results uploaded by node i after completing local training in batch t using the method described in step 4 (B final ·A final ) i，t and the training cost C i，t ; Step 5-2: Based on the results of the t batch local training of node i (B final ·A final ) i，t , according to the preset questions in the node evaluation data set, generate the corresponding answers ans i，t ; Step 5-3: Answer ans i，t The referee model will score the node, with the lowest score set to L and the highest score set to H. When the referee model makes a judgment, it will conduct two rounds of evaluation on the node's answer, and finally get the node score s i，t ; Step 5-4: Benchmark and score all node parameters involved in training to get U t ={s i，t , |i is the node number participating in round t training}, and the parameters of the last round of global aggregation are benchmarked and scored to obtain {s last,t-1 }; Calculate the set U t ∪{s last,t-1 The standard deviation sd t and the average score avg t ; Step 5-5: Calculate the Z score of each node in batch t, If the node Z score is greater than 2, it is considered that the node has made a positive contribution in this round of training, and the node reputation value is adjusted upward to rep i +h rep , and accept the training results of the node in this round of global aggregation; if the Z score is less than -2, it is considered that the node may have attack behavior, and the reputation value of the node is lowered to rep i -h rep , and refuse to aggregate the training results of this node in this round of global aggregation.

5. The benchmark-based large language model fine-tuning federated learning incentive method according to claim 1, characterized in that: In Step 5, globally aggregate the training results of the nodes through the federated average algorithm FedAvg.

6. The method for fine-tuning federated learning based on benchmarking of a large language model according to claim 1, characterized in that: In Step 5, distribute the incentives for the nodes through the method of distribution according to work, specifically as follows: For nodes that have completed local training and are accepted into the global aggregation, calculate the distribution that node i should have received in batch t training in, represents the total incentive proposed by the central node in batch t, ​​S t represents the sum of the contribution values ​​of all nodes participating in the global aggregation in batch t, ​​con i，t represents the contribution value of node i in t batch training, con i，t satisfy Rank is the ranking of the benchmark scores of the nodes participating in the global aggregation in the second screening from high to low. satisfy C i，t represents the training cost of local training of node i in batch t that passes the first screening, is the sum of the training costs of the nodes participating in the global aggregation.

7. The method for fine-tuning federated learning based on benchmarking of a large language model according to claim 1, characterized in that: In Step 6, the preset end condition of the federated learning for large language model fine-tuning includes using the maximum number of iterations, and terminate the training when the number of training reaches the maximum number of iterations.

Images