Differential privacy-based heterogeneous federal fine tuning language model construction method and system

Abstract

The invention discloses a heterogeneous federal fine tuning language model construction method and system based on differential privacy, and the method comprises the steps: a parameter server firstly carries out the clustering of clients with similar capabilities, carries out the sensing clustering according to resources, obtains a larger-rank adaptive matrix for each class according to the class with the large computing power of the computing resources, and carries out the calculation of the resources; therefore, the flexible rank of the low-rank adaptive matrix is realized, and the fine tuning performance is enhanced. In each class, privacy federal fine tuning is adopted, including two parts of noise distribution with sensitivity perception and regular term optimization, so that better training efficiency and reduced noise influence are obtained. Between each class, heterogeneous low rank adaptive matrix aggregation is performed through a knowledge distillation technique, including knowledge aggregation and selection of high quality knowledge to enhance the local model. According to the method, strict privacy guarantee can be provided, the influence of differential privacy on fine tuning of the low-rank adaptive matrix can be relieved in a heterogeneous environment, and higher efficiency and accuracy are realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer network security technology and cryptography technology, and relates to a method and system for constructing an efficient heterogeneous federated fine-tuning language model, and is specifically designed to construct a heterogeneous federated fine-tuning method and system based on sensitivity-aware differential privacy noise allocation. Background Art

[0002] Pre-trained language models are first trained on a large number of public datasets, and then fine-tuned on private datasets in specific fields. They have shown great potential in many scenarios. However, due to the huge parameter scale, directly fine-tuning these pre-trained models will bring high computational overhead. LoRA low-rank adaptive technology, as a parameter-efficient fine-tuning method, can provide similar performance to fine-tuning all parameters while fine-tuning a small number of parameters. However, LoRA also needs to collect rich data for fine-tuning, which will undoubtedly bring privacy risks. Federated low-rank adaptive fine-tuning allows multiple clients to collaboratively fine-tune pre-trained language models on downstream tasks without exposing private data, but this paradigm has two inherent problems. On the one hand, due to the heterogeneity of client capabilities and data distribution, the model performance will be reduced, and the efficiency of model fine-tuning will be significantly reduced. On the other hand, due to the shared model gradient, sending model parameters or model gradients to the parameter server for aggregation will bring the risk of privacy leakage, which has been proven to be unsafe in many works. Although differential privacy provides a solution to the problem of privacy leakage by adding random noise to the transmitted parameters or model updates, theoretically ensuring that the adversary cannot perform reasoning attacks, this privacy protection technology will sacrifice model performance due to random noise, and it will be more significant under low-rank adaptive fine-tuning. Fine-tuning a small number of parameters is very sensitive to changes in the model, which will amplify the negative impact of differential privacy noise on accuracy.

[0003] There are still several challenges in the current federated fine-tuning of differential privacy protection. The values ​​of low-rank adaptive parameters are different, which means they have different sensitivities to the added noise. The existing differential privacy scheme adds a uniform noise level to all parameters. Secondly, the local model is trained on a non-IID dataset, and low-rank adaptive matrices of different levels also perform differently on non-IID datasets. Low-rank models are more sensitive due to their lower generalization ability. In addition, the existing federated low-rank adaptive fine-tuning directly combines parameter-efficient methods with federated learning, while ignoring the limited and heterogeneous capabilities of clients in practice. Existing methods achieve the aggregation of heterogeneous low-rank adaptive matrices through singular value decomposition, etc., but there is still room for improvement in performance. Therefore, how to design a flexible and efficient differential privacy protection scheme under the federated fine-tuning of heterogeneous low-rank adaptive matrices and allocate adaptive matrices of the rank that adapts to resource capabilities to clients is a challenge that needs to be solved urgently. Summary of the invention

[0004] In order to solve the technical problems of low-rank adaptive matrix noise sensitivity and client heterogeneity, the present invention provides a sensitivity-aware differential privacy to realize a flexible and efficient heterogeneous federated fine-tuning system construction method and system.

[0005] The technical solution adopted by the method of the present invention is: a method for constructing a heterogeneous federated fine-tuning language model based on differential privacy; the heterogeneous federated fine-tuning language model, whose client local model is obtained by training; the training process specifically includes the following steps:

[0006] Step 1: The parameter server clusters all participating N clients into C categories according to the resource capabilities of each client, and assigns a low-rank adaptive matrix LoRA of the corresponding level (the level corresponds to the size of the rank) to each category;

[0007] Step 2: At the beginning of each round of training of the client local model, the parameter server selects a set of participating clients P for each class c c and a clustering client p c , each selected client trains and updates the local model multiple times; the selected client participates in this round of training, performs low-rank adaptive matrix LoRA fine-tuning on the local model on its own private dataset, and only updates the local trainable parameters w i , freeze the pre-training weight W0;

[0008] Step 3: After the selected client completes the local model update, it performs privacy fine-tuning, adds parameter-sensitive differential privacy noise and regularization term optimization to the local model update, and sends the privacy-preserving privacy model update to the clustering client PC;

[0009] Step 4: Perform homogeneous aggregation within the class; for each class c, the clustering client pc of each class receives the information of other clients i∈P in the same class. c Privacy model update in Include only the LoRA trainable parameters, exclude the parameters of the frozen part, and perform the federated average aggregation of FedAvg to obtain the clustering model w c =[A c , B c ]; among them, A c , B c are respectively the aggregated w c The corresponding two low-rank matrices (LoRA is a combination of two low-rank matrices A and B);

[0010] Step 5: Perform heterogeneous aggregation between classes; the clustering client of each class is in the public dataset Computational Knowledge c , and sends the knowledge to the parameter server for subsequent heterogeneous LoRA aggregation;

[0011] Step 6: For the received knowledge of each class, the parameter server first calculates the aggregation weight ω based on the correlation between the knowledge c , and then perform weight aggregation to obtain global knowledge f;

[0012] Step 7: The parameter server selects the knowledge f whose confidence of each public dataset sample is greater than the threshold φ from the aggregated knowledge s , and sent to each clustering client;

[0013] Step 8: Each cluster client p c , c≤C for the received global selected knowledge f s , perform the knowledge distillation process on the public dataset and update the clustering model w c , and sent to each client P in the class c ;

[0014] Step 9: Each client in each cluster receives the cluster client p c The clustering model w issued c Parameters, update your local LoRA module w i And the local model W i =W0+B c A c .

[0015] Preferably, in step 1, the parameter server first performs resource-aware clustering on all clients and defines the resource feature vector χ i =[χ i，1 , χ i，2 , χi，3 , χ i，4 ], where χ i，1 represents the data volume feature, χ i，2 represents the computing capability characteristics, χ i3 represents the storage capacity characteristic, χ i，4 Represents the communication capability feature; for features of different dimensions, first normalize them using Min-Max, then randomly select C cluster centers, and calculate the fuzzy membership between each client i and the cluster center c represents the probability that the client belongs to cluster c, d(χ i , v c ) represents the distance between client i and cluster center v c distance, m is a fuzzy factor greater than 1; then client i is assigned to the cluster with the highest membership; then the cluster center c is updated for each class, and the above process is repeated until the cluster center no longer changes.

[0016] Preferably, in step 3, after the local fine-tuning is completed, the client updates the local model by Δw i Each element of adds sensitivity-aware differential privacy, and each value adds a noise size related to its own sensitivity.

[0017] Preferably, the sensitivity-aware differential privacy is achieved by first estimating the corresponding clipping threshold s of the m-th local model update of each client i i，m , and then estimate the sensitivity of the model update And calculate the noise level corresponding to each model update where M is the number of parameter updates, σ * is the overall noise level.

[0018] Preferably, in step 6, the parameter server calculates the relevance of the knowledge sent by each clustering client, where the Pearson correlation coefficient is used as a weight measurement indicator, and the average knowledge of all clusters is used as a reference. The greater the correlation, the greater the aggregation weight assigned.

[0019] Preferably, the Pearson correlation coefficient in, From public datasets Sample Computed knowledge, is the average clustering knowledge of all samples, For the sample The average knowledge of all C clusters; the average knowledge of all classes And the correlation coefficient ρ cThrough normalization as the weight of knowledge aggregation; then, the parameter server normalizes the correlation coefficient. Calculate the weight ω c After that, perform weight aggregation on the knowledge

[0020] Preferably, in step 7, the parameter server calculates the confidence of each knowledge after aggregation in Representation sample The corresponding knowledge Represents the probability distribution after probability normalization; when the confidence φi>φ, φ is the confidence threshold.

[0021] As a preferred embodiment, in step 8, each clustering client pc receives the selected global knowledge f fed back by the parameter server s , perform the knowledge distillation process, and the loss function is in Yes s Corresponding to the selected data set, ξ is the adjustment factor, τ is the distillation temperature, and σ is the activation function; Represents a sample, represents the sample, W c are the model parameters of cluster c, is the cross entropy loss function, is the global knowledge of server selection, is the cluster c in the sample Knowledge on D KL () represents the KL divergence loss function.

[0022] The technical solution adopted by the system of the present invention is: a heterogeneous federated fine-tuning language model construction system based on differential privacy, comprising:

[0023] one or more processors;

[0024] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the heterogeneous federated fine-tuning language model construction method based on differential privacy.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] (1) The present invention provides an efficient differentially private heterogeneous federated fine-tuning language model (DP-FedLoRA). The prior art does not fully consider the differences in client resource capabilities, ignores the differences in noise sensitivity of low-rank adaptive matrices and the impact of non-independent and identically distributed data on the performance of federated fine-tuning. The present invention integrates resource-aware client clustering, intra-class privacy federated fine-tuning, and aggregation of heterogeneous low-rank adaptive matrices between classes for the first time, solving the challenges caused by noise and heterogeneity in the actual use of federated devices, and achieving both privacy protection and efficiency.

[0027] (2) The present invention proposes an efficient, resource-aware clustering method. The parameter server clusters clients according to their different resource levels and assigns clients with similar resource levels to the same class. The prior art usually clusters based on data similarity, but it is difficult to ensure data privacy while evaluating data similarity. Moreover, data similarity clustering only considers the data as a resource feature, and ignores important resource features such as computing power, communication bandwidth, and storage capacity in the federated fine-tuning process. The present invention comprehensively considers different resource attributes and combines them into resource vectors. During clustering, fuzzy membership is used to approximate the probability that the client belongs to the current class, and the client is assigned to the class with the highest probability. Such a clustering method is also much more efficient than the usual calculation of cosine similarity.

[0028] (3) The intra-class regularization optimization proposed in the present invention is used to limit the update direction and step size of the local model so as not to deviate too much from the model of the current class, so as to alleviate the aggregation bias caused by non-independent and identically distributed data, and the restriction of the regularization term is combined when the local model calculates the loss function. At the same time, the regularization term of the present invention is based on the rank size of each class. For smaller ranks, the generalization ability of the low-rank adaptive matrix is ​​poor, and more restrictions on the regularization term are required. For larger ranks, the restrictions on the regularization term can be reduced, and the model is less affected by non-independent and identically distributed data. The problem of performance degradation under non-independent and identically distributed conditions is alleviated, and rank-related regularization term restrictions are added for different ranks to give full play to the generalization ability of higher-rank models.

[0029] (4) The present invention adds sensitivity-aware noise to each model update, estimates the local model update through past gradient updates, and uses it as an estimate of the clipping threshold for the model update, and calculates the noise level corresponding to each model update. The adaptive noise is added at a scalar granularity. The present invention adds sensitivity-aware noise to each element of the model update. Model updates with less sensitivity do not require excessive noise protection, while model updates with greater sensitivity add greater noise to protect privacy. The noise distribution scheme of the present invention can greatly alleviate the degradation of model performance caused by noise under the same privacy protection strength, especially for low-rank adaptive matrices.

[0030] (5) The present invention provides a knowledge distillation method for the aggregation of heterogeneous low-rank adaptive matrices. Existing technologies usually solve the problem of heterogeneous aggregation by cutting or filling, but cannot give full play to the performance of heterogeneous aggregation. The present invention transmits knowledge of different classes between classes, and the parameter server performs weighted aggregation on the knowledge, calculates the Pearson correlation coefficient between the knowledge, and uses it as the aggregation weight of the knowledge. The knowledge with higher correlation has higher consistency and is given a higher aggregation weight. The aggregated knowledge is selected through a knowledge selection scheme to select some high-quality knowledge, that is, knowledge with greater confidence. The global client has high confidence in this type of knowledge and can predict accurately, and the selected knowledge is fed back to each class for the knowledge distillation process. In addition, knowledge distillation, as a smoothing technology, can also better handle the problem of non-independent and identically distributed data, and realizes efficient knowledge migration between clients, clustering clients, and parameter servers with different model structures between classes, and provides higher heterogeneous aggregation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The technical solution of this invention is further described below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of describing the technical solution. For those skilled in the art, other drawings and the intention of the invention can also be obtained based on these drawings without paying creative labor.

[0032] Figure 1 This is a principle diagram of heterogeneous federated fine-tuning language model training for differential privacy according to an embodiment of the present invention;

[0033] Figure 2 A flow chart of resource-aware clustering according to an embodiment of the present invention;

[0034] Figure 3 This is a flow chart of heterogeneous aggregation within a class and knowledge migration between classes according to an embodiment of the present invention;

[0035] Figure 4This is a flow chart of sensitivity-aware differential privacy noise allocation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0037] Federated fine-tuning of language models enables multiple clients to collaboratively fine-tune a pre-trained language model without sharing local data. To avoid privacy leakage during the aggregation process, differential privacy is widely used in federated learning. Local model updates are pruned and noise is added, but uniform random noise can lead to significant performance loss. At the same time, in order to give full play to the different resource capabilities of the clients, heterogeneous low-rank adaptive modules are used for heterogeneous clients in federated learning. However, how to efficiently aggregate heterogeneous fine-tunable modules while ensuring model performance is a major challenge.

[0038] This embodiment proposes an efficient differential privacy heterogeneous federated fine-tuning language model, such as Figure 1 As shown, compatible resource-aware client clustering, rank-related regularization optimization, sensitivity-aware noise distribution, and inter-class weight knowledge distillation and knowledge selection are used to solve performance degradation and heterogeneity problems. The technologies involved in this embodiment include resource-aware client clustering, regularization optimization, sensitivity-aware differential privacy technology, weighted knowledge aggregation of correlation, and knowledge selection mechanism of confidence.

[0039] First, the formal definition of the t-th round symbol of this system is given:

[0040] Knowledge distillation loss function

[0041] Client set {i|i=1, 2, 3, ..., N};

[0042] Client Parameters

[0043] Client-side gradient updates

[0044] Client-side clipped gradient updates

[0045] Client-side gradient update after adding noise

[0046] Clustering Collection

[0047] Aggregate weight for each class

[0048] The number of clusters C;

[0049] Each class c is selected to the client

[0050] Each clustering client c is selected

[0051] The knowledge sent by each class c

[0052] The knowledge received by the parameter server

[0053] The knowledge f aggregated by the parameter server t ;

[0054] Knowledge after parameter server selection

[0055] The probability κ of client selection;

[0056] Confidence threshold φ.

[0057] Please see Figure 1 This embodiment provides a method for constructing a heterogeneous federated fine-tuning language model based on differential privacy. The heterogeneous federated fine-tuning language model (DP-FedLoRA) with differential privacy is obtained by training a client local model. The training process specifically includes the following steps:

[0058] Step 1: The parameter server clusters all participating clients according to the resource capabilities of each client and assigns a low-rank adaptive matrix module of corresponding rank to each cluster;

[0059] Step 2: At the beginning of each round of training of the client local model, the server selects a set of participating clients and corresponding clustering clients for each class, and each selected client trains and updates the local model multiple times;

[0060] Step 3: After the selected client completes the local model update, it performs privacy fine-tuning, adds sensitivity-aware noise and regularization optimization to the model update, and sends the privacy-preserving model update to the clustering client;

[0061] Step 4: Perform homogeneous aggregation within the class. The clustering client of each class receives the privacy model update of other clients in the same class, including only the trainable parameters of the low-rank adaptive matrix, excluding the parameters of the frozen part, and performs federated average aggregation to obtain the clustering model.

[0062] Step 5: Perform heterogeneous aggregation between classes. The clustering client of each class calculates on the public dataset and sends the knowledge to the parameter server.

[0063] Step 6: The parameter server uses the correlation between the knowledge received from each class as the aggregation weight, performs weight aggregation, and obtains global knowledge;

[0064] Step 7: The parameter server selects the knowledge whose confidence of each public dataset sample is greater than the threshold from the aggregated knowledge and sends it to each clustering client;

[0065] Step 8: Each clustering client performs a knowledge distillation process on the public dataset for the globally selected knowledge received, updates the clustering model, and sends it to each client in the class.

[0066] Step 9: Each client in each class receives the model parameters sent by the clustering client pc and updates its local low-rank adaptive module and local model.

[0067] Step 10: When the preset conditions are met, the training ends and the trained client local model is obtained and applied to the heterogeneous federated privacy fine-tuning system.

[0068] In this embodiment, the specific implementation process of the DP-FedLoRA construction method is as follows:

[0069] Server

[0070] 1: Cluster all clients into C categories based on resource feature vectors.

[0071] 2: Assign adaptive matrix initialization of different sizes to each class according to the resource score of each class;

[0072] 3: Traverse each round t∈{1,…,T} and do the following:

[0073] 4: For each class To traverse, do the following:

[0074] 5: Sampling clients with probability κ And clustering clients And distribute it to the clients of the corresponding class;

[0075] 6: Receive from cluster client Knowledge

[0076] 7: Perform weighted knowledge aggregation to obtain global knowledge

[0077] 8: Execute knowledge selection and obtain the selected knowledge And send it to all cluster clients;

[0078] 9: End the loop;

[0079] Client

[0080] 10: Traverse each round t∈{1,…,T} and do the following:

[0081] 11: Receive the client selected in this round And clustering clients

[0082] 12: If the current client is selected

[0083] 13: Execute the privacy fine-tuning of Algorithm 2 to get the model update

[0084] 14: If the current client is not a cluster client

[0085] 15: Send privacy-preserving model updates To cluster clients

[0086] 16: If the current client is a cluster client

[0087] 17: Perform federated average aggregation on the received model updates to obtain

[0088] 18: Update clustering model And calculate and send model knowledge To the parameter server;

[0089] 19: Receive globally aggregated knowledge

[0090] 20: Perform knowledge distillation and update the model And distribute it to clients within the class;

[0091] 21: Other clients receive the global model sent by the clustering client

[0092] 22: Update local model

[0093] 23: End the loop;

[0094] Another specific implementation is provided here. In order to make full use of the heterogeneous resource capabilities of the client, clustering is usually used as an effective way to enhance resource utilization and is combined with heterogeneous aggregation between classes to improve the efficiency of training. The present invention designs a resource-aware clustering method such as Figure 2 As shown, clustering is based on available resources rather than relevant attributes of privacy data. A resource feature vector is used to integrate features such as data volume, computing power, storage capacity, and communication capacity. Clients are classified by fuzzy membership, and clients with similar resource capability levels are classified into the same class. Uncertainty may arise in the clustering process. When the difference between several highest memberships of a client is less than a threshold value, cluster overlap will occur. In order to avoid this uncertainty, the present invention adopts a load balancing strategy, that is, assigning clients to classes with fewer clients. Then, the resource score of each category is calculated. The class with a higher score represents a class with higher resource capability, and a larger rank adaptive matrix is ​​allocated accordingly.

[0095] The specific implementation process is as follows: the parameter server first performs resource-aware clustering on all clients and defines the resource feature vector χ i =[χ i，1 , χ i，2 , χ i，3 , χ i，4 ], where χ i，1 represents the data volume feature, usually the size of the privacy dataset used for fine-tuning, χ i，2 Represents the computing capability characteristics, usually the number of GPU CUDA cores, χ i，3 Represents the storage capacity characteristics, usually the GPU memory size (GB), i，4 Represents the communication capability feature, usually bandwidth (Mbps); for features of different dimensions, first normalize them through Min-Max, then randomly select C cluster centers, and calculate the fuzzy membership of each client i and cluster center c This reflects the probability that the client belongs to cluster c, d(χ i , v c ) represents the distance between client i and cluster center v c distance, m represents the fuzzy factor; then client i is assigned to the cluster with the highest membership; then the cluster center c is updated for each class, and the above process is repeated until the cluster center no longer changes.

[0096] In addition, knowledge distillation is a commonly used method for heterogeneous aggregation. Directly performing fully heterogeneous knowledge distillation, that is, each client is heterogeneous from each other, will lead to poor model performance due to the low level of knowledge. Figure 3As shown, the approach taken is intra-class aggregation and then heterogeneous aggregation for knowledge transfer between classes. The clustering model is built by aggregating the model updates of each class, which enhances the quality of knowledge to a certain extent. Then, knowledge is transferred between classes. The parameter server uses weight aggregation and knowledge selection to enhance the model performance of the local client.

[0097] Another specific implementation is provided here. When aggregating, the parameter server calculates different weights according to the relevance of the knowledge of each class. The weight is determined by the Pearson correlation coefficient between the knowledge. The higher the correlation, the stronger the consistency of the knowledge. The correlation is calculated as in, From public datasets Sample Computed knowledge, Average knowledge of all classes And the correlation coefficient ρ c Normalization is used as the weight of knowledge aggregation. After that, the parameter server normalizes the correlation. After calculating the weights, perform weight aggregation on the knowledge

[0098] Another specific implementation is provided here. By aggregating the weights of knowledge, the aggregated knowledge can be obtained on the basis of ensuring consistency. However, the aggregated knowledge may also contain some low-quality content, which will mislead the client's training and lead to poor knowledge distillation effect. The present invention proposes a knowledge selection scheme based on confidence, which selects some knowledge with higher confidence for client training instead of sending all knowledge. The calculation of confidence is in When the confidence φi>φ, the current knowledge i is selected, where φ is the confidence threshold. The parameter server integrates the selected knowledge into fs and sends it to the client for the client's knowledge distillation process.

[0099] In this embodiment, the specific implementation process of the privacy fine-tuning algorithm of client i is:

[0100] Client i

[0101] 1:

[0102] 2: For the local round number k = 1, ..., K, traverse and do the following:

[0103] 3: Sampling data sample b t，k ∈D i ;

[0104] 4: Regularization term

[0105] 5: Regularization term optimizes loss function

[0106] 6: Calculate the gradient

[0107] 7: If the current local round number k = 1, do the following:

[0108] 8:

[0109] 9: Otherwise, do the following:

[0110] 10:

[0111] 11: Update the model

[0112] 12: Local model update

[0113] 13: Calculate the clipping threshold

[0114] 14: For each locally updated element m=1,...,M, do the following:

[0115] 15:

[0116] 16: Cutting model update

[0117] 17: Adding noise

[0118] 18: End the loop;

[0119] 19: Return privacy-preserving model updates

[0120] Differential privacy is a widely used privacy protection technology that can theoretically provide strong privacy protection capabilities and resist inference attacks. However, adding noise to the low-rank adaptive matrix will cause the model performance to degrade more seriously due to the higher sensitivity between parameters. The fundamental reason is that it ignores the difference in sensitivity of different parameters. Parameters with lower sensitivity require less noise to protect privacy, and too much noise will lead to deviation in the model update direction and performance loss. This paper proposes a sensitivity-aware differential privacy noise allocation scheme, such as Figure 4 As shown, less noise is allocated to parameters with lower sensitivity, and more noise is allocated to parameters with higher sensitivity to protect privacy. Overall, (∈, δ)-differential privacy is still satisfied.

[0121] Another specific implementation is provided herein, where the noise of sensitivity perception is determined by first estimating the m-th model update corresponding to the clipping threshold s for each client i. i，m , and then estimate the sensitivity of the model update And calculate the noise level corresponding to each model update where M is the number of parameter updates, σ * is the overall noise level. Overall, for each client, Without destroying the strength of privacy protection, the noise is redistributed, and the sensitivity of different parameters is combined at the scalar granularity to add adaptive noise. This protects privacy while taking into account model performance, especially in the low-rank adaptive matrix scenario.

[0122] This embodiment also provides an efficient heterogeneous federated fine-tuning language model construction device based on differential privacy, including:

[0123] one or more processors;

[0124] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the efficient heterogeneous federated fine-tuning language model construction method based on differential privacy.

[0125] The resource-aware clustering, sensitivity-aware differential privacy, and heterogeneous weighted knowledge aggregation and selection proposed in the present invention can also be extended to other heterogeneous algorithms or differential privacy protection algorithms, and have good scalability.

[0126] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A method for constructing a heterogeneous federated fine-tuning language model based on differential privacy, characterized by: The heterogeneous federated fine-tuning language model, whose client local model is obtained through training; the training process specifically includes the following steps: Step 1: The parameter server clusters all participating N clients into C categories according to the resource capabilities of each client, and assigns a low-rank adaptive matrix LoRA of the corresponding level to each category; Step 2: At the beginning of each round of training of the client local model, the parameter server selects a set of participating clients P for each class c c and a clustering client p c , each selected client trains and updates the local model multiple times; the selected client participates in this round of training, performs low-rank adaptive matrix LoRA fine-tuning on the local model on its own private dataset, and only updates the local trainable parameters w i , freeze the pre-training weight W0; Step 3: After the selected client completes the local model update, it performs privacy fine-tuning, adds parameter-sensitive differential privacy noise and regularization term optimization to the local model update, and sends the privacy-preserving privacy model update to the clustering client p c ; Step 4: Perform homogeneous aggregation within the class; for each class c, the clustering client p of each class c Received from other clients i∈P in the same class c Privacy model update in Include only the LoRA trainable parameters, exclude the parameters of the frozen part, and perform the federated average aggregation of FedAvg to obtain the clustering model w c =[A c , B c ]; among them, A c , B c are respectively the aggregated w c The corresponding two low-rank matrices; Step 5: Perform heterogeneous aggregation between classes; the clustering client of each class is in the public dataset Computational Knowledge c , and sends the knowledge to the parameter server for subsequent heterogeneous LoRA aggregation; Step 6: For the received knowledge of each class, the parameter server first calculates the aggregation weight ω based on the correlation between the knowledge c , and then perform weight aggregation to obtain global knowledge F; Step 7: The parameter server selects the knowledge f whose confidence of each public dataset sample is greater than the threshold φ from the aggregated knowledge s , and sent to each clustering client; Step 8: Each cluster client p c , c≤C for the received global selected knowledge f s , perform the knowledge distillation process on the public dataset and update the clustering model w c , and sent to each client P in the class c ; Step 9: Each client in each cluster receives the cluster client p c The clustering model w issued c Parameters, update your local LoRA module w i And the local model W i =W o +B c A c .

2. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to claim 1, characterized in that: In step 1, the parameter server first performs resource-aware clustering on all clients and defines the resource feature vector χ i =[χ i，1 , χ i，2 , χ i，3 , χ i，4 ], where χ i，1 represents the data volume feature, χ i，2 represents the computing capability characteristics, χ i，3 represents the storage capacity characteristic, χ i，4 represents the communication capability characteristics; For features of different dimensions, we first normalize them using Min-Max, then randomly select C cluster centers, and calculate the fuzzy membership between each client i and the cluster center c. represents the probability that the client belongs to cluster c, d(χ i , v c ) represents the distance between client i and cluster center v c The distance, m is a fuzzy factor greater than 1; Then client i is assigned to the cluster with the highest membership; then the cluster center c is updated for each class, and the above process is repeated until the cluster center no longer changes.

3. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to claim 1, characterized in that: In step 3, after the local fine-tuning is completed, the client updates the local model by Δw i Each element of adds sensitivity-aware differential privacy, and each value adds a noise size related to its own sensitivity.

4. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to claim 3 is characterized in that: The sensitivity-aware differential privacy is achieved by first estimating the corresponding clipping threshold s of the m-th local model update of each client i i，m , and then estimate the sensitivity of the model update And calculate the noise level corresponding to each model update where M is the number of parameter updates, σ * is the overall noise level.

5. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to claim 1, characterized in that: In step 6, the parameter server calculates the relevance of the knowledge sent by each cluster client. Here, the Pearson correlation coefficient is used as the weight measurement indicator, and the average knowledge of all clusters is used as a reference. The greater the correlation, the greater the aggregation weight assigned.

6. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to claim 5, characterized in that: The Pearson correlation coefficient in, From public datasets Sample Computed knowledge, is the average clustering knowledge of all samples, For the sample The average knowledge of all C clusters; the average knowledge of all classes And the correlation coefficient ρ c Through normalization as the weight of knowledge aggregation; then, the parameter server normalizes the correlation coefficient. Calculate the weight ω c After that, perform weight aggregation on the knowledge 7. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to claim 1, characterized in that: In step 7, the parameter server calculates the confidence of each knowledge after aggregation in Representation sample The corresponding knowledge Represents the probability distribution after probability normalization; when the confidence φi>φ, φ is the confidence threshold.

8. The method for constructing a heterogeneous federated fine-tuning language model based on differential privacy according to any one of claims 1 to 7, characterized in that: In step 8, each clustering client pc receives the selected global knowledge f fed back by the parameter server s , perform the knowledge distillation process, and the loss function is in Yes s Corresponding to the selected data set, ξ is the adjustment factor, τ is the distillation temperature, and σ is the activation function; Represents a sample, represents the sample, W c are the model parameters of cluster c, is the cross entropy loss function, is the global knowledge of server selection, is the cluster c in the sample Knowledge on D KL () represents the KL divergence loss function.

9. A heterogeneous federated fine-tuning language model construction system based on differential privacy, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for constructing a heterogeneous federated fine-tuning language model based on differential privacy as described in any one of claims 1 to 8.

Images