Federal learning-based privacy protection method for high-efficiency fine-tuning large language model

Abstract

The invention relates to the technical field of large language model fine tuning, and discloses a federated learning-based privacy protection method for an efficient fine-tuning large language model, which comprises the following steps: a server side distributes a base large model to a participant, and then the participant performs fine-tuning training on the base large model and uploads an updated parameter or gradient to the server side; after the server receives the updated parameter or gradient, performing parameter updating or gradient updating by using a federal aggregation algorithm, and returning an aggregated result to the participant; identifying and processing sensitive information in the data of the participant through a named entity identification method; authenticating the identity of the participant by using an asymmetric encryption mode; an IA3 method is adopted, and a set number of learnable parameters are added on each layer of the pre-trained large model; in a data communication process, training parameters of a plurality of layers of the large model are integrated and then parameter transmission is carried out. According to the method, the privacy security of the data set participating in training can be further ensured, and the time for training the model is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model fine-tuning, and in particular to a privacy protection method for efficiently fine-tuning a large language model based on federated learning. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have become a major breakthrough in the field of natural language processing (NLP). These large language models, such as ChatGPT, not only perform well in understanding and generating human language, but also play an important role in many fields such as search and translation.

[0003] The main reason why large language models can cover many fields is that they use massive high-quality data sets. In certain fields, such as healthcare, finance, and law, high-quality data sets are very scarce. These data sets usually have high commercial value, and it is difficult to aggregate them for training large language models. Federated learning provides an effective solution to this problem. By jointly participating in the training of the model and sharing the results of the trained model, large language models in vertical fields can be obtained while protecting the data, which has attracted widespread attention from the academic community.

[0004] In the existing technology, most of the federated learning methods are used to fine-tune large language models, such as the efficient fine-tuning method Low-Rank Adaptation (LoRA). Compared with the full parameter fine-tuning model, it uses two low-rank matrices to replace the update of the original model parameters, which can not only reduce the amount of training parameters, but also indirectly accelerate the federated learning process, and can ensure that the trained model is almost the same as the full parameter training. This method can successfully solve the problem of data scarcity and difficulty in training large language models in vertical fields.

[0005] In terms of data privacy protection, the federated learning method can protect the privacy data of the participants to a certain extent. The principle is that the participants do not need to share local privacy data, but instead share the training parameters. In addition, most of the existing privacy protection methods are based on differential privacy and homomorphic encryption. For example, the differential privacy protection method adds a small amount of Gaussian noise during the parameter training process, which can effectively prevent reasoning attacks from other members. The method is effective in the following process: After aggregating the model parameters, the server returns the gradient to the participants. A member can calculate the gradient update of other members through the local gradient update and the returned gradient update, and then infer the privacy data of other members. If the differential privacy method is used and noise is added, then the above process cannot be used for reasoning and recovery. The homomorphic encryption algorithm uses encrypted data for gradient update, which more effectively ensures the security of the data of the participants.

[0006] Currently, federated learning has become the mainstream solution for privacy protection of large language models, but there are also two problems: the large number of training parameters leads to long training time, and the federated learning privacy protection method is not fully adapted to large language models.

[0007] The time consumed by jointly training a large language model is in the following two aspects: one is the time-consuming training of the large language model, and the other is the time-consuming communication between the server and the participants. The use of the efficient fine-tuning method LoRA can reduce the required number of parameters. It can be known that the number of parameters occupied by fine-tuning using the LoRA method is 0.1% to 1% of the original model parameters, but the number of parameters of the large language model is in the billions, so even for the LoRA method, the number of parameters required in this process is still very large. Compared with using two low-rank matrices to replace the update of the original parameters, using a learnable adaptation vector (such as IA3) to adjust the model's attention mechanism can greatly reduce the number of model parameters, which accounts for 0.01% of the original model parameters, and the number of parameters required for its training is 1% to 10% of the LoRA method.

[0008] As more efficient fine-tuning methods are used, the number of parameters in the training model is reduced, the number of parameters in its communication is also reduced, and the communication time is also improved.

[0009] In the process of fine-tuning a large language model by federated learning, if differential privacy protection methods are used and noise is added during parameter updates, it will cause the model to be difficult to converge and the performance of the training model to be low. The reason is that large language models are almost all based on the Transformer structure. By stacking encoder-decoder modules, the added noise will also spread as the depth of the model increases, resulting in more noise in the last few layers, and the last few layers have the greatest impact on the training results of the model. If the differential privacy method is used directly, there will be two results: one is that more noise is added and the training model effect is very poor; the other is that very little noise is added, and its privacy protection characteristics are obvious. Therefore, if differential privacy is used to further protect the privacy data of the participants, it becomes a very difficult problem to determine the amount of noise to add. If a homomorphic encryption scheme is used so that the training process is carried out in the form of ciphertext, then its computational cost will increase significantly, and its training time may be unbearable. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides a privacy protection method for efficiently fine-tuning a large language model based on federated learning.

[0011] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0012] A privacy protection method for efficiently fine-tuning a large language model based on federated learning, including: an overall training process, a data privacy security protection step, and a communication efficiency improvement step;

[0013] The overall training process includes: the server distributes the base large model to the participants, and then the participants fine-tune the base large model and upload the updated parameters or gradients to the server; when the server receives the updated parameters or gradients, it uses the federated aggregation algorithm to update the parameters or gradients, and returns the aggregated results to the participants; repeat the above process until the large model converges;

[0014] Data privacy and security protection steps include:

[0015] Data desensitization operations, identifying and processing sensitive information in the data of participants through named entity recognition methods;

[0016] Multi-party secure computing uses asymmetric encryption to authenticate the identities of participants and symmetric encryption algorithms to encrypt data communicated during training;

[0017] Steps to improve communication efficiency include:

[0018] An efficient fine-tuning method uses the IA3 method to add a set number of learnable parameters to each layer of the pre-trained large model to scale some weight values ​​and adjust the output of this layer of the large model, thereby reducing the model parameters of the large model training;

[0019] In the sparse communication method, during the data communication process, the training parameters of each layer in the large model are not transmitted, but the training parameters of several layers of the large model are integrated before the parameters are transmitted.

[0020] Furthermore, the sensitive information includes the user's real name, mobile phone number, ID number, and bank card number.

[0021] Compared with the prior art, the beneficial technical effects of the present invention are:

[0022] The invention has made a remedy for the privacy leakage problem in the federated learning process. The scheme uses identity authentication and key negotiation technology, and uses symmetric keys to encrypt communication information, which can prevent member reasoning attacks and reverse gradient reasoning attacks, and screen out sensitive information before participants participate in training, which can further ensure the privacy security of the training data set.

[0023] The invention improves the efficiency of fine-tuning large language models through federated learning, uses a more efficient fine-tuning method (IA3), further reduces the number of parameters required for fine-tuning the model, reduces the video memory of fine-tuning large models, and uses a sparse communication method to reduce the number of communications and save the time for training the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall framework diagram of the present invention;

[0025] Figure 2 It is a schematic diagram of the desensitization mechanism of the present invention;

[0026] Figure 3 A schematic diagram of identity authentication and key negotiation of the present invention;

[0027] Figure 4 Schematic diagram of efficient fine-tuning and sparse communication of the present invention. DETAILED DESCRIPTION

[0028] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0029] In order to aggregate isolated data and build large models in vertical fields, using federated learning methods to fine-tune large language models has become a new solution. Some of the problems it brings include two aspects: data privacy and security, and training efficiency. The solution mainly solves the above two problems. The invention content mainly includes two aspects: data security protection and communication efficiency improvement.

[0030] The general process of fine-tuning a large language model through federated learning is that the server first distributes the base model to the participants, and then the participants perform fine-tuning training locally and upload the parameters or updated gradients. When the server receives the updated parameters or gradients, it uses the federated aggregation algorithm to update the total gradient and returns the aggregated results to the server. After the client receives the updated parameters or gradients, it updates them locally. The above process will continue until the fine-tuned model converges, such as Figure 1 shown.

[0031] In terms of data privacy and security, the invention sets up two defense mechanisms for the risk of privacy leakage in the training process, which can effectively protect the privacy data of the participants. The first mechanism is data desensitization. In the process of federated learning, the data of the participants often contain sensitive information, such as the user's real name, mobile phone number, ID number, bank card number, etc. If this information is used directly for model training without processing, there will be a great risk of privacy leakage. These data can be strictly desensitized to remove or replace very sensitive data to ensure that the data can safely participate in model training. You can use named entity recognition methods to find the location containing private information, and use replacement, invalidation, shuffling and other operations to desensitize the data, such as Figure 2 shown.

[0032] The second mechanism is multi-party secure computing. In order to ensure the identity of the participants, asymmetric encryption can be used to authenticate the identity, and symmetric encryption algorithms can be used to encrypt the data communicated during the training process to further ensure the security of the data. Figure 3 shown.

[0033] In terms of training efficiency, the invention mainly optimizes the large number of parameters and data communication in the training process. Using the IA3 method, a small number of learnable parameters are added to each layer of the pre-trained large model. These parameters are used to scale some weight values ​​and then adjust the output of the layer. Compared with the full fine-tuning of the large language model, this method only requires one ten-thousandth of the original number of parameters to adapt to downstream tasks, and the performance is almost the same as the full parameter fine-tuning. Using this method, the model parameters of large model training can be greatly reduced, the video memory occupied by training is reduced, and the speed of efficient fine-tuning is increased. In the data communication process, sparse communication is adopted. Instead of transferring training parameters for each layer in the model, several layers can be integrated for parameter transfer, which can reduce the number of communications exponentially to reduce the time consumption of communication. Figure 4 shown.

[0034] In summary, the present invention uses a data desensitization system to screen out sensitive data of the participants, uses identity authentication and key negotiation methods to obtain symmetric keys, and uses them in the encryption process of communication messages between the server and the client, thereby protecting the privacy and security of communication data; uses a more efficient fine-tuning scheme (IA3) to reduce the number of parameters required for fine-tuning the large language model, increases the training rate, and uses a diluted communication method to reduce the number of communications and reduce communication time.

[0035] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0036] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A privacy protection method for efficiently fine-tuning a large language model based on federated learning, characterized in that: include: The overall training process, steps to protect data privacy and security, and steps to improve communication efficiency; The overall training process includes: the server distributes the base large model to the participants, and then the participants fine-tune the base large model and upload the updated parameters or gradients to the server; when the server receives the updated parameters or gradients, it uses the federated aggregation algorithm to update the parameters or gradients, and returns the aggregated results to the participants; repeat the above process until the large model converges; Data privacy and security protection steps include: Data desensitization operations, identifying and processing sensitive information in the data of participants through named entity recognition methods; Multi-party secure computing uses asymmetric encryption to authenticate the identities of participants and symmetric encryption algorithms to encrypt data communicated during training; Steps to improve communication efficiency include: An efficient fine-tuning method uses the IA3 method to add a set number of learnable parameters to each layer of the pre-trained large model to scale some weight values ​​and adjust the output of this layer of the large model, thereby reducing the model parameters of the large model training; In the sparse communication method, during the data communication process, the training parameters of each layer in the large model are not transmitted, but the training parameters of several layers of the large model are integrated before the parameters are transmitted.

2. According to claim 1, a privacy protection method for efficiently fine-tuning a large language model based on federated learning is characterized in that: The sensitive information includes the user’s real name, mobile phone number, ID number, and bank card number.

Images