Federal learning data privacy protection method and system based on correlation pruning

Abstract

The invention discloses a federal learning data privacy protection method and system based on correlation pruning, and belongs to the technical field of information security. According to the invention, by evaluating the correlation between neurons and performing targeted pruning on the model, the complexity of the model is reduced, the communication overhead is reduced, and the amount of noise required to be added subsequently is also reduced, so that the overall model performance is improved. Meanwhile, data privacy protection is achieved through differential privacy, by calculating the correlation LRP of each neural layer and output, reasonably distributing privacy budget, distributing more budget to an important layer and distributing less budget to a secondary important layer, on the premise that the overall privacy budget is ensured, the accuracy performance of the model is improved, unnecessary privacy budget waste is avoided, and the method is suitable for being popularized and applied. And the limited budget is better utilized. According to the method, the problems of model complexity and differential privacy protection in federated learning can be effectively solved, and good balance is achieved between model efficiency and privacy protection.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and more specifically to a method and system for protecting privacy of federated learning data based on correlation pruning. Background Art

[0002] In today's digital age, data has become an extremely important asset and is widely used in various fields, including medical, financial, and scientific research. Data contains a large amount of sensitive information about individuals, enterprises, and society, and its protection is crucial. Once data is leaked or abused, it may lead to serious consequences such as exposure of personal privacy and damage to corporate trade secrets. Therefore, ensuring the security, integrity, and confidentiality of data is a key task in data management and application.

[0003] As an innovative distributed machine learning method, federated learning has unique advantages in processing data. It allows clients to independently train models using their own data while maintaining data locality, thereby ensuring data privacy to a certain extent. However, federated learning still faces many challenges in its development. Although it attempts to solve the problem of data privacy, malicious third parties may still be able to infer sensitive information by analyzing shared gradients and parameters, which poses a potential threat to user privacy. In terms of privacy protection technology, verifiable privacy-preserving cryptographic technology (homomorphic encryption and secure multi-party computation) can provide privacy protection, but due to its high computational cost, it is difficult to be widely used in federated learning.

[0004] In the privacy protection of federated learning, differential privacy is a commonly used method. The current privacy budget allocation strategy is not flexible enough and difficult to adapt to the needs of different scenarios. At the same time, the application of model pruning technology in federated learning also has many problems, such as privacy issues are not fully considered, pruning strategies have poor adaptability, and it is difficult to balance communication efficiency and model performance. All of the above problems will lead to a weakening of data protection effects.

[0005] Therefore, designing a data protection method and system that can effectively balance privacy protection and model accuracy is a technical problem that needs to be urgently solved in this field. Summary of the invention

[0006] In view of this, the present invention provides a method and system for protecting the privacy of federated learning data based on correlation pruning, which reduces the complexity of the model through model pruning, and uses differential privacy to ensure the privacy of federated learning while improving the model accuracy.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] On the one hand, the present invention proposes a method for protecting privacy of federated learning data based on correlation pruning, comprising the following steps:

[0009] Obtaining a federated learning model, wherein the federated learning model includes clients distributed in various locations and a central server;

[0010] The client obtains an initial model from the central server, and performs model pruning based on correlation on the initial model to obtain a pruned model;

[0011] The pruned model is trained using client privacy data, differential privacy is used to add noise to parameters during the training process to achieve privacy protection, a privacy budget is allocated using LPR, and noise is added according to the privacy budget; the federated learning model is optimized using the pruned model after adding noise;

[0012] The client's real-time privacy data is input into the optimized federated learning model, and noise is added to the real-time privacy data to protect the real-time privacy data.

[0013] Preferably, the initial model is subjected to correlation-based model pruning to obtain a pruned model, comprising:

[0014] The weights are used to calculate the correlation between each neuron in the initial model and the weights of other neurons in the same layer, and the importance of each neuron is obtained based on the correlation;

[0015] In each layer, neurons are sorted according to the importance to obtain a sorted sequence;

[0016] The initial model is pruned according to the pruning ratio q on the sorted sequence to obtain a pruned model.

[0017] Preferably, the correlation between each neuron and the weights of all other neurons in the same layer is normalized, and the average of the normalized correlation data is calculated as the importance of the corresponding neuron.

[0018] Preferably, the pruning model is trained using client privacy data, differential privacy is used to add noise to parameters during training to achieve privacy protection, a privacy budget is allocated using LPR, and noise is added according to the privacy budget; and the pruning model after adding noise is used to optimize the federated learning model, including:

[0019] Calculate the gradient of each batch of client private data;

[0020] Performing gradient descent to update the parameters of the pruning model, performing parameter pruning on the updated parameters, and adding noise to the pruned parameters;

[0021] The pruned model with added noise performs parameter aggregation on the central server, and the aggregated model is sent back to the client until the federated learning model converges.

[0022] Preferably, using LPR to allocate a privacy budget, and adding noise according to the privacy budget, includes:

[0023] Use LRP to calculate the layer importance of neurons in the pruned model;

[0024] Allocate the total privacy budget according to the layer importance to obtain the privacy budget of each layer of neurons;

[0025] Add noise of corresponding size according to the privacy budget of each layer of neurons.

[0026] Preferably, using LRP to calculate the layer importance of neurons in the pruning model includes:

[0027] LRP is used to calculate the correlation of neurons in each layer, and the correlation of each neuron in each layer is averaged to obtain the layer importance of each layer.

[0028] On the other hand, the present invention also proposes a federated learning data privacy protection system based on correlation pruning, which is used to implement the above-mentioned federated learning data privacy protection method based on correlation pruning. The system includes:

[0029] Data acquisition module, to obtain client privacy data;

[0030] A model processing module, in which the client obtains an initial model based on a central server, and performs model pruning based on correlation on the initial model to obtain a pruned model;

[0031] A model training module, which trains the pruned model using client privacy data, uses differential privacy to add noise to parameters during training to achieve privacy protection, allocates a privacy budget using LPR, and adds noise according to the privacy budget; and optimizes the federated learning model using the pruned model after adding noise;

[0032] The data privacy protection module inputs the client's real-time privacy data into the optimized federated learning model, adds noise to the real-time privacy data, and completes the protection of the real-time privacy data.

[0033] Preferably, the model processing module includes:

[0034] An importance calculation unit, which uses the weights to calculate the correlation between each neuron in the initial model and the weights of other neurons in the same layer, and obtains the importance of each neuron based on the correlation;

[0035] A sorting unit, which sorts neurons in each layer according to the importance to obtain a sorted sequence;

[0036] The pruning model acquisition unit prunes the initial model according to the pruning ratio q on the sorted sequence to obtain a pruning model.

[0037] Preferably, the model training module includes:

[0038] Gradient calculation unit, which calculates the gradient of each batch of client privacy data;

[0039] A parameter clipping unit, performing gradient descent to update the parameters of the pruning model, performing parameter clipping on the updated parameters, and adding noise to the clipped parameters;

[0040] The parameter aggregation iteration unit performs parameter aggregation on the pruned model after adding noise on the central server side, and sends the aggregated model back to the client until the federated learning model converges.

[0041] Preferably, the parameter clipping unit includes:

[0042] The layer importance calculation subunit uses LRP to calculate the layer importance of neurons in the pruned model;

[0043] The privacy budget allocation subunit allocates the total privacy budget according to the layer importance to obtain the privacy budget of each layer of neurons;

[0044] The noise allocation subunit adds noise of corresponding size according to the privacy budget of each layer of neurons.

[0045] Through the above technical solutions, it can be seen that the present invention discloses a method and system for protecting data privacy in federated learning based on correlation pruning, which can effectively solve the model complexity and differential privacy protection problems in federated learning and achieve a good balance between model efficiency and privacy protection. It has the following beneficial effects:

[0046] 1. By evaluating the correlation between neurons, the model is pruned in a targeted manner, which reduces the model complexity, communication overhead, and the amount of noise that needs to be added later, thereby improving the overall model performance.

[0047] 2. By calculating the correlation LRP between each neural layer and the output, the privacy budget is allocated reasonably, with more budget allocated to important layers and less budget allocated to less important layers. Under the premise of ensuring the overall privacy budget, the accuracy of the model is improved, unnecessary waste of privacy budget is avoided, and the limited budget is better utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A flow chart of the method provided by the present invention;

[0050] Figure 2 A schematic diagram of pruning provided by the present invention;

[0051] Figure 3 This is a schematic diagram of the LPR rule provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The specific algorithm of the federated learning data privacy protection method based on correlation pruning proposed in the present invention is shown in Table 1:

[0054] Table 1 Pseudo code of the data privacy protection method for federated learning based on correlation pruning

[0055]

[0056] For the initial model of the server, pruning is performed using Algorithm 2 based on the set model sparsity ratio (line 4), and then local training is performed on the client using local data (lines 5-8). On the client, the gradient of each batch of data is first calculated, and then gradient descent is performed to update the model, and the updated parameters are pruned, and noise is added to the pruned parameters (lines 13-21). Finally, parameter aggregation is performed on the server side (lines 9-11). The aggregated model is sent back to the client until the federated learning model converges.

[0057] Specifically, an embodiment of the present invention proposes a method for protecting privacy of federated learning data based on correlation pruning, comprising the following steps:

[0058] S1. Obtain a federated learning model, which includes clients distributed in various locations and a central server.

[0059] In the present invention, on the client side, firstly, the model pruning is initialized before training to delete redundant elements from the model; secondly, after pruning, the model is sent to the client side, and each client uses local data for training. At the same time, in order to protect the privacy of the model, differential privacy noise is injected into the model parameters during training. After noise addition, these privacy-enhanced parameters are sent to the central server for aggregation.

[0060] S2. The client obtains the initial model from the central server, performs correlation-based model pruning on the initial model, and obtains a pruned model.

[0061] In this step, first, the server sends down an initial model. In order to speed up training, avoid fine-tuning, and prevent any changes to the architecture during or after training, this work focuses on pre-training pruning. Therefore, the initial model is pruned before client training, which includes removing redundant elements from the model.

[0062] The present invention adopts a structured pruning method rather than an unstructured pruning method, and in order to avoid introducing hierarchical hyperparameters, each layer uses the same pruning ratio. Neurons in the same layer are sorted according to their importance scores. Given a pruning rate, neurons with lower importance scores will be deleted from the layer. In this embodiment, the correlation between neurons is selected to obtain the importance score of the neuron. Specifically, by calculating a neuron a in the current layer i The importance of a neuron is measured by the average of the correlation between its weight and the weights of all other neurons in the same layer. The calculation is as follows:

[0063]

[0064] in, Respectively and The mean and standard deviation of Pearson correlation are between -1 and 1, so it needs to be normalized for easy calculation. The formula is as follows:

[0065]

[0066] is the normalized correlation. The average of these correlations is taken as the importance of the neuron, and the formula is as follows:

[0067]

[0068] Figure 2 This is a simple pruning example. The weights of neurons a1, a2, and a3 are The correlation between each neuron and other neurons is calculated using the neuron weights through the above formula, and then the average is taken as the importance of the neuron. In this embodiment, neurons with low importance are pruned because the weight relationship between neurons with low importance and other neurons is considered to be weak, which means that these neurons will contribute less to the overall function and performance of the network. In general, this pruning strategy aims to determine the importance of each neuron through a simple and effective metric, and remove those parts that are considered less important from the model according to the pruning ratio q to achieve network sparsification. The pseudo code of the model pruning process is shown in Table 2:

[0069] Table 2 Pseudocode of correlation-based model pruning

[0070]

[0071]

[0072] For Algorithm 2, first, for each neuron, the weight is used to calculate the correlation between it and the weights of other neurons in the same layer (lines 1-3), and the average is taken as the importance of each neuron after normalization (lines 4-7). Each layer sorts the neurons according to the importance obtained to obtain a sorted sequence. Then, the neurons with low importance are pruned by taking the percentage q of the sequence (lines 8-9). For the neurons to be pruned, their parameters are set to 0, otherwise they are set to 1, thereby generating a mask matrix consisting only of 0 and 1 (lines 10-14). Finally, the final sparse model (pruned model) is obtained by performing element-wise multiplication between the current parameters and the mask matrix (line 15).

[0073] S3. Use the client's private data to train the pruned model. During the training process, use differential privacy to add noise to the parameters to achieve privacy protection. Use LPR to allocate a privacy budget and add noise based on the privacy budget. Use the pruned model with added noise to optimize the federated learning model.

[0074] In this embodiment, client privacy data refers to data that needs to protect privacy information and is distributed on multiple clients, such as financial risk control model construction data or patient health data on medical clients.

[0075] The processing process of the pruning model is as follows: calculate the gradient of each batch of client privacy data; perform gradient descent to update the parameters of the pruning model, prune the updated parameters, and add noise to the pruned parameters.

[0076] For the pruned model, Layer-wise relevance propagation (LRP) is used to allocate the privacy budget.

[0077] In LRP, the correlation of neurons in each layer is calculated by propagating layer by layer, and the correlation of all neurons in the next layer determines the correlation of neurons in the previous layer. Therefore, for a given input x, the correlation score R of a given layer l is calculated by propagating from the output layer to the input x. j Propagate to the previous layer l-1, the calculation formula is as follows:

[0078]

[0079] Where j and k are the neuron indices in two consecutive layers (l and l-1) of the neural network. kj It is used to measure the influence of neuron k on neuron j. kj The calculation method is as follows:

[0080]

[0081] Among them, a k is the activation value of neuron k, w kj is the connection weight between neuron k and neuron j.

[0082] Therefore, Figure 3 For example, The correlation calculation method is as follows:

[0083]

[0084] After calculating the correlation of each neuron, take the average as the correlation of this layer. The correlation of each layer can reflect the importance of this layer. The higher the correlation, the greater the impact on the model output. Based on the correlation between layers, the privacy budget allocated to each layer is obtained by the following formula.

[0085]

[0086] Among them, ε n The privacy budget allocated to the nth client in this round, The privacy budget allocated to the nth client at layer l in this round.

[0087] Table 3 Pseudo code for privacy budget allocation based on LRP

[0088]

[0089] Table 3 summarizes the algorithm implementation process of obtaining the privacy budget of each client layer. First, the layer importance of each neuron is calculated by using LRP (rows 1-3), and then the total privacy budget is allocated according to the importance of each layer to obtain the privacy budget of each layer (rows 4-7). The more important the layer, the more privacy budget it will get, which can maximize the effect of privacy protection.

[0090] Then, according to the allocated privacy budget, noise of corresponding size is added.

[0091] Furthermore, the pruning model after adding noise is used to optimize the federated learning model, including:

[0092] The pruned model with added noise is uploaded to the server; the server aggregates the uploaded local model and then sends it to the client; the above steps are repeated until the federated learning model converges.

[0093] S4. Input the client's real-time privacy data into the optimized federated learning model, add noise to the real-time privacy data, and complete the protection of the real-time privacy data.

[0094] On the other hand, the present invention also proposes a federated learning data privacy protection system based on correlation pruning, which is used to implement the above-mentioned federated learning data privacy protection method based on correlation pruning. The system includes:

[0095] Data acquisition module, to obtain client privacy data;

[0096] Model processing module: the client obtains the initial model based on the central server, performs model pruning based on correlation on the initial model, and obtains the pruned model;

[0097] The model training module uses the client's privacy data to train the pruned model. During the training process, differential privacy is used to add noise to the parameters to achieve privacy protection. The privacy budget is allocated using LPR, and noise is added according to the privacy budget. The pruned model with added noise is used to optimize the federated learning model.

[0098] The data privacy protection module inputs the client's real-time privacy data into the optimized federated learning model, adds noise to the real-time privacy data, and completes the protection of the real-time privacy data.

[0099] Preferably, the model processing module includes:

[0100] The importance calculation unit uses the weights to calculate the correlation between each neuron in the initial model and the weights of other neurons in the same layer, and obtains the importance of each neuron based on the correlation;

[0101] Sorting unit, each layer sorts neurons according to importance to obtain a sorted sequence;

[0102] The pruning model acquisition unit prunes the initial model according to the pruning ratio q on the sorted sequence to obtain a pruning model.

[0103] Preferably, the model training module includes:

[0104] Gradient calculation unit, which calculates the gradient of each batch of client privacy data;

[0105] The parameter clipping unit performs gradient descent to update the parameters of the pruned model, clips the updated parameters, and adds noise to the clipped parameters;

[0106] The parameter aggregation iteration unit performs parameter aggregation on the pruned model after adding noise on the central server side, and sends the aggregated model back to the client until the federated learning model converges.

[0107] Preferably, the parameter clipping unit includes:

[0108] The layer importance calculation subunit uses LRP to calculate the layer importance of neurons in the pruned model;

[0109] The privacy budget allocation subunit allocates the total privacy budget according to the layer importance to obtain the privacy budget of each layer of neurons;

[0110] The noise allocation subunit adds noise of corresponding size according to the privacy budget of each layer of neurons.

[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0112] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for protecting data privacy in federated learning based on correlation pruning, characterized in that: The following steps are involved: Obtaining a federated learning model, wherein the federated learning model includes clients distributed in various locations and a central server; The client obtains an initial model from the central server, and performs model pruning based on correlation on the initial model to obtain a pruned model; The pruned model is trained using client privacy data, differential privacy is used to add noise to parameters during the training process to achieve privacy protection, a privacy budget is allocated using LPR, and noise is added according to the privacy budget; the federated learning model is optimized using the pruned model after adding noise; The client's real-time privacy data is input into the optimized federated learning model, and noise is added to the real-time privacy data to protect the real-time privacy data.

2. According to claim 1, a method for protecting privacy of federated learning data based on correlation pruning is characterized in that: The initial model is subjected to correlation-based model pruning to obtain a pruned model, including: The weights are used to calculate the correlation between each neuron in the initial model and the weights of other neurons in the same layer, and the importance of each neuron is obtained based on the correlation; In each layer, neurons are sorted according to the importance to obtain a sorted sequence; The initial model is pruned according to the pruning ratio q on the sorted sequence to obtain a pruned model.

3. According to claim 2, a method for protecting privacy of federated learning data based on correlation pruning is characterized in that: The correlation between each neuron and the weights of all other neurons in the same layer is normalized, and the average of the normalized correlation data is used as the importance of the corresponding neuron.

4. According to claim 1, a method for protecting privacy of federated learning data based on correlation pruning is characterized in that: The pruning model is trained using the client privacy data, differential privacy is used to add noise to the parameters during the training process to achieve privacy protection, a privacy budget is allocated using LPR, and noise is added according to the privacy budget; Optimizing the federated learning model using the pruning model after adding noise, including: Calculate the gradient of each batch of client private data; Performing gradient descent to update the parameters of the pruning model, performing parameter pruning on the updated parameters, and adding noise to the pruned parameters; The pruned model with added noise performs parameter aggregation on the central server, and the aggregated model is sent back to the client until the federated learning model converges.

5. According to claim 4, a method for protecting privacy of federated learning data based on correlation pruning is characterized in that: LPR is used to allocate a privacy budget, and noise is added according to the privacy budget, including: Use LRP to calculate the layer importance of neurons in the pruned model; Allocate the total privacy budget according to the layer importance to obtain the privacy budget of each layer of neurons; Add noise of corresponding size according to the privacy budget of each layer of neurons.

6. A method for protecting privacy of federated learning data based on correlation pruning according to claim 5, characterized in that: LRP is used to calculate the layer importance of neurons in the pruned model, including: LRP is used to calculate the correlation of neurons in each layer, and the correlation of each neuron in each layer is averaged to obtain the layer importance of each layer.

7. A federated learning data privacy protection system based on correlation pruning, characterized in that: include: Data acquisition module, to obtain client privacy data; A model processing module, in which the client obtains an initial model based on a central server, and performs model pruning based on correlation on the initial model to obtain a pruned model; A model training module, which trains the pruned model using client privacy data, uses differential privacy to add noise to parameters during training to achieve privacy protection, uses LPR to allocate a privacy budget, and adds noise according to the privacy budget; Optimizing the federated learning model using the pruning model after adding noise; The data privacy protection module inputs the client's real-time privacy data into the optimized federated learning model, adds noise to the real-time privacy data, and completes the protection of the real-time privacy data.

8. The data privacy protection system for federated learning based on correlation pruning according to claim 7, characterized in that: The model processing module includes: An importance calculation unit, which uses the weights to calculate the correlation between each neuron in the initial model and the weights of other neurons in the same layer, and obtains the importance of each neuron based on the correlation; A sorting unit, which sorts neurons in each layer according to the importance to obtain a sorted sequence; The pruning model acquisition unit prunes the initial model according to the pruning ratio q on the sorted sequence to obtain a pruning model.

9. The data privacy protection system for federated learning based on correlation pruning according to claim 7, characterized in that: The model training module includes: Gradient calculation unit, which calculates the gradient of each batch of client privacy data; A parameter clipping unit, performing gradient descent to update the parameters of the pruning model, performing parameter clipping on the updated parameters, and adding noise to the clipped parameters; The parameter aggregation iteration unit performs parameter aggregation on the pruned model after adding noise on the central server side, and sends the aggregated model back to the client until the federated learning model converges.

10. A federated learning data privacy protection system based on correlation pruning according to claim 9, characterized in that: The parameter cutting unit comprises: The layer importance calculation subunit uses LRP to calculate the layer importance of neurons in the pruned model; The privacy budget allocation subunit allocates the total privacy budget according to the layer importance to obtain the privacy budget of each layer of neurons; The noise allocation subunit adds noise of corresponding size according to the privacy budget of each layer of neurons.

Images