Federated learning model layer pruning method and system based on differential privacy

Abstract

The present invention provides a method and system for pruning federated learning model layers based on differential privacy, wherein the method includes: measuring the correlation between the local model and the global model in the federated learning model based on the gradient sign and gradient size in the local model and the global model, screening the relevant layers in the local model with correlation higher than a preset correlation, and pruning the relevant layers with correlation lower than the preset correlation; using a Top‑K-based gradient sparsification method to screen the parameters with gradient values greater than the preset gradient value in the relevant layers with higher than the preset correlation for double pruning; considering the historical parameter information of the server-side global model in the previous round when aggregating the global model on the server side; using a preset local differential privacy mechanism to perturb the parameters of the local model and inject noise into the local model. This application reduces the uploaded model parameters and ensures that a small amount of noise is injected into the model, thereby achieving higher communication efficiency while ensuring that the model has good privacy and accuracy.

Description

Technical Field

[0001] This document relates to the field of federated learning model technology, and in particular to a federated learning model layer pruning method and system based on differential privacy. Background Art

[0002] Amid the rapid development of big data and machine learning technologies, artificial intelligence (AI), a science and technology that analyzes and simulates human behavior and expands and extends human intelligence, has been widely and effectively applied in areas such as smart cities, smart homes, smart healthcare, and autonomous driving. However, these applications raise concerns about data security and user privacy. For example, machine learning models used to train disease predictions can potentially leak large amounts of patient medical data. To overcome the constraints of data silos on the development of AI technology, the Federated Learning Framework was released in 2017 to create a data-sharing machine learning platform while protecting sensitive data.

[0003] Existing federated learning technologies struggle to strike a good balance between privacy, model practicality, and communication efficiency. Deep neural networks have high-dimensional model parameters, often numbering in the tens of thousands or even millions. In federated learning, adding differential privacy noise to a user's local model effectively protects privacy. However, the amount of injected model noise is proportional to the parameter dimensionality. Therefore, injecting noise into all parameters can lead to excessive model noise, resulting in a range of issues such as low model accuracy. Furthermore, in federated learning, the server and users primarily interact through model parameters. If a large number of users continuously transmit large amounts of data during training, this can lead to excessive system communication overhead, reducing training efficiency and ultimately impacting model convergence and performance.

[0004] Wang et al. proposed an orthogonal scheme, CMFL, which can identify irrelevant updates from clients and exclude them to reduce the number of clients, thereby alleviating communication overhead. Building on Wang's work, Lian et al. proposed COFEL, a novel federated learning system that can both reduce communication time by making fine-grained selections of layer-based parameters and enhance privacy protection by applying a local differential privacy protection mechanism to the selected parameters. However, CMFL only considers pruning of the entire model, which is not conducive to fine-grained reduction of network parameters. COFEL prunes in a more fine-grained manner, but the factors considered when pruning parameters are not comprehensive enough. Therefore, an effective communication optimization method is needed to achieve better model performance and communication efficiency at a low privacy cost. Summary of the Invention

[0005] The present invention provides a federated learning model layer pruning method and system based on differential privacy, aiming to solve the above problems.

[0006] An embodiment of the present invention provides a federated learning model layer pruning method based on differential privacy, comprising:

[0007] S1. Measure the correlation between the local model and the global model based on the gradient sign and gradient size in the federated learning model. This is used to filter out the relevant layers in the local model whose correlation is higher than the preset correlation and to trim the relevant layers whose correlation is lower than the preset correlation.

[0008] S2, using the Top-K-based gradient sparsification method to screen out parameters with gradient values greater than the preset gradient value in the relevant layer that are higher than the preset correlation, and then perform double pruning;

[0009] S3. When aggregating the global model on the server side, the historical parameter information of the server-side global model in the previous round is considered;

[0010] S4. Use the preset local differential privacy mechanism to perturb the parameters of the local model, and the preset local differential privacy mechanism injects noise into the local model.

[0011] An embodiment of the present invention provides a federated learning model layer pruning system based on differential privacy, including:

[0012] The correlation clipping module is used to measure the correlation between the local model and the global model in the federated learning model based on the gradient sign and gradient size in the local model and to filter out the correlation layers in the local model with higher correlation than the preset correlation and clip the correlation layers with lower correlation than the preset correlation.

[0013] The Top-K gradient sparsification module is used to use the Top-K-based gradient sparsification method to screen parameters with gradient values greater than the preset gradient value in the relevant layer with higher than the preset correlation for double pruning;

[0014] The historical information aggregation module is used to consider the historical parameter information of the server-side global model in the previous round when aggregating the global model on the server side;

[0015] The local differential privacy module is used to perturb the parameters of the local model using a preset local differential privacy mechanism, which injects noise into the local model.

[0016] The embodiment of the present invention selects high-correlation layers and prunes low-correlation layers according to the designed correlation calculation formula to reduce the amount of communication. At the same time, a gradient sparsification scheme based on Top-K is used to select parameters with large gradient values within the layer for double pruning, further reducing the number of uploaded parameters. Secondly, in order to solve the global sparsity problem during aggregation, the information of the previous round of global models is taken into account when aggregating user parameters to prevent the accuracy of the model from decreasing due to sparsification. The local differential privacy mechanism is used to perturb the model parameters. This mechanism injects noise into the client's model locally, which can solve the problem of untrustworthy third-party servers and provide stricter privacy protection. The present invention achieves higher communication efficiency while ensuring that the model has good privacy and accuracy. It aims to improve communication efficiency and model accuracy while achieving privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flowchart of a federated learning model layer pruning method based on differential privacy according to an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of a federated learning model layer tailoring system based on differential privacy according to an embodiment of the present invention;

[0020] Figure 3 This is a diagram of the federated learning framework of an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of differential privacy according to an embodiment of the present invention;

[0022] Figure 5 This is a method for pruning federated learning model layers based on gradient correlation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0024] Federated learning is an emerging distributed learning technology designed to address data silos. By storing data locally on devices and leveraging distributed computing to train machine learning models, federated learning protects data privacy and improves model performance.

[0025] Federated learning is essentially a distributed machine learning technology consisting of two main entities: clients and a central server. Multiple clients, coordinated by a central server, collaborate to train a global model. Clients train their local data to generate local models, while the central server aggregates these models, ultimately achieving a global model after multiple iterations. This learning approach effectively addresses data silos and provides a degree of protection for user privacy.

[0026] Differential privacy is a privacy-preserving method that obscures sensitive data by adding noise to computations. In federated learning based on differential privacy, participants add noise to model parameters when training the model locally, thus achieving a degree of privacy protection. This approach allows participants to collaboratively learn the model while preserving privacy, thereby improving the model's accuracy and generalization capabilities.

[0027] Differential privacy was originally proposed to address the privacy issue of dataset query results. It describes a privacy definition in which the impact of the addition or deletion of a single data record on the statistical query results cannot be identified by attackers. It is a standard privacy model for statistical analysis with provable privacy guarantees.

[0028] Next, we will outline the concepts and definitions related to differential privacy.

[0029] Definition 1: Adjacent datasets. If two datasets D and D' have the same attribute structure and differ by only one data record, then datasets D and D' are called adjacent datasets.

[0030] Define 2ε-differential privacy. Given a random algorithm M, whose range is Range(M), if any output of algorithm M on adjacent datasets D and D' satisfies the following inequality,

[0031] Pr[M(D)∈S]≤e ε ·Pr[M(D ′ )∈S] (1)

[0032] Then the random algorithm M is said to satisfy ε-differential privacy, which means that the data after being disturbed by the algorithm M cannot be changed by a value greater than e εThe query results are provided based on the probability distinguishability of . Pr[·] is the probability of the set of all possible outputs of M, and the privacy budget ε is an indicator to measure the strength of privacy protection. The smaller the value of ε, the higher the privacy protection strength and the lower the data availability, and vice versa.

[0033] Definition 3: ε-local differential privacy. A randomized algorithm M satisfies ε-local differential privacy if and only if for any input v and v' in M, and any output Y after perturbation by M,

[0034] Pr [M(v)=Y]≤e ε ·Pr [M(v′)=Y] (2)

[0035] As can be seen from Definition 3, local differential privacy is an independent perturbation to a single data point. The privacy budget is used to control the similarity of the output between two data points, thereby ensuring that the algorithm M satisfies e ε - Localized differential privacy.

[0036] Figure 3 This is a diagram of the federated learning framework of an embodiment of the present invention. Its training process mainly includes the following four steps:

[0037] (1) The server first initializes the global model parameters w t and network structure, and then distribute the global model to each client participating in the federated learning training.

[0038] (2) Each client k downloads the global model distributed by the server and uses the model for local training to obtain the local model

[0039] (3) Client uploads local model to the central server.

[0040] (4) The server receives the local model of each client, performs model aggregation, and obtains a new round of global model.

[0041] Federated learning continuously cycles through the four steps above until the model meets convergence criteria and training is complete. Each client ultimately obtains a converged global model. As can be seen from the above steps, each user's local data in federated learning is always stored locally. This prevents privacy leaks caused by users uploading data to untrusted servers and addresses the issue of traditional centralized training, which only allows for model distribution but cannot be trained locally.

[0042] Figure 4This is a diagram of differential privacy in an embodiment of the present invention. Differential privacy uses a privacy budget to limit the impact of a single data record on the perturbation algorithm's output. This ensures that even if a single record is added or deleted from the dataset, the algorithm's output remains within a controlled range. Therefore, differential privacy prevents attackers from inferring sensitive information about a particular piece of data from the output, effectively preventing background knowledge attacks.

[0043] Researchers working on differential privacy in federated learning, such as Abadi et al., first proposed a federated learning privacy-preserving method combining differential privacy with asynchronous gradient descent (DP-SGD). This algorithm uses gradient clipping to limit the sensitivity of samples and then adds noise to the client-uploaded gradients for local updates. Subsequent research has largely improved upon DP-SGD. Geyer et al. proposed a federated learning framework with user-level differential privacy, focusing on the user's perspective. The authors proposed an adaptive differential privacy scheme that balances privacy and accuracy by adjusting the amount of noise added. This approach considers differences in data distribution and training data size between each client, thereby maximizing model accuracy while ensuring privacy. Wei et al. proposed the NbAFL scheme, which adjusts the noise variance to ensure that global data meets global privacy requirements under a certain level of Gaussian noise perturbation. Liu proposed a two-stage framework, FedSel, that privately selects the top-k dimensions based on the contribution of gradient parameters in each iteration and employs gradient accumulation techniques to stabilize the learning process with noisy updates. Considering unreliable participants in federated learning, Zhao et al. proposed a new scheme called SecProbe, which allows participants to share model parameters and handles unreliable participants by leveraging an exponential mechanism for differential privacy. Li et al. proposed an Efficient Privacy Preserving FL (EPPFL) scheme with Unreliable Users. This scheme mitigates the negative impact of unreliable users by excluding irrelevant model parameter components and weighted aggregation, and ensures that the model is updated with high-quality data.

[0044] In addition to privacy issues, federated learning is based on large-scale distributed learning scenarios, and communication problems seriously hinder the trade-off between privacy and utility. To improve the communication efficiency of federated learning, Wang et al. proposed an orthogonal scheme CMFL, which can identify irrelevant updates from clients and exclude irrelevant updates to reduce the number of clients, thereby alleviating communication overhead. Based on Wang, Lian et al. proposed COFEL, a novel federated learning system that can reduce communication time by making fine-grained selections of layer-based parameters and strengthen privacy protection by applying local differential privacy protection mechanisms to the selected parameters. Liu et al. proposed an efficient communication scheme FedCPF, which greatly improves communication efficiency by reducing the number of iterations of local clients, the number of participating clients, and limiting the time for clients to upload parameters.

[0045] This invention improves upon CMFL and COFEL. CMFL, as previously mentioned, only considers pruning the entire model, which is not conducive to fine-grained reduction of network parameters. COFEL prunes in a more fine-grained manner, but the parameters are not considered comprehensively. This invention combines the strengths and weaknesses of CMFL and COFEL, making corresponding improvements. In terms of communication efficiency, this invention outperforms both of these algorithms in terms of pruning efficiency and parameter reduction. In terms of model utility and privacy, this invention achieves improved model performance and communication efficiency at a low privacy cost, making it an effective communication optimization method.

[0046] Method Example

[0047] The embodiment of the present invention provides a federated learning model layer pruning method based on differential privacy, Figure 1 This is a flowchart of the federated learning model layer pruning method based on differential privacy according to an embodiment of the present invention. Figure 1 As shown, the federated learning model layer pruning method based on differential privacy in an embodiment of the present invention specifically includes:

[0048] Step S1: measuring the correlation between the local model and the global model based on the gradient sign and gradient size in the local model and the global model in the federated learning model, so as to filter out the relevant layers in the local model whose correlation is higher than the preset correlation and trim the relevant layers whose correlation is lower than the preset correlation;

[0049] In federated learning, a global model is obtained by aggregating a large number of local models of users. Since local models are trained with the local data of a specific user, there are usually differences between the local model and the global model. Specifically, given a parameter in the update, its sign and magnitude determine the degree of change (increase or decrease) in the direction in which the model should improve along the dimension of that parameter. In federated learning, the server hopes that users will upload parameters that are beneficial to the global model (i.e., the signs and magnitudes of the global model parameters are roughly the same as those of the local model) to improve the accuracy and convergence speed of the model. Therefore, by comparing the parameters in the local model and the global model, the degree of alignment of the two updates can be measured.

[0050] By comparing the differences between the local model and the global model, the highly correlated models are uploaded to the server for aggregation. Models with low correlations are not uploaded because they would compromise the accuracy of the global model. In practice, neural network models at different layers have different characteristics, and also differ in correlation and training performance. The present invention considers quantifying correlations from a layer-based perspective.

[0051] Specifically, let g = {g1,2,…g c} is a local model with C model parameters, is the global model with C model parameters. The correlation between the local model in a specific layer and the global model is measured as:

[0052]

[0053] Where g={g1,2,… c} is a local model containing C model parameters, is a global model containing C model parameters, D() represents the correlation based on the gradient sign, j represents the jth layer of the neural network, C j Represents the number of elements in the j-th layer of the neural network, S() represents the symbolic function, if g j,c and The signs are the same, then is 1, otherwise 0;

[0054] In fact, in a neural network, the sign and magnitude of the gradient jointly determine the optimization direction of the gradient. Therefore, the embodiment of the present invention also designs a correlation calculation formula based on the gradient sign, namely:

[0055]

[0056] Among them, V() represents the correlation based on the gradient size, Represented as the local gradient value g j, and the global gradient value If the distance between them meets certain conditions, it is 1, otherwise it is 0.

[0057] Formulas 3 and 4 allow us to measure the correlation between the local model and the global model in a specific layer based on the gradient sign and gradient magnitude, respectively. To evaluate the correlation between models from a more comprehensive perspective, this embodiment of the present invention considers both the gradient sign and magnitude to measure the correlation between models.

[0058] For the parameters of a certain layer of a neural network, the signs and sizes of all the parameters in that layer determine the overall optimization direction of the model at that layer. However, even parameters at different layers from the same user can contribute significantly to the global model. Specifically, uploading network layer parameters that are beneficial to the global model will improve the accuracy and convergence speed of the global model, while uploading some irrelevant network layer parameters will harm the performance of the global model. In the actual training process, most parameters are still involved in training, so even if some irrelevant parameters are excluded, the model can still maintain good accuracy.

[0059] Step S2: Using a Top-K-based gradient sparsification method to select parameters with gradient values greater than a preset correlation in related layers and greater than a preset gradient value for double pruning;

[0060] In deep neural networks, larger gradients of model parameters indicate that the parameters in the current dimension have not yet stabilized and converged, moving away from their local optimum and accommodating more noise. Therefore, parameters with larger gradients are crucial for neural network convergence, often dominating the network's optimization direction and determining model performance. Parameters with smaller gradients, on the other hand, indicate near-convergence, with minimal fluctuations in parameter values. Even slight perturbations can significantly impact model accuracy. Therefore, the ability of parameters with smaller gradients to accommodate noise is more sensitive and requires more precise measurement. The above analysis shows that parameters with larger gradients dominate network optimization and have a greater ability to accommodate noise than parameters with smaller gradients. Adding noise to these parameters only slightly impacts model performance. Therefore, when pruning a model, it's wise to select parameters with larger gradients and exclude those with smaller gradients. The following describes the specific implementation process.

[0061] Based on the layer pruning method in step S1, the C j Select the first K dimensions with the largest absolute value of gradient to upload (K <C j ). First, define the scoring function graScore(), score according to the absolute value of each dimension, and get the score set of all gradients Then use the sorting algorithm to sort the above scoring results in descending order to obtain the sorted result set Then from the sorted Cj The gradient parameter g′ of the dimension k,j In , the first K dimensions are selected as the pruned model. Specifically, The cropped model From C j The dimension is reduced to K. For each layer j of all users k participating in the training, the same steps are used to sparse the gradient to obtain the clipped gradient in It contains all layers j selected by the user and the K parameters with the largest gradient values in layer j. That is to say, the parameters uploaded to the server by the user are the K parameters with the largest gradients in the selected layer.

[0062] Step S3: when aggregating the global model on the server side, the historical parameter information of the server side global model in the previous round is considered;

[0063] If the correlation threshold is set too high, the network layers of most users will not meet the selection criteria, and the user's overall model will be pruned. When the server aggregates incomplete local models, it may cause the global model to become sparse, ultimately compromising the accuracy of the global model. To address this issue, the embodiments of the present invention not only consider the client's aggregation results during aggregation, but also consider the historical parameter information of the previous round of global models, as shown in Formulas 5 and 6.

[0064]

[0065]

[0066] where w t is the historical global model of the previous round t, w t+1 is the next round of global model to be updated, It is the global model updated after aggregating the client's gradients through the federated averaging algorithm in the current round t+1. is the mean of all client gradient parameters, and α is the learning rate of the update algorithm.

[0067] Step S4: Use the preset local differential privacy mechanism to perturb the parameters of the local model, and the preset local differential privacy mechanism injects noise into the local model.

[0068] This embodiment of the present invention uses a local differential privacy mechanism to perturb model parameters. This mechanism locally injects noise into the client's model, addressing the issue of untrustworthy third-party servers and providing stricter privacy protection. The perturbation result returned by this mechanism is a binary real number.

[0069] First, given a model gradient parameter G, the perturbation algorithm randomizes each dimension of G and returns a perturbed gradient parameter G *Assume that M is the perturbation mechanism. For each dimension g∈G in G, assume that g∈[c-,c+]. Where c is the center of the range of g and r is the radius of the range. c and r are preset parameters that determine the range of values of the input data. The setting method depends on how to clip the gradient parameter g. Specifically, for the original gradient parameter g, the following LDP mechanism is used for perturbation:

[0070]

[0071] Among them G * It is the noise parameter after the LDP mechanism perturbation, including d dimensions, “wp” represents the corresponding probability value, e ε represents the privacy budget.

[0072] The solution of the present invention mainly includes a single server and multiple users. The server's main task is to distribute the global model and aggregate the local model. The main task of the client is to use the federated learning model layer pruning method based on gradient correlation proposed in this chapter to prune the local model.

[0073] In this solution, the neural network structures of the global model and the local model are consistent, but the parameters of the network layers (such as convolutional layer, pooling layer, fully connected layer, etc.) of the user's local model are different and have certain personalized characteristics. Figure 5 As shown in the figure, the user first downloads the global model and performs local training to obtain a local model. Then, for each layer j of the local model, the gradient correlation formula is used to calculate its correlation with the global model. If the correlation of layer j is greater than the set correlation threshold Q, the layer is selected for upload; otherwise, the layer is pruned. Finally, the user performs double pruning on the selected layer j using a Top-K-based gradient sparsification scheme, selects the K-dimensional parameters with the largest gradient value in the layer, adds local differential privacy noise to them, and uploads the perturbed model to the server. The server finally aggregates the perturbed model using an aggregation method based on historical information and sends the new global model to the user in a new round of updates. The above steps are repeated until the convergence condition is met. Finally, through relevant experimental analysis and verification, this method ensures that the model has a certain level of privacy protection while also having good communication efficiency and accuracy.

[0074] By adopting the embodiments of the present invention, the following beneficial effects are achieved:

[0075] The embodiment of the present invention selects high-correlation layers and prunes low-correlation layers according to the designed correlation calculation formula to reduce the amount of communication. At the same time, a gradient sparsification scheme based on Top-K is used to select parameters with large gradient values within the layer for double pruning, further reducing the number of uploaded parameters. Secondly, in order to solve the global sparsity problem during aggregation, the information of the previous round of global models is taken into account when aggregating user parameters to prevent the accuracy of the model from decreasing due to sparsification. The local differential privacy mechanism is used to perturb the model parameters. This mechanism injects noise into the client's model locally, which can solve the problem of untrustworthy third-party servers and provide stricter privacy protection. The present invention achieves higher communication efficiency while ensuring that the model has good privacy and accuracy. It aims to improve communication efficiency and model accuracy while achieving privacy protection.

[0076] System Example

[0077] The embodiment of the present invention provides a federated learning model layer pruning system based on differential privacy, Figure 2 This is a schematic diagram of a federated learning model layer pruning system based on differential privacy according to an embodiment of the present invention. Figure 2 As shown, the federated learning model layer pruning system based on differential privacy in an embodiment of the present invention specifically includes:

[0078] A correlation clipping module 20 is configured to measure the correlation between the local model and the global model based on the gradient sign and gradient magnitude in the local model and the global model in the federated learning model, and to filter out the correlation layers in the local model whose correlation is higher than a preset correlation and to clip the correlation layers whose correlation is lower than the preset correlation;

[0079] The relevance tailoring module 20 is specifically used for:

[0080] The correlation between the local model and the global model is measured by the gradient sign and gradient magnitude through formulas 1 and 2, respectively:

[0081]

[0082] Where g={g1,2,… c} is a local model containing C model parameters, is a global model containing C model parameters, D() represents the correlation based on the gradient sign, j represents the jth layer of the neural network, C j Represents the number of elements in the j-th layer of the neural network, S() represents the symbolic function, if g j,c and The signs are the same, then is 1, otherwise 0;

[0083]

[0084] Among them, V() represents the correlation based on the gradient size, Represented as the local gradient value g j, and the global gradient value If the distance between them meets the preset conditions, it is 1, otherwise it is 0.

[0085] A Top-K gradient thinning module 22 is used to use a Top-K-based gradient thinning method to screen parameters with gradient values greater than a preset gradient value in the relevant layer with a higher than preset correlation and perform double pruning;

[0086] The Top-K gradient sparsification module 22 is specifically used for:

[0087] In the model parameter C j Select the first K dimensions with the largest absolute value of gradient from the dimension and upload them to the server, where K <C j .

[0088] A historical information aggregation module 24 is configured to consider the historical parameter information of the server-side global model in the previous round when aggregating the global model on the server side;

[0089] The historical information aggregation module 24 is specifically used for:

[0090] Use Formula 3 and Formula 4 to obtain the global model of the server in this round:

[0091]

[0092]

[0093] Among them, w t is the historical global model of the previous round t, w t+1 is the next round of global model to be updated, It is the global model updated after aggregating the client's gradients through the federated averaging algorithm in the current round t+1. is the mean of all client gradient parameters, and α is the learning rate of the update algorithm.

[0094] The local differential privacy module 26 is used to perturb the parameters of the local model using a preset local differential privacy mechanism, and the preset local differential privacy mechanism injects noise into the local model.

[0095] The local differential privacy module 26 is specifically used to:

[0096] For the original gradient parameter g, the LDP mechanism is used to perturb it through formula 5:

[0097]

[0098] Among them, G is the gradient parameter of the model, M is the perturbation mechanism, for each dimension g∈G in G, let g∈[cr,c+r], where c is the range center of g and r is the radius of the range. c and r are preset parameters that determine the value range of the input data. The setting method depends on how to clip the gradient parameter g, G * It is the noise parameter after the LDP mechanism perturbation, including d dimensions, wp represents the corresponding probability value, e ε represents the privacy budget.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning model layer pruning method based on differential privacy, characterized by: include: S1. Measure the correlation between the local model and the global model based on the gradient sign and gradient size in the federated learning model. This is used to filter out the relevant layers in the local model whose correlation is higher than the preset correlation and to trim the relevant layers whose correlation is lower than the preset correlation. S2. Using a Top-K-based gradient sparsification method to screen parameters with gradient values greater than a preset gradient value in the relevant layer having a higher than preset correlation, and then performing double pruning; S3. When aggregating the global model on the server side, the historical parameter information of the server-side global model in the previous round is considered; S4. Using a preset local differential privacy mechanism to perturb parameters of the local model, wherein the preset local differential privacy mechanism injects noise into the local model; Said S1 specifically includes: The correlation between the local model and the global model is measured by the gradient sign and gradient magnitude through formulas 1 and 2, respectively: Formula 1: in, is included local model with model parameters, is included A global model with model parameters, represents the correlation based on the gradient sign, j The neural network layer, Indicates the The number of elements of the layer neural network, S () represents the symbol function, if and The signs are the same, then is 1, otherwise 0, sgn() is used to extract the gradient sign; Formula 2: in, V ( ) represents the correlation based on the gradient size, Represented as local gradient value and the global gradient value If the distance between them meets the preset conditions, it is 1, otherwise it is 0. val() represents the value function, which is used to extract the specific value of the gradient; The S3 specifically includes: Use Formula 3 and Formula 4 to obtain the global model of the server in this round: Formula 3: Formula 4: in, It's the last round The historical global model of is the next round of global model to be updated, Is the current round The global model is updated after aggregating the client's gradients through the federated averaging algorithm. is the mean of all client gradient parameters, is the learning rate of the update algorithm.

2. The method according to claim 1, characterized in that The S2 specifically includes: In the model parameters Select the first dimension with the largest absolute value of gradient Dimensions uploaded to the server, where .

3. The method according to claim 1, characterized in that The S4 specifically includes: For the original gradient parameters , use the LDP mechanism to perform perturbations using Formula 5: Formula 5: in, is the gradient parameter of the model, is a disturbance mechanism. Each dimension in ,set up ,in yes The center of the range, is the radius of the range, and It is a preset parameter that determines the value range of the input data. The setting method depends on how to clip the gradient parameter. , It is the noise parameter after the LDP mechanism disturbance, including dimension, Represents the corresponding probability value represents the privacy budget.

4. A federated learning model layer pruning system based on differential privacy, characterized by: include: The correlation clipping module is used to measure the correlation between the local model and the global model in the federated learning model based on the gradient sign and gradient size in the local model and to filter out the correlation layers in the local model with higher correlation than the preset correlation and clip the correlation layers with lower correlation than the preset correlation. A Top-K gradient sparsification module is used to use a Top-K-based gradient sparsification method to screen the parameters with gradient values greater than the preset gradient value in the relevant layer that are higher than the preset correlation and perform double clipping; The historical information aggregation module is used to consider the historical parameter information of the server-side global model in the previous round when aggregating the global model on the server side; A local differential privacy module, configured to perturb the parameters of a local model using a preset local differential privacy mechanism, wherein the preset local differential privacy mechanism injects noise into the local model; The correlation cutting module is specifically used for: The correlation between the local model and the global model is measured by the gradient sign and gradient magnitude through formulas 1 and 2, respectively: Formula 1: in, is included local model with model parameters, is included A global model with model parameters, represents the correlation based on the gradient sign, j The neural network layer, express The number of elements in the layer, S( ) represents the sign function, if and The signs are the same, then is 1, otherwise 0, sgn() is used to extract the gradient sign; Formula 2: in, V ( ) represents the correlation based on the gradient size, Represented as local gradient value and the global gradient value If the distance between them meets the preset conditions, it is 1, otherwise it is 0. val() represents the value function, which is used to extract the specific value of the gradient; The historical information aggregation module is specifically used to: Use Formula 3 and Formula 4 to obtain the global model of the server in this round: Formula 3: Formula 4: in, It's the last round The historical global model of is the next round of global model to be updated, Is the current round The global model is updated after aggregating the client's gradients through the federated averaging algorithm. is the mean of all client gradient parameters, is the learning rate of the update algorithm.

5. The system according to claim 4, characterized in that The Top-K gradient sparsification module is specifically used for: In the model parameters Select the first dimension with the largest absolute value of gradient Dimensions uploaded to the server, where .

6. The system according to claim 4, characterized in that The local differential privacy module is specifically used to: For the original gradient parameters , use the LDP mechanism to perform perturbations using Formula 5: Formula 5: in, is the gradient parameter of the model, is a disturbance mechanism. Each dimension in ,set up ,in yes The center of the range, is the radius of the range, and It is a preset parameter that determines the value range of the input data. The setting method depends on how to clip the gradient parameter. , It is the noise parameter after the LDP mechanism disturbance, including dimension, represents the corresponding probability value, represents the privacy budget.

Images