A federated learning method and system based on local differential privacy
By using tensor similarity algorithms and adaptive noise adjustment methods in federated learning, the problem of imbalance in computing resource consumption and privacy performance is solved, and the application feasibility and efficiency of federated learning are improved.
Patent Information
- Application Number
- CN202410700416.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-05-31
AI Technical Summary
The existing federated learning method based on local differential privacy occupies a large amount of computing resources of terminal devices, limiting the feasibility of federated learning and the efficiency of global model training, and there is a problem of imbalance between privacy protection and model performance.
Use the tensor similarity algorithm to judge the similarity of global model parameter updates. If the similarity is less than the threshold, local training will be carried out to obtain the model parameter update and add noise. If the similarity is greater than or equal to the threshold, select similar parameters from the historical update to update. Adaptively tuned noise is added to balance privacy protection and model performance.
Saves the computing resources and training time on the user side, dynamically adjusts the noise addition to improve the level of privacy protection, and balances privacy protection and model performance.
Smart Images

Figure CN118504712B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of federated learning technology, and in particular to a federated learning method and system based on local differential privacy. Background Art
[0002] Federated learning (FL) as a distributed machine learning technology can solve the problem of data silos, allowing institutions to collaborate across regions without leaving the local data, and the models built by multi-party cooperation can more accurately predict various problems. It aims to upload the model parameters trained on the local data set without directly uploading local data. The server receives the model parameters uploaded by multiple users and aggregates them, repeating multiple times to obtain a global model.
[0003] Since the federated learning training process requires the server and some users to exchange multiple rounds of global models, global model parameter updates, or local model parameter updates, malicious attackers can use gradient inversion attacks to reconstruct part of a user's original data, or use the parameters exchanged in the middle to perform member attacks on user data. In order to protect user privacy gradients or model parameter updates, and prevent gradient inversion attacks, a series of gradient update protection methods based on localized differential privacy (LDP) have been proposed.
[0004] Reference Figure 1 As shown in the figure, in localized differential privacy, data is randomized locally by the user, and by adding noise, perturbations, or mapping data, it is ensured that the data has been randomized before being transmitted to the server. The responsibility and control of privacy protection are handed over to the user, and the risk of data leakage is reduced by performing randomization on the user's device, thereby improving the level of privacy protection. At the same time, the server is allowed to perform data analysis without directly accessing the original data, achieving a balance between privacy protection and data availability.
[0005] Although the existing federated learning method based on local differential privacy improves the level of privacy protection, the user-end devices involved in model training may require a large amount of computing resources. Especially for some user-end devices with weaker performance, it will lead to excessive consumption of computing resources and even affect the user's normal device usage experience, limiting the feasibility of federated learning based on local differential privacy in practical applications and the efficiency of global model training.
[0006] In addition, most of the existing federated learning methods based on local differential privacy directly add Gaussian noise to the model parameter update without considering the changes of the model parameter update before and after the training process. The same privacy protection level is used for the model parameter updates with high sensitivity (large changes during training) and the model parameter updates with low sensitivity (small changes during training). This will result in the protection strength of the highly sensitive model parameter updates being too weak, increasing the risk of privacy leakage, and the protection strength of the low-sensitivity model parameter updates being too strong, reducing the model performance, that is, there is an imbalance between privacy protection and model performance. Summary of the invention
[0007] The present invention aims to at least solve the technical problems of existing federated learning methods based on local differential privacy, such as the need to occupy a large amount of computing resources of terminal devices, limiting the practical application feasibility of federated learning and the efficiency of global model training, as well as the imbalance between privacy protection and model performance, and provide a federated learning method and system based on local differential privacy.
[0008] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a federated learning method based on local differential privacy, corresponding to the user end, the method comprising: the user end obtains the global model and global model parameter update of this round from the server end; uses the tensor similarity algorithm to obtain the similarity between the global model parameter update of this round and the global model parameter update of the previous round; if the similarity is less than the backtracking threshold, the global model of this round is trained by local data to obtain the local model parameter update of this round, and the local model parameter noise update of this round is obtained by data perturbation on the local model parameter update of this round; if the similarity is greater than or equal to the backtracking threshold, the historical global model parameter update that is most similar to the global model parameter update of this round is selected from the historical global model parameter update set, and the local model parameter noise update of the round to which the selected historical global model parameter update belongs is used as the local model parameter noise update of this round; and the local model parameter noise update of this round is uploaded to the server end.
[0009] Further preferably, the execution process of the tensor similarity algorithm includes: respectively calculating the data point density of the global model parameter update of the tth round and the t'th round; weighting the global model parameter update of the tth round using the data point density of the global model parameter update of the tth round to obtain the weighted global model parameter update of the tth round; weighting the global model parameter update of the t'th round using the data point density of the global model parameter update of the t'th round to obtain the weighted global model parameter update of the t'th round; obtaining the cosine similarity of the weighted global model parameter update of the tth round and the weighted global model parameter update of the t'th round, and using the cosine similarity as the similarity between the global model parameter update of the tth round and the global model parameter update of the t'th round; wherein t and t' are both rounds, and t≠t'.
[0010] More preferably,
[0011] Data point density for global model parameter update in round t The calculation formula is:
[0012]
[0013] Among them, Δ t represents the global model parameter update of the tth round, n represents the number of elements of the global model parameter update of the tth round, and the global model parameter update of the tth round j represents the index of the element where the global model parameters are updated, x j Represents the jth element value of the global model parameter update in the tth round, and the jth tensor is constructed h represents the smoothing parameter, and K(·) represents the Gaussian kernel function.
[0014] Further preferably, the selecting the historical global model parameter update that is most similar to the global model parameter update of this round from the historical global model parameter update set includes: using the tensor similarity algorithm to calculate the similarity between the global model parameter update of this round and each historical global model parameter update in the historical global model parameter update set; selecting the historical global model parameter update with the greatest similarity as the most similar historical global model parameter update.
[0015] Further preferably, the data perturbation of the local model parameter update of this round to obtain the local model parameter noise update of this round includes: obtaining the dynamic noise coefficient based on the similarity between the global model parameter update of this round and the global model parameter update of the previous round; adding Gaussian noise to the local model parameter update of this round to obtain the local model parameter noise update of this round, wherein the variance of the Gaussian noise is the product of a preset variance and the dynamic noise coefficient.
[0016] Further preferably, the calculation formula of the dynamic noise coefficient β is:
[0017] β=e -ks
[0018] Among them, k represents the decay rate; s represents the similarity between the global model parameter update of this round and the global model parameter update of the previous round.
[0019] Further preferably, the adding of Gaussian noise to the local model parameter update of this round to obtain the local model parameter noise update of this round comprises: performing trimming processing on the local model parameter update of this round based on a trimming threshold to obtain a trimmed local model parameter update; adding Gaussian noise to the trimmed local model parameter update to obtain the local model parameter noise update of this round, wherein the variance of the Gaussian noise is the product of a preset variance, a dynamic noise coefficient, the square of the trimming threshold and a unit matrix.
[0020] Further preferably, the local model parameters of the tth round are updated δ based on the clipping threshold according to the following clipping formula: t To perform cropping:
[0021]
[0022] in, represents the clipping local model parameter update, C represents the clipping threshold, ||δ t ||2 means δ t 2-norm of , max(,) represents the maximum value function.
[0023] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a federated learning method based on local differential privacy, which is jointly executed by a server and multiple user terminals. The method iteratively executes multiple rounds of training until the training stop condition is reached, and executes in each round of training: at least one user terminal obtains the local model parameter noise update of this round by executing the method steps described in the first aspect of the present invention, and uploads the local model parameter noise update of this round to the server terminal; the server terminal aggregates the local model parameter noise updates of this round uploaded by multiple user terminals to obtain the global model and global model parameter update of the next round.
[0024] In order to achieve the above-mentioned object of the present invention, according to the third aspect of the present invention, the present invention
[0025] A model training system based on the local differential privacy-based federated learning method provided in the second aspect of the present invention is provided, comprising a server side, and a plurality of user sides respectively connected to and communicating with the server side.
[0026] The beneficial technical effect of the present invention is as follows: in each round of training, the user terminal adopts a historical local backtracking update mechanism, which compares the similarity between the global model parameter update of the current round and the global model parameter update of the previous round with the backtracking threshold. If the similarity is less than the backtracking threshold, it indicates that the global model parameter update of the current round and the previous round are quite different, and it is necessary to use local data to train the global model. If the similarity is greater than or equal to the backtracking threshold, the local model parameter noise update of the corresponding round of the historical global model parameter update that is most similar to the global model parameter update of the current round is directly selected from the historical global model parameter update set, and uploaded, which can avoid unnecessary local training process and save time. Client computing resources and federated learning training time; adaptively perform differential privacy, and adaptively adjust the Gaussian noise added to the local model parameter update based on the similarity between the global model parameter update of this round and the global model parameter update of the previous round. When the similarity is high, it means that the model has not changed much during the training process, and the sensitivity of the local model parameter update is relatively low, so smaller noise is added. When the similarity is low, it means that the model has changed significantly during the training process, and the sensitivity of the local model parameter update is high. Larger noise is added to improve the level of privacy protection. The dynamic noise coefficient is used to achieve real-time adjustment of noise addition, balancing privacy protection and model performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is an existing federated learning framework combined with localized differential privacy;
[0028] Figure 2 It is a flowchart of a federated learning method based on local differential privacy in a preferred embodiment of the present invention;
[0029] Figure 3 This is a flow chart of a preferred implementation of the present invention in which the historical local retrospective update mechanism is not started;
[0030] Figure 4 It is an overall block diagram of a model training system in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0032] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0033] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.
[0034] The present invention discloses a federated learning method based on local differential privacy. Corresponding to the user end, in a preferred embodiment, referring to Figure 2 , the method comprising:
[0035] Step S1: The user end obtains the global model and global model parameter update of this round from the server end.
[0036] Specifically, the acquisition method can be that the server broadcasts the global model and global model parameter updates to all or part of the user terminals, or the user terminal downloads the global model and global model parameter updates of this round from the server. The global model parameter update is the difference between the global model parameters of the previous round and the global model parameters of the previous round calculated by the server. In order to improve the level of privacy protection, noise is added to the global model parameter update, and the noise is preferably Gaussian noise.
[0037] Step S2, using a tensor similarity algorithm to obtain the similarity between the global model parameter update of this round and the global model parameter update of the previous round.
[0038] The user side stores the global model parameter updates obtained in each round, forming a historical global model parameter update set, and also stores the local model parameter updates uploaded to the server side in each round. Each historical global model parameter update in the historical global model parameter update set corresponds one-to-one with the local model parameter update uploaded to the server side in the round to which it belongs, so as to perform similarity calculation and implement the historical local backtracking update mechanism.
[0039] Since the global model parameter update is essentially a tensor and has directionality, the tensor similarity algorithm is preferably but not limited to the existing cosine similarity calculation method, or the Euclidean distance similarity calculation method, or a combination of the cosine similarity calculation method and the Euclidean distance similarity calculation method (the combination is not limited to the weighted sum of the calculation results output by the two similarity calculation methods).
[0040] Step S3, if the similarity is less than the backtracking threshold, the global model of this round is trained with local data to obtain the local model parameter update of this round, and the local model parameter update of this round is perturbed with data to obtain the local model parameter noise update of this round. The backtracking threshold can be set in advance based on experience.
[0041] If the similarity is less than the backtracking threshold, it means that the global model parameter update in this round is quite different from that in the previous round. This round does not meet the requirements of the historical local backtracking update mechanism and needs to execute the local training process. Figure 3 ,include:
[0042] ① The user uses the local data set to train the global model obtained in this round. The data set is divided into multiple batches for training, and the following operations are performed on each batch:
[0043]
[0044] Where η is the local learning rate, b i Calculate the gradient for the number of samples contained in the i-th batch And use the local learning rate η to update the local model parameters W of the i-th batch in this round i .W i-1 W represents the local model parameters of the i-1th batch. 0 represents the initial local model parameters of this round, and is the global model parameters obtained in this round.
[0045] ② Calculate the local model parameter update δ for this round t , the specific calculation method is as follows:
[0046] δ t =W t -W t-1
[0047] W t represents the local model parameters obtained after this round of local training, W t-1 Represents the local model parameters obtained after the last round of local training.
[0048] ③ Update the local model parameters δ t Perform data perturbation to obtain local model parameter noise update The data perturbation is preferably but not limited to adding Gaussian noise.
[0049] ④ Update the local model parameter noise Save it locally and upload it to the server.
[0050] Step S4: If the similarity is greater than or equal to the backtracking threshold, the historical global model parameter update that is most similar to the global model parameter update of this round is selected from the historical global model parameter update set, and the local model parameter noise update of the round to which the selected historical global model parameter update belongs is used as the local model parameter noise update of this round.
[0051] If the similarity is greater than or equal to the backtracking threshold, it indicates that the global model parameter update in this round is slightly different from that in the previous round. This round meets the requirements of the historical local backtracking update mechanism and does not require local training, which can save computing resources on the user side.
[0052] In step S4, the historical global model parameter update that is most similar to the global model parameter update of this round is selected from the historical global model parameter update set, including:
[0053] Step S41, using the tensor similarity algorithm to calculate the similarity between the global model parameter update of this round and each historical global model parameter update in the historical global model parameter update set. Let the global model parameter update of this round be expressed as Δ t , and obtain a series of similarities: s(Δ t ,Δ t-1 ),...,s(Δ t ,Δ k )..., Δ t-1 represents the global model parameter update of round t-1, Δ k represents the global model parameter update of round k. t ,Δ t-1 ) represents Δ t With Δ t-1 The similarity of s(Δ t ,Δ k ) represents Δ t With Δ k The similarity.
[0054] Step S42, selecting the historical global model parameter update with the greatest similarity in step S41 as the most similar historical global model parameter update, expressed as:
[0055] max{s(Δ t ,Δ t-1 ),...,s(Δ t ,Δ k )...
[0056] Suppose the global model parameter update Δ in the kth round k And the global model parameter update Δ t If the kth round is the most similar, the local model parameter noise update uploaded to the server is As local model parameter noise update
[0057]
[0058] At the same time, the obtained Save locally.
[0059] Step S5, upload the local model parameter noise update of this round to the server.
[0060] In this embodiment, the user terminal performs the above steps S1 to S5 in each round of training of federated learning. The above steps S2 to 4 constitute a historical local backtracking update mechanism, which first calculates the similarity of global model parameter updates, compares the similarity with the size of the backtracking threshold, and selects local training according to the comparison result, or directly uploads the previous local model parameter noise update without conducting local training. This method can avoid unnecessary local training processes, save computing resources and time, and reduce potential errors that may be introduced by local training.
[0061] In a preferred embodiment, existing data similarity calculation methods (such as cosine similarity calculation method, Euclidean distance similarity calculation method, etc.) usually do not fully consider the diversity and complexity of data. When the data is affected by Gaussian noise, the similarity calculation result may not be accurate enough, affecting the reliability and effectiveness of further analysis and application. Considering that the global model parameter update of this application is data with Gaussian noise added, in order to reduce the impact of noise in the process of calculating similarity, this application introduces a data point density weighting method to preprocess the global model parameter update.
[0062] Assuming that the similarity of the global model parameter updates of the tth round and the t'th round is calculated using the tensor similarity algorithm, the execution process of the tensor similarity algorithm of the present application includes:
[0063] Step A1, respectively calculate the data point density for the global model parameter update in round t and round t'. Data points in high-density areas often reflect the main features or patterns in the data set, while data points in low-density areas may represent noise or outliers, which have less impact. Therefore, when calculating similarity, giving data points in high-density areas a greater weight can make the result more consistent with the overall trend of the data, effectively reduce the impact of noise on similarity calculations, and reduce the misleading caused by outliers.
[0064] Further preferably, the data point density of the global model parameter update in the tth round is The calculation formula is:
[0065]
[0066] Among them, Δ t represents the global model parameter update of the tth round, n represents the number of elements of the global model parameter update of the tth round, and the global model parameter update of the tth round j represents the index of the element where the global model parameters are updated, x j Represents the jth element value of the global model parameter update in the tth round, and the jth tensor is constructed h represents the smoothing parameter, and K(·) represents the Gaussian kernel function. Gaussian kernel function:
[0067]
[0068] Here, the Gaussian function variable n represents the data dimension of Gaussian function variable y. ||y|| represents the norm of y.
[0069] Similarly, the data point density of the global model parameter update in round t' is obtained according to the above formula:
[0070] Step A2: Update the data point density using the global model parameters of the tth round Weighted processing of the global model parameter update Δ in round t t , obtain the weighted global model parameter update g for round t t :
[0071]
[0072] Step A3: Update the data point density using the global model parameters of round t' Weighted processing of the global model parameter update Δ in round t' t′ , obtain the weighted global model parameters of round t'
[0073] Number of updates t′ :
[0074]
[0075] Step A4, obtain the weighted global model parameter update g for round t t and the weighted global model parameter update g in round t' t′ The cosine similarity cos(g t ,g t′ ), the cosine similarity cos(g t ,g t′) as the similarity s(g) between the global model parameter update in round t and the global model parameter update in round t' t ,g t′ ), where t and t' are both rounds, and t≠t'.
[0076]
[0077] The similarity s(g t ,g t′ ), the result range is [-1, 1]. s(g t ,g t′ ) is closer to 1, indicating a higher similarity, that is, the two tensors are more similar; and the closer to -1, indicating a lower similarity, that is, the two tensors are less similar.
[0078] In this implementation, data points in high-density areas often reflect the main features or patterns in the data set. In comparison, data points in low-density areas may represent noise or outliers and have less impact. Therefore, by preprocessing the global model parameter update using the data point density weighting method, the weight of high-density data points can be increased, which can effectively reduce the impact of noise on similarity calculations, reduce the misleading caused by outliers, and improve the accuracy and stability of the calculation results.
[0079] In a preferred embodiment, if a large change occurs during model training, it means that the model has learned more information during training, including privacy or sensitive data. In this case, adding a large amount of noise can blur the data and reduce the risk of data privacy leakage. On the contrary, when the model changes less, it means that it is relatively stable, which means that the model has learned some laws and patterns in the data. In this case, it is not necessary to introduce a large amount of noise to protect private data. Therefore, in order to balance the relationship between privacy protection and model performance and improve model performance, this application introduces adaptive differential privacy. To achieve adaptive differential privacy, preferably, the local model parameter update of this round is perturbed to obtain the local model parameter noise update of this round, including:
[0080] Step B1: Based on the similarity between the global model parameter update of this round and the global model parameter update of the previous round (which can be expressed as s(Δ t ,Δ t-1 ))Get the dynamic noise figure.
[0081] Combined similarity s(Δ t ,Δ t-1) can reflect the changes of the model during the training stage. A high similarity indicates that the model has not changed much during the training process, and a low similarity indicates that the model has changed significantly during the training process. The present application designs a dynamic noise coefficient β, which can realize the function of real-time adjustment of noise addition. Further preferably, the calculation formula of the dynamic noise coefficient β is:
[0082] β=e -ks
[0083] Where k represents the decay rate, k∈(0,1); s represents the similarity s(Δ t ,Δ t-1 ).
[0084] Step B2, adding Gaussian noise to the local model parameter update of this round to obtain the local model parameter noise update of this round, wherein the variance of the Gaussian noise is the preset variance σ 2 The product of the dynamic noise coefficient, the Gaussian noise is expressed as: N(0,βσ 2 ).
[0085] In this embodiment, in order to prevent gradient explosion, the upper limit of the gradient (model parameter update) is set to ensure that the size of the gradient (model parameter update) does not exceed the set clipping threshold C. Further preferably, step B2, adding Gaussian noise to the local model parameter update of this round to obtain the local model parameter noise update of this round, includes:
[0086] Step B21, based on the clipping threshold, the local model parameter update of this round is clipped to obtain the clipped local model parameter update. Specifically, the local model parameter update δ of the tth round is clipped according to the following clipping formula: t To perform cropping:
[0087]
[0088] in, represents the clipping local model parameter update, C represents the clipping threshold, ||δ t ||2 means δ t 2-norm of , max(,) represents the maximum value function.
[0089] Step B22, add Gaussian noise to the cropped local model parameter update to obtain the local model parameter noise update of this round, where the variance of the Gaussian noise is the product of the preset variance, the dynamic noise coefficient, the square of the cropping threshold, and the unit matrix. Suppose the current round is t, then the local model parameter noise update of the tth round is It is expressed as:
[0090]
[0091] Where I is the identity matrix.
[0092] In this embodiment, adaptive differential privacy can be performed in the order of steps B1, B21, and B22, or in the order of steps B21, B1, and B22.
[0093] The present invention also discloses a federated learning method based on local differential privacy. The method is executed by a server and multiple user terminals in cooperation, referring to Figure 4 , this method iterates to perform multiple rounds of training until the training stop condition is reached, and executes in each round of training:
[0094] Step 1: At least one user terminal obtains the local model parameter noise update of this round by executing the above-mentioned federated learning method based on local differential privacy applied to the user terminal, and uploads the local model parameter noise update of this round to the server terminal. All or part of the multiple user terminals execute the above-mentioned federated learning method based on local differential privacy applied to the user terminal.
[0095] Step 2: The server aggregates the local model parameter noise updates of this round uploaded by multiple users to obtain the global model and global model parameter updates of the next round.
[0096] The training stop condition is preferably, but not limited to, the number of training rounds reaching a round number threshold.
[0097] The present invention also discloses a model training system based on the above-mentioned federated learning method based on local differential privacy. In a preferred embodiment, the system includes a server side and multiple user sides respectively connected to and communicating with the server side.
[0098] In this embodiment, the user end is preferably but not limited to a user terminal device, a personal computer, a tablet computer, etc. The server end is preferably but not limited to a computer cluster or a cloud server.
[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0100] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
Claims
1. A federated learning method based on local differential privacy, It is characterized in that Corresponding to the user end, the method includes: The user side obtains the global model and global model parameter updates of this round from the server side; The tensor similarity algorithm is used to obtain the similarity between the global model parameter update of this round and the global model parameter update of the previous round; If the similarity is less than the backtracking threshold, the global model of this round is trained by local data to obtain the local model parameter update of this round, and the local model parameter update of this round is perturbed by data to obtain the local model parameter noise update of this round, including: The dynamic noise coefficient is obtained based on the similarity between the global model parameter update of this round and the global model parameter update of the previous round. The calculation formula of the dynamic noise coefficient β is: β=e -ks Where k represents the decay rate; s represents the similarity between the global model parameter update of this round and the global model parameter update of the previous round; Adding Gaussian noise to the local model parameter update of this round obtains the local model parameter noise update of this round: including: Performing clipping processing on the local model parameter update of this round based on the clipping threshold to obtain clipped local model parameter update; Adding Gaussian noise to the cropped local model parameter update to obtain the local model parameter noise update of this round, wherein the variance of the Gaussian noise is the product of a preset variance, a dynamic noise coefficient, a square of a cropping threshold, and a unit matrix, or the variance of the Gaussian noise is the product of a preset variance and a dynamic noise coefficient; If the similarity is greater than or equal to the backtracking threshold, the historical global model parameter update that is most similar to the global model parameter update of this round is selected from the historical global model parameter update set, and the local model parameter noise update of the round to which the selected historical global model parameter update belongs is used as the local model parameter noise update of this round; Upload the local model parameter noise update of this round to the server.
2. A federated learning method based on local differential privacy as claimed in claim 1, It is characterized in that The execution process of the tensor similarity algorithm includes: Calculate the data point density of global model parameter updates for round t and round t′ respectively; The global model parameter update of the tth round is weighted by using the data point density of the global model parameter update of the tth round to obtain the weighted global model parameter update of the tth round; The global model parameter update of the t′th round is weighted by using the data point density of the global model parameter update of the t′th round to obtain the weighted global model parameter update of the t′th round; Obtaining the cosine similarity between the weighted global model parameter update of the tth round and the weighted global model parameter update of the t′th round, and using the cosine similarity as the similarity between the global model parameter update of the tth round and the global model parameter update of the t′th round; Where t and t′ are both rounds, and t≠t′; Among them, the data point density of the global model parameter update in round t is The calculation formula is: Among them, Δ t represents the global model parameter update of the tth round, n represents the number of elements of the global model parameter update of the tth round, and the global model parameter update of the tth round j represents the index of the element where the global model parameters are updated, x j Represents the jth element value of the global model parameter update in the tth round, and the jth tensor is constructed h represents the smoothing parameter, and K(·) represents the Gaussian kernel function.
3. A federated learning method based on local differential privacy as claimed in claim 1, It is characterized in that The selecting, from the historical global model parameter update set, the historical global model parameter update that is most similar to the global model parameter update of this round includes: Calculate the similarity between the global model parameter update of this round and each historical global model parameter update in the historical global model parameter update set using the tensor similarity algorithm; The historical global model parameter update with the greatest similarity is selected as the most similar historical global model parameter update.
4. A federated learning method based on local differential privacy as described in any one of claims 1 to 3, It is characterized in that Based on the clipping threshold, the local model parameters of the tth round are updated according to the following clipping formula: t To perform cropping: in, represents the clipping local model parameter update, C represents the clipping threshold, ||δ t || 2 Denotes δ t The 2-norm of , max(,) represents the maximum value function.
5. A federated learning method based on local differential privacy, It is characterized in that The method is executed by the cooperation of a server and multiple client terminals, and iteratively executes multiple rounds of training until a training stop condition is reached, and executes in each round of training: At least one user terminal obtains the local model parameter noise update of this round by executing the method steps of one of claims 1 to 4, and uploads the local model parameter noise update of this round to the server terminal; The server aggregates the local model parameter noise updates of this round uploaded by multiple users to obtain the global model and global model parameter updates for the next round.
6. A model training system based on the federated learning method based on local differential privacy as described in claim 5, It is characterized in that It includes a server end and a plurality of user ends respectively connected to and communicating with the server end.
Citation Information
Patent Citations
Federal learning method and system combined with personalized differential privacy
CN113344217A
Federal recommendation method and device for improving privacy and robustness, equipment and storage medium
CN115455471A
Hospital privacy data noise adding and optimization protection method based on federated learning
CN117493877A
Personalized federal learning method and system based on shuffling differential privacy protection
CN117932683A
Federal learning security and privacy method oriented to user behaviors
CN118013574A
