Trusted compressive sensing and shuffling differential privacy federated learning method

Abstract

The application provides a trusted compression sensing and shuffle differential privacy federated learning method, which comprises the following steps: each client dynamically updates a probability according to a self-sampling strategy, and independently and randomly decides whether to participate in the training round according to the probability; the client participating in the training round updates a local model in combination with a local data set and global model parameters of the last round; the client performs adaptive disturbance on the updated local model parameters to obtain disturbed local model parameter update values; compression sensing is performed on the local model parameter update values to obtain compressed data; the compressed data is sent to a shuffler, the shuffler uploads the shuffled compressed data to a center server; the center server aggregates the received data, reconstructs a model according to the aggregated data to obtain a global model of the current training round, and then distributes the global model to relevant clients. The application can reduce communication overhead while protecting user privacy, and is particularly suitable for solving distributed large-scale problems.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and more specifically to a trusted compressed sensing and shuffling differential privacy federated learning method. Background Technology

[0002] With the rapid development of big data, mobile internet, and artificial intelligence technologies, many organizations and institutions face the challenge of processing massive amounts of data, and data privacy protection is another important issue they face. The data silo problem is one manifestation of this, referring to the phenomenon where data held by different institutions or individuals is separated, hindering effective data interaction and sharing. This situation makes it difficult to integrate and analyze data across institutions and fields.

[0003] To address this problem, federated learning emerged. Federated learning is a machine learning approach that allows multiple participants to collaboratively train a model without sharing the original data. Instead, each participant trains a local model using their own data and then only shares model updates with other participants in the network or on a central server. This approach is particularly suitable for data-sensitive or geographically dispersed scenarios because it allows participants to benefit from the network's collective knowledge while maintaining control over their own data. Federated learning has been applied across various fields, including healthcare, finance, and telecommunications, and has the potential to transform how machine learning is practiced.

[0004] However, existing technologies still present other privacy and security issues in federated learning, such as gradient leakage. In federated learning, each participant calculates gradients and shares them with other participants for global model updates; however, since gradients contain private information, attackers can analyze them to infer the original data, thus compromising privacy protection.

[0005] To address this issue, existing technologies have proposed various methods, such as differential privacy, secure multi-party computation, and homomorphic encryption, to protect privacy in federated learning. Among these, differential privacy is one of the most commonly used privacy protection techniques and has been widely applied in federated learning. Differential privacy mainly comes in two forms: centralized differential privacy and local differential privacy. Compared to centralized differential privacy, local differential privacy is more decentralized, requiring no trusted server as a central node for privacy protection. In local differential privacy, participants locally add noise or perturbation to their data, making the data obfuscated and thus protecting its privacy. Furthermore, local differential privacy offers good flexibility and scalability, making it suitable for different data processing scenarios and application areas.

[0006] However, for commonly used Gaussian noise, Laplace noise, etc., since the noise source follows a random distribution, the sampling noise is random, and its magnitude is unpredictable and uncontrollable; this may cause some weights to be over-perturbed, which will hinder model training and impair the accuracy of the model. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a trustworthy compressed sensing and shuffling differential privacy federated learning method to solve the technical problem in existing technologies where weights are excessively perturbed during federated learning, hindering model training and impairing model accuracy.

[0008] The technical solution adopted in this invention is as follows:

[0009] Firstly, a trustworthy compressed sensing and shuffling differential privacy federated learning method is provided, including:

[0010] Each client dynamically updates its probabilities according to a self-sampling strategy and independently and randomly decides whether to participate in this round of training based on the probabilities. Clients participating in this round of training update their local models by combining their local datasets and the global model parameters from the previous round. The clients adaptively perturb the updated local model parameters to obtain perturbed local model parameter update values. The local model parameter update values ​​are then compressed using compression sensing to obtain compressed data. The compressed data is sent to a shuffling unit, which shuffles the compressed data and uploads it to the central server. The central server aggregates the received data, reconstructs the model based on the aggregated data to obtain the global model for this round of training, and then distributes it to the relevant clients.

[0011] Furthermore, the probabilities are dynamically updated according to the self-sampling strategy, including:

[0012] The client calculates the cosine similarity and Manhattan distance between its own model parameters and the updated global model parameters in the previous round; and dynamically updates the probability based on the cosine similarity and Manhattan distance.

[0013] Furthermore, based on cosine similarity and Manhattan distance, the probabilities are dynamically updated as follows:

[0014]

[0015] In the above formula, cos represents cosine similarity, and norm_dis represents normalized Manhattan distance.

[0016] Furthermore, the clients participating in this training round update their local models by combining their local datasets with the global model parameters from the previous round, including:

[0017] The client performs secondary sampling on the local dataset to obtain a subset of the dataset;

[0018] Based on the subset of data and the current gradient, gradient descent is performed to obtain the updated local model.

[0019] Furthermore, the client adaptively perturbs the updated local model parameters, including perturbing the local model parameters according to the range of model weight changes in each layer of the deep neural network.

[0020] Furthermore, based on the range of model weight changes in each layer of the deep neural network, the local model parameters are perturbed, including:

[0021] Find the maximum value of the weights in the l-th layer of a deep neural network. l and minimum value min l ;

[0022] Calculate center c according to the following formula l and r l :

[0023]

[0024] r l =max l -c l

[0025] Calculate w and center c l Offset μ:

[0026] μ = wc l

[0027] The following formula is used to perturb w:

[0028]

[0029] Where w* is the perturbed updated model parameter value, wp means "with probability", and ∈ represents the privacy budget.

[0030] Furthermore, compressed sensing is applied to the local model parameter update values, including:

[0031] TopK sparsification is applied to the updated parameters of the local model.

[0032] The model parameters are randomly projected using a Gaussian random matrix to obtain compressed data after compressed sampling.

[0033] Furthermore, after the shredder shreds the compressed data, it first anonymizes it before uploading it to the central server.

[0034] Furthermore, the central server receives data sent by the shuffler, uses the FedAvg algorithm to update the global model, and uses an iterative hard thresholding algorithm to reconstruct the model.

[0035] Secondly, an electronic device is provided, comprising:

[0036] One or more processors;

[0037] Storage device for storing one or more programs;

[0038] When one or more programs are executed by one or more processors, the one or more processors implement the federated learning method provided in the first aspect.

[0039] As can be seen from the above technical solution, the beneficial technical effects of the present invention are as follows:

[0040] It can protect user privacy while reducing communication overhead, helping to reduce data transmission and communication costs, and ensuring the security and efficiency of privacy during collaborative model training across multiple clients; it is particularly suitable for solving distributed large-scale problems, thereby protecting the model from privacy threats. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0042] Figure 1 This is a flowchart illustrating the federated learning method in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the accuracy comparison of different CR values ​​on MNIST / Fashion-MNIST during experiments in this embodiment of the invention. Figure 1 ;

[0044] Figure 3 This is a schematic diagram illustrating the accuracy comparison of different CR values ​​on MNIST / Fashion-MNIST during experiments in this embodiment of the invention. Figure 2 ;

[0045] Figure 4 This is a schematic diagram illustrating the accuracy comparison of different CR values ​​on MNIST / Fashion-MNIST during experiments in this embodiment of the invention. Figure 3 ;

[0046] Figure 5 This is a schematic diagram comparing the impact of the experiment on convergence speed on MNIST / Fashion-MNIST in the embodiments of the present invention. Figure 4 . Detailed Implementation

[0047] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0048] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by those skilled in the art to which this invention pertains.

[0049] Example

[0050] This embodiment provides a trustworthy compressed sensing and shuffling differential privacy federated learning method, the hardware and software facilities used in this method include:

[0051] N clients: Each client i (i∈[1,N]) has a local dataset Di, and obtains model updates by training using the local dataset and the global model from the previous round. These updates are uploaded to the shuffler after perturbation and compressed sampling.

[0052] Washer: The washer is responsible for receiving data uploaded by users. Its main responsibility is to wash the data uploaded by the client by random arrangement, and then send the washed data to the central server.

[0053] Central server: The central server receives the shuffled data sent by the shuffler, reconstructs the model, and then sends it to the relevant clients.

[0054] A trustworthy compressed sensing and shuffled differential privacy federated learning method includes the following steps:

[0055] S1. Each client dynamically updates its probability q according to the self-sampling strategy, and independently and randomly decides whether to participate in this round of training based on probability q.

[0056] In each round of training in federated learning, to avoid potential privacy threats from the server specifying which clients participate in each round, each client first performs self-sampling, independently and randomly deciding whether to participate in the current round based on probability q. To focus on clients whose contributions are more consistent with the global model and improve the overall convergence and accuracy of the FL process, this embodiment proposes a client self-sampling strategy with dynamically updated probabilities. This strategy adjusts the probabilities based on the similarity between locally uploaded model parameters and global parameters. The value of q is dynamically updated throughout the training process based on the correlation between the model parameters uploaded by the client in the previous rounds and the global parameters. Clients with higher similarity between their uploaded parameters and the global parameters are more likely to be selected to participate.

[0057] The probability q is dynamically updated according to the self-sampling strategy, including: the client calculates the cosine similarity and Manhattan distance between its own model parameters and the updated global model parameters in the previous round, and updates the probability q dynamically based on the cosine similarity and Manhattan distance.

[0058] Cosine similarity measures the similarity between two vectors based on their directions, ranging from -1 to 1. The cosine similarity reaches its maximum value of 1 when the two vectors are in the same direction; it reaches its minimum value of -1 when their directions are completely opposite; and it is 0 when their directions are perpendicular. Therefore, cosine similarity can be used to evaluate the similarity between two vectors and their directional relationship.

[0059] The Manhattan distance is used to measure the difference between two model parameter vectors. The Manhattan distance measures the difference between two vectors by calculating the absolute distance between them; it is calculated as the sum of the absolute differences between their corresponding elements. A smaller Manhattan distance indicates that the two vectors are closer in all dimensions, while a larger Manhattan distance indicates that they are more different in all dimensions.

[0060] For each client i∈U t-1 We calculate separately and ΔW t Cosine similarity between them:

[0061]

[0062] In the above formula, This indicates that client i updated the model parameters in the previous upload, ΔW. t-1 This represents the global parameter update in the (t-1)th round.

[0063] And the distance to Manhattan:

[0064]

[0065] In the above formula, n represents the model parameters. and ΔW t-1 The corresponding vector dimension. j represents the j-th layer in the model parameters. This represents the parameter value of layer j in the model parameter update values ​​uploaded by client i in the previous round. This represents the parameter value of the j-th layer in the global model update of round (t-1).

[0066] The Manhattan distance is normalized by limiting the distance to the range [0,1], resulting in the normalized Manhattan distance norm_dis:

[0067]

[0068] Based on cosine similarity and Manhattan distance, the probabilities are dynamically updated as follows:

[0069]

[0070] In the above formula, cos represents cosine similarity, and norm_dis represents normalized Manhattan distance. At the beginning of each training round, each client randomly generates a 0 or 1 based on probability; if it is 1, it participates in this round of training. Let U be the set of clients participating in training in round t. t .

[0071] When dynamically updating probabilities using a self-sampling strategy, cosine similarity and Manhattan distance are used to jointly calculate the similarity between the two parameters. Cosine similarity and Manhattan distance measure the similarity and difference between model parameters from different perspectives. Combining the two methods can comprehensively consider the directional and numerical differences of model parameters, providing a more comprehensive evaluation result. Furthermore, the performance of the two indicators may differ under different circumstances. At the same time, using two indicators can increase the robustness to differences in model parameters and reduce the bias that a single indicator may introduce, thereby improving the accuracy and reliability of the evaluation.

[0072] S2. Clients participating in this training round update their local models by combining their local datasets with the global model parameters from the previous round.

[0073] For client i that is determined to participate in this round of training, firstly, client i∈U t The local dataset Di is sampled a second time to obtain a subset Ci. Based on the subset Ci and the current gradient W... t-1 W t-1 Gradient descent is performed to obtain the updated local model. The expression is:

[0074]

[0075] In the above formula, W t-1 Di represents the global model parameters from the previous round, and Di represents the client's local dataset.

[0076] S3. The client adaptively perturbs the updated local model parameters to obtain the perturbed updated values ​​of the local model parameters.

[0077] Considering that common sampling noises such as Gaussian noise and Laplace noise are random and have unpredictable and uncontrollable magnitudes, and that the model weights of different layers in a deep neural network can change significantly, this embodiment perturbs the local model parameters according to the range of model weight changes for each layer. By perturbating the trained local model The model parameter update for the i-th client in round t was obtained. As shown below:

[0078]

[0079] for First, find the maximum weight of layer l, max. l and minimum value min l Then calculate center c l and r l :

[0080]

[0081] r l =max l -c l

[0082] Each w in the range [c l -r l ,c l +r l [Inside.] Then on Perform the following operations:

[0083] First, calculate w and the center c. l Offset μ:

[0084] μ = wc l

[0085] Then perturb w:

[0086]

[0087] w* represents the perturbed model parameter update value, wp stands for "with probability", ∈ represents the privacy budget, e is a mathematical constant e = 2.71828, and μ represents the offset.

[0088] By implementing the technical solution in this step, the client-side training results are perturbed layer by layer in the neural network, thereby achieving local differential privacy. This method can perturb the model weights of different layers of the neural network layer by layer, adjusting them to the same range, thus protecting privacy while reducing the impact on accuracy.

[0089] S4. Perform compressed sensing on the local model parameter update values ​​to obtain compressed data.

[0090] In addition to privacy issues, federated learning also faces the problem of communication overhead. This is because federated learning requires the transmission of model parameters among multiple devices for model training, and these devices are located in different positions and may be owned by different organizations or individuals, so communication is required to transmit data. This communication process consumes a large amount of bandwidth and time, especially when federated learning involves large-scale datasets and complex models, the communication overhead becomes even more significant. Directly transmitting data to the server will result in a large communication volume, and the compressive sensing technology is used in this embodiment to reduce the communication volume.

[0091] Compressive sensing is a signal processing technology that aims to recover the original signal by using measurements far less than the traditional sampling rate. The core idea of compressive sensing is to utilize the sparsity of a signal and reconstruct the complete signal by randomly measuring a part of the signal. For a sparse signal, in this embodiment, let a one-dimensional discrete signal x with a length of N be regarded as an N×1 column vector in an N-dimensional space. If this column vector contains K non-zero elements and K<<N, the signal x is referred to as a K-sparse signal with sparsity.

[0092] The compressive sampling process obtains measurement values through an observation matrix, and the specific expression is:

[0093] y=Φx

[0094] Where Φ is the observation matrix with a dimension of M×N; x is the original signal with a dimension of N×1; and y is the observation signal with a dimension of M×1.

[0095] To implement compressive sensing, signal data needs to be sparsified to meet the requirement of signal sparsity for compressive sensing. In a specific implementation, the value of the K-th element sorted by the absolute value of parameters from large to small is calculated first, all parameter values greater than or equal to this value are retained, and the remaining positions are set to zero, so as to obtain the sparsified

[0096]

[0097] In the compressive sampling process, random projection is performed on the original data using a Gaussian random matrix to obtain a set of sampling values. These sampling values can be regarded as linear combinations of the original data. By selecting an appropriate sampling matrix, useful information of data can be obtained with high sampling efficiency. A random measurement matrix is used at the client to compress the perturbed model parameter update values into measurement values, and the expression of this process is:

[0098]

[0099] Through this step, each client, while protecting its privacy, passes its local model update information to the shuffling unit, providing data for the next step of federated learning. This method not only protects user data privacy but also reduces the dimensionality of model parameters, improves computational efficiency, and provides useful information for the generation of the global model.

[0100] S5. The compressed data is sent to the washing unit, which then washes the compressed data and uploads it to the central server.

[0101] When the shuffler receives updated model parameter values ​​uploaded from multiple clients, it performs a series of operations to protect user data privacy. First, the shuffler shuffles this data, that is, it rearranges their order to remove identifiers associated with specific users.

[0102] Furthermore, in some embodiments, the shredder also anonymizes the data to further protect user privacy. This process removes any personal information or characteristics that could be used to identify a particular user.

[0103] Finally, the shredder uploads the processed data to the central server, making it impossible for the central server to determine which data comes from specific users.

[0104] By implementing this approach, user data privacy is effectively protected, and centralized model training and updates are possible while maintaining privacy.

[0105] S6. The central server aggregates the received data, reconstructs the model based on the aggregated data to obtain the global model for this round of training, and then distributes it to the relevant clients.

[0106] The central server receives data from the shuffling unit and uses the FedAvg algorithm to update the global model during aggregation, obtaining the compressed global model parameter update Y. t :

[0107]

[0108] Among them, U t It is the client that participates in this round of training through the self-sampling mechanism in round t, {y i :i∈U t} is the collection of data uploaded by these clients that has been shuffled.

[0109] During the model reconstruction process, the Iterative Hard Thresholding (IHT) algorithm is used to evaluate Y. t Reconstruct the data to obtain R. t This leads to the updated global model W. t .

[0110] Rt =IHT(y,Φ)

[0111] W t =W t-1 +R t

[0112] By implementing this step, the central server can integrate the model parameter update values ​​from various clients to generate a global model for further training and application.

[0113] To illustrate the effectiveness of the technical solution provided in this embodiment, the following experiment is used for explanation:

[0114] The MNIST handwritten digit dataset and the Fashion-MNIST clothing image dataset were selected as experimental datasets to evaluate the accuracy of the model in different scenarios.

[0115] In the experiment, a Convolutional Neural Network (CNN) was used as the model architecture. This model includes two convolutional layers, a ReLU activation function, a max pooling layer, and two fully connected layers for classification. The dataset was standardized before the experiment. The number of clients N was set to 100, the self-sampling probability q = 0.3, and the learning rate was set to 0.1.

[0116] Accuracy tests were conducted on two datasets under different compression ratios and privacy budgets. Experimental results show that accuracy decreases to varying degrees as the compression ratio and privacy budget decrease. For example, with a compression ratio of 0.5, setting the privacy budget ∈ to 1, 2, and ∞ respectively... Figure 2 , Figure 3 , Figure 4 As shown, the accuracies on the MNIST dataset are 98.30%, 98.34%, and 98.48%, respectively; and the accuracies on the Fashion-MNIST dataset are 88.31%, 88.37%, and 88.44%, respectively. This indicates that the present invention can maintain high accuracy even after privacy protection operations using adaptive perturbation technology, demonstrating high practicality.

[0117] To address the communication overhead issue in federated learning, the impact of different algorithms and compression rates on convergence performance was tested. Figure 5 In the diagram, the horizontal axis represents the cumulative number of uploaded parameters when reaching that level of precision, which is the communication overhead. The impact of communication overhead on convergence speed is statistically analyzed: This invention exhibits significantly faster convergence speed; with the accumulation of rounds, only a relatively smaller number of uploaded parameters are needed to achieve convergence.

[0118] By employing the technical solution provided in this embodiment, a self-sampling strategy is first used to determine the clients participating in each round of training, ensuring that client selection does not raise privacy issues. Next, adaptive perturbations are applied to the model parameters to achieve privacy protection. Then, compressed sampling is performed to reduce communication overhead. Finally, shuffling is performed to prevent the server from linking to clients across different rounds. This technical solution can protect user privacy while reducing communication overhead, helping to reduce data transmission and communication costs, and ensuring the security and efficiency of privacy during collaborative model training across multiple clients. It is particularly suitable for solving distributed, large-scale problems, thereby protecting the model from privacy threats.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A trustworthy compressed sensing and shuffling differential privacy federated learning method, characterized in that, For use in a distributed image recognition terminal, the images include handwritten digit images or clothing images; The method includes: Each client dynamically updates its probabilities based on a self-sampling strategy, including: each client calculating the cosine similarity and Manhattan distance between its own model parameters from the previous round and the updated global model parameters; and dynamically updating the probabilities based on the cosine similarity and Manhattan distance in the following manner: In the above formula, cos represents the cosine similarity. Represents the normalized Manhattan distance; Whether to participate in this round of training is determined independently and randomly based on probability. The clients participating in this round of training update their local models by combining the local dataset with the global model parameters from the previous round. The client adaptively perturbs the updated local model parameters, adjusting the local model parameters according to the range of weight changes in each layer of the deep neural network, including: Find the maximum weight of the l-th layer of a deep neural network. and minimum value ; Calculation center according to formula and : calculate With the center offset : According to the formula Perform a disturbance: in, These are the updated model parameters after the perturbation, where wp stands for "with probability"; Indicates privacy budget; Obtain the updated values ​​of the local model parameters after the perturbation; Compressed sensing is applied to the updated values ​​of the local model parameters to obtain compressed data; The compressed data is sent to the washing machine, which then washes the compressed data and uploads it to the central server. The central server aggregates the received data, reconstructs the model based on the aggregated data to obtain the global model for this round of training, and then sends it to the relevant clients.

2. The federated learning method according to claim 1, characterized in that, The clients participating in this training round update their local models by combining their local datasets with the global model parameters from the previous round, including: The client performs secondary sampling on the local dataset to obtain a subset of the dataset; Based on the subset of data and the current gradient, gradient descent is performed to obtain the updated local model.

3. The federated learning method according to claim 1, characterized in that, Compressed sensing is applied to the updated local model parameters, including: TopK sparsification is applied to the updated parameters of the local model. The model parameters are randomly projected using a Gaussian random matrix to obtain compressed data after compressed sampling.

4. The federated learning method according to claim 1, characterized in that, The shredder shreds the compressed data, anonymizes it, and then uploads it to the central server.

5. The federated learning method according to claim 1, characterized in that, The central server receives data sent by the shuffler, updates the global model using the FedAvg algorithm, and reconstructs the model using an iterative hard thresholding algorithm.

6. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the federated learning method according to any one of claims 1-5.

Images