Adaptive privacy-preserving distributed learning method and apparatus based on attenuated noise disturbance

Abstract

The present invention relates to the technical field of distributed machine learning, and specifically relates to an adaptive privacy-preserving distributed learning method and apparatus based on attenuated noise disturbance. The method comprises: on the basis of a clipped sample gradient of a node, acquiring a local gradient of the node, wherein a clipping threshold of the node decreases with the increase of iteration rounds; injecting Gaussian noise into the local gradient, wherein the intensity of the Gaussian noise decreases stepwise with the increase of the iteration rounds; aggregating the local gradient of the node after injecting the Gaussian noise in each round of iteration, using the aggregated gradient to update a local model parameter, and broadcasting the updated local model parameter to an adjacent node for parameter update; and then aggregating the updated model parameter of the adjacent node for next iteration. The present invention effectively protects data privacy by adding noise, and ensures data accuracy by reducing the noise error.

Description

Adaptive privacy-preserving distributed learning method and device based on decaying noise perturbation TECHNICAL FIELD

[0001] The present application belongs to the technical field of distributed machine learning, and more particularly relates to an adaptive privacy-preserving distributed learning method and device based on decaying noise perturbation. BACKGROUND

[0002] With the rapid development and wide application of artificial intelligence technology, machine learning has become a powerful tool for solving various complex problems, and a large amount of personal data is used to train and optimize machine learning models. However, as the amount of data and the complexity of models continue to increase, people need more powerful computing resources to train models. Distributed machine learning allows simultaneous model training on multiple computing nodes, so it is a method that can meet the needs of large-scale data processing and complex model training.

[0003] However, since personal data often contains sensitive information such as identity information and medical records, using traditional distributed model training methods may face the risk of privacy leakage. For users, as data is shared and analyzed, personal privacy is at increasing risk.

[0004] In practice, computing nodes involved in distributed machine learning usually maintain isolation of their local data and cooperate only by sharing model parameters or gradient information. However, this information sharing poses a potential risk of data privacy leakage, as malicious parties can use this information to infer the sensitivity of the original data. Although this method helps protect user privacy to some extent, attackers may still be able to infer the original data from shared model parameters or gradients, posing a potential risk of privacy leakage. Therefore, how to strike a balance between data sharing and privacy protection has become an important research topic.

[0005] Chinese patent document CN116911382A discloses an asynchronous aggregation and privacy protection method in resource-limited federated edge learning. The method improves the delay compensation mechanism, compensates for model parameters within the model staleness threshold range, and uses a bell-shaped curve function for the decay coefficient. The larger the staleness, the faster the decay. For clients exceeding the staleness threshold, synchronization with the current global parameters is forced, and the next round of local training is entered.

[0006] The Chinese patent document CN115983598A discloses a microgrid privacy protection and energy scheduling method based on distributed deep reinforcement learning. First, the action network interacts with the local environment to obtain the corresponding action strategy, generates corresponding noise and independent power generation unit power, and then judges whether the constraint condition is met according to the environmental parameters and the selected action, and calculates the reward value. Subsequently, the action neural network and the value neural network extract historical data for learning, and finally the learned model is used to feedback the action selected by the action network, guiding the action network to pursue higher reward values.

[0007] The above scheme can realize privacy protection of parameters in the model training process.

[0008] Differential privacy (DP) is a data privacy protection method, which principle is to introduce controlled random perturbation to data to ensure that personal data will not leak sensitive information in the data analysis process. The core idea of this method is to add noise so that the specific content of the original data cannot be accurately inferred from the analysis results, thereby achieving effective protection of user privacy. DP provides a solution to balance the relationship between privacy protection and data analysis, allowing artificial intelligence systems to learn and extract valuable information from personal data while ensuring the security of personal privacy. In the privacy protection method based on perturbation, differential privacy is considered to have the highest security level.

[0009] In the field of machine learning, differential privacy is usually implemented by perturbing the gradient, and a common method is to add Gaussian noise proportional to the sensitivity of the data to introduce perturbation. Therefore, the size of the perturbation strength directly affects the accuracy of data analysis. Higher perturbation strength can provide better privacy protection effect, but will reduce the accuracy of data analysis, and vice versa, the level of privacy protection is closely related to the perturbation strength. However, due to the differences in characteristics and attributes of different data samples, two data samples may have a large sensitivity, resulting in excessive noise and seriously affecting the effect of model training. Therefore, how to find a balance between privacy protection effect and accuracy has become an urgent problem to be solved.

[0010] To solve this problem, differential privacy usually adopts gradient clipping technique to limit the sensitivity of gradient in the field of machine learning, that is, to adjust the gradient according to the pre-set clipping boundary. Through gradient clipping, the introduction of noise can be controlled to balance privacy protection and model training effect. However, when setting the clipping boundary, the relationship between noise variance and clipping bias needs to be balanced. A larger clipping boundary will introduce a larger noise variance, while a smaller boundary may cause deviation in the gradient direction. Since the errors caused by noise and gradient deviation cannot be avoided, a suitable balance point needs to be found when setting the clipping threshold, so that differential privacy can provide sufficient privacy protection without significantly reducing the utility of the model.

[0011] In the field of machine learning, this trade-off is an important issue in the application of differential privacy. Privacy protection, data utility and model convergence need to be considered comprehensively, but it is difficult to determine the appropriate clipping threshold in practical applications, and there is currently no effective method to solve this problem. Most of the existing researches mainly focus on symmetric undirected networks and only consider adding noise with constant variance in model updating, ignoring the possibility of time-varying adjustment of variance to reduce noise error.

[0012] Based on this, the present application designs an adaptive privacy-preserving distributed learning method based on decaying noise perturbation to solve the above problems. SUMMARY

[0013] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and provides an adaptive privacy-preserving distributed learning method based on decaying noise perturbation, which adds noise to effectively protect data privacy while reducing noise error to ensure data accuracy.

[0014] The present application also discloses a device loaded with an adaptive privacy-preserving distributed learning method based on decaying noise perturbation.

[0015] The detailed technical solutions of the present application are as follows:

[0016] An adaptive privacy-preserving distributed learning method based on decaying noise perturbation is applied to a distributed distributed learning system, the distributed learning system comprises a self-node and its adjacent nodes, and each node has a local data set for image classification and recognition task; the method comprises:

[0017] S1, based on the local data set, the sample gradient of the self-node in the current iteration round is obtained, and the sample gradient is clipped using the clipping threshold of the current iteration round, and the initial local gradient is calculated using the clipped sample gradient, wherein the clipping threshold is attenuated in a stepwise manner based on a set time interval during iteration;

[0018] S2, injecting Gaussian noise to the initial local gradient of the self-node to obtain an intermediate local gradient, wherein the strength of the injected Gaussian noise is adjusted by a noise strength coefficient, and the strength of the injected Gaussian noise is stepwise attenuated based on a set time interval in the iteration process;

[0019] S3, aggregating the intermediate local gradient of the self-node in the current iteration round and the target local gradient in the historical iteration round by using an adaptive aggregated history gradient method to obtain a target local gradient of the self-node in the current iteration round;

[0020] S4, calculating a learning rate in the current iteration round based on the noise strength coefficient, and updating the local model parameters of the self-node by using the aggregated target local gradient, and then transmitting the updated local model parameters to the adjacent node for model parameter updating;

[0021] S5, receiving the updated model parameters of the adjacent node by the self-node and aggregating to obtain the local model parameters in the next iteration round.

[0022] According to the application, in S1, the sample gradient of the self-node i in the tth iteration round is wherein, represents the debiased parameter of the self-node i in the tth iteration round, represents the sample of the self-node i in the tth iteration round sampled from the local data set D i of the self-node i;

[0023] The sample gradient is clipped by using the clipping threshold value of the current iteration round, specifically: when , there is:

[0024] In formula (1): represents the clipped sample gradient; represents the clipping threshold value of the tth iteration round; τ1 represents the first time interval;

[0025] wherein, the clipping threshold value is stepwise attenuated based on the set first time interval τ1 in the iteration process, that is:

[0026] In formula (2): represents the clipping threshold value in the next first time interval τ1; ψ represents the attenuation coefficient.

[0027] According to the application, in S1, the initial local gradient is calculated by using the clipped sample gradient:

[0028] In formula (3): denotes the initial local gradient of the ego node i in the tth iteration round; B denotes the total number of samples in the local dataset D of the ego node i i ; b denotes the bth sample in the local dataset D i .

[0029] According to the application, in S2, Gaussian noise is injected into the initial local gradient of the ego node to obtain an intermediate local gradient:

[0030] In formula (4), denotes the intermediate local gradient of the ego node i in the tth iteration round; denotes the second noise intensity coefficient of the ego node i in the tth iteration round, T denotes the total number of iterations, τ2 denotes the second time interval, and the second noise intensity coefficient decreases in a stepped manner based on the set second time interval τ2 during the iteration process; denotes Gaussian noise with an expectation of 0 and a variance of .

[0031] According to the application, S3 specifically includes:

[0032] determining whether the second noise intensity coefficient in the current iteration round is equal to the second noise intensity coefficient in the last iteration round, if then:

[0033] In formula (6), denotes the target local gradient of the ego node i in the tth iteration round; θ is a hyperparameter and takes a value in the range of [0, 1); denotes the target local gradient of the ego node i in the (t-1)th iteration round.

[0034] According to the application, in S4, the learning rate in the current iteration round is calculated based on the noise intensity coefficient:

[0035] In formula (9), η t denotes the learning rate in the tth iteration round; η denotes an initial learning rate; denotes the learning rate coefficient of the ego node i in the tth iteration round; denotes the first noise intensity coefficient of the ego node i in the tth iteration round, and the first noise intensity coefficient increases in a stepped manner based on the set second time interval τ2 during the iteration process;

[0036] and the target local gradient obtained by aggregation is used to update the local model parameters of the ego node, that is:​​

[0037] In formula (10), represents the updated intermediate model parameter of the self-node i;

[0038] According to the application, in the S5, the self-node receives the updated model parameters of the adjacent nodes and aggregates them to obtain the local model parameters in the next iteration round, in particular:

[0039] In formula (11), represents the local model parameter of the self-node i in the t+1th iteration round; n represents the number of nodes;

[0040] In another aspect of the application, a device for implementing an adaptive privacy-preserving distributed learning method based on decaying noise perturbation is provided, and the device comprises:

[0041] A gradient obtaining module is configured to obtain sample gradients of a self-node in a current iteration round based on a local data set, clip the sample gradients using a clipping threshold of the current iteration round, and calculate an initial local gradient using the clipped sample gradients, wherein the clipping threshold is attenuated in a stepwise manner based on a set time interval during the iteration process.

[0042] A gradient noise adding module is configured to add Gaussian noise to the initial local gradient of the self-node to obtain an intermediate local gradient, wherein the intensity of the added Gaussian noise is adjusted by a noise intensity coefficient, and the intensity of the added Gaussian noise is attenuated in a stepwise manner based on a set time interval during the iteration process.

[0043] A gradient aggregation module is configured to aggregate the intermediate local gradient of the self-node in the current iteration round and the target local gradient in the historical iteration round using an adaptive aggregation history gradient method to obtain the target local gradient of the self-node in the current iteration round.

[0044] ​​​​​​​​a parameter updating module, configured to calculate a learning rate in a current iteration round based on the noise intensity coefficient, update local model parameters of the self-node by using the aggregated target local gradient, and transmit the updated local model parameters to a neighboring node for model parameter updating;

[0045] a parameter aggregation module, configured to aggregate the updated model parameters of the neighboring node to obtain local model parameters of the self-node in a next iteration round.

[0046] In another aspect of the present application, an electronic device is also provided, comprising:

[0047] at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the adaptive privacy-preserving distributed learning method based on decaying noise perturbation as described above.

[0048] In another aspect of the present application, a machine-readable storage medium storing executable instructions is also provided, the instructions, when executed, causing the machine to perform the adaptive privacy-preserving distributed learning method based on decaying noise perturbation as described above.

[0049] Compared with the prior art, the present application has the following beneficial effects:

[0050] (1) The adaptive privacy-preserving distributed learning method based on decaying noise perturbation provided by the present application adopts noise variance and clipping threshold with time variation, and in the iteration process, the intensity of injected noise is reduced step by step, which can reduce the negative impact on the gradient direction, and the adaptive aggregation history gradient (AA) method is used to further reduce the noise error and improve the overall performance; at the same time, the noise variance and the learning rate are adjusted to provide personalized privacy protection level for each node.

[0051] (2) The method of the present application combines the Push-Sum technology and is suitable for general time-varying communication topology.

[0052] (3) The method of the present application first gives the convergence analysis of noise error and gradient staleness bias in a general communication topology with direction (asymmetry), sparsity and time variation. BRIEF DESCRIPTION OF DRAWINGS

[0053] Fig. 1 is a flowchart of the adaptive privacy-preserving distributed learning method based on decaying noise perturbation according to the present application.

[0054] Fig. 2(a) is a graph of the relationship between the value of parameter θ and the convergence effect when σ i = 5.

[0055] Fig. 2(b) is a graph of the relationship between the value of parameter θ and the convergence effect when σ iFig. 2(c) is a graph of the relationship between the value of parameter θ and the convergence effect when σ = 10.

[0056] Fig. 2(c) is a graph of the relationship between the value of parameter θ and the convergence effect when σ = 10. i Fig. 2(c) is a graph of the relationship between the value of parameter θ and the convergence effect when σ = 10.

[0057] Fig. 3(a) is a graph of the relationship between the value of second time interval τ2 and the convergence effect when σ = 5. i Fig. 3(a) is a graph of the relationship between the value of second time interval τ2 and the convergence effect when σ = 5.

[0058] Fig. 3(b) is a graph of the relationship between the value of second time interval τ2 and the convergence effect when σ = 10. i Fig. 3(b) is a graph of the relationship between the value of second time interval τ2 and the convergence effect when σ = 10.

[0059] Fig. 3(c) is a graph of the relationship between the value of second time interval τ2 and the convergence effect when σ = 15. i Fig. 3(c) is a graph of the relationship between the value of second time interval τ2 and the convergence effect when σ = 15.

[0060] Fig. 4 is a comparison experiment graph of the method of the present application and other methods. DETAILED DESCRIPTION

[0061] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0062] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0063] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a reference to the presence of a feature, step, operation, device, component, and / or combinations thereof.

[0064] The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0065] In view of the defects of the prior art, the present application provides an adaptive privacy-preserving distributed learning method based on decaying noise perturbation, named DP-ASGP, which can be used to solve non-convex optimization problems. The method can realize individualized privacy protection level for each node and is suitable for general time-varying network topology.

[0066] The adaptive privacy-preserving distributed learning method and device based on decaying noise perturbation of the present application will be further described below in conjunction with specific embodiments.

[0067] Embodiment 1,

[0068] Referring to FIG. 1, the embodiment provides an adaptive privacy-preserving distributed learning method based on decaying noise perturbation, which is applied to a distributed learning system. The distributed learning system includes a self node and its neighboring nodes, and each node has a local dataset for an image classification and recognition task.

[0069] For the distributed distributed learning system, there is no separate server, and each node included in the system simultaneously plays the role of a server and a worker node, and the nodes can directly interact with each other. In the embodiment, a time-varying directed communication network can be described, and the nodes can deliver information based on the communication network.

[0070] It is assumed that there is a time-varying directed graph G(t). For the time-varying directed graph G(t) = (V, ε(t)), where V represents a node set, and ε(t) represents a directed edge set at the tth iteration. A non-negative mixing matrix Q(t) ∈ R n×n is used to represent the time-varying directed graph G(t) at the tth iteration. i represents a self node, j represents a neighboring node of the self node i, and (i, j) ∈ ε(t), represents the weight of the neighboring node j to the self node i at the tth iteration, if represents that the neighboring node j sends a message to the self node i at the tth iteration; if represents that the neighboring node j does not send a message to the self node i at the tth iteration. For a non-negative mixing matrix Q(t), it is set to be column stochastic (i.e., the sum of each column is 1), that is, I d represents a unit vector. R n×n represents a set of all n × n order real matrices, and n represents the number of nodes.

[0071] The embodiment proposes a random gradient pushing method based on a time-varying directed topology structure. This method allows each node to arbitrarily select its mixing weight without being affected by other nodes. During the learning process, each node needs to maintain three variables: a model parameter x, a scalar pushing weight w, and a debiasing parameter z.

[0072] Based on the above communication environment, the method of the embodiment includes:

[0073] S1, based on the local dataset, a sample gradient of the self node in the current iteration round is obtained, and the sample gradient is clipped using a clipping threshold of the current iteration round, and an initial local gradient is calculated using the clipped sample gradient, wherein the clipping threshold is tapered in a stepwise manner based on a set time interval during iteration.

[0074] In the distributed learning system, each node has a local dataset Di In this embodiment, the local dataset D i contains B samples, which can be obtained from the CIFAR-10 dataset.

[0075] The CIFAR-10 dataset is a standard dataset widely used for image classification tasks. It was created by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton at the University of Toronto, Canada. The dataset contains 10 different classes of color images, with 6000 images per class, totaling 60000 color images of 32x32 pixels. The image classes include airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.

[0076] The CIFAR-10 dataset is divided into a training set and a test set, with the training set containing 50000 images and the test set containing 10000 images. Each image has pixel values ranging from 0 to 255, representing color intensity in the RGB color space. Each image has a corresponding label, which is an integer between 0 and 9, indicating the class it belongs to.

[0077] In specific application scenarios, the CIFAR-10 dataset can be used for distributed learning, such as collaborative learning among multiple research institutions or companies, jointly training an image classification model for recognizing and classifying images of common objects. Each participant trains locally using their own data subset and shares model updates through distributed learning without exchanging actual data. In this process, differential privacy techniques can be used to introduce noise to protect the data privacy of each participant, ensuring that individual or company-level data is not leaked, thereby improving the overall performance and security of the model. This method can be used to develop intelligent monitoring systems, object recognition systems in autonomous driving, and real-time image classification functions in augmented reality devices.

[0078] For the ego node i, it can use the local dataset D i to obtain its sample gradient in each iteration round. Let the total number of iterations be T, and in the tth iteration (t ∈ T), the sample gradient of ego node i in the tth iteration is where, denotes the de-biasing parameter of ego node i in the tth iteration, denotes the sample drawn by ego node i in the tth iteration from its local dataset D i .

[0079] Considering that the intensity of noise is related to the sensitivity of sample data, the sensitivity of sample data in the local data set is usually measured by using the two-norm of the gradient when adding Gaussian noise. Let G represent the maximum value of the gradient two-norm, and let Δ represent the clipping threshold after applying the clipping technique. In the case where the clipping technique is not applied, the added noise variance is (G / Δ) 2. In the case where the clipping technique is applied, the added noise variance is (G / Δ) 2 / 2. 2 The noise added to the gradient will significantly increase by a factor of 2, so it is usually necessary to use a clipping technique to control it. The clipping technique is essential in differential privacy.

[0080] In this embodiment, gradient clipping is only performed when the obtained sample gradient is greater than the clipping threshold in the current iteration round. That is, for the sample gradient of node i in the tthiteration round, the clipping threshold thereof is τ1 represents the first time interval; when there is:

[0081] In formula (1): denotes the clipped sample gradient; denotes the clipping threshold in the tthiteration round; and τ1 represents the first time interval.

[0082] Further, the clipping threshold in this embodiment is stepwise attenuated based on the set first time interval τ1 during iteration, that is, given the initial clipping threshold the clipping threshold is attenuated once every first time interval τ1, and thus there is:

[0083] In formula (2): denotes the clipping threshold in the next first time interval τ1; and ψ represents the attenuation coefficient.

[0084] It should be understood that multiple iterations can occur within each first time interval τ1.

[0085] In this step, the initial local gradient is calculated using the clipped sample gradient as:

[0086] In formula (3): denotes the initial local gradient of node i in the tthiteration round; B denotes the total number of samples in the local data set D i of node i; and b denotes the bthsample in the local data set D i .

[0087] S2, injecting Gaussian noise to the initial local gradient of the self-node to obtain an intermediate local gradient, wherein the strength of the injected Gaussian noise is adjusted by a noise strength coefficient, and the strength of the injected Gaussian noise is stepwise attenuated based on a set time interval during the iteration process.

[0088] In the differential privacy technology, a fixed noise positively correlated with the clipping threshold is usually added to the gradient during the iteration process. However, it can be observed that adding a noise with a larger strength to a smaller gradient value will affect the direction of the gradient. With the increase of the iteration number, the model parameters gradually approach the optimal value, and the gradient value also tends to zero. At this stage, if a noise with a larger strength is added to the gradient, the direction of the gradient will be affected to some extent, thereby increasing the negative impact of the noise error on the model accuracy.

[0089] To alleviate this problem, the embodiment proposes a learning method for attenuating noise disturbance. In the early stage of iteration, the value of the gradient is larger, and therefore a noise with a larger strength is added to the gradient. With the iteration of the model parameters gradually approaching the optimal value, the value of the gradient gradually tends to zero. Therefore, in this stage, a noise with a smaller strength is added to the gradient, which can reduce the negative impact on the direction of the gradient.

[0090] The Gaussian noise strength is defined as wherein denotes the first noise strength coefficient of the self-node i in the tth iteration, and the first noise strength coefficient is stepwise increased based on a set second time interval τ2 during the iteration process; and σ i denotes the standard deviation of the Gaussian noise added by the self-node i. By setting the value of the first noise strength coefficient , the Gaussian noise strength with time attenuation is introduced during the iteration process wherein denotes the second noise strength coefficient of the self-node i in the tth iteration, and the second noise strength coefficient is stepwise decreased based on a set second time interval τ2 during the iteration process. Then, the attenuated Gaussian noise strength

[0091] The embodiment gives an attenuation strategy: stepwise attenuation. The stepwise attenuation is updated according to the change interval of the step size.

[0092] In order to ensure the security of local privacy data and avoid the risk of leakage, Gaussian noise is injected into the gradient before the self-node client sends the gradient information obtained by local training to the server or adjacent node, so as to effectively protect the gradient data. That is, Gaussian noise is injected into the initial local gradient of the self-node to obtain the intermediate local gradient as follows:

[0093] In formula (4), represents the intermediate local gradient of the self-node i in the tth iteration; represents the second noise intensity coefficient of the self-node i in the tth iteration, T represents the total number of iterations, τ2 represents the second time interval, and the second noise intensity coefficient is reduced in a stepped manner based on the set second time interval τ2 during the iteration process; represents the Gaussian noise with an expectation of 0 and a variance of .

[0094] Based on the above formula, the strength of the injected Gaussian noise is also attenuated in a stepped manner.

[0095] Under a given privacy level, adding stronger noise at the beginning of training can accelerate the convergence of the model, because adding random perturbations to stochastic gradient descent during training can make the gradient quickly escape the saddle point. As the gradient becomes smaller and smaller, the influence of noise on the model gradually increases. Therefore, in the later stage of training, a smaller noise scale is needed to approach the optimization of the model.

[0096] Most documents assume that each client has the same privacy protection requirement, but considering the differences in different countries, laws or work backgrounds, this assumption is unreasonable in actual application. In actual situation, the privacy requirements of clients may be different. In addition, the same privacy protection level means that a large amount of privacy budget is wasted for some clients, which usually has a negative impact on the accuracy of the model. Therefore, the present embodiment proposes a reasonable noise allocation method without clipping, which allows each node to have individualized privacy protection level (∈ i , δ i )-DP, ∈ i is the privacy budget of the self-node i, δ i is the failure probability of the self-node i, and DP is the differential privacy.

[0097] Differential privacy is an effective technology to protect personal privacy and has been rigorously theoretically proved. However, when using differential privacy technology, a method needs to be found to balance the effect of privacy protection and the accuracy of data analysis. In differential privacy, Gaussian noise proportional to the sensitivity of data is generally added to achieve a preset privacy protection level, but this will reduce the accuracy of analysis. ​

[0098] To alleviate this contradiction, the method disclosed in the embodiment allows each node to have a personalized privacy protection level (ε i , δ i )-DP, through which the error caused by noise can be reduced as much as possible, so that the training process of the model can achieve ideal convergence effect.

[0099] Specifically, a randomized algorithm M satisfies (ε i , δ i )-DP if for any adjacent data sets D and D', and any subset S of the output set of the algorithm M, it has: Pr[M(D)∈S≤e ∈ ×Pr[M(D')∈S+δ, ∈ is the privacy budget, which controls the trade-off between privacy and accuracy, and smaller ∈ provides stronger privacy guarantee; δ is the failure probability.

[0100] That is, there exist constants c1 and c2, N is the total sample size of a given image data set (i.e. CIFAR-10 data set), and given the total number of iterations T, for any δ i >0, to achieve a privacy protection level (ε i , δ i )-DP, must satisfy σ i value, that is:

[0101] That is, only when the standard deviation σ i of the Gaussian noise added by the self-node satisfies the above formula, the required privacy protection level (ε i , δ i )-DP can be achieved.

[0102] S3, using the adaptive aggregation history gradient method to aggregate the intermediate local gradient of the self-node in the current iteration round and the target local gradient in the historical iteration round, to obtain the target local gradient of the self-node in the current iteration round.

[0103] In order to further reduce the error caused by noise and accelerate the convergence speed, the embodiment adopts the method of aggregating history gradient (AA) to perform iterative update, which can further reduce the noise variance through aggregation.

[0104] Generally, the way of aggregating historical gradients is to add or subtract small noise variance by aggregating historical gradients, however, historical gradients have staleness, that is, the farther away from the current time, the higher the staleness of the gradient, and the closer to the current time, the lower the staleness of the gradient. Stale gradients can damage model utility and even cause the training process to diverge. The embodiment proposes a ladder decay strategy to adaptively update stale gradients and reduce the impact of historical gradients on the algorithm.

[0105] Specifically, first, it is judged whether the second noise intensity coefficient in the current iteration round is equal to the second noise intensity coefficient in the last iteration round If , then:

[0106] In formula (6): represents the target local gradient of node i in the tth iteration; θ is a hyperparameter, and the value is [0, 1); represents the target local gradient of node i in the (t-1)th iteration.

[0107] It should be understood that if , then: That is, no aggregation operation is performed on the gradient, because if the noise intensity coefficients are not equal, the noise may become larger after aggregation.

[0108] The effect of adaptively updating stale gradients to reduce the impact of noise is significant. Specifically, in the tth iteration, it is assumed that the second time interval τ2 to which the iteration belongs contains (t-1) mod τ2 historical gradients. In this iteration, the gradient calculated in the current tth iteration (intermediate local gradient) is aggregated with the (t-1) mod τ2 historical gradients (target local gradient) in the time interval. Specifically, it is assumed that in the tth iteration, the second time interval τ2 belonging to the tth iteration includes (t-1) mod τ2 historical gradients, and the (t-1) mod τ2 historical gradients are aggregated with the gradient of the tth iteration to reduce the impact of noise error.

[0109] For the gradient aggregated in the tth iteration, the following analysis is performed: the AA method aggregates the (t-1) mod τ2 historical gradients with the gradient calculated in the tth iteration, which reduces the noise error. However, due to the staleness effect, each gradient updated by the node client contains stale error. Therefore, the historical gradients can be aggregated by setting a reasonable time interval, and the impact of stale gradients on the convergence effect can be controlled by setting the second time interval τ2.

[0110] In the theoretical derivation part, a method for setting the second time interval τ2 is given. It is assumed that the error between the historical gradient and the current gradient satisfies The gradient can be decomposed into three parts:

[0111] In formula (7), (8): denotes the estimated value of the aggregated target local gradient in the tth iteration; denotes the estimated value of the clipped intermediate local gradient in the tth iteration; denotes the deviation due to the staleness effect; denotes the error due to noise; h denotes the magnitude of noise reduction when the adaptive aggregation of historical gradient method is used.

[0112] It can be seen that there is a trade-off relationship between the staleness deviation error and the noise error. When the hyperparameter θ is small, the weight of the current gradient is large, that is, the staleness deviation error is small. When the hyperparameter θ is large, the noise error is small. When the hyperparameter θ = 0, the algorithm degenerates into ordinary gradient descent, in which the staleness deviation error is 0, and the noise error is maximum. When the hyperparameter θ is fixed, the larger the value of τ2, the smaller the noise error, but the staleness deviation error increases, and vice versa.

[0113] S4, based on the noise intensity coefficient, calculate the learning rate in the current iteration round, and update the local model parameters of the self-node using the aggregated target local gradient, and then pass the updated local model parameters to the adjacent node for model parameter update.

[0114] Given the initial learning rate η. Further consider the size of the learning rate coefficient η t , which has an important influence on the noise error. In this embodiment, the learning rate in the current iteration round is calculated based on the noise intensity coefficient as follows:

[0115] In formula (9): η t denotes the learning rate in the tth iteration; η denotes the initial learning rate; denotes the learning rate coefficient of the self-node i in the tth iteration.

[0116] It needs to be further explained here that by setting the learning rate η t to a reasonable value, it can be ensured that the learning rate coefficient β i t is greater than the first noise intensity coefficient , which can reduce the error value caused by noise, thereby optimizing the final convergence result under the same number of iteration rounds.

[0117] Set the learning rate coefficient In order to better reduce the noise error, according to the first noise intensity coefficient The learning rate coefficient is set to The theoretical analysis is as follows:

[0118] For In the proof process, the variance of the noise is Where d represents the dimension of the model parameter, and n represents the number of nodes; it can be seen that:

[0119] When Or , the variance of the noise is

[0120] And when , the variance of the noise is

[0121] Since the first noise intensity coefficient is a stepwise increasing function, it has:

[0122] Through the above theoretical analysis, it can be seen that through this setting, the error of the noise can be better reduced.

[0123] Then based on the stochastic gradient descent, the target local gradient obtained by aggregation is used to update the local model parameters of the self node, that is:

[0124] In equation (10): Indicates the updated intermediate model parameters of the self node i, which is used for exchange between nodes before each round of parameter aggregation, and is used for subsequent model parameter update. Indicates the local model parameters of the self node i before update.

[0125] S5, the self node receives the updated model parameters of its adjacent nodes and aggregates to obtain the local model parameters of the next iteration round.

[0126] It is known that in the learning process, each node needs to maintain three variables: model parameters x, scalar push weight w and debiasing parameter z.

[0127] Given the initial parameters, including: initial scalar push weight Initial model parameters Initial debiasing parameter The update rules of the three variables maintained by the node in the learning process are as follows:

[0128] ①Node push and receive parameters, including the self node i pushing parameters push to its neighboring node j, and receive from node i the parameters sent by its neighboring node j

[0129] ②Node updates parameters, that is, the node receives the updated model parameters of its neighboring nodes and aggregates to obtain the local model parameters of the next iteration round, specifically:

[0130] In formula (11): denotes the local model parameters of node i in the t+1th iteration round; n denotes the number of nodes; denotes the weight of neighboring node j to node i in the tth iteration round; denotes the intermediate model parameters of neighboring node j updated; denotes the scalar push weight of node i in the t+1th iteration round; denotes the scalar push weight of neighboring node j updated; denotes the debiasing parameter of node i in the t+1th iteration round.

[0131] It should be understood that the Push-Sum technique is applied here, and the push weight w can be used to control the size of the model parameters x, since is not necessarily 1, and can be a large value, so that is also large, so the debiasing parameter is introduced to control.

[0132] Through the above steps, assuming that the loss function of the local sample satisfies the following conditions: Lipschitz continuity and boundedness

[0133] Let the first noise intensity coefficient be proportional to the total number of iterations T, where p is a set value and

[0134] The learning rate coefficient η is the initial learning rate, set to O(·) represents an omitted constant factor.

[0135] When , L is the Lipschitz constant, and has:

[0136] In formula (12): denotes the loss function f on the local data set iGradient on the debiasing parameter; K is a constant; F0 represents the gap between the initial function value F(x0) and the optimal solution f * ; m represents the upper limit of the unbiased error of the local gradient; represents the initialized model parameters; a represents the bound between the local gradient and the global gradient; L represents the Lipschitz constant, which is used to represent the bound of the gradient change, that is, the maximum change rate of the gradient that does not change too fast; U is the noise error.

[0137] For the last two items of the convergence result (i.e. ), it can be seen that they are in equilibrium, when the second time interval τ2 is larger, the amplitude h will be smaller, and the error will increase, but the amplitude h will be limited by , so the size of the second time interval τ2 should be reasonably set.

[0138] Given a θ value, assuming a reasonable second time interval τ2 value, the amplitude h is fixed, and the goal now is to reduce the impact of the error as much as possible, but it is difficult to analyze the error directly, and its size is also difficult to define.

[0139] Assuming that the error d τ2 between the historical gradient and the current gradient satisfies , that is, the error is controlled by , and combining formulas (1) and (3), it can be seen that the size of the clipping threshold can be reasonably controlled to control and , thereby controlling , and further controlling the size of the error . For example, when and are limited (or clipped) by the same clipping threshold, it will be limited within the clipping threshold (or 0), greatly reducing the error caused by the error . In this way, the unquantifiable error can be converted into a quantifiable clipping threshold.

[0140] The effectiveness of the method is verified by the following specific experiment.

[0141] The method trains a convolutional neural network (CNN) model for image classification on the Cifar-10 dataset.

[0142] First, an ablation experiment is performed to test the relationship between the value of the hyperparameter θ and the convergence effect, respectively in σ i = 5, σi = 10, σ i = 15. The experimental results are shown in Figures 2(a), 2(b), and 2(c), respectively, where Figure 2(a) is a graph of the relationship between the value of the parameter θ and the convergence effect when σ i = 5; Figure 2(b) is a graph of the relationship between the value of the parameter θ and the convergence effect when σ i = 10; and Figure 2(c) is a graph of the relationship between the value of the parameter θ and the convergence effect when σ i = 15.

[0143] As can be seen from the above figures, the effect is better when the parameter θ = 0.7, θ = 0.5, and θ = 0.3.

[0144] Further, the influence of gradient staleness is analyzed, and a strategy for setting the second time interval τ2 is given. For It can be seen that the larger the second time interval τ2, the smaller the amplitude h, and the smaller the error. However, the amplitude h is limited by the constant term , so it is unreasonable to simply increase the second time interval τ2 under the limitation of the constant term. Therefore, the method gives a strategy for setting the second time interval τ2, that is, when the value of the given parameter θ is given, the limit is , and when the value of the second time interval τ2 satisfies , the value of the second time interval τ2 is taken as the time interval.

[0145] Assuming θ = 0.5, the magnitudes of and can be further calculated, and it can be obtained that when the second time interval τ2 ≥ 5, h remains basically unchanged, and at this time, if the second time interval τ2 is increased, the influence of gradient staleness will be aggravated. Therefore, the method performs experiments based on the second time interval τ2 under the conditions of σ i = 5, σ i = 10, and σ i = 15, and verifies the theoretical analysis. The experimental results are shown in Figures 3(a), 3(b), and 3(c), respectively, where Figure 3(a) is a graph of the relationship between the value of the second time interval τ2 and the convergence effect when σ i = 5; Figure 3(b) is a graph of the relationship between the value of the second time interval τ2 and the convergence effect when σ i = 10; and Figure 3(c) is a graph of the relationship between the value of the second time interval τ2 and the convergence effect when σ i = 15.

[0146] Finally, the method DP-ASGP is compared with the MAPA (Multi-Phase Asynchronous Federated Learning and Adaptive Differential Privacy) method and the A(DP)2 SGD (stochastic decentralized parallel stochastic gradient descent algorithm with differential privacy) method.

[0147] Set n = 10, p = 0.2, δ = 10 -5 , θ = 0.5, σ i = 10, ψ = 0.9, τ2 = 5, and the clipping threshold decay is performed every 100 rounds. The noise system of each time interval is set to be the first iteration of the time interval. The test accuracy is used to evaluate the performance of distributed learning. The comparison experiment results are shown in Figure 4. The experimental results show that, under the same number of iterations, the method DP-ASGP shows good convergence effect.

[0148] Embodiment 2,

[0149] The embodiment provides a device for implementing an adaptive privacy-preserving distributed learning method based on decaying noise perturbation, and the device comprises:

[0150] A gradient obtaining module is configured to obtain a sample gradient of a self-node in a current iteration round based on a local data set, clip the sample gradient using a clipping threshold of the current iteration round, and calculate an initial local gradient using the clipped sample gradient, wherein the clipping threshold is decayed in a stepwise manner based on a set time interval during iteration.

[0151] A gradient noise adding module is configured to add Gaussian noise to the initial local gradient of the self-node to obtain an intermediate local gradient, wherein the strength of the added Gaussian noise is adjusted by a noise strength coefficient, and the strength of the added Gaussian noise is decayed in a stepwise manner based on a set time interval during iteration.

[0152] A gradient aggregation module is configured to aggregate the intermediate local gradient of the self-node in the current iteration round and a target local gradient in a historical iteration round using an adaptive aggregation history gradient method to obtain a target local gradient of the self-node in the current iteration round.

[0153] A parameter updating module is configured to calculate a learning rate in the current iteration round based on the noise strength coefficient, update the local model parameters of the self-node using the aggregated target local gradient, and pass the updated local model parameters to a neighboring node for model parameter updating.

[0154] A parameter aggregation module is configured to aggregate the updated model parameters of the neighboring node to obtain local model parameters of the self-node in a next iteration round.

[0155] Embodiment 3,

[0156] ​The embodiments also provide an electronic device, comprising:

[0157] at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the adaptive privacy-preserving distributed learning method based on decaying noise perturbation as described above.

[0158] In the embodiments, the electronic device can include, but is not limited to, a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile computing device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable computing device, a consumer electronic device, and the like.

[0159] Embodiment 4,

[0160] The embodiments also provide a machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the adaptive privacy-preserving distributed learning method based on decaying noise perturbation as described above.

[0161] In particular, a system or apparatus equipped with a readable storage medium on which a software program code implementing the functions of any of the above embodiments is stored, and causing the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium can be provided.

[0162] In this case, the program code read from the readable medium itself can implement the functions of any of the above embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present specification.

[0163] Embodiments of the readable storage medium include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as a CD ROM, a CD R, a CD RW, a DVD ROM, a DVD RAM, a DVD RW, a DVD RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer or the cloud over a communication network.

[0164] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) embodying computer-readable program code.

[0165] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in the flowchart one or more blocks and / or in the block or blocks of the block diagram.

[0166] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in the flowchart one or more blocks and / or in the block or blocks of the block diagram.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in the flowchart one or more blocks and / or in the block or blocks of the block diagram.

[0168] Obviously, the above-described embodiments of the present application are only examples for clearly illustrating the technical solutions of the present application, and are not intended to limit the specific embodiments of the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An adaptive privacy-preserving distributed learning method based on decaying noise perturbation, applied to a distributed distributed learning system, the distributed learning system comprising a self-node and its neighboring nodes, and each node having a local dataset for an image classification and recognition task; characterized in that, The method comprises: S1, based on the local data set, obtaining the sample gradient of the self node in the current iteration round, and using the clipping threshold of the current iteration round to clip the sample gradient, and using the clipped sample gradient to calculate the initial local gradient, wherein the clipping threshold is stepwise decayed based on the set time interval in the iteration process; S2, injecting Gaussian noise into the initial local gradient of the self node to obtain an intermediate local gradient, wherein the strength of the injected Gaussian noise is adjusted by a noise intensity coefficient, and the strength of the injected Gaussian noise is stepwise decayed based on the set time interval in the iteration process; S3, using an adaptive aggregation history gradient method to aggregate the intermediate local gradient of the self node in the current iteration round and the target local gradient in the historical iteration round to obtain the target local gradient of the self node in the current iteration round; S4, calculating the learning rate in the current iteration round based on the noise intensity coefficient, and updating the local model parameters of the self node using the aggregated target local gradient, and then transmitting the updated local model parameters to the adjacent node for model parameter updating; S5, the self node receives the updated model parameters of the adjacent node and aggregates them to obtain the local model parameters of the next iteration round; In the S1, the sample gradient of the node i in the tth iteration is wherein denotes the debiasing parameter of the node i in the tth iteration round, denotes the sample drawn from the local data set D i of node i in the t-th iteration of the round clipping the sample gradient with the clipping threshold of the current iteration round, specifically: when At times, there are: In formula (1): denotes the clipped sample gradient; The clipping threshold of the tth iteration is represented by τt; τ1 represents the first time interval; wherein the clipping threshold is tapered in a stepwise manner based on a set first time interval τ1 during the iteration, i.e. In formula (2): The clipping threshold in the next first time interval τ1 is represented by τt+1; ψ represents the decay coefficient; In the S1, the initial local gradient is calculated using the clipped sample gradient as: in formula (3): denotes the initial local gradient of node i in the tth iteration; B denotes the total number of samples in the local dataset D of node i i ; b denotes the bth sample in the local dataset D i of node i In S2, the initial local gradient of the self-node is injected with Gaussian noise to obtain an intermediate local gradient: In formula (4): denotes the intermediate local gradient of node i in iteration t; denotes the second noise intensity coefficient of the node i in the tth iteration, T denotes the total number of iterations, τ2 denotes the second time interval, and the second noise intensity coefficient decrease stepwise based on a set second time interval τ2 during the iteration process; denotes that the expectation is 0, the variance is Gaussian noise of the self node i in the tth iteration. The S3 specifically comprises: determining a second noise intensity coefficient in the current iteration round the second noise intensity coefficient in the previous iteration round whether equal, if Then we have: In formula (6): denotes the target local gradient of node i in the tth iteration; θ is a hyperparameter, and takes a value in [0, 1); The target local gradient of the self node i in the tth iteration is represented by i.

2. The adaptive privacy-preserving distributed learning method based on decaying noise perturbation of claim 1, wherein, In the S4, the learning rate in the current iteration round is calculated based on the noise intensity coefficient as: In formula (9): η t denotes the learning rate in the tth iteration; η denotes an initial learning rate; denotes the learning rate coefficient of the node i in the tth iteration round; denotes the first noise intensity coefficient of the node i in the tth iteration round, and the first noise intensity coefficient The strength of the injected Gaussian noise is adjusted by a noise intensity coefficient, and the strength of the injected Gaussian noise is stepwise decayed based on the set time interval in the iteration process; And the target local gradient obtained by aggregation is used to update the local model parameters of the self-node, that is, In formula (10): representing updated intermediate model parameters of the self-node i; The local model parameters of the self node i before updating are represented by i.

3. The adaptive privacy-preserving distributed learning method based on decaying noise perturbation of claim 1, wherein, In S5, the node receives the updated model parameters of its neighboring nodes and aggregates them to obtain the local model parameters of the next iteration round, specifically: In formula (11): denotes the local model parameters of node i in the t+1th iteration; n denotes the number of nodes; Wtj(i) represents the weight of the edge from node j to node i in the tth iteration; denotes the updated intermediate model parameter of the neighboring node j; denotes the scalar push weight of node i in the t+1th iteration; denotes the updated scalar push weight of the neighboring node j; The debiased parameters of the self node i in the t+1th iteration are represented by i.

4. An apparatus for implementing a privacy-preserving distributed learning method based on adaptive noise perturbation, characterized in that, The device comprises: A gradient acquisition module is configured to obtain the sample gradient of the self node in the current iteration round based on the local data set, and clip the sample gradient using the clipping threshold of the current iteration round, and calculate the initial local gradient using the clipped sample gradient, wherein the clipping threshold is stepwise decayed based on the set time interval in the iteration process; A gradient noise injection module is configured to inject Gaussian noise into the initial local gradient of the self node to obtain an intermediate local gradient, wherein the strength of the injected Gaussian noise is adjusted by a noise intensity coefficient, and the strength of the injected Gaussian noise is stepwise decayed based on the set time interval in the iteration process; A gradient aggregation module is configured to use an adaptive aggregation history gradient method to aggregate the intermediate local gradient of the self node in the current iteration round and the target local gradient in the historical iteration round to obtain the target local gradient of the self node in the current iteration round; A parameter updating module is configured to calculate the learning rate in the current iteration round based on the noise intensity coefficient, and update the local model parameters of the self node using the aggregated target local gradient, and then transmit the updated local model parameters to the adjacent node for model parameter updating; A parameter aggregation module is configured to aggregate the updated model parameters of the adjacent node to obtain the local model parameters of the self node in the next iteration round. In the S1, the sample gradient of the node i in the tth iteration is wherein denotes the debiasing parameter of the node i in the tth iteration round, denotes the sample drawn from the local data set D i of node i in the t-th iteration of the round; clipping the sample gradient with the clipping threshold of the current iteration round, specifically: when At times, there are: In formula (1): denotes the clipped sample gradient; denotes the clipping threshold of the t-th iteration; τ1 denotes the first time interval; wherein the clipping threshold is stepwise attenuated during the iteration process based on a set first time interval τ1, i.e. In formula (2): denotes the clipping threshold of the next first time interval τ1; ψ denotes the decay coefficient; In the S1, the initial local gradient is calculated using the clipped sample gradient as: In formula (3): denotes the initial local gradient of node i in the tth iteration; B denotes the total number of samples in the local dataset D of node i i ; b denotes the bth sample in the local dataset D of node i i . In S2, the initial local gradient of the self-node is injected with Gaussian noise to obtain an intermediate local gradient: In formula (4): denotes the intermediate local gradient of node i in iteration t; denotes the second noise intensity coefficient of the node i in the tth iteration, T denotes the total number of iterations, τ2 denotes the second time interval, and the second noise intensity coefficient decrease stepwise based on a set second time interval τ2 during the iteration process; denotes that the expectation is 0, the variance is Gaussian noise with variance The S3 specifically comprises: determining a second noise intensity coefficient in the current iteration round the second noise intensity coefficient in the previous iteration round whether equal, if Then we have: In formula (6): denotes the target local gradient of node i in the tth iteration; θ is a hyperparameter, and takes a value in [0, 1); denotes the target local gradient of node i in the t-1-th iteration.

5. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the adaptive privacy-preserving distributed learning method based on the decaying noise perturbation as claimed in any one of claims 1 to 3.

6. A machine-readable storage medium, characterized in that, The machine-readable storage medium has stored thereon executable instructions that, when executed, cause the machine to perform the adaptive privacy-preserving distributed learning method based on the decaying noise perturbation as claimed in any one of claims 1 to 3.

Images