Asynchronous aggregation and privacy protection method in resource-limited federated edge learning

Abstract

The application discloses an asynchronous aggregation and privacy protection method in resource-limited federated edge learning, relates to the field of distributed machine learning, and comprises the following modules: system initialization, client selection, asynchronous federated training, personalized differential privacy protection and resource monitoring. The delay compensation mechanism is improved, model parameters in the model obsolescence threshold range are subjected to delay compensation, a bell-shaped curve function is adopted for the attenuation coefficient, and the greater the obsolescence is, the faster the attenuation is; for the client exceeding the obsolescence threshold, synchronization with the current global parameter is forced, and the next round of local training is entered.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning, and in particular to asynchronous aggregation and privacy protection methods in resource-constrained federated edge learning. Background Technology

[0002] As research into federated learning techniques in edge computing environments increases, researchers are referring to it as federated edge learning. In this environment, a global model can be collaboratively trained using local clients, ensuring data remains on-premises and achieving the goal of "data stays put, model moves; data is usable but not visible." Furthermore, due to the increasing emphasis on data and the growing awareness of privacy among data owners, privacy-preserving computing has received strong support and development in recent years. Federated edge learning provides powerful support for privacy-preserving computing scenarios.

[0003] There are many challenges in applying federated edge learning. This chapter mainly considers and addresses the following issues:

[0004] The laggard problem. The synchronous federated learning training process can lead to inconsistent completion times due to edge heterogeneity, causing some clients to become laggards, thereby increasing the waiting time of edge servers and reducing system efficiency.

[0005] Resource constraints are a significant issue. Most local training clients are mobile devices or IoT devices, which have limited power. Furthermore, since most use wireless communication, too many clients uploading data simultaneously can lead to insufficient bandwidth. Besides power and bandwidth, other resources are also limited, such as storage and computing power.

[0006] Model parameter leakage issue. Model parameters may be vulnerable to internal collusion attacks, external malicious attacks, or other attacks during transmission, thereby allowing the acquisition of client-related private information. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides an asynchronous aggregation and privacy protection method for resource-limited federated edge learning.

[0008] The technical solution adopted by this invention to solve its technical problem is: an asynchronous aggregation and privacy protection method in resource-limited federated edge learning, including the following modules: system initialization, client selection, asynchronous federated training, personalized differential privacy protection, and resource monitoring.

[0009] Preferably, the system initialization module is performed by an edge server;

[0010] Initialize a global model and broadcast the global model parameter w0 to all N clients;

[0011] Then set the global privacy budget ∈ and the resource budget D for the k-th resource. k If resources are consumed beyond the resource budget during the learning process, the system will terminate.

[0012] Broadcast the initialization parameter pruning threshold C to all clients;

[0013] The edge server sets an initial model obsolescence threshold τ0. If it detects that the obsolescence of a client model exceeds this threshold, it sends the updated global parameters of the current round to the client for forced synchronization and enters a new communication round for retraining.

[0014] Preferably, the client selection module:

[0015] The buffered asynchronous aggregation method is adopted. A cache is set up on the edge server. Before global aggregation, the local parameters received in advance are cached, and global update is performed only when the aggregation condition is triggered.

[0016] Assuming the buffer length is L (l∈[1,L]), in each communication round, an appropriate number of clients M (M≤L) or a client ratio α is selected based on the model training results. t (α t =M / N).

[0017] Preferably, the asynchronous federated training module:

[0018] The asynchronous federated training module includes client-side asynchronous parallel training and edge server-side globally buffered asynchronous aggregation processes; for client v i Using local private datasets Local training is performed using the gradient descent (SGD) algorithm to obtain local model parameters. The update process is as follows:

[0019]

[0020] Where η is the learning rate. This is the local iteration time; local iteration completes. After that, Then it is uploaded to the edge server via a wireless link;

[0021] Once the edge server receives the local parameters, it calculates the client's model staleness. If the model is stale, it corrects it according to the latency compensation mechanism to obtain the corrected model parameters.

[0022]

[0023] Where β∈(0,1] is a function of model staleness, wt-1 To save the global model parameters from the previous round;

[0024] Once the edge server cache has M local parameters, global aggregation is performed immediately, using the federated average method.

[0025]

[0026] Where p i Indicates client v i The proportion of data volume, i.e.

[0027] Preferably, the personalized differential privacy protection module uses differential privacy technology to protect the transmitted parameters.

[0028] Its definition is as follows:

[0029] ((∈,δ)-DP)

[0030] When two adjacent databases And output space At that time, random mechanism It is (∈,δ)-DP and satisfies:

[0031]

[0032] Where ∈ represents the privacy budget, and δ represents the probability that ∈ does not satisfy strict differential privacy;

[0033] The model parameters are fuzzed using a Gaussian mechanism by adding Gaussian noise with a mean of 0 and a standard deviation of σ.

[0034]

[0035] The Gaussian mechanism used is in the case where ∈ (0,1), σ≥c△2f / ∈ and the constant is constant. Differential privacy (∈,δ) can be guaranteed at this time;

[0036] in It is a function L2 sensitivity.

[0037] Set the global privacy budget to ∈;

[0038] Preferably, assuming the client has exhausted its privacy budget after T rounds of training, then the relationship is satisfied. Where ∈ t This represents the privacy budget consumed in each round; if the global privacy budget is evenly distributed, the average privacy budget consumed in each round is... In this scenario, the privacy budget consumed per client

[0039] If client v i Having more valid data allows for a lower privacy budget to be allocated to it. i , making v i It offers better privacy protection.

[0040] In this scheme, it is assumed that each client consumes a different privacy budget, then the privacy budget set is {∈ i According to the differential privacy composition theory, the privacy budget consumed in each communication round is ∈ t =max{∈ i For each client v i If its parameters contribute more to the model, more noise is added to prevent leakage.

[0041] After each local training session, the model parameters need to be pruned, with a pruning threshold of C, i.e., ||w i ||≤C; According to the definition of sensitivity, we can obtain The standard deviation of Gaussian noise σ i =v△2f / ∈ i ,Right now

[0042]

[0043] For asynchronous federated training, an appropriate privacy budget needs to be allocated to each client for personalized privacy protection; however, during the training process, if... The training will then be terminated immediately.

[0044] Preferably, the resource monitoring module: assumes there are K types of resources in the learning system; for each resource type k∈{1,2,...,K}, let g k This indicates the resource consumption during the client's local update process, b k D represents the resource consumption for a single exchange of model parameters between the edge server and the client. k For global resource budgeting; after T global aggregations, the resource consumption of local updates is g. k The resource consumption resulting from model swapping is 2b. k ·N·T;

[0045] During federated training, the edge server continuously monitors resources and checks resource consumption after each communication round. If the consumed resources are less than the global resource budget, training continues. In other words, the following conditions must be met during training:

[0046] (g k +2b k )·N·t≤D k .

[0047] Preferably, a delay compensation mechanism is used: during the global aggregation process, the received local model parameters are corrected according to the staleness of each client's model;

[0048] Assuming the client's model staleness is τ, when τ = 0, the aggregation process does not need to decay the model, i.e., Z(0) = 1; when τ is small, the decay is slow, and when τ is large, the decay is fast.

[0049] Based on this property, a bell-shaped curve is designed, and its functional expression is shown below:

[0050]

[0051] As a model decay factor, where It is a hyperparameter for adjustable decay rate;

[0052] After each communication round, calculate the model obsoleteness for each client. If τ i If the obsolescence level exceeds the set obsolescence threshold τ0, a forced synchronization is performed on the client, sending the current global model parameters to the client, and then training restarts.

[0053]

[0054] Preferably, the problem is constructed as follows:

[0055]

[0056]

[0057] Preferably, the model training process is subject to several constraints:

[0058] The first constraint indicates that (∈,δ)-differential privacy can be guaranteed after adding noise to the local model parameters;

[0059] The second constraint states that during the T-round communication process, the privacy budget consumed must be less than the total budget set.

[0060] The third constraint is a resource-constrained constraint, which means that the resources consumed by local computation and model exchange during the T-round communication process must be less than the set total resource budget;

[0061] The fourth constraint is the model obsolescence constraint, which means that the obsolescence of the model for each client cannot exceed the obsolescence threshold.

[0062] Of all constraints, the privacy budget consumed by each client ∈ i and the number of aggregated clients M in each communication round tThese two parameters are not fixed and can significantly affect the model's convergence speed; therefore, it is necessary to adjust ∈ i and M t Estimate and optimize to make the model converge faster and more accurate.

[0063] Advantages of this invention:

[0064] An improved latency compensation mechanism is implemented to compensate for the latency of model parameters within the model staleness threshold range. The decay coefficient adopts a bell-shaped curve function, with faster decay as the staleness increases. For clients exceeding the staleness threshold, synchronization with the current global parameters is forced, and the client enters the next round of local training. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only eight of the drawings in this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0066] Figure 1 This is an asynchronous federated edge learning system according to an embodiment of the present invention;

[0067] Figure 2 These are system components of an embodiment of the present invention;

[0068] Figure 3 This is the model staleness decay function in this embodiment of the invention;

[0069] Figure 4 This is an asynchronous federated architecture for deep reinforcement learning, as described in this embodiment of the invention.

[0070] Figure 5 The following are the DRL training results of this embodiment of the invention: (a) training cycle loss; (b) reward changes with training cycle;

[0071] Figure 6 The impact of the number of aggregated clients without privacy protection in this embodiment of the invention: (a) test accuracy; (b) test loss;

[0072] Figure 7 The impact of the number of aggregated clients in this embodiment of the invention on: (a) test accuracy; (b) test loss;

[0073] Figure 8 A comparison of different algorithms in embodiments of the present invention: (a) test accuracy; (b) test loss. Detailed Implementation

[0074] To enhance understanding of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are only used to explain the invention and do not limit the scope of protection of the invention.

[0075] Example

[0076] The symbols and definitions used in this scheme are shown in Table 1.1.

[0077] 1 System Model

[0078] 1.1 System Framework

[0079] Suppose there is an asynchronous federated edge learning system, such as Figure 1 As shown, there is one edge server and N clients, with the client set being... Each client v i A private dataset of individuals (i∈{1,2,...,N}) is given by Data size is In the federated learning process, each client's local learning model is F. i (w), using a private dataset Local training and local updates are performed. The global update process on the edge server adopts an asynchronous aggregation scheme. Clients that do not participate in the aggregation in a timely manner will generate stale models. Therefore, each client has a staleness parameter τ. i This is the difference between the current communication round and the last communication round when the global parameters were received. This parameter is calculated and saved by the edge server before each global aggregation.

[0080] Table 1.1 List of main symbols used

[0081]

[0082]

[0083] If the differential privacy federated edge system is modularized, it mainly consists of five modules: system initialization, client selection, asynchronous federated training, personalized differential privacy protection, and resource monitoring. The model composition is as follows: Figure 1 As shown.

[0084] 1) System Initialization

[0085] The initialization of the federated edge system is primarily handled by the edge server. First, a global model is initialized, and its parameters w0 are broadcast to all N clients. Then, the global privacy budget ∈ and the resource budget D for the k-th resource class are set. kDuring the learning process, if resource consumption exceeds the resource budget, the system will terminate. Furthermore, the initial parameter pruning threshold C will be broadcast to all clients. The edge server will also set an initial model staleness threshold τ0. If a client's model staleness exceeds this threshold, the global parameters updated in the current round will be sent to that client for forced synchronization, initiating a new communication round for retraining.

[0086] 2) Client selection

[0087] To reduce latency on edge servers while waiting for a large number of clients to participate in global aggregation, and to minimize resource consumption caused by frequent communication during global aggregation by individual clients, selecting an appropriate number of clients for global aggregation is crucial. This scheme employs a buffered asynchronous aggregation approach. A buffer is set up on the edge server to cache pre-received local parameters before global aggregation, and then globally updates the parameters only when the aggregation condition is triggered. Assuming the buffer length is L (l∈[1,L]), the appropriate number of clients M (M≤L) or client ratio α is selected in each communication round based on the model training results. t (α t =M / N).

[0088] 3) Asynchronous Federated Training

[0089] The asynchronous federated training module includes client-side local asynchronous parallel training and edge server-side globally buffered asynchronous aggregation processes. For client v i Using local private datasets Local model parameters are obtained by performing local training using the stochastic gradient descent (SGD) algorithm. The update process is as follows:

[0090]

[0091] Where η is the learning rate. This is the local iteration time; local iteration completes. After that, Then it is uploaded to the edge server via a wireless link.

[0092] Once the edge server receives the local parameters, it calculates the client's model staleness. If the model is stale, it corrects it according to the latency compensation mechanism to obtain the corrected model parameters.

[0093]

[0094] Where β∈(0,1] is a function of model staleness, w t-1 This is to save the global model parameters from the previous round. The detailed process of the delay compensation mechanism is shown in Section 1.2.

[0095] Once the edge server cache has M local parameters, global aggregation is performed immediately, using the federated average method.

[0096]

[0097] Where p i Indicates client v i The proportion of data volume, i.e.

[0098] 4) Personalized differential privacy protection

[0099] Since updated parameters are transmitted wirelessly through a federated edge learning system, there is still a possibility of privacy leakage. Therefore, this scheme employs differential privacy technology to protect the transmitted parameters. Differential privacy possesses several properties, including compositionality and post-processing, making it suitable for the federated edge learning system in this scheme. We adopt (∈,δ)-DP, which is defined as follows:

[0100] Definition 1 ((∈,δ)-DP). When two adjacent databases And output space At that time, random mechanism It is (∈,δ)-DP and satisfies:

[0101]

[0102] Where ∈ represents the privacy budget, and δ represents the probability that ∈ does not satisfy strict differential privacy.

[0103] To ensure (∈,δ)-DP, a Gaussian mechanism is used to fuzz the model parameters, mainly by adding Gaussian noise with a mean of 0 and a standard deviation of σ.

[0104]

[0105] The Gaussian mechanism used is in the case where ∈ (0,1), σ≥c△2f / ∈ and the constant is constant. Differential privacy (∈,δ) can be guaranteed at this time. It is the function Ψ: L2 sensitivity.

[0106] To protect the privacy of model parameters in the system, a global privacy budget is set to ∈. Assuming the client consumes all of its privacy budget after T rounds of training, the following relationship is satisfied: Where ∈ t This represents the privacy budget consumed in each round. If the global privacy budget is evenly distributed, the average privacy budget consumed in each round is... In this scenario, the privacy budget consumed per client

[0107] According to formula (1.4), if client v i Having more valid data allows for a lower privacy budget to be allocated to it. i , making v i It offers better privacy protection. In this scheme, it is assumed that each client consumes a different privacy budget, then the privacy budget set is {∈ i According to the differential privacy composition theory, the privacy budget consumed in each communication round is ∈ t =max{∈ i For each client v i If its parameters contribute more to the model, more noise is added to prevent leakage.

[0108] In addition, to prevent problems caused by excessively large parameters, they need to be pruned after each local training session. The pruning threshold is C, i.e., ||w||. i ||≤C. According to the definition of sensitivity, we can obtain... The standard deviation of Gaussian noise σ i =c△2f / ∈ i ,Right now

[0109]

[0110] For asynchronous federated training, an appropriate privacy budget (Gaussian noise standard deviation) needs to be allocated to each client for personalized privacy protection. However, during training, if... The training will then be terminated immediately.

[0111] 5) Resource monitoring

[0112] In asynchronous federated edge learning systems, the training process consumes significant resources, including computational and communication resources. Edge servers typically have ample resources, so their resource consumption is negligible. However, due to system heterogeneity, clients have varying resource availability. If resources are exhausted during training, communication with the edge servers will be lost.

[0113] Suppose that the learning system has K types of resources (such as energy, network bandwidth, etc.). For each resource type k∈{1,2,...,K}, let g k This indicates the resource consumption during the client's local update process, b k D represents the resource consumption for a single exchange of model parameters between the edge server and the client. k This is a global resource budget. Therefore, after T global aggregations, the resource consumption of a local update is g. k The resource consumption resulting from model swapping is 2b. k ·N·T.

[0114] During federated training, the edge server continuously monitors resources and checks resource consumption after each communication round. If the consumed resources are less than the global resource budget, training continues. In other words, the following conditions must be met during the training process:

[0115] (g k +2b k )·N·t≤D k (1.7)

[0116] 1.2 Delay Compensation Mechanism

[0117] Since the federated edge learning system proposed in this scheme uses asynchronous federated training, the client may have a model staleness problem. During the global aggregation process, we correct the received local model parameters according to the model staleness of each client. The correction method is shown in (1.2), where β=Z(·) is the delay decay coefficient, which is a function of model staleness.

[0118] Assuming the client's model staleness is τ, we know that when τ = 0, the aggregation process does not need to decay the model, i.e., Z(0) = 1; when τ is small, the decay is slow, and when τ is large, the decay is fast.

[0119] Based on this property, the comparison Figure 3 We designed a bell curve, the function expression of which is shown below.

[0120]

[0121] As a model decay factor, where It is a hyperparameter for adjusting the decay rate.

[0122] The remaining function expressions in the diagram are shown below.

[0123] The following lists some commonly used staleness functions Z(τ), where x, y > 0:

[0124] Constant functions:

[0125] Z(τ)=1 (1.9)

[0126] Polynomial functions:

[0127] Z x (τ)=(τ+1) -x (1.10)

[0128] Piecewise function:

[0129]

[0130] • Exponential function:

[0131] z x (τ)=e -xτ (1.12)

[0132] After each communication round, calculate the model obsoleteness for each client. If τ i If the obsolescence level exceeds the set obsolescence threshold τ0, a forced synchronization is performed on the client, sending the current global model parameters to the client, and then training restarts.

[0133]

[0134] 1.3 Problem Construction

[0135] Based on the asynchronous federated edge learning scheme proposed above, our main goal is to achieve privacy protection of the system model parameters under resource-constrained conditions, ultimately leading to global model convergence. Therefore, our optimization problem is constructed as follows:

[0136]

[0137]

[0138] Several constraints apply during model training. The first constraint ensures (∈,δ)-differential privacy even after adding noise to the local model parameters. The second constraint requires that the privacy budget consumed during T rounds of communication be less than the total budget. The third constraint is a resource-constrained constraint, meaning that the resources consumed for local computation and model exchange during T rounds of communication must be less than the total resource budget. The fourth constraint is a model staleness constraint, meaning that the model staleness of each client cannot exceed the staleness threshold.

[0139] Of all constraints, the privacy budget consumed by each client ∈ i and the number of aggregated clients M in each communication round t These two parameters are not fixed and can significantly affect the model's convergence speed; therefore, it is necessary to adjust ∈ i and M t Estimate and optimize to make the model converge faster and more accurate.

[0140] In general, finding the optimal solution to the above problem is an NP-hard problem. Therefore, we consider using deep reinforcement learning algorithms to solve the optimization problem.

[0141] 2. Resource-constrained and privacy-preserving asynchronous aggregation algorithms

[0142] This solution focuses on asynchronous aggregation algorithms that prioritize resource availability and privacy protection. As the system model shows, the main computations are performed by client and edge server entities, with each communication round consisting of two main steps: local training and global aggregation.

[0143] Each communication round begins with shared global model parameters, therefore, when t=0, the edge server initializes the global parameters w. 0 And a list of model obsolescence, with the model obsolescence τ for each client. i =0, then broadcast the global parameters to all clients for initial synchronization, and notify the clients to start local training.

[0144] After receiving the global model, the client uses the SGD algorithm to perform local updates and obtain local model parameters. To prevent the parameters from becoming too large, they are cropped, with an upper bound set to C, and then Gaussian noise is added to obtain the noise parameters. This is to prevent parameters from being intercepted during transmission and thus allowing the original data information to be deduced. After the client completes the above calculation process locally, it immediately uploads the data to the edge server via a wireless link.

[0145] The edge server first performs pre-aggregation preparation, including latency compensation for the received local noise model parameters.

[0146]

[0147] Where β i =Z(τ) i ).

[0148] When the number of model parameters temporarily stored in the buffer reaches M, global aggregation is performed immediately.

[0149]

[0150] When the client data volume is the same, then

[0151] After aggregation is complete, the model obsolescence list needs to be updated. If the client did not participate in the current round of aggregation, then τ i =τ i +1. If the model staleness exceeds the threshold, the client needs to synchronize the current global model parameters. Due to limited system resources and privacy budget constraints, each communication round requires calculation of the consumed privacy budget and resources. If the set threshold is exceeded, the system is interrupted; if there are remaining resources, model training continues, sharing the current global model parameters with the aggregation client and clients exceeding the staleness threshold. Then, the next communication round begins iteratively.

[0152] The system training process is shown in Algorithm 1.1.

[0153] Algorithm 1.1: Resource-constrained and privacy-preserving asynchronous aggregation algorithm

[0154] 3. Client selection and personalized privacy protection

[0155] In this section, we briefly introduce agent deep reinforcement learning (DRL) techniques and then use DRL to solve the optimization problem described above. To align with federated edge learning systems, we choose to use the A3C algorithm to design a strategy that executes corresponding actions based on the acquired state, enabling the proposed framework to achieve rapid global model convergence while protecting the privacy of model parameters.

[0156] 3.1 Design Concept

[0157] The original federated edge learning system mainly consists of two parts: a local update process and a global update process, such as... Figure 1 As shown. The following content mainly addresses two issues that arise in these two processes: 1) How much noise (and how much privacy budget) needs to be added in the local update to ensure model parameters are usable while protecting user privacy? 2) In the global update process, how many client-side local model parameters need to be received for asynchronous aggregation to ensure rapid model convergence and reduce edge server waiting time?

[0158] The two issues mentioned above correspond to the client selection module and the personalized differential privacy protection module in terms of system module composition. Therefore, the following design addresses these two modules to solve the problems. 1) For the personalized differential privacy protection module, since the data from different clients is distributed non-IID and the importance of each client's data varies, it is necessary to design different noise privacy budgets for personalized privacy protection of each client. 2) For the device selection module, the global update requires short waiting time and rapid model convergence while reducing communication resource consumption. Therefore, different clients can be adaptively selected to participate in global aggregation in each communication round during training. Through module design, the federated edge system can achieve rapid model convergence without significant loss of accuracy under the constraints of limited resources and privacy protection. Through analysis, we design a learning system based on DRL, adaptively selecting the proportion α of client participation. t ,(t∈{1,...,T}) and the personalized privacy budget for a single client ∈ i Two parameter values.

[0159] To address the privacy budget decision-making and client number selection issues in the target problem, federated edge learning is combined with a deep reinforcement learning framework. This can be viewed as deploying a global network on the edge server and sub-networks on the clients. Each network contains an actor-critic (AC) network, and the network framework employs a neural network model. The architecture diagram is shown below. Figure 4 As shown, each sub-network interacts with the environment to obtain its state, and then executes corresponding actions through a policy to obtain rewards from the environment. The global network does not interact directly with the environment. The decision-making process for the two problems will be described below.

[0160] 3.2 Privacy Budget Decision Algorithm

[0161] At each client, Gaussian noise needs to be added after local model training to protect the privacy of model parameters. Due to a limited total privacy budget, the amount of noise added after each training iteration is also limited. Furthermore, because client data is non-IID distributed, the contribution of data to model updates is inconsistent; therefore, we believe that parameters that contribute more to model training should have more noise added (with less privacy budget). When using DRL for decision-making, the corresponding parameters need to be designed according to the objective.

[0162] (1) DRL Model

[0163] First, we focus on the privacy budget decision-making algorithm during the local update process, using deep reinforcement learning frameworks such as... Figure 4 As shown. A standard reinforcement learning model is an agent that continuously learns based on rewards or penalties received during interactions with its environment in order to output the optimal action. The following introduces several important concepts used in the DRL model.

[0164] Environment. The environment of the DRL system we are referring to here is the designed federated edge learning framework, which includes both global and local learning models.

[0165] An intelligent agent refers to a client that interacts with the environment by performing actions.

[0166] state It is a feature vector describing the state of the agent at time t.

[0167] action Indicates client v i The privacy budget consumed at time t. Given the current state, the DRL agent will perform an action based on a policy, denoted as...

[0168] award Indicates the action at time t After being executed, the AI ​​will receive a reward from the environment. This will be used to judge whether the action is good or bad.

[0169] In each communication round, the policy network receives the state s from the previous time step. t-1 The probability of outputting a certain action (such as completion time, loss function, and resource consumption) is called the policy π, which is the state space. To the action space The mapping typically uses a convolutional neural network with a softmax output layer. Then, actions are selected from the action space according to policy π. Next, the agent receives the current state. and reward value The goal of an intelligent agent is to select the best action through a strategy to maximize expected returns.

[0170] Based on the analysis of the design modules, we employ the Asynchronous Advantage Actor-Critic (A3C) algorithm to implement the DRL network structure. At each client, a composite neural network is deployed, taking the current state of each client as input and outputting a policy π and a state value function V(s). The actor network determines the privacy budget used by the client, while the critic network evaluates the benefits of taking the current action.

[0171] (2) Subnetwork status and rewards

[0172] After completing local training, the local sub-network agent obtains its current state by interacting with the local environment. Here, t represents the current communication round. This represents the global model parameters for the current communication round. ε represents the parameters of the local model trained using local private data. t D represents the remaining privacy budget for the current communication round. t This indicates the remaining resource budget for the current communication round.

[0173] The client executes the actions as required. Choose an appropriate privacy budget, assuming ∈ i ∈{∈ j},j∈[1,J], where J is the number of actions in the local action space, set as a discrete value. To clarify, at time t in each communication round, the client performs a privacy budget decision process, in the current state... Next, perform all actions and receive the corresponding reward. When j=1, assign a value.

[0174] The agent selects actions according to a policy, which is represented as follows: This represents the probability distribution of actions. Here, we use a neural network to represent policy learning, with policy parameters θ. Therefore, our policy can be expressed as: Indicates the state Next, execute action a i,j The probability of.

[0175] When commentators observe action a i,j After execution, the reward value r will be calculated. i,j The quality of the current state can be judged by the reward; therefore, the reward we set is related to changes in model parameters, resource consumption, and privacy budget consumption.

[0176]

[0177] The first part This indicates the difference in parameters of the local model before and after the update. Indicates the execution of action a i,j The local model parameters after that, The smaller the difference, the greater the benefit of the current action, and the greater the reward value. The second part represents the impact of changes in resource consumption on the environment; if the local computation consumes resources g... i,j The more [actions], the smaller the reward value. The goal of local updates is to obtain the current action that maximizes the cumulative reward, and then upload the local update parameters to the global network. The cumulative reward is calculated as follows:

[0178]

[0179] The discount factor γ∈(0,1], and q is the time step index value when the current time step j reaches the termination state.

[0180] (3) Model Training

[0181] Because this system uses the A3C framework for model training, a master agent manages the global network, and multiple sub-agents manage local networks. All sub-agents undergo asynchronous parallel training. During local updates, the actor network follows a policy... The commentator network estimates the state-value function by selecting an action. Here, θ is the policy parameter, θ v These are the parameters of the state value function. The state value function is estimated based on the neural network function as follows:

[0182]

[0183] The policy and value functions are updated after J actions, or until the terminal state is reached (e.g., model convergence or resource exhaustion). The local update process updates the policy function and the estimated state-value function, updating them to... in This is the dominant function.

[0184] Therefore, the loss function for model updates can be obtained, where the value function loss is the minimum mean square error of the advantage function.

[0185]

[0186] The policy function loss is:

[0187]

[0188] Where H is the entropy of the policy distribution. Then, the accumulated gradients are used to update the policy parameters θ and the value function parameters θ. v .

[0189] 3.3 Client Quantity Decision Algorithm

[0190] After the local client completes its asynchronous update, the updated parameters are uploaded to the edge server for a global update, resulting in new global parameters. If a new client begins training, parameters are retrieved from the global network for the next local sub-network update. Before the next communication round begins, the edge server needs to determine the number of clients participating in the next aggregation round. The privacy budget decision and the client number decision after the local update are interrelated, and these decisions may not be made within the same communication round.

[0191] An A3C global network is deployed on the edge server, and the main intelligent agent obtains the current state s. t The current global state includes the current communication round t and the local model parameters uploaded by each client. Remaining resource budget D t At time t, the global network in the edge server selects an action according to the policy, and the action executed is represented by a. j This indicates that j is the time step index for a given stage, used to select an appropriate number of clients to participate in global aggregation. The critic network calculates rewards based on the actions performed, and the reward setting is related to the convergence status of the current global model. Here F * ΔF represents the optimal value when the global model converges. t This represents the difference between the current loss value and the optimal loss value F. * The difference, i.e. ΔF t =F * -F t At this point, each client uploads local model parameters after adding Gaussian noise. The corrected noise parameters are used when performing global aggregation.

[0192] The policy network parameter of the global network is θ', and the state-value network parameter is θ'.v When selecting clients, the initial parameters are set to the aggregated global network parameters. After completing the global network settings, a global update is performed, similar to the local network update process. Finally, the optimal number of clients is selected to maximize the expected return for global aggregation of the asynchronous federated learning system. The algorithm implementation is shown in Algorithm 1.2.

[0193] 4. Experiment and Performance Evaluation

[0194] Next, we present the experimental and performance evaluation scheme. We use a public dataset to compare the proposed scheme under different parameter settings. The experimental results show that federated edge learning based on limited resources and differential privacy can effectively protect the privacy of local model parameters when deep reinforcement learning technology is used to assist training, and can enable the model to converge quickly.

[0195] Algorithm 1.2: Personalized Privacy Protection and Client Quantity Selection Algorithm

[0196]

[0197] 4.1 Experimental Environment

[0198] To verify the privacy protection scheme proposed in this paper, a local computer was used for simulation, and the PyTorch framework was employed to implement the differential privacy buffer asynchronous aggregation process. The computer used in the experiment had an Intel Core i7-10700 processor (CPU@2.90GHz) and 32GB of RAM.

[0199] The primary dataset for evaluation is the MNIST handwritten digit recognition dataset, which consists of 60,000 training samples and 10,000 test samples. Each sample is a 28×28 pixel grayscale image representing a digit from 0 to 9. During training, the batch size for the training dataset is set to 64, and the batch size for the test set is set to 1000. For clients in the federated edge learning environment, a non-IID data partitioning method is assumed, assigning digits 0-9 to different clients. Each client contains samples from only one or a few digit categories, with different data distributions but the same data volume. The global model for client-co-training is a convolutional neural network (CNN) using two convolutional layers and a fully connected layer. Rectified linear units (ReLUs) are chosen as the activation function, and dropout layers are used for regularization.

[0200] During collaborative training and asynchronous parallel updates on the client side, the following metrics are used to evaluate the proposed algorithm: (1) Test accuracy. This is the most commonly used performance metric in the classification training process, representing the ratio of the number of correctly identified data samples in the test dataset to the total number of test data samples. (2) Test loss. This represents the magnitude of the error between the predicted value and the true value during training, usually using cross-entropy loss and NLL loss as loss functions. (3) DRL reward. This represents the reward calculated according to the reward function during DRL training. (4) Communication rounds. This represents the number of communication rounds required for the global model to converge, with each communication round representing the entire process from the distribution of global model parameters to the completion of global updates.

[0201] In the comparative experiments, we selected relevant benchmark algorithms for comparison. The first was DP-SGD, which was used to compare the effect of adding Gaussian noise to the federated learning process to protect the privacy of model parameters. The second was NbAFL, which adds Gaussian noise to both local and global model parameters during the federated learning training process.

[0202] 4.2 Experimental Results

[0203] (1) DRL Training

[0204] Deep reinforcement learning (DRL) training is primarily performed by edge servers and clients. The deployed A3C network handles client selection on the edge servers and personalized privacy protection on the clients. The experiments mainly tested DRL performance, including training loss and reward. In this part of the experiment, the total number of clients was set to N=10. As shown in the graph, the loss value drops rapidly in the early training phase, mainly because the agent lacks information about the environment at the beginning. After a period of training, the agent acquires sufficient information about the environment, and the loss value begins to stabilize, indicating that the DRL agent gradually adapts to the federated edge learning system. The reward value gradually accumulates during training, as the agent continuously explores and chooses better strategies to complete actions, resulting in a higher reward value. When the training cycle reaches 200 iterations, the reward value only changes slightly.

[0205] (2) Parameter Influence

[0206] Regarding the buffered asynchronous aggregation algorithm proposed in this solution, its test performance varies as the number of clients aggregated in a single communication round changes. Figure 6As shown, it can be observed that the fewer the number of clients in each aggregation round, the more communication rounds are required for convergence, and the accuracy is lower than with multi-client aggregation. Since the data exhibits a non-IID distribution, fewer clients participating in aggregation result in fewer data samples, leading to significant fluctuations in the global model accuracy during aggregation. Only a large number of frequent communications can evenly utilize the samples from all clients, ultimately achieving convergence. The figure also indicates that if an asynchronous aggregation algorithm is used, and resource consumption is considered, an appropriate number of participating clients needs to be set.

[0207] To enable the model to converge quickly, we choose to use a dynamically changing number of participating clients for global aggregation. The number required in each round is determined according to the A3C algorithm, and the determination process needs to take into account the current state of the model.

[0208] Adjusting the number of clients during the aggregation process has a significant impact on model convergence performance, such as... Figure 7 This demonstrates the impact of varying client numbers on model performance, compared to a fixed client number of M=4. Due to the non-IID distribution of the data and asynchronous aggregation, the accuracy curve fluctuates significantly. Using the adaptive client number obtained from the A3C decision algorithm allows for the selection of more clients for aggregation even with low accuracy, thereby improving accuracy and accelerating model convergence.

[0209] (3) Algorithm Comparison

[0210] Figure 8 The comparison results between different algorithms are presented. DRL-DPAFL represents our proposed scheme that uses DRL to determine the privacy budget consumption of different clients and the number of clients aggregated in each communication round. DP-SGD and NbAFL were used in experiments where 4 participants were selected from 10 clients for aggregation, and the data distribution was set to non-IID. As shown in the figure, our DRL-DPAFL achieves similar results to DP-SGD. When using differential privacy mechanisms for privacy protection, the accuracy fluctuates around 90%, and both outperform NbAFL.

[0211] 5. Summary

[0212] This proposal offers a privacy-preserving scheme for asynchronous federated learning with limited resources, providing personalized privacy protection for local clients under resource constraints. Specifically, during local training, different levels of privacy protection are provided to each client based on the importance of their data. This is primarily achieved by setting personalized privacy budgets and adding varying amounts of noise. If local data contributes significantly, a smaller privacy budget is allocated to the local model parameters, meaning more noise is added for higher privacy protection. After the local parameters reach the edge server, they are attenuated based on the model staleness of each client and then temporarily stored in a cache. If the cache size equals the optimal value obtained by the deep reinforcement learning algorithm, global aggregation is performed. The global parameters are distributed to participating clients for the next training iteration and sent to clients whose model staleness exceeds a set threshold for forced synchronization. Extensive experiments in the performance evaluation phase validate the proposed algorithm, demonstrating that the proposed scheme outperforms comparative algorithms with more lenient conditions while addressing the laggard problem, limited resources, and model parameter leakage.

[0213] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An asynchronous aggregation and privacy protection method in resource-limited federated edge learning, characterized in that, It includes the following modules: system initialization, client selection, asynchronous federated training, personalized differential privacy protection module, and resource monitoring module; The asynchronous federated training module: The asynchronous federated training module includes client-side local asynchronous parallel training and edge server-side globally buffered asynchronous aggregation processes; for the client... Using local private datasets Local model parameters are obtained by performing local training using the stochastic gradient descent (SGD) algorithm. The update process is as follows: in For learning rate, This is the local iteration moment; local iteration completes. After that, Then it is uploaded to the edge server via a wireless link; Once the edge server receives the local parameters, it calculates the client's model staleness. If the model is stale, it corrects it according to the latency compensation mechanism to obtain the corrected model parameters. , in Let be a function of model staleness. To save the global model parameters from the previous round; When the edge server cache has After determining the local parameters, global aggregation is performed immediately, using the federated average method: in Indicates the client The proportion of data volume, i.e. .

2. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 1, characterized in that: The system initialization module is completed by the edge server. Initialize a global model and set the global model parameters. Broadcast to all One client; Then set the global privacy budget. and the Resource budget for resource class If resources are consumed beyond the resource budget during the learning process, the system will terminate. Initialize the model parameter clipping threshold Broadcast to all clients; Setting model obsolescence threshold on edge servers If the model obsolescence of a client exceeds the threshold, the global parameters updated in the current round will be sent to that client for forced synchronization, and a new communication round will be entered for retraining.

3. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 1, characterized in that: The client selection module: The buffered asynchronous aggregation method is adopted. A cache is set up on the edge server. Before global aggregation, the local parameters received in advance are cached, and the global update is performed only when the aggregation condition is triggered. Assuming the cache length is In each communication round, an appropriate number of clients is selected based on the model training results. Or client ratio .

4. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 1, characterized in that: The personalized differential privacy protection module uses differential privacy (DP) technology to protect transmitted parameters. Its definition is as follows: When two adjacent databases and output space At that time, random mechanism yes Yes, satisfied: in Indicates privacy budget, Indicates dissatisfaction The probability of strictly differential privacy; The model parameters are fuzzed using a Gaussian mechanism, by adding parameters with a mean of 0 and a standard deviation of 0. Gaussian noise, i.e. The Gaussian mechanism used in the current , and constant Time can guarantee Differential privacy; in It is a function of Sensitivity; Set the global privacy budget to .

5. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 4, characterized in that: Assuming the client is training If the privacy budget is used up exactly after one round, then the relationship is satisfied. ,in This represents the privacy budget consumed in each round; if the global privacy budget is evenly distributed, the average privacy budget consumed in each round is... In this case, the privacy budget consumed by each client ; If the client Having more valid data will result in a lower privacy budget being allocated to it. ,make It offers better privacy protection; In this scheme, it is assumed that each client consumes a different privacy budget, then the privacy budget set is: According to the differential privacy composition theory, the privacy budget consumed in each communication round is... For each client If its parameters contribute more to the model, more noise is added to prevent leakage. The model parameters need to be pruned after each local training session, with a pruning threshold of [value missing]. ,Right now According to the definition of sensitivity, we can obtain Then the standard deviation of Gaussian noise ,Right now For asynchronous federated training, appropriate privacy budgets need to be allocated to each client for personalized privacy protection; however, during the training process, if... If this happens, the training will be terminated immediately.

6. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 1, characterized in that: The resource monitoring module: assuming the learning system has a total of Types of resources; for each resource type ,make This indicates the resource consumption during the client's local update process. This represents the resource consumption required for one exchange of model parameters between the edge server and the client. For global resource budgeting; in the process After the global aggregation, the resource consumption of the local update is The resource consumption resulting from model exchange is ; During federated training, the edge server continuously monitors resources and checks resource consumption after each communication round. If the consumed resources are less than the global resource budget, training continues. In other words, the following conditions must be met during the training process. .

7. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 1, characterized in that: Delay compensation mechanism: During the global aggregation process, the received local model parameters are corrected according to the staleness of each client's model; Assume the client's model is obsolete. ,when At that time, the aggregation process does not require attenuation of the model, i.e. ;when When the size is small, the decay is slower. The larger the value, the faster the decay. Design a bell-shaped curve with the following function expression: As a model decay factor, where It is a hyperparameter for adjustable decay rate; After each communication round, the model obsoleteness of each client is calculated. Greater than the set obsolescence threshold If this happens, a forced synchronization is performed on the client, sending the current global model parameters to the client, and then training restarts. .

8. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 7, characterized in that: Problem Construction: .

9. The asynchronous aggregation and privacy protection method in resource-limited federated edge learning according to claim 8, characterized in that: Several constraints need to be met during model training: The first constraint indicates that after adding noise to the local model parameters, it can guarantee... -Differential privacy; The second constraint is expressed in During round-robin communication, the privacy budget consumed must be less than the total budget set. The third constraint is a resource-constrained constraint, meaning that in The resources consumed by local computation and model exchange during round-robin communication should be less than the total resource budget set. The fourth constraint is the model obsolescence constraint, which means that the obsolescence of the model for each client cannot exceed the obsolescence threshold. Privacy budget consumed by each client under all constraints and the number of aggregated clients in each communication round These two parameters are not fixed and can significantly affect the model's convergence speed, therefore they need to be adjusted. and Estimate and optimize to make the model converge faster and more accurate.

Images