A federated learning method and system

Abstract

The application discloses a kind of federal learning method and system, federal learning server sends gradient weighted aggregation strategy with byzantine robustness to all federal learning participants;Wherein, the gradient weighted aggregation strategy is generated by deep reinforcement learning;Federal learning server utilizes gradient weighted aggregation strategy and weights and aggregates perturbed model parameter update, utilizes the model parameter update after weighting and aggregation and updates global model;Federal learning server carries out quality assessment to the global model after updating according to own standard verification set, based on the evaluation result, utilizes deep reinforcement learning and outputs the gradient weighted aggregation strategy of next round.The purpose of the present application is to resist the various byzantine attacks including customized byzantine attack in federal learning, while realizing the defense of privacy inference attack to honest but curious server, so as to realize the federal learning of safe privacy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine learning security, and particularly relates to a federated learning method and system. BACKGROUND

[0002] Federated learning is an emerging collaborative machine learning paradigm that aims to solve the data silo problem while protecting the privacy of local data of participants. Participants with various devices (e.g., Internet of Things devices, smartphones, unmanned vehicles, drones, or high-performance computing clusters) jointly train a global model without sharing their local data with a central server. However, a participant controlled by an attacker can perform a Byzantine attack by uploading malicious updates, which prevents the normal convergence of the global model and reduces the performance of the model; at the same time, an honest but curious server can honestly perform the steps of the federated learning process, but is curious about the private data of the participants and tries to infer sensitive information.

[0003] In recent years, secure multi-party computation technology and trusted computing environment have been used to implement privacy Byzantine robust federated learning. However, (1) secure multi-party computation technology and trusted computing environment usually use statistical-based Byzantine robust solutions, which can still be successfully attacked by carefully constructed Byzantine attacks, reducing the accuracy of the global model, and the privacy Byzantine robust federated learning solution still needs to implement effective defense against customized Byzantine attacks. (2) The trusted computing environment has a hardware bottleneck, and existing trusted computing environments usually have limited memory and cannot implement trusted computing on neural networks with numerous parameters and complex structures. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a federated learning method and system, which aims to resist various Byzantine attacks including customized Byzantine attacks in federated learning, and at the same time implement defense against privacy inference attacks of honest but curious servers, thereby realizing secure and private federated learning.

[0005] To solve the above technical problems, the present application is implemented by the following technical solutions:

[0006] A federated learning method, comprising:

[0007] The federated learning server sends a gradient weighted aggregation strategy with Byzantine robustness to all federated learning participants; wherein the gradient weighted aggregation strategy is generated by deep reinforcement learning;

[0008] The federal learning server utilizes a gradient weighting aggregation strategy to perform weighted aggregation on the perturbed model parameter update, and utilizes the model parameter update after the weighted aggregation to update the global model; wherein the perturbed model parameter update is specifically: each federal learning participant utilizes local data to train a local model to obtain a model parameter update, utilizes a Lagrange interpolation method to generate a privacy perturbation according to the received gradient weighting aggregation strategy, and adds the privacy perturbation on the model parameter update obtained by the training, that is, the perturbed model parameter update is obtained.

[0009] The federal learning server performs quality evaluation on the updated global model according to a self-standard verification set, and based on the evaluation result, utilizes deep reinforcement learning to output a gradient weighting aggregation strategy of a next round.

[0010] Further, the step of generating the gradient weighting aggregation strategy through deep reinforcement learning comprises:

[0011] The generation process of the gradient weighting aggregation strategy by the federal learning server utilizing deep reinforcement learning is defined as a Markov decision process, and the specific corresponding relationship is:

[0012] Markov state space Wherein, the state S of the deep reinforcement learning agent is: t l r is the loss of the global model on the standard verification set, is the current gradient weighting aggregation strategy, is auxiliary information, and the subscript t is the tth communication round;

[0013] Markov action space The action a is an n-dimensional vector, n is the number of federal learning participants, and the action a represents the behavior that the deep reinforcement learning agent can take at each state, which can be converted into a gradient weighting aggregation strategy through standardization;

[0014] The reward function R is the immediate reward obtained by the deep reinforcement learning agent after taking an action at each state, which reflects the change of the state of the deep reinforcement learning agent and provides a clue for the update direction of the gradient weighting aggregation strategy, and the reward r of the tth round t is set as:

[0015]

[0016] In the formula, l r (*) is a loss function, θ' is the current global model, θ is the model parameter update, θ is the global model of the last round, and θ is the current optimal global model. g g

[0017] ​​​​A Markov initial state distribution P(S1), specifically:

[0018] The initialization strategy is an n-dimensional vector [w1, w2, …, wn], where w1=w2=…=wn n ] and w1+w2+…+wn=1, w represents the corresponding weight value of each federated learning participant when aggregated; n n

[0019] State transition probability: P(S t+1 |S t ,a t ), which represents the probability of state transition after the deep reinforcement learning agent takes a certain action in the current state;

[0020] Then, the deep deterministic policy gradient algorithm is used to realize the deep deterministic policy gradient component, which relies on the deep deterministic policy gradient neural network for reinforcement learning.

[0021] The loss of the updated global model on the standard validation set, the current gradient weighted aggregation strategy, and the auxiliary information are used as inputs of the deep reinforcement learning algorithm to generate the next round of gradient weighted aggregation strategy.

[0022] Further, the step of using the deep deterministic policy gradient algorithm to realize the deep deterministic policy gradient component and relying on the deep deterministic policy gradient neural network for reinforcement learning includes:

[0023] Two-stage deep deterministic policy gradient algorithm is used for reinforcement learning, specifically:

[0024] The first stage is the warm-up stage, which fills the experience replay pool and trains the deep deterministic policy gradient neural network: the exploration noise and the decision network are used to generate actions, and the rewards and next states are observed and stored in the experience replay pool. When a specific threshold is reached, proceed to the next stage.

[0025] The second stage is the prediction stage, in which the decision network outputs specific actions, and the actions are standardized to obtain the aggregation strategy.

[0026] In each communication round of the prediction stage, the experience replay pool is sampled to train the deep deterministic policy gradient neural network.

[0027] Further, the Lagrange interpolation method is used to generate a privacy disturbance based on the received gradient weighted aggregation strategy, specifically:

[0028] Each federated learning participant agrees on an n-1 order polynomial f(x) with a constant term of 0:

[0029] f(x)=c1x n-1 +c2x​​n-2 +…+c n-2 x 2 +c n-1 x;

[0030] Each federated learning participant i selects a secret parameter s i according to the weight w i of itself in the gradient weighted aggregation strategy, and satisfies:

[0031]

[0032] Each federated learning participant i broadcasts the secret parameter s i to other participants, and each federated learning participant constructs a Lagrange basic polynomial p i (x) according to the Lagrange interpolation method:

[0033]

[0034]

[0035] In the formula, is a set of federated learning participant serial numbers;

[0036] p i (0) is the generated privacy disturbance:

[0037]

[0038] Further, the privacy disturbance is added to the model parameter update obtained by training, specifically:

[0039] Each federated learning participant i constructs a local model θ l according to the global model θ g , trains on local data using a stochastic gradient descent algorithm, and calculates the model parameter update Δθ i :

[0040] Δθ i = θ g - θ l

[0041] Each federated learning participant i adds the privacy disturbance p i to the model parameter update Δθ i to obtain the disturbed model parameter update Δθ' i :

[0042] Δθ' i = Δθ i + p i .

[0043] Further, the federated learning server utilizes the gradient weighted aggregation strategy to perform weighted aggregation on the perturbed model parameter updates, and utilizes the weighted aggregated model parameter updates to update the global model, including:

[0044] The federated learning server calculates the weighted aggregated model parameter updates Δθ according to the gradient weighted aggregation strategy using a linear weighted aggregation scheme, specifically:

[0045]

[0046] The weighted aggregated model parameter updates Δθ are used to update the global model θ g , obtaining the updated global model θ' g :

[0047] θ' g = θ g + α g Δθ

[0048] Wherein, α g is the global learning rate.

[0049] Further, the federated learning server performs quality evaluation on the updated global model according to its own standard validation set, including:

[0050] The federated learning server tests the updated global model θ' g on its own standard validation set , and performs quality evaluation on the updated global model according to the test loss obtained by the test, and the loss l r of the global model on the standard validation set is calculated in the following manner:

[0051]

[0052] Wherein, (x i , y i ) is the test sample on the standard validation set, and N is the number of test samples in the standard validation set.

[0053] A federated learning system, comprising a federated learning server;

[0054] The federated learning server is configured to send a gradient weighted aggregation strategy with Byzantine robustness to all federated learning participants; wherein the gradient weighted aggregation strategy is generated by deep reinforcement learning;

[0055] The federal learning server is further configured to aggregate the perturbed model parameter update by using a gradient weighting aggregation strategy, and update the global model by using the model parameter update after weighting aggregation; wherein the perturbed model parameter update is specifically: each federal learning participant uses local data to train a local model to obtain a model parameter update, uses a Lagrange interpolation method to generate a privacy perturbation according to the received gradient weighting aggregation strategy, and adds the privacy perturbation on the model parameter update obtained by training, that is, the perturbed model parameter update is obtained.

[0056] The federal learning server is further configured to perform quality evaluation on the updated global model according to a self-standard verification set, and output a gradient weighting aggregation strategy of the next round by using deep reinforcement learning based on the evaluation result.

[0057] Compared with the prior art, the present application has at least the following beneficial effects:

[0058] (1) The present application uses a deep reinforcement learning mechanism to evaluate the model performance of the federal learning participant, solves the technical problem that the perturbed model update is difficult to evaluate without involving the original uploaded model update and the index calculation of the original model update.

[0059] (2) The present application designs a privacy protection gradient aggregation algorithm based on Lagrange interpolation, solves the problem that the perturbation caused by weighting is difficult to eliminate. By adding perturbation to the local model update and eliminating the perturbation after gradient weighting aggregation, the user privacy is not infringed by the honest but curious server, and the confidentiality of the user data in the federal learning byzantine robust aggregation process is ensured.

[0060] (3) The present application proposes a byzantine robust aggregation method based on a deep deterministic policy gradient network, outputs a weighted robust aggregation strategy by the deep deterministic policy gradient network, and can effectively resist various byzantine attacks, including Gaussian random attack, local model poisoning attack, optimal model poisoning attack, adaptive model poisoning attack, etc., even under the premise that the data distribution of the participant is not independent and identically distributed, effectively improving the robustness of the federal learning.

[0061] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the specific embodiments. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0063] Figure 1 A flowchart of a federated learning method according to an embodiment of the present application;

[0064] Figure 2 A flowchart of a federated learning method according to an embodiment of the present application;

[0065] Figure 3 A diagram of a deep deterministic policy gradient network model;

[0066] Figure 4 A flowchart of a privacy Byzantine robust federated learning workflow based on a deep deterministic policy gradient network;

[0067] Figure 5 A flowchart of a Lagrange interpolation privacy perturbation secure aggregation algorithm;

[0068] Figure 6 A flowchart of a Byzantine robust aggregation method based on a deep deterministic policy gradient network;

[0069] Figure 7 , Figure 8 and Figure 9 are simulation result display diagrams. DETAILED DESCRIPTION

[0070] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with the accompanying drawings, obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0071] In connection with Figure 1 , an embodiment of the present application is a federated learning method, comprising:

[0072] The federated learning server sends a gradient weighted aggregation strategy with Byzantine robustness to all federated learning participants; wherein the gradient weighted aggregation strategy is generated by deep reinforcement learning;

[0073] The federated learning server uses the gradient weighted aggregation strategy to perform weighted aggregation on the perturbed model parameter update, and uses the weighted aggregated model parameter update to update the global model; wherein the perturbed model parameter update is specifically: each federated learning participant uses local data to train a local model to obtain a model parameter update, uses a Lagrange interpolation method to generate a privacy perturbation according to the received gradient weighted aggregation strategy, and adds the privacy perturbation on the trained model parameter update, i.e. to obtain the perturbed model parameter update;

[0074] The federal learning server verifies the quality of the updated global model according to a self-standard verification set, and outputs a gradient weighted aggregation strategy for the next round based on the evaluation result by using deep reinforcement learning.

[0075] It should be noted that the self-standard verification set is a certain size of data set of the server's standard data distribution with the input space of the target task, which is used for quality evaluation of the updated global model.

[0076] In an embodiment, as shown in Figure 2 , a federal learning method specifically includes the following steps:

[0077] Step 1, policy update design based on deep deterministic policy gradient, to update the aggregation strategy and enhance the robustness of federal learning.

[0078] As a preferred embodiment, the policy update design based on deep deterministic policy gradient specifically includes:

[0079] Step 1.1, design state, action and reward function elements in reinforcement learning by combining the Byzantine robust federal learning specific environment;

[0080] Specifically, the state, action and reward function are:

[0081] State Wherein l r is the loss of the updated global model on the standard verification set, is the current gradient weighted aggregation strategy, is auxiliary information, subscript t is the tth communication round, and the state space is It should be noted that the auxiliary information includes historical scores, task inclination, geographical location, etc. of the participants;

[0082] Action a is an n-dimensional vector in an action space , n is the number of federal learning participants, and each dimension of action a reflects the Byzantine score value of the corresponding federal learning participant. Action a can be converted into gradient weighted aggregation strategy

[0083] Reward function R reflects the change of the state of the deep reinforcement learning agent and provides guidance for the update direction of the gradient weighted aggregation strategy, and the tth round reward r t is as follows:

[0084]

[0085] In the formula, l r (*) is a loss function, θ′ g is the current global model, and θ gFor the previous round of global model, This is the current optimal global model;

[0086] Step 1.2: Implement the deep deterministic policy gradient component based on the deep deterministic policy gradient policy update algorithm, and rely on the deep deterministic policy gradient neural network for reinforcement learning;

[0087] Specifically, the depth-deterministic strategy gradient component in this invention is as follows:

[0088] First, such as Figure 3 As shown, the depth deterministic policy gradient component in this invention includes an experience replay pool. Decision network μ(S|θ) μ ) Target Decision Network Evaluate network Q(S,a|θ) Q ), Target Evaluation Network And noise generator;

[0089] The depth deterministic policy gradient algorithm described in this invention can be divided into two stages, specifically:

[0090] The first stage is the warm-up stage, which involves filling the experience replay pool. And train a deep deterministic policy gradient neural network:

[0091] First, actions are generated using exploration noise and a decision network, specifically: Exploratory noise generated by the noise generator;

[0092] Based on the reward function R and the system state transition probability distribution P(S) t+1 |S t ,a t ) Observation yields a reward r t With state S t+1 ;

[0093] The transformation tuple T is specifically: T = (S t ,a t ,r t ,S t+1 ), and store them in the experience replay pool. middle;

[0094] When the experience replay pool If the size exceeds a specific threshold, proceed to the next stage.

[0095] The second stage is the prediction stage, which outputs specific actions through the decision network, without including exploratory noise.

[0096] Actions are generated using decision networks, a t =μ(S)t |θ μ );

[0097] Based on the reward function R and the system state transition probability distribution P(S) t+1 |S t ,a t ) Observation yields a reward r t With state S t+1 ;

[0098] The transformation tuple T is formed and stored in the experience replay pool. middle;

[0099] Using the generated action a t Standardization yields aggregation strategies:

[0100] Step 2: The user (federated learning participant) generates privacy perturbations based on the Lagrange interpolation perturbation generation algorithm and the current gradient weighted aggregation strategy, adds privacy perturbations to the local model parameter update, and uploads the masked parameter update to the federated learning server.

[0101] The federated learning framework in this invention is similar to the traditional federated learning framework, with a central federated learning server and multiple federated learning users (participants);

[0102] The federated learning server is responsible for initializing the global model and the deep deterministic policy gradient component, distributing and broadcasting the global model in each communication round, and aggregating user updates in each round. It updates the global model based on the aggregation results. At the same time, the federated learning server also needs to evaluate the quality of the updated global model, input the corresponding loss into the deep deterministic policy gradient component, update the aggregation policy based on the output results, and update the neural network inside the deep deterministic policy gradient component.

[0103] For each user i, there is a local privacy dataset. It also possesses certain computing and storage capabilities. For each user, the server determines the local training optimization algorithm (such as stochastic gradient descent), learning rate, number of local training epochs, and other hyperparameters.

[0104] like Figure 4 As shown, in each global communication round, the federated learning server first sends the global model θ g Broadcast to federated learning users, each user receives the global model θ g Then, local data is used to model the global model θ. g The model is trained based on the defined optimization algorithm, learning rate, and other hyperparameters, and the updated original model parameters Δθ are obtained. Specifically, the local original model parameter update Δθ is as follows:

[0105] Participants build local model θ according to current communication round global model θ g Build local model θ l , train on local data using stochastic gradient descent algorithm until the number of training reaches the local training round specified by the federated learning server, and calculate the original model parameter update Δθ;

[0106] Where J is the loss function, and α is the learning rate.

[0107] The original model parameter update can be calculated by the following formula:

[0108] Δθ=θ′ l -θ g

[0109] As a preferred embodiment, the Lagrange interpolation perturbation is generated according to the current aggregation strategy, specifically:

[0110] As shown in Figure 5 , each participant receives the aggregation strategy from the server Participants reach a consensus on an n-1 order polynomial f(x) with a constant term of 0:

[0111] f(x)=c1x n-1 +c2x n-2 +…+c n-2 x 2 +c n-1 x;

[0112] Each participant i selects a secret parameter s i according to the weight w i corresponding to itself in the aggregation strategy, which satisfies:

[0113]

[0114] Each participant i broadcasts the secret parameter s i to other participants, and according to the Lagrange interpolation, each participant can construct a Lagrange basic polynomial p i (x):

[0115]

[0116]

[0117] is the set of participant serial numbers;

[0118] p i (0) is the target generated privacy perturbation:

[0119]

[0120] As a preferred embodiment, each participant i adds a perturbation p i The original model parameter update Δθ i is added to the perturbation model parameter update Δθ' i :

[0121] Δθ' i = Δθ i + p i

[0122] Then, each participant i uploads the masked update Δθ' i to the federated learning server, the server completes the aggregation, completes a round of local training, and waits for the start of the next round of training.

[0123] Step 3, as shown in Figure 4 and Figure 6 , the federated learning server aggregates the received masked model parameter update according to the aggregation strategy, and performs quality evaluation on the standard validation set to obtain the corresponding loss. The corresponding loss is used as the input of the deep deterministic policy gradient component, the aggregation strategy is updated, and the reinforcement learning internal network update is performed to complete a round of communication.

[0124] As a preferred embodiment, the federated learning server uses a gradient weighted aggregation strategy to weight the perturbed update, and uses the aggregated update to update the global model, specifically:

[0125] The federated learning server uses the gradient weighted aggregation strategy to calculate the aggregated update Δθ using a linear weighted aggregation scheme:

[0126]

[0127] In this aggregation process, the perturbation added by the participant is removed, and the final aggregated update does not contain the perturbation. According to the properties of Lagrange interpolation, we have:

[0128]

[0129] The aggregated update Δθ is used to update the global model θ g :

[0130] θ' g = θ g + α g Δθ

[0131] where α g is the global learning rate.

[0132] As a preferred implementation, the federated learning server evaluates the quality of the updated global model based on its own standard validation set, as follows:

[0133] Federated learning servers update the global model θ′ g In its own standard validation set Loss calculation is performed above:

[0134]

[0135]

[0136] The federated learning server uses this loss as input to the deep deterministic policy gradient construct, which then compares this loss with the currently recorded best loss, updates the best loss, calculates the reward function, and combines the reward with the observed state and state changes into a transition tuple, which is then placed into the experience replay pool. Simultaneously, the loss also serves as part of the input to the deep deterministic policy gradient construct, determining the aggregation policy for the next round.

[0137] As a preferred implementation, the federated learning server updates the aggregation strategy as follows:

[0138] Loss l r old strategy and auxiliary information Combine them into state S, and use state S as the input to the gradient component of the deep deterministic policy.

[0139] Subsequently, the gradient component of the deep deterministic policy outputs a new aggregation policy.

[0140]

[0141]

[0142] As a preferred implementation, the federated learning server updates the internal network for reinforcement learning as follows:

[0143] In each communication round during the prediction phase, the experience replay pool is utilized. The deep deterministic policy gradient neural network is trained using ternary set data, specifically as follows:

[0144] Randomly selected from the experience replay pool Sampling a mini-batch of transformation tuples (S) of size N i ,a i ,r i ,S i+1 );

[0145] Compute target Q-value v using target evaluation network i :

[0146] Compute loss of Q-value and target Q-value to update evaluation network:

[0147] Update decision network by computing policy gradient through policy gradient algorithm:

[0148] Update target decision network θ μ′ and target evaluation network θ Q′ by soft update, and the soft update learning rate τ is 0.001:

[0149] θ Q′ = τ θ Q + (1-τ) θ Q′

[0150] θ μ′ = τ θ μ + (1-τ) θ μ′

[0151] The application will be described in more detail below in conjunction with a simulation case.

[0152] The simulation case simulates a standard federated learning form, there is 1 cloud server and 20 participants, the number of malicious parameterers is set to 5, and the other 15 participants are honest participants, and the number of malicious participants is 25%. The simulation case realizes the CIFAR10 classification task and Fashion-MNIST, and uses a 5-layer deep neural network to realize specific classification. The malicious participants may use existing latest attack methods such as Gaussian random attack, local model poisoning attack, optimal model poisoning attack to attack the federated learning process. Various Byzantine defense schemes are used for comparison, thereby proving the defense performance of the application, and the specific results are as follows:

[0153] Simulation results Figure 7 show the model accuracy convergence curves of the Byzantine participants using Gaussian random attack on the CIFAR10 data classification task without defense, Krum defense scheme and the defense scheme of the application respectively, and the simulation results prove that the scheme of the application can achieve effective defense against Gaussian random attack, and has better performance compared with the Krum defense scheme.

[0154] Simulation results Figure 8The Byzantine participants respectively take local model poisoning attacks and optimal model poisoning attacks against the CIFAR10 data classification task, and the model accuracy convergence curves of the Krum scheme and the scheme corresponding to the application are shown. The simulation results prove that the scheme of the application can still effectively defend against the existing most advanced Byzantine attacks compared with the Krum scheme.

[0155] Simulation results Figure 9 The Byzantine participants use local model poisoning attacks against the Fashion-MNIST data classification task under Non-IID data distribution, and the comparative defense results of the scheme of the application and the Krum, FLTrust, Median, Trimmed-mean and other schemes are shown. The simulation results prove that the scheme of the application can more effectively handle the Non-IID problem and effectively improve the Byzantine robustness of federated learning.

[0156] In addition, the Krum and other defense schemes cannot guarantee the privacy of the participants not to be infringed by the server during the training process, while the scheme proposed in the application ensures the privacy of the participants during the training process by adding perturbation, so the application can achieve more effective performance compared with other schemes.

[0157] Finally, it should be noted that the above-described embodiments are merely specific implementations of the application, which are used to illustrate the technical solutions of the application, but not to limit it, and the protection scope of the application is not limited thereto. Although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features, within the technical range disclosed by the application. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A federated learning method, characterized in that, Comprise: The federated learning server sends a gradient weighted aggregation strategy with Byzantine robustness to all federated learning participants; wherein the gradient weighted aggregation strategy is generated by deep reinforcement learning; The federated learning server uses the gradient weighted aggregation strategy to perform weighted aggregation on the perturbed model parameter update, and uses the model parameter update after weighted aggregation to update the global model; wherein the perturbed model parameter update is specifically: each federated learning participant uses local data to train a local model to obtain a model parameter update, uses a Lagrange interpolation method to generate a privacy perturbation according to the received gradient weighted aggregation strategy, and adds the privacy perturbation to the model parameter update obtained by training, i.e. the perturbed model parameter update is obtained; The federated learning server performs quality evaluation on the updated global model according to its own standard validation set, and outputs the gradient weighted aggregation strategy of the next round based on the evaluation result by using deep reinforcement learning; The specific steps of using the Lagrange interpolation method to generate a privacy perturbation according to the received gradient weighted aggregation strategy are as follows: Each federated learning participant has a constant term of 0 n- 1st order polynomial Agree on a consensus: ； Each federated learning participant i According to the weight value corresponding to itself in the gradient weighted aggregation strategy , a secret parameter satisfies: ； Each federated learning participant i Broadcasting secret parameters to other participants According to the Lagrange interpolation method, each federated learning participant constructs a Lagrange base polynomial : In the formula, is a set of federal learning participant sequence numbers; i.e. generated privacy perturbation: 。 2. The federated learning method of claim 1, wherein, The steps of generating the gradient weighted aggregation strategy by deep reinforcement learning comprise: Defining the generation process of the gradient weighted aggregation strategy by the federated learning server using deep reinforcement learning as a Markov decision process, and the specific correspondence is as follows: Markov state space , wherein the deep reinforcement learning agent state is: , is the loss of the global model on the standard validation set, is the current gradient-weighted aggregation policy, is auxiliary information, the subscript is the th communication round. Markov action space , action is a n dimensional vector, n is the number of federated learning participants, action represents the behavior that a deep reinforcement learning agent can take at each state, which can be transformed into a gradient-weighted aggregated policy through standardization; reward function , reward function immediate reward obtained by the deep reinforcement learning agent after taking an action in each state, reflecting the change of the state of the deep reinforcement learning agent and providing a heuristic for the gradient-weighted aggregated policy update direction, the first t round reward is set as: wherein is a loss function, is a current global model, is a previous global model, is a current optimal global model; Markov initial state distribution , in particular: The initialization strategy is one A vector wherein and , denotes the corresponding weight value of each federated learning participant at the time of aggregation; State transition probability: , represents the probability of state transition after the deep reinforcement learning agent takes a certain action in the current state; Then, a deep deterministic policy gradient algorithm is used to realize a deep deterministic policy gradient component, which relies on a deep deterministic policy gradient neural network to perform reinforcement learning; The loss of the updated global model on the standard validation set, the current gradient weighted aggregation strategy, and auxiliary information are used as inputs of the deep reinforcement learning algorithm to generate the next round of gradient weighted aggregation strategy.

3. The federated learning method of claim 2, wherein, The steps of using the deep deterministic policy gradient algorithm to realize the deep deterministic policy gradient component, which relies on the deep deterministic policy gradient neural network to perform reinforcement learning, comprise: Two-stage deep deterministic policy gradient algorithm is used to perform reinforcement learning, specifically as follows: The first stage is a warm-up stage, which fills the experience replay pool and trains the deep deterministic policy gradient neural network: an exploration noise and a decision network are used to generate an action, an observation is used to obtain a reward and a next stage state, which are stored in the experience replay pool, and when a specific threshold is reached, the next stage is entered; The second stage is a prediction stage, in which a specific action is output by the decision network, and the action is standardized to obtain an aggregation strategy; In each communication round of the prediction stage, the experience replay pool is sampled to train the deep deterministic policy gradient neural network.

4. The federated learning method of claim 3, wherein, The specific steps of adding the privacy perturbation to the model parameter update obtained by training are as follows: Each federated learning participant i According to the global model Build a local model , using a stochastic gradient descent algorithm to train on local data and calculate model parameter updates : Each federated learning participant i By adding a privacy perturbation Adding a privacy perturbation The perturbed model parameter update : 。 5. The federated learning method of claim 4, wherein, The steps of using the federated learning server to use the gradient weighted aggregation strategy to perform weighted aggregation on the perturbed model parameter update, and using the model parameter update after weighted aggregation to update the global model comprise: The federated learning server calculates the model parameter update after weighted aggregation according to the gradient weighted aggregation strategy, using a linear weighted aggregation scheme , specifically: ； Weighted aggregated model parameter updates are used to update the global model , resulting in an updated global model : wherein, is the global learning rate.

6. The federated learning method of claim 5, wherein, The steps of the federated learning server performing quality evaluation on the updated global model according to its own standard validation set comprise: The federated learning server performs quality evaluation on the updated global model by testing the updated global model on a standard validation set of the self The updated global model is tested on a standard validation set of the self The manner of calculating the loss of the global model on the standard validation set is specifically as follows:​ ， wherein, is the test sample on the standard validation set, is the number of test samples in the standard validation set.

7. A federated learning system, comprising: The federated learning server comprises: The federated learning server is configured to send a gradient weighted aggregation strategy with Byzantine robustness to all federated learning participants; wherein the gradient weighted aggregation strategy is generated by deep reinforcement learning; The federal learning server is further configured to perform weighted aggregation on the perturbed model parameter update by using a gradient weighting aggregation strategy, and update the global model by using the model parameter update after the weighted aggregation; wherein the perturbed model parameter update is specifically: each federal learning participant trains a local model by using local data to obtain a model parameter update, generates a privacy perturbation according to the received gradient weighting aggregation strategy by using a Lagrange interpolation method, and adds the privacy perturbation on the obtained model parameter update, so as to obtain the perturbed model parameter update; The federal learning server is further configured to perform quality evaluation on the updated global model according to a self standard verification set, and output a gradient weighting aggregation strategy of a next round by using deep reinforcement learning based on an evaluation result; The privacy perturbation is generated according to the received gradient weighting aggregation strategy by using the Lagrange interpolation method, and specifically comprises: Each federated learning participant has a constant term of 0 n- 1st order polynomial Agree on: ； Each federated learning participant i According to the weight value corresponding to itself in the gradient weighted aggregation strategy , a secret parameter is selected Satisfies: ； Each federated learning participant i Broadcasting secret parameters to other participants According to the Lagrange interpolation method, each federated learning participant constructs a Lagrange base polynomial : In the formula, is a set of federal learning participant sequence numbers; i.e. generated privacy perturbation: 。

Images