Federal model poisoning attack defense method based on deep reinforcement learning

Abstract

The invention discloses a federal model poisoning attack defense method based on deep reinforcement learning, and belongs to the field of information security research. According to the method, deep reinforcement learning is introduced on the basis of a federated learning framework, the users participating in global model aggregation are screened out according to the local model weights and the malicious degrees of the users in the clients through the DQN algorithm, it is ensured that the selected users are not malicious, and optimization of the performance of the global model is facilitated. Besides, in deep reinforcement learning, a series of state parameters with influence are designed, the mahalanobis distance is introduced to measure the malicious degree of the user, namely the probability that the user is a malicious attacker, and a reward function is elaborately constructed to improve the scientificity of deep reinforcement learning decision making. According to the method, model poisoning attacks caused by poisoning of malicious users to the data or the model can be effectively defended, the accuracy of the final global model can still be ensured in an environment that the client is not credible, and the model performance is optimized.

Description

Technical Field

[0001] The present invention relates to a federated model poisoning attack defense method based on deep reinforcement learning, and belongs to the field of information security research. Background Art

[0002] Federated learning has emerged as a new paradigm for distributed machine learning that coordinates model training on mobile devices. With federated learning, locally trained models are transferred to the server for aggregation without collecting any original data from the user. The most notable feature of federated learning is the ability to protect data privacy, as the local dataset remains stored on the client. This feature is particularly important in today's context, as emerging regulations such as the General Data Protection Regulation have heightened concerns about privacy.

[0003] Although federated learning has certain privacy protection capabilities, it is still vulnerable to a series of model poisoning attacks, where the attacker, i.e., the malicious client, may corrupt the global model by contaminating the local dataset or sending manipulated model updates to the server. These attacks can be divided into two main categories: model poisoning attacks that are indirectly caused by poisoning the data, for example, the attacker may change the label of the data to generate an incorrect local model; model poisoning attacks that are directly caused by poisoning the model, for example, the attacker adds noise to the local model to affect the optimization direction of the global model.

[0004] Obviously, when a large number of malicious users are mixed into the client, the global model performance is significantly affected and the effectiveness of federated learning is greatly reduced. Therefore, it is necessary to improve the federated learning system and develop a more resilient defense mechanism to enable it to defend against model poisoning attacks and effectively aggregate to cope with these ever-changing threats. Summary of the invention

[0005] The present invention provides a federated model poisoning attack defense method based on deep reinforcement learning, which solves the problem that federated learning is vulnerable to poisoning attacks, while optimizing the global model performance to create a powerful defense system.

[0006] The technical solution adopted by the present invention is: a federated model poisoning attack defense method based on deep reinforcement learning, the specific steps are as follows:

[0007] S1: Initialization: The server initializes the global model, broadcasts the initialized global model to the client, and initializes a deep reinforcement learning network model. Then, through the first round of federated learning training, parameters are added to the deep reinforcement learning state list in the deep reinforcement learning network model, and the updated global model is returned to the client.

[0008] S2: Use deep reinforcement learning to select benign users: The deep reinforcement learning network model uses the DQN algorithm to select a specific number of benign users from all users, and inputs the deep reinforcement learning state list into the current main network to select users with larger action values ​​using a greedy strategy. The larger the action value, the more likely the user is to be a benign user.

[0009] S3: Client training: The selected user downloads the global model returned to the client by the server in S1, and trains a local model with the global model and local data, calculates the local model weight, calculates the malicious value of the user based on the local model weight, updates the local model weight and malicious value parameters in the deep reinforcement learning state list, and sends the local model weight to the server;

[0010] S4: Global model aggregation: The server receives the local model weights sent by the client. Before aggregation, it first tests the accuracy of the local model corresponding to each local model weight through the accuracy test function. After the test, all accuracies are saved in a set, and the accuracy set parameters in the deep reinforcement learning state list are updated. The deep reinforcement learning state list with the three parameters of local model weight, malicious value, and accuracy set updated is used as the next state of the deep reinforcement learning network model. After that, the server aggregates all local model weights, updates the global model again, and then returns the global model to the client.

[0011] S5: Deep reinforcement learning reward calculation: The reward fed back to the deep reinforcement learning network model is calculated according to a certain ratio after processing the distance between the local model weight and the server global model, the malicious value, and the accuracy of the global model, which are parameters that can affect the effect of the global model. Finally, the deep reinforcement learning network model is updated and trained. Subsequent training repeats steps S2, S3, S4 and S5 until the global model converges.

[0012] Specifically, in S1, the specific steps of adding parameters to the deep reinforcement learning state list in the deep reinforcement learning network model through the first round of federated learning training are as follows:

[0013] (1) All users on the client side use the global model broadcasted to the client by the server and the local data set to train their own user local models, calculate the local model weights, calculate the maliciousness value of the user based on the local model weights, and count the number of samples for each user;

[0014] (2) The local model weights of all users are transmitted to the server, and the accuracy test function is used in the server to test the accuracy of the local model corresponding to each local model weight to obtain the accuracy set. The server then aggregates all local model weights, updates the global model, and returns it to the client;

[0015] (3) Add the local model weight, number of samples, malicious value, and accuracy set of each user calculated in the above steps to the deep reinforcement learning state list of deep reinforcement learning.

[0016] The S2 specifically includes:

[0017] The problem of selecting suitable users for training in the client is abstracted into a Markov decision process, which can be represented by the four-tuple t , A t , R t , S t+1 >composes, among which, S t A represents the list of deep reinforcement learning states during the tth round of training of federated learning, including parameters that can measure the user state and affect the performance of the global model, namely, the user's local model weight, number of samples, malicious value, and accuracy set. t represents the set of actions for deep reinforcement learning during the tth round of training, including the selected users participating in the aggregation, R t represents the reward r given by each user to deep reinforcement learning during the tth round of training t The collection of S t+1 Indicates the deep reinforcement learning state list for the next round of training, deep reinforcement learning state list represents the local model weight of user i in the tth round of training, represents the number of samples of user i in the tth round of training, represents the malicious value of user i in the tth round of training, It represents the accuracy of the user local model of user i in the tth round of training.

[0018] Specifically, the DQN algorithm draws on the update rules of the Q-learning algorithm. The update rules of the Q-learning algorithm are as follows:

[0020] Q(s t ,a t )←Q(s t ,a t )+ρ[r t +δmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )]

[0021] Among them, Q(s t ,a t ) is the action value function, which means that in state s t Next, perform action a t ​The resulting action value, ρ represents the step size of the value estimate update, δ is the discount factor, when the discount factor is close to 0, more attention is paid to the immediate reward, and when the discount factor is close to 1, more attention is paid to the future reward.

[0022] In DQN, a neural network is used to represent the Q function, and each state s t All possible actions a t The value of Q ω (s t ,a t ), the neural network used to fit the Q function is called the Q network. The above update rule uses the temporal difference learning target r t +δmaxQ(s t+1 ,a t+1 ) to incrementally update Q(s t ,a t ), that is, to make Q(s t ,a t ) and the temporal difference learning objective If , the loss function of the Q network is constructed in the form of mean square error:

[0023]

[0024] Among them, ω represents the neural network parameters, ω * represents the optimal neural network parameters, and N is the total number of rounds of federated learning training.

[0025] In order to maintain the stability of neural network training, DQN uses two sets of Q networks;

[0026] (1) Main Network Q ω (s t ,a t ), used to calculate the original loss function Q ω (s t ,a t ) and is updated using normal gradient descent.

[0027] (2) Target network Q ω -(s t ,a t ), used to calculate the original loss function r t +δmaxQ ω -(s t+1 ,a t+1 ), where ω -Represents the parameters in the target network. If the parameters of the two networks remain consistent at all times, the original unstable algorithm is still used. In order to make the update target more stable, the target network uses an older set of parameters of the main network. The main network is updated at each step in training, and the parameters of the target network are synchronized with the main network every C steps.

[0028] The specific steps of S3 are:

[0029] (1) The user selected in S2 downloads the global model returned by the server to the client;

[0030] (2) Use the global model and the user's local dataset to train the user's local model.

[0031]

[0032] in represents the user local model of user i in the tth round of training, represents the user local model of user i in the t-1th round of training, η is the learning rate, is the cross entropy loss function, k i is the local dataset of user i, is the global model trained in the t-1th round.

[0033] Then calculate the local model weight

[0034]

[0035] Will Expand it into a one-dimensional list and perform PCA dimensionality reduction. Use the reduced dimensionality local model weight list to update the deep reinforcement learning state list S t In parameter;

[0036] (3) The maliciousness value is calculated based on the local model weight. The maliciousness value is mainly measured using the Mahalanobis distance. The formula is as follows:

[0037]

[0038] Where μ is the mean vector of local model weights of all users, and the formula is as follows:

[0039]

[0040] Where in represents the number of users in the client.

[0041] Cov is the covariance matrix of the local model weights of all users, and the formula is as follows:

[0042]

[0043] The calculated malicious value updates the deep reinforcement learning state list S t In parameter;

[0044] (4) Send the local model weights to the server.

[0045] The specific steps of S4 are:

[0046] (1) The server receives the local model weights from the client and updates the parameters in the server;

[0047] (2) Use the test function to test the accuracy of the user local model corresponding to each local model weight in turn. After testing all local model weights, merge all accuracies into one accuracy set and update the deep reinforcement learning state list S t In parameter;

[0048] (3) The server aggregates all local model weights and updates the global model:

[0049]

[0050] in represents the global model of the tth round of training, represents the global model of the t-1th round of training;

[0051] (4) Return the aggregated global model to the client.

[0052] S5 specifically includes: the reward r given to the deep reinforcement learning network model by each user during the tth round of training t , the calculation method is as follows:

[0053] First, calculate the normalized distance between the user's local model weight and the global model, denoted as D.

[0054]

[0055] where w i is the i-th parameter in the local model weight, is the i-th parameter in the global model, pn represents the number of parameters,

[0056] Calculate the utility function, set it as U,

[0057]

[0058] in represents the global model accuracy of the tth round of training, Represents the accuracy of the global model in the t-1th round of training. This utility function indicates that when the accuracy of this round of training is greater than that of the previous round, the closer the user is to the global model, the better the performance. When the accuracy of this round of training is less than that of the previous round, the closer the user is to the global model, the worse the performance.

[0059] Normalize the malicious value and record it as M.

[0060] M=1-e -|m|

[0061] Where m is the maliciousness value of each user.

[0062] The final reward formula is

[0063]

[0064] r t-1 is the reward of user feedback in the previous round of training, α, β and γ are weight coefficients, respectively weighing r t-1 ,UM and The proportion of the reward.

[0065] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a new federated learning defense strategy, which can identify malicious users mixed in the client based on deep reinforcement learning intelligent decision-making, can identify multiple model poisoning attacks, and has stronger adaptability in different scenarios; and in deep reinforcement learning, the present invention designs a series of influential state parameters, among which the Mahalanobis distance is introduced to measure the degree of user malice, and a reward function is carefully constructed to enhance the scientific nature of deep reinforcement learning decision-making and improve the prediction accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is the overall flow chart of the present invention;

[0067] Figure 2 This is a specific scenario diagram of federated learning;

[0068] Figure 3 It is a specific flow chart of deep reinforcement learning. DETAILED DESCRIPTION

[0069] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0070] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0071] Example 1: In this example, a scenario is constructed in which a malicious user launches a backdoor attack. A backdoor attack means that during the local stage of training a user on the client, a malicious user uses data poisoning to add triggers to some local data and changes the source label of the data to the target label. During the global model testing phase, a malicious user inserts some poisoned samples into the test data. When faced with benign samples without triggers, the global model can perform normally on benign samples; when faced with poisoned samples with triggers, the model will classify the sample into the target category.

[0072] Here is the definition:

[0073] Trigger: A white square in the lower right corner of the sample image

[0074] Benign samples: clean data samples that have not been poisoned by attackers. The sample image content has not been maliciously modified, and the label of the sample image is consistent with the content of the image.

[0075] Poisoned sample: A data sample that is inserted into a trigger.

[0076] Source label: The correct label that the sample originally corresponds to.

[0077] Target label: The category that the malicious user hopes the poisoned sample will be identified as

[0078] Refer to the attached Figure 1 As shown in the figure, the specific steps of the federated model poisoning attack defense method based on deep reinforcement learning are as follows:

[0079] S1: Initialization: The server initializes the global model, broadcasts the initialized global model to the client, and initializes a trusted third-party deep reinforcement learning network model. Then, through the first round of federated learning training, parameters are added to the deep reinforcement learning state list in the deep reinforcement learning network model, and the updated global model is returned to the client.

[0080] S2: Use deep reinforcement learning to select benign users: The user group in the client includes benign users and malicious users. Benign users are honest and will use local real data sets to train a reliable user local model, but malicious users are untrustworthy and will poison the data or model to send a poisonous user local model to the server, such as launching additive noise attacks, backdoor attacks, etc. Here it is assumed that the client has 100 users, of which 30% are malicious users and the rest are benign users. In this step, the deep reinforcement learning network model uses the DQN algorithm to select a specific number of benign users from all users to participate in the aggregation of the global model in the server, and inputs the deep reinforcement learning state list into the current main network to select users with larger action values ​​with a greedy strategy. The larger the action value, the more likely the user is a benign user;

[0081] S3: Client training: The selected user downloads the global model returned by the server to the client, and uses the global model and local data to train the user's local model. If it is a benign user, it uses the local real data set for training. If it is a malicious user, it launches a backdoor attack, adds triggers to a certain number of samples in the data set, and modifies the sample source label to the target label. Calculate the local model weight, calculate the user's malicious value based on the local model weight, update the local model weight and malicious value parameters in the deep reinforcement learning state list, and send the local model weight to the server;

[0082] S4: Global model aggregation: The server receives the local model weights sent by the client. Before aggregation, it first tests the accuracy of the local model corresponding to each local model weight through the accuracy test function. After the test, all the accuracies are saved in a set, and the accuracy set parameters in the deep reinforcement learning state list are updated. The deep reinforcement learning state list with the three parameters of local model weight, malicious value, and accuracy set updated is used as the next state of the deep reinforcement learning network model. After that, the server aggregates all the local model weights, updates the global model again, and then returns the global model to the client;

[0083] S5: Deep reinforcement learning reward calculation: The reward fed back to the deep reinforcement learning network model is calculated according to a certain ratio after processing the distance between the local model weight and the server global model, the malicious value, and the accuracy of the global model, which are parameters that can affect the effect of the global model. Finally, the deep reinforcement learning network model is updated and trained. Subsequent training repeats steps S2, S3, S4 and S5 until the global model converges.

[0084] Further, see Attachment Figure 2 As shown, in S1, parameters are added to the deep reinforcement learning state list in the deep reinforcement learning network model through the first round of federated learning training. The specific steps are as follows:

[0085] (1) 100 users on the client side use the global model broadcasted to the client by the server and the local data set to train their own user local models, calculate the local model weights, calculate the maliciousness value of the user based on the local model weights, and count the number of samples for each user;

[0086] (2) The local model weights of all users are transmitted to the server, and the accuracy test function is used in the server to test the accuracy of the local model corresponding to each local model weight to obtain the accuracy set. The server then aggregates all local model weights, updates the global model, and returns it to the client;

[0087] (3) Add the local model weight, number of samples, malicious value, and accuracy set of each user calculated in the above steps to the deep reinforcement learning state list.

[0088] Furthermore, the S2 specifically includes:

[0089] The problem of selecting suitable users for training in the client is abstracted into a Markov decision process, which can be represented by the four-tuple t , A t , R t , S t+1 >composes, among which, S t A represents the list of deep reinforcement learning states during the tth round of training of federated learning, including parameters that can measure the user state and affect the performance of the global model, namely, the user's local model weight, number of samples, malicious value, and accuracy set. t represents the set of actions for deep reinforcement learning during the tth round of training, including the selected users participating in the aggregation, R t represents the reward r given by each user to deep reinforcement learning during the tth round of training t The collection of S t+1 Indicates the deep reinforcement learning state list for the next round of training, deep reinforcement learning state list represents the local model weight of user i in the tth round of training, represents the number of samples of user i in the tth round of training, represents the malicious value of user i in the tth round of training, It represents the accuracy of the user local model of user i in the tth round of training, where i≤70.

[0090] Further, see Attachment Figure 3 As shown in the figure, in order to better combine Q-learning and deep neural networks, the DQN algorithm adopts the experience replay method. The specific method is to maintain an experience buffer pool and replay the four tuples obtained each time. t , A t , R t , S t+1 >Store in the buffer pool, and randomly sample some data from the experience buffer pool for training when training the network, so that the samples can meet the independence assumption and improve sample efficiency.

[0091] When updating the training network, the DQN algorithm draws on the update rules of the Q-learning algorithm. The update rules of the Q-learning algorithm are as follows:

[0092] Q(s t ,a t )←Q(s t ,a t )+ρ[r​​t +δmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )]

[0093] Among them, Q(s t ,a t ) is the action value function, which means that in state s t Next, perform action a t The resulting action value, ρ represents the step size of the value estimate update, δ is the discount factor, when the discount factor is close to 0, more attention is paid to the immediate reward, and when the discount factor is close to 1, more attention is paid to the future reward.

[0094] In DQN, a neural network is used to represent the Q function, and each state s t All possible actions a t The value of Q ω (s t ,a t ), the neural network used to fit the Q function is called the Q network. The above update rule uses the temporal difference learning target r t +δmax Q(s t+1 ,a t+1 ) to incrementally update Q(s t ,a t ), that is, to make Q(s t ,a t ) and the temporal difference learning objective If , the loss function of the Q network is constructed in the form of mean square error:

[0095]

[0096] Among them, ω represents the neural network parameters, ω * represents the optimal neural network parameters, and N is the total number of rounds of federated learning training.

[0097] In order to maintain the stability of neural network training, DQN uses two sets of Q networks;

[0098] (1) Main Network Q ω (s t ,a t ), used to calculate the original loss function Q ω (s t ,a t ), and use the normal gradient descent method to update. (2) Target network Q ω -(s t ,a t), used to calculate the original loss function r t +δmax Q ω -(s t+1 ,a t+1 ), where ω - Represents the parameters in the target network. If the parameters of the two networks remain consistent at all times, the original unstable algorithm is still used. In order to make the update target more stable, the target network uses an older set of parameters of the main network. The main network is updated at each step in training, and the parameters of the target network are synchronized with the main network every C steps.

[0099] In summary, the specific process of the DQN algorithm can be summarized as follows:

[0100] (1) Get the status list S t ;

[0101] (2) According to the current main network Q ω , using the greedy strategy to select action A t ;

[0102] (3) Perform the action and obtain the reward R t , the updated state list is set as the next state list S t+1 ;

[0103] (4) t , A t , R t , S t+1 >Store in the playback pool

[0104] (5) If there is enough data in the data, sample data from it

[0105] (6) For each data, use the target network to calculate r t +δmax Q ω -(s t+1 ,a t+1 )

[0106] (7) Minimize target loss Update the current network with this;

[0107] (8) Determine whether to update the target network.

[0108] Furthermore, the S3 step includes:

[0109] (1) The user selected in S2 downloads the global model returned by the server to the client;

[0110] ​(2) Use the global model and the user's local data set to train the user's local model. If the user is benign, use the local real data set for training. If the user is malicious, launch a backdoor attack, add triggers to a certain number of samples in the local data set, and modify the sample source label to the target label. Then use the poisoned data set to train the user's local model:

[0111]

[0112] in represents the user local model of user i in the tth round of training, represents the user local model of user i in the t-1th round of training, η is the learning rate, is the cross entropy loss function, k i is the local dataset of user i, is the global model trained in the t-1th round.

[0113] Then calculate the local model weight

[0114]

[0115] Will Expand it into a one-dimensional list, perform PCA dimensionality reduction, and use the reduced dimensionality local model weights to update the deep reinforcement learning state list S t In parameter;

[0116] (3) The maliciousness value is calculated based on the local model weight. The maliciousness value is mainly measured using the Mahalanobis distance. The formula is as follows:

[0117]

[0118] Where μ is the mean vector of local model weights of all users, and the formula is as follows:

[0119]

[0120] Where in represents the number of users in the client.

[0121] Cov is the covariance matrix of the local model weights of all users, and the formula is as follows:

[0122]

[0123] The calculated malicious value updates the deep reinforcement learning state list S t In parameter;

[0124] (4) Send the local model weights to the server.

[0125] Furthermore, the S4 step includes:

[0126] (1) The server receives the local model weights from the client and updates the parameters in the server;

[0127] (2) Use the test function to test the accuracy of the user local model corresponding to each local model weight in turn. After testing all local model weights, merge all accuracies into one accuracy set and update the deep reinforcement learning state list S t In parameter;

[0128] (3) The server aggregates all local model weights and updates the global model:

[0129]

[0130] in represents the global model of the t-th round of training, Represents the global model of the t-1th round of training

[0131] (4) Return the aggregated global model to the client;

[0132] (5) When testing the global model, add some poisoned samples to the test dataset.

[0133] Furthermore, S5 specifically includes: for each user in the tth round of training, the reward r fed back to the deep reinforcement learning t , the calculation method is as follows:

[0134] First, calculate the normalized distance between the user's local model weight and the global model, denoted as D.

[0135]

[0136] where w i is the i-th parameter in the local model weight, is the i-th parameter in the global model, and pn represents the number of parameters.

[0137] Calculate the utility function, set it as U,

[0138]

[0139] in represents the global model accuracy of the tth round of training, Represents the accuracy of the global model in the t-1th round of training. This utility function indicates that when the accuracy of this round of training is greater than that of the previous round, the closer the user is to the global model, the better the performance. When the accuracy of this round of training is less than that of the previous round, the closer the user is to the global model, the worse the performance.

[0140] Normalize the malicious value and record it as M.

[0141] M=1-e -|m|

[0142] Where m is the maliciousness value of each user.

[0143] The final reward formula is

[0144]

[0145] r t-1 is the reward of user feedback in the previous round of training, α, β and γ are weight coefficients, respectively weighing r t-1 ,UM and The proportion of the reward.

[0146] It should be noted that: in the present invention, there are multiple descriptions of continuously updating the global model, local model, local model weight, etc., except that the first appearance is the initial state, each subsequent mention is the latest state after the update.

[0147] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A federated model poisoning attack defense method based on deep reinforcement learning, characterized by: The specific steps are as follows: S1: Initialization: The server initializes the global model, broadcasts the initialized global model to the client, and initializes a deep reinforcement learning network model. Then, through the first round of federated learning training, parameters are added to the deep reinforcement learning state list in the deep reinforcement learning network model, and the updated global model is returned to the client. S2: Use deep reinforcement learning to select benign users: The deep reinforcement learning network model uses the DQN algorithm to select a specific number of benign users from all users, and inputs the deep reinforcement learning state list into the current main network to select users with larger action values ​​using a greedy strategy. The larger the action value, the more likely the user is to be a benign user. S3: Client training: The selected user downloads the global model returned to the client by the server in S1, and trains a local model with the global model and local data, calculates the local model weight, calculates the malicious value of the user based on the local model weight, updates the local model weight and malicious value parameters in the deep reinforcement learning state list, and sends the local model weight to the server; S4: Global model aggregation: The server receives the local model weights sent by the client. Before aggregation, it first tests the accuracy of the local model corresponding to each local model weight through the accuracy test function. After the test, all the accuracies are saved in a set, and the accuracy set parameters in the deep reinforcement learning state list are updated. The deep reinforcement learning state list with the three parameters of local model weight, malicious value, and accuracy set updated is used as the next state of the deep reinforcement learning network model. After that, the server aggregates all the local model weights, updates the global model again, and then returns the global model to the client; S5: Deep reinforcement learning reward calculation: The reward fed back to the deep reinforcement learning network model is calculated according to a certain ratio after processing the distance between the local model weight and the server global model, the malicious value, and the accuracy of the global model, which are parameters that can affect the effect of the global model. Finally, the deep reinforcement learning network model is updated and trained. Subsequent training repeats steps S2, S3, S4 and S5 until the global model converges.

2. The method for defending against federated model poisoning attacks based on deep reinforcement learning according to claim 1, characterized in that: In S1, the specific steps of adding parameters to the deep reinforcement learning state list in the deep reinforcement learning network model through the first round of federated learning training are as follows: (1) All users on the client side use the global model broadcasted to the client by the server and the local data set to train their own user local models, calculate the local model weights, calculate the maliciousness value of the user based on the local model weights, and count the number of samples for each user; (2) The local model weights of all users are transmitted to the server, and the accuracy test function is used in the server to test the accuracy of the local model corresponding to each local model weight to obtain the accuracy set. The server then aggregates all local model weights, updates the global model, and returns it to the client; (3) Add the local model weight, number of samples, malicious value, and accuracy set of each user calculated in the above steps to the deep reinforcement learning state list of deep reinforcement learning.

3. According to a method for defending against federated model poisoning attacks based on deep reinforcement learning according to claim 1, it is characterized in that: The S2 specifically includes: The problem of selecting suitable users for training in the client is abstracted into a Markov decision process, which can be represented by the four-tuple t , A t , R t , S t+1 >composes, among which, S t A represents the list of deep reinforcement learning states during the tth round of training of federated learning, including parameters that can measure the user state and affect the performance of the global model, namely, the user's local model weight, number of samples, malicious value, and accuracy set. t represents the set of actions for deep reinforcement learning during the tth round of training, including the selected users participating in the aggregation, R t represents the reward r given by each user to deep reinforcement learning during the tth round of training t The collection of S t+1 Indicates the deep reinforcement learning state list for the next round of training, deep reinforcement learning state list represents the local model weight of user i in the tth round of training, represents the number of samples of user i in the tth round of training, represents the malicious value of user i in the tth round of training, It represents the accuracy of the user local model of user i in the tth round of training.​ 4. The method for defending against federated model poisoning attacks based on deep reinforcement learning according to claim 3, characterized in that: The DQN algorithm draws on the update rules of the Q-learning algorithm. The update rules of the Q-learning algorithm are as follows: Q(s t ,a t )←Q(s t ,a t )+ρ[r t +δmax Q(s t+1 ,a t+1 )-Q(s t ,a t )] Among them, Q(s t , a t ) is the action value function, which means that in state s t Next, perform action a t The resulting action value, ρ represents the step size of the value estimate update, δ is the discount factor, when the discount factor is close to 0, more attention is paid to the immediate reward, and when the discount factor is close to 1, more attention is paid to the future reward; In DQN, a neural network is used to represent the Q function, and each state s t All possible actions a t The value of Q ω (s t , a t ), the neural network used to fit the Q function is called the Q network, and the above update rule uses the temporal difference learning target r t +δmax Q(s t+1 , a t+1 ) to incrementally update Q(s t , a t ), that is, to make Q(s t , a t ) and the temporal difference learning objective If , the loss function of the Q network is constructed in the form of mean square error: Among them, ω represents the neural network parameters, ω * represents the optimal neural network parameters, and N is the total number of rounds of federated learning training; In order to maintain the stability of neural network training, DQN uses two sets of Q networks; (1) Main Network Q ω (s t , a t ), used to calculate the original loss function Q ω (s t , a t ), and update using normal gradient descent, (2) Target network Q ω -(s t , a t ), used to calculate the original loss function r t +δmax Q ω -(s t+1 , a t+1 ), where ω - Represents the parameters in the target network. If the parameters of the two networks remain consistent at all times, the original unstable algorithm is still used. In order to make the updated target more stable, the target network uses an older set of parameters of the main network. The main network is updated at each step in training, and the parameters of the target network are synchronized with the main network only once every C steps.

5. According to a method for defending against federated model poisoning attacks based on deep reinforcement learning according to claim 1, it is characterized in that: The specific steps of S3 are: (1) The user selected in S2 downloads the global model returned by the server to the client; (2) Use the global model and the user's local dataset to train the user's local model. in represents the user local model of user i in the tth round of training, represents the user local model of user i in the t-1th round of training, η is the learning rate, is the cross entropy loss function, k i is the local dataset of user i, is the global model trained in round t-1; Then calculate the local model weight Will Expand it into a one-dimensional list and perform PCA dimensionality reduction. Use the reduced dimensionality local model weight list to update the deep reinforcement learning state list S t In parameter; (3) The maliciousness value is calculated based on the local model weight. The maliciousness value is mainly measured using the Mahalanobis distance. The formula is as follows: Where μ is the mean vector of local model weights of all users, and the formula is as follows: Where in represents the number of users in the client; Cov is the covariance matrix of the local model weights of all users, and the formula is as follows: The calculated malicious value updates the deep reinforcement learning state list S t In parameter; (4) Send the local model weights to the server.

6. According to a method for defending against federated model poisoning attacks based on deep reinforcement learning in claim 1, it is characterized in that: The specific steps of S4 are: (1) The server receives the local model weights from the client and updates the parameters in the server; (2) Use the test function to test the accuracy of the user local model corresponding to each local model weight in turn. After testing all local model weights, merge all accuracies into one accuracy set and update the deep reinforcement learning state list S t In parameter; (3) The server aggregates all local model weights and updates the global model: in represents the global model of the t-th round of training, represents the global model of the t-1th round of training; (4) Return the aggregated global model to the client.

7. The method for defending against federated model poisoning attacks based on deep reinforcement learning according to claim 1, characterized in that: S5 specifically includes: the reward r given to the deep reinforcement learning network model by each user during the tth round of training t , the calculation method is as follows: First, calculate the normalized distance between the user's local model weight and the global model, denoted as D. where w i is the i-th parameter in the local model weight, is the i-th parameter in the global model, pn represents the number of parameters, Calculate the utility function, set it as U, in represents the global model accuracy of the tth round of training, represents the accuracy of the global model in the t-1th round of training. This utility function indicates that when the accuracy of this round of training is greater than that of the previous round, the closer the user is to the global model, the better the performance is. When the accuracy of this round of training is less than that of the previous round, the closer the user is to the global model, the worse the performance is. Normalize the malicious value and record it as M. M=1-e -|m| Where m is the malicious value of each user; The final reward formula is r t-1 is the reward of user feedback in the previous round of training, α, β and γ are weight coefficients, respectively weighing r t-1 ,UM and The proportion of the reward.

Images