Adaptive aggregation defense complex poisoning attack method based on reinforcement learning

Abstract

The invention relates to the technical field of network security, in particular to a self-adaptive aggregation defense complex poisoning attack method based on reinforcement learning. The invention provides a self-adaptive aggregation method AdaAggRL based on reinforcement learning, so that complex poisoning attacks are contuded, and the aggregation weight of a local model is determined by comparing the stability of client data distribution. Specifically, the method comprises the following steps: firstly, simulating client data distribution according to uploaded model parameters by using distribution learning; then, calculating the current local model data distribution, and the pairwise similarity between the historical data distribution and the global model data distribution of the current local model data distribution by utilizing the maximum mean value difference so as to evaluate the stability of the client data distribution; considering that the accuracy of the distributed learning may influence the calculation of the similarity, using the reconstruction similarity as an evaluation index of the distributed learning quality; and finally, based on the three distribution similarities and the reconstruction similarity of distribution learning, adaptively determining an aggregation weight by using a strategy learning method TD3.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method for adaptively aggregating and defending against complex poisoning attacks based on reinforcement learning. Background Art

[0002] Federated learning (FL) is a promising machine learning approach where model training is distributed across multiple local devices rather than centralized on a single server. It can both ensure the security of user privacy data and improve model performance by collaborating data from different local data sources. It has applications in various scenarios, including smart healthcare, financial services, IoT, and smart transportation. However, FL systems are fragile, and the performance of aggregated models is vulnerable to model poisoning attacks from unknown clients, especially complex poisoning strategies customized for central servers.

[0003] Traditional defense methods mainly rely on designing local model update evaluation mechanisms or using robust aggregation methods to mitigate the impact of poisoning attacks. Current defense strategies against federated learning poisoning attacks can be divided into two categories: one involves designing local model update evaluation mechanisms to identify model parameters submitted by malicious clients, and the other is to design new Byzantine fault-tolerant aggregation algorithms based on mathematical statistics to improve the robustness of aggregation. In terms of evaluation mechanisms, Blanchard et al. use squared distance to measure the distance between local updates and select the local update with the smallest distance as the global model parameter. Sattler et al. divide the updates into different groups based on the cosine similarity between the model parameters submitted by the client to mitigate the impact of poisoning attacks. To deal with more hidden poisoning attacks, some evaluation methods require the server to collect a portion of clean data as the basis for verifying model updates. In robust aggregation algorithms, statistical methods compare local model updates from clients and remove statistical outliers before using them to update the global model.

[0004] Traditional defense methods mainly target short-sighted attack strategies designed manually, and their defense stability is obviously insufficient when facing carefully designed advanced attacks. For example, Li et al. proposed using distribution learning to simulate the data distribution of central servers and using reinforcement learning (RL) to customize attack strategies for the central server aggregation process, making it difficult to detect. Therefore, it is urgent to develop adaptive defense measures against such learnable advanced poisoning attacks.

[0005] For example, Yin et al. sorted the parameter values ​​in all local model updates and took the median of each parameter as the corresponding parameter value in the global model update. Sun et al. performed gradient clipping on parameter updates before aggregation. Since statistical estimates are susceptible to outliers, existing aggregation methods cannot guarantee accuracy well and are still vulnerable to local model poisoning attacks by malicious clients. These defense methods mainly rely on local model parameters. For complex attacks, such as custom attacks based on reinforcement learning, it is difficult to accurately identify malicious updates from parameter information alone. In addition, statistically based robust aggregation is not flexible enough. Summary of the invention

[0006] The present invention proposes an adaptive aggregation method AdaAggRL based on reinforcement learning to thwart complex poisoning attacks and determine the aggregation weight of the local model by comparing the stability of the client data distribution. Specifically, firstly, the client data distribution is simulated according to the uploaded model parameters using distribution learning; then, the pairwise similarity of the current local model data distribution, its historical data distribution and the global model data distribution is calculated using the maximum mean discrepancy (MMD) to evaluate the stability of the client data distribution; considering that the accuracy of distribution learning may affect the calculation of the above similarity, the reconstruction similarity is used as the evaluation index of the distribution learning quality; finally, based on the above three distribution similarities and the reconstruction similarity of distribution learning, the policy learning method TD3 is used to adaptively determine the aggregation weight.

[0007] The technical solution adopted by the present invention is: an adaptive aggregation defense method for complex poisoning attacks based on reinforcement learning, comprising the following steps:

[0008] Step 1, Task Definition: Based on Federated Learning (FL), design a system consisting of K clients, each of which has a fixed local dataset. where n k D k The size of customer k is Where l is the loss function, (x kj ,y kj ) is from a distribution P k The jth data sample in iid extracted from Indicates that from P k n extracted from k The empirical distribution of data samples is obtained by the clients through collaborative learning under the coordination of the central server to obtain a global model.

[0009] Step 2: Distributed learning: For each epoch t≥0, the server receives the model parameters updated locally by the client and calculates the corresponding batch-level gradients. Then, the server starts solving a batch of the following optimization problems with randomly generated dummy data and dummy labels β is a fixed parameter, B' is the size of the virtual data batch, TV calculates the total variation function;

[0010] Step 3, Environmental indicators: Use the pre-trained convolutional neural network to extract sample distribution and transform the image sample D rec Convert to a feature vector set V; for each customer k, use the maximum mean difference (MMD) to calculate the difference d between the feature distribution and the historical distribution k，cl , in The feature vector representing the current data distribution of client k, represents the historical feature vector;

[0011] Step 4, strategy learning: The server dynamically adjusts the aggregation weight according to the three similarity indicators obtained and the reconstruction similarity associated with the client. By leveraging the experience sampled from the simulated environment, the server participates in learning a collaborative defense strategy with the goal of minimizing the experience loss.

[0012] As a further improvement of the present invention, in step 1, the optimization target of FL is defined as: Among them, p k is the weight of customer k, usually p k =n k / N, where is the total number of samples.

[0013] As a further improvement of the present invention, the FL algorithm process is as follows: at each epoch t≥0, the server randomly selects a subset C from all clients t Participate in the current round of training and set the latest global model parameters θ t Distribute to these selected clients;

[0014] The selected clients perform synchronous training on their respective local datasets and update their local model parameters to Where α is the learning rate;

[0015] Then, customer k∈C t Upload the updated local model parameters to the server;

[0016] The server aggregates the received model parameters according to a specific aggregation rule, denoted as Aggr, to obtain a new global model parameter

[0017] As a further improvement of the present invention, in step 2, in the process of solving the optimization problem, Ddummy Continuously update, the optimization terminates after max_iter iterations, and then outputs the updated data as the reconstructed data sample D rec , the reconstruction similarity of client k is expressed as

[0018] As a further improvement of the present invention, in step 3, the characteristic vector of the global model data distribution is denoted as V g , obtained by calculating the mean of the feature vectors of all clients participating in the training process, and then, the dissimilarity between the current data distribution of client k and the global data distribution is expressed as Similarly, the difference between the historical data distribution and the global data distribution is recorded as Current data distribution and historical data distribution S k，cl The similarity calculation formula is as follows: S k，cl ∈(0, 1], and as the current data distribution obtained by gradient inversion is more consistent with the historical data distribution, S k，cl The value increases.

[0019] As a further improvement of the present invention, in step 4, the learning process adopts deep reinforcement learning algorithm TD3.

[0020] As a further improvement of the present invention, in order to simulate the training process of FL, including malicious clients and their behaviors, a reinforcement learning environment is established. For each epoch t in federated learning, Simulate the environment state for reinforcement learning, where k j The identifier of the client participating in the training. The state of client k is represented by the metrics obtained by reconstructing the similarity and tree;

[0021] Through reinforcement learning, the training server is trained according to the current FL environment state s t Make a decision t =(a t , b t ) T , a t is a four-dimensional vector, in Represents four environmental parameters (S k，R , S k，cl , S k，cg , S k，lg ), b t ∈[0, 1] represents the threshold value, and the FL system changes according to the server's decision;

[0022] Introducing vectors into the FL system environment To record the malicious behavior of each client, when otherwise. Malicious behavior occurs on behalf of client k. Based on these results, the FL system determines the aggregation strategy for the new round of global model parameters as follows: is a hyperparameter that indicates the severity of the penalty for malicious behavior;

[0023] The larger the λ, the stronger the penalty effect. Based on the newly obtained global model parameter θ t+1 , the FL system calculates the reward at step t as r: =f(θ t )-f(θ t+1 ), in the subsequent epoch t+1, the FL system selects a new customer subset for training, denoted as C t+1 , and the updated model parameters θ t+1 Sent to clients, each client then performs local training on its dataset to obtain local parameters This creates a new state s t+1 .

[0024] As a further improvement of the present invention, the FL system first obtains That is, weighting the environmental parameters. represents the fraction of client k, then the function g(·) will is mapped to the interval [0, 1] and normalized, resulting in make definition: Function f δ The client with a lower score is considered to exhibit malicious behavior and is excluded from the aggregation. The generation process of the server policy reveals that the threshold δ is δ It is also adaptively adjusted according to the environmental status.

[0025] Beneficial effects of the present invention: AdaAggRL, an adaptive aggregation method based on reinforcement learning, is used to defend against complex non-targeted model poisoning attacks on servers in federated learning, and plays an important role in promoting the development of defense systems.

[0026] Over time, the distribution of training data in benign clients is stable, while the irregular distribution caused by malicious clients' sabotage attempts is the opposite. The present invention proposes four indicators as clues for the reinforcement learning environment, and uses policy learning to determine the local model aggregation weights based on the observed clue changes.

[0027] Experimental results on four datasets demonstrate that the AdaAggRL defense approach maintains more stable global model accuracy than baseline methods even when applying more advanced customized attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a graph of experimental results of a method for adaptive aggregation defense against complex poisoning attacks based on reinforcement learning of the present invention (the x-axis represents the number of client update rounds, and the y-axis represents the similarity between the current data distribution and its historical data distribution). DETAILED DESCRIPTION

[0029] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The present invention provides a method for adaptively aggregating and defending against complex poisoning attacks based on reinforcement learning.

[0031] Step 1, Task Definition: Based on Federated Learning (FL), design a system consisting of K clients, each of which has a fixed local dataset. where n k D k The size of customer k is Where l is the loss function, (x kj ,y kj ) is from a distribution P k The jth data sample in iid extracted from Indicates that from P k n extracted from k The client performs collaborative learning under the coordination of the central server to obtain the global model.

[0032] The optimization objective of FL is defined as: Among them, p k is the weight of customer k, usually p k =n k / N, where is the total number of samples.

[0033] The FL algorithm process is as follows: At each epoch t≥0, the server randomly selects a subset C from all clients t Participate in the current round of training and set the latest global model parameters θ t Distribute to these selected clients;

[0034] The selected clients perform synchronous training on their respective local datasets and update their local model parameters to Where α is the learning rate;

[0035] Then, customer k∈C t Upload the updated local model parameters to the server;

[0036] The server aggregates the received model parameters according to a specific aggregation rule, denoted as Aggr, to obtain a new global model parameter

[0037] Step 2: Distributed learning: For each epoch t≥0, the server receives the model parameters updated locally by the client and calculates the corresponding batch-level gradients. Then, the server starts solving a batch of the following optimization problems with randomly generated dummy data and dummy labels β is a fixed parameter, B' is the size of the virtual data batch, TV computes the total variation function.

[0038] In the process of solving the optimization problem, D dummy Continuously update, the optimization terminates after max_iter iterations, and then outputs the updated data as the reconstructed data sample D rec , the reconstruction similarity of client k is expressed as

[0039] Step 3, Environmental indicators: Use the pre-trained convolutional neural network to extract sample distribution and transform the image sample D rec Convert to a feature vector set V; for each customer k, use the maximum mean difference (MMD) to calculate the difference d between the feature distribution and the historical distribution k，cl , in The feature vector representing the current data distribution of client k, represents the historical feature vector.

[0040] The eigenvector of the global model data distribution is denoted by V g , obtained by calculating the mean of the feature vectors of all clients participating in the training process, and then, the difference between the current data distribution of client k and the global data distribution is expressed as Similarly, the difference between the historical data distribution and the global data distribution is recorded as Current data distribution and historical data distribution S k,cl The similarity calculation formula is as follows: S k,cl ∈(0, 1], and as the current data distribution obtained by gradient inversion is more consistent with the historical data distribution, S k,cl The value increases.

[0041] Step 4: Strategy learning: The server dynamically adjusts the aggregation weight based on the three similarity indicators obtained and the reconstruction similarity associated with the client. By utilizing the experience sampled from the simulated environment, the server participates in learning a collaborative defense strategy with the goal of minimizing the experience loss. The learning process uses the deep reinforcement learning algorithm TD3.

[0042] In order to simulate the training process of FL, including malicious clients and their behaviors, a reinforcement learning environment is established. For each epoch t in federated learning, let Simulate the environment state for reinforcement learning, where k j The identifier of the client participating in the training. The state of client k is represented by the metrics obtained by reconstructing the similarity and tree;

[0043] Through reinforcement learning, the training server is trained according to the current FL environment state s t Make a decision t =(a t , b t ) T , a t is a four-dimensional vector, in Represents four environmental parameters (S k，R , S k，cl , S k，cg , S k，lg ), b t ∈[0, 1] represents the threshold value, and the FL system changes according to the server's decision;

[0044] Specifically, the FL system first obtains That is, weighting the environmental parameters. represents the fraction of client k, then the function g(·) will is mapped to the interval [0, 1] and normalized, resulting in make definition: Function f δ The client with a lower score is considered to exhibit malicious behavior and is excluded from the aggregation. The generation process of the server policy reveals that the threshold δ is δ It is also adaptively adjusted according to the environmental status.

[0045] Introducing vectors into the FL system environment To record the malicious behavior of each client, when otherwise, Malicious behavior occurs on behalf of client k. Based on these results, the FL system determines the aggregation strategy for the new round of global model parameters as follows: is a hyperparameter that indicates the severity of the penalty for malicious behavior;

[0046] The larger the λ, the stronger the penalty effect. Based on the newly obtained global model parameter θ t+1 , the FL system calculates the reward at step t as r: =f(θ t )-f(θ t+1 ), in the subsequent epoch t+1, the FL system selects a new customer subset for training, denoted as C t+1 , and the updated model parameters θ t+1 Sent to clients, each client then performs local training on its dataset to obtain local parameters This creates a new state s t+1 .

[0047] The present invention proposes an adaptive aggregation method AdaAggRL based on reinforcement learning to defend against complex poisoning attacks. Figure 1 The experimental results on four real-world datasets shown in the paper demonstrate that the proposed defense model significantly outperforms the current state-of-the-art defense models in terms of defense performance against complex attacks.

[0048] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive aggregation defense method for complex poisoning attacks based on reinforcement learning, characterized in that: The following steps are involved: Step 1, Task Definition: Based on Federated Learning (FL), design a system consisting of K clients, each of which has a fixed local dataset. where n k D k The size of customer k is Where l is the loss function, (x kj ,y kj ) is from a distribution P k The iid-th data sample extracted from Indicates that from P k n extracted from k The empirical distribution of data samples is obtained by the clients through collaborative learning under the coordination of the central server to obtain a global model. Step 2: Distributed learning: For each epoch t≥0, the server receives the model parameters updated locally by the client and calculates the corresponding batch-level gradients. Then, the server starts to solve a batch of the following optimization problems with randomly generated dummy data and dummy labels β is a fixed parameter, B' is the size of the virtual data batch, TV calculates the total variation function; Step 3, Environmental indicators: Use the pre-trained convolutional neural network to extract sample distribution and transform the image sample D rec Convert to a feature vector set V; for each customer k, use the maximum mean difference (MMD) to calculate the difference d between the feature distribution and the historical distribution k，cl , in The feature vector representing the current data distribution of client k, represents the historical feature vector; Step 4, strategy learning: The server dynamically adjusts the aggregation weight according to the three similarity indicators obtained and the reconstruction similarity associated with the client. By leveraging the experience sampled from the simulated environment, the server participates in learning a collaborative defense strategy with the goal of minimizing the experience loss.

2. According to claim 1, the adaptive aggregation defense method for complex poisoning attacks based on reinforcement learning is characterized in that: In step 1, the optimization objective of FL is defined as: Among them, p k is the weight of customer k, usually p k =n k / N, where is the total number of samples.

3. According to claim 2, the adaptive aggregation defense method for complex poisoning attacks based on reinforcement learning is characterized in that: The FL algorithm process is as follows: At each epoch t≥0, the server randomly selects a subset C from all clients t Participate in the current round of training and set the latest global model parameters θ t Distribute to these selected clients; The selected clients perform synchronous training on their respective local datasets and update their local model parameters to Where α is the learning rate; Then, customer k∈C t Upload the updated local model parameters to the server; The server aggregates the received model parameters according to a specific aggregation rule, denoted as Aggr, to obtain a new global model parameter 4. According to claim 1, the adaptive aggregation defense method for complex poisoning attacks based on reinforcement learning is characterized in that: In the step 2, in the process of solving the optimization problem, D dummy Continuously update, the optimization terminates after max_iter iterations, and then outputs the updated data as the reconstructed data sample D rec , the reconstruction similarity of client k is expressed as 5. According to claim 1, the method for adaptive aggregation defense against complex poisoning attacks based on reinforcement learning is characterized in that: In step 3, the characteristic vector of the global model data distribution is denoted as V g , obtained by calculating the mean of the feature vectors of all clients participating in the training process, and then, the dissimilarity between the current data distribution of client k and the global data distribution is expressed as Similarly, the difference between the historical data distribution and the global data distribution is recorded as Current data distribution and historical data distribution S k，cl The similarity calculation formula is as follows: And as the current data distribution obtained by gradient inversion is more consistent with the historical data distribution, S k，cl The value increases.

6. According to claim 1, the adaptive aggregation defense method for complex poisoning attacks based on reinforcement learning is characterized in that: In step 4, the learning process adopts the deep reinforcement learning algorithm TD3.

7. The method of adaptive aggregation defense against complex poisoning attacks based on reinforcement learning according to claim 6 is characterized in that: In order to simulate the training process of FL, including malicious clients and their behaviors, a reinforcement learning environment is established. For each epoch t in federated learning, let Simulate the environment state for reinforcement learning, where k j The client identifier for the training. The state of client K is represented by the metrics obtained by reconstructing the similarity and tree; Through reinforcement learning, the training server is trained according to the current FL environment state s t Make a decision t =(a t , b t ) T , a t is a four-dimensional vector, in Represents four environmental parameters (S k，R , S k，cl , S k，cg , S k，lg ), b t ∈[0, 1] represents the threshold value, and the FL system changes according to the server's decision; Introducing vectors into the FL system environment To record the malicious behavior of each client, when otherwise, Malicious behavior occurs on behalf of client k. Based on these results, the FL system determines the aggregation strategy for the new round of global model parameters as follows: is a hyperparameter that indicates the severity of the penalty for malicious behavior; The larger the λ, the stronger the penalty effect. Based on the newly obtained global model parameter θ t+1 , the FL system calculates the reward at step t as r: =f(θ t )-f(0 t+1 ), in the subsequent epoch t+1, the FL system selects a new customer subset for training, denoted as C t +1 , and the updated model parameters θ t+1 Sent to clients, each client then performs local training on its dataset to obtain local parameters This creates a new state s t+1 .

8. The method of adaptive aggregation defense against complex poisoning attacks based on reinforcement learning according to claim 7 is characterized in that: The FL system first obtains That is, weighting the environmental parameters. represents the fraction of client k, then the function g(·) will Mapped to the interval [0,1] and normalized, yielding make definition: Function f δ The client with a lower score is considered to exhibit malicious behavior and is excluded from the aggregation. The generation process of the server policy reveals that the threshold δ is δ It is also adaptively adjusted according to the environmental status.

Images