A federated learning-oriented adversarial sample poisoning attack method

Abstract

A kind of attack method of poison attack of facing federated learning to confrontation sample, define the following scene, suppose that m participants participate in training, m >= 2, suppose the kth participant is attacker, its attack target is that local model parameter after participating in aggregation, so that the performance of global model of federated learning on test set is as bad as possible;First, the attacker generates "poisoned" adversarial samples by adding some human eye imperceptible adversarial perturbations to local private training samples, and carries out local training based on these samples;Second, in order to dominate the training process of global model, the attacker increases the training learning rate during local training to accelerate the generation of malicious model parameters;Finally, the attacker uploads its local model parameters to the server side to participate in aggregation to affect the global model.Under the attack of the application, the performance of global federated model decreases significantly, and the attack method of the application shows good generalization performance.

Description

Technical Field

[0001] This invention belongs to the field of federated learning security technology, specifically relating to an adversarial sample poisoning attack method for federated learning. Background Technology

[0002] Although machine learning technology has been widely applied in various fields, data silos and data privacy issues remain two major challenges hindering its development. For example, in medical applications, training a high-performance machine learning model requires a large amount of information describing patient symptoms from various medical institutions or departments, but medical data is often highly private and sensitive. Similarly, a city's emergency response, logistics, and security information departments generate a large amount of heterogeneous data, which exists in the form of data silos and cannot be integrated and utilized. To address these issues, federated learning technology has emerged. Unlike machine learning models, it adopts a distributed architecture, eliminating the need to centrally store data before model training. Instead, it transfers the process to local training participants, protecting user privacy by submitting local model parameters to a central server.

[0003] However, research shows that although federated learning effectively solves the data silo and data privacy problems in machine learning, it still has many security issues, such as malicious actors attacking the federated learning system during the training phase. This can lead to the failure of the global federated learning model and the leakage of participant privacy. Depending on the malicious actor's attack objective, attacks can be categorized into inference attacks and poisoning attacks. Inference attacks by malicious actors aim to infer information from the training process, such as the training samples and labels of other participants. Poisoning attacks can be achieved by controlling local model parameter updates or local training samples, causing the global model to make incorrect predictions. Specifically, these can be divided into model poisoning attacks and data poisoning attacks. Currently, research on data poisoning attacks in federated learning mainly focuses on label inversion attacks and backdoor attacks, both of which are targeted attacks aimed at making the global model predict specific targets. Another method to modify training samples to achieve an attack is to add noise to the training samples; however, such attacks are mainly carried out during the model testing phase. Currently, there is relatively little research applying such attacks to the training phase of federated learning. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes the following steps: S1. The attacker provides local private training samples. Add some adversarial perturbations that are imperceptible to the human eye To generate “toxic” adversarial examples and to perform local training based on these examples; S2. In order to dominate the training process of the global model, the attacker increases the training learning rate during local training to accelerate the generation of malicious model parameters. S3. Attackers upload their local model parameters to the server to participate in aggregation in order to affect the global model.

[0005] The beneficial effects of this invention are as follows: Unlike traditional machine learning, federated learning systems involve two key aspects. First, the server-side performs federated averaging and aggregation of local parameters, which narrows down the parameters of malicious actors, thus weakening their toxicity. Second, the aggregation process also allows other non-malicious actors to cause the final model parameters to deviate from the attacker's model parameters, further reducing the attack's effectiveness.

[0006] This invention demonstrates through adversarial poisoning attacks against federated learning systems that adversarial examples can attack federated learning systems not only during the testing phase but also during the training phase. Furthermore, this invention reveals that the learning rate is a crucial factor influencing attack success rate. Experiments show that malicious actors using adversarial examples and a large learning rate for local training can effectively attack federated learning systems, significantly reducing the global model's test accuracy. Simultaneously, experimental results show that the attack method of this invention has excellent generalization performance, maintaining good attack effectiveness even when the training actor's model changes. Compared to machine learning, federated learning systems are more vulnerable to attacks presented by this invention. Attached Figure Description

[0007] Figure 1 The images shown are the original images and adversarial sample images under targeted and untargeted attacks in the CIFAR10 dataset of this invention. The first row in the figure represents the original samples; the second row represents the adversarial samples under targeted attacks; and the third row represents the adversarial samples under untargeted attacks. Figure 2 The images shown in the figure are the original images and adversarial sample images under targeted attacks and untargeted attacks based on the MNIST dataset of this invention. The first row of the figure represents the original samples; the second row represents the adversarial samples under targeted attacks; and the third row represents the adversarial samples under untargeted attacks. Figure 3 This invention uses the CIFAR10 dataset to generate adversarial examples in the IID scenario with targeted attacks. The bar chart shows the test results of the global model when different numbers of malicious participants are involved in the training. Figure 4 This invention uses the MNIST dataset to generate adversarial examples in the IID scenario with targeted attacks. The bar chart shows the test results of the global model when different numbers of malicious participants participate in the training. Figure 5This invention generates adversarial examples using untargeted attacks in the IID scenario based on the CIFAR10 dataset. The bar chart shows the test results of the global model when different numbers of malicious participants are involved in the training. Figure 6 This invention generates adversarial examples using untargeted attacks in the IID scenario based on the MNIST dataset. The bar chart shows the test results of the global model when different numbers of malicious participants are involved in the training. Figure 7 This invention uses the CIFAR10 dataset to generate adversarial examples in a Non-IID scenario with targeted attacks. The bar chart shows the test results of the global model when different numbers of malicious participants are involved in the training. Figure 8 This invention uses the MNIST dataset to generate adversarial examples in a Non-IID scenario with targeted attacks. The bar chart shows the test results of the global model when different numbers of malicious participants are involved in the training. Figure 9 This is a line graph showing the test accuracy of the global model after amplifying the local training learning rate of the malicious participant based on the CIFAR10 dataset to different degrees. Figure 10 This is a line graph showing the test accuracy of the global model after amplifying the local training learning rate of the malicious participant based on the MNIST dataset to different degrees. Figure 11 This is a line graph showing the test accuracy of the global model based on the CIFAR10 dataset for non-malicious participants under different local training learning rates. Figure 12 This is a line graph showing the test accuracy of the global model based on the MNIST dataset for non-malicious participants under different local training learning rates. Detailed Implementation

[0008] An adversarial example poisoning attack method for federated learning includes the following steps: S1. The attacker provides local private training samples. Add some adversarial perturbations that are imperceptible to the human eye To generate “toxic” adversarial examples and to perform local training based on these examples; S2. In order to dominate the training process of the global model, the attacker increases the training learning rate during local training to accelerate the generation of malicious model parameters. S3. Attackers upload their local model parameters to the server to participate in aggregation in order to affect the global model.

[0009] In step 1, the following scenario is defined: assuming there are m participants in the training, where m>=2, and the k-th participant is the attacker, in the federated learning system, each participant's local training is considered a traditional machine learning model training process. The attacker obtains malicious local model parameters by solving the following two-layer optimization problem, thereby achieving a poisoning attack against the federated learning system: in, and They represent Each training sample and its corresponding label, = It is the set of all training samples. = It is a collection of tags. For the attacker, select an optional set of noise that meets the perturbation constraints. Each sample Added perturbation Limit disturbances , This represents the upper limit of the perturbation. The set of perturbations added to all training samples. For the local training learning rate of the malicious participant, For loss function, For model parameters, for each Each of these will have a corresponding optimal model parameter obtained by training under "toxic" adversarial examples. The attacker's goal is to [address the issue of] the collection. Find an optimal set of perturbations. Make its corresponding optimal classifier exist Spatial distribution of samples The generalization performance should be as poor as possible.

[0010] In step 1, the adversarial example poisoning attack against federated learning is achieved through the following process: First, the model is trained using original clean samples. This model is then used to generate adversarial examples, and the model parameters remain constant during the adversarial example generation process. Based on the aforementioned fixed, pre-trained model, adversarial examples are generated as "toxic" training samples. The classic adversarial example generation method, PGD, is then used to generate these "toxic" adversarial examples. Initialization noise is introduced before generating the adversarial examples to disrupt the distribution of the original image. Multi-step gradient updates are then performed to generate the adversarial examples. The following optimizations generate adversarial examples using both untargeted and targeted attack methods, respectively. in, Indicates the process Samples generated after iterative attack Indicates the degree of single-step perturbation. Indicates in - The projection onto the sphere, meaning that when the disturbance amplitude is too large, it will be confined within the sphere. in, It is the target tag generation function. The choice is not fixed and varies depending on the sample, model, and federated learning data distribution. The experiment defines... , Malicious actors use adversarial examples for local training, obtaining malicious local model parameters. They then upload these parameters to the server to participate in aggregation, thereby launching an attack on the global model. in, For adversarial sample sets.

[0011] In step 2, the idea of ​​learning rate amplification is adopted to maliciously increase the learning rate of local training during local training to accelerate gradient descent and promote the rapid generation of local malicious model parameters. This allows for the submission of stronger "poison" to the server in each round of aggregation, making the global model more dependent on the local model of the malicious participant in each round of aggregation. in, This is a factor that amplifies the local training learning rate for malicious participants.

[0012] The detailed process is shown in Algorithm 1. Experimental setup This invention conducted experiments on the MNIST and CIFAR10 datasets. MNIST contains 10 classes of 28x28 grayscale images, with 60,000 images in the training set and 10,000 in the test set. The CIFAR10 dataset contains 10 classes of 32x32 color images, with 50,000 images in the training set and 10,000 in the test set. In the collaborative learning scenario, for the CIFAR10 dataset, the learning rate for non-malicious participants was set to 0.01, with 5 local training rounds per participant and 60 global aggregations. For the MNIST dataset, the learning rate for non-malicious participants was set to 0.001, with 1 local training round and 100 global aggregations. In this paper, ε was set to 0.032 and 0.3 on CIFAR10 and MNIST, respectively.

[0013] This invention discusses the effectiveness of attacks in federated learning data scenarios with independent and identically distributed (IID) and non-independent and identically distributed (Non-IID) data. We consider the attack effects when different numbers of malicious participants participate in training, with 2, 3, and 4 participants respectively. In the IID scenario, each participant has all categories, and each category is evenly distributed among the participants. ResNet18 is used as the classification network for both the MNIST and CIFAR10 datasets. In the Non-IID scenario, to ensure a fair comparison with the Fed-DeepConfuse method proposed by Feng et al., the same data partitioning method and classification network are used. The MNIST dataset uses a simple convolutional neural network with two convolutional layers and 20 and 50 channels respectively. The data partitioning method is shown in Table 1.

[0014] Table 1 Data Distribution of Federated Learning Non-IID Scenarios (i) The performance of the poisoning attack method of the present invention, under federated independent identically distributed (IID) and non-independent identically distributed (Non-IID) scenarios, the attack performance when different numbers of malicious participants participate in training; Adversarial sample poisoning attack performance This paper considers the impact of different numbers of attackers participating in training on the performance of the global model in federated learning IID and Non-IID scenarios. Figure 1 and Figure 2 The original images and adversarial sample images are shown under targeted and untargeted attacks, respectively.

[0015] Under the federated data IID distribution, malicious actors use targeted attack methods to generate adversarial examples, resulting in attacks such as... Figure 3 and Figure 4 As shown, in this invention, the local training learning rate of the malicious participant is set to 0.3. Figure 3 and Figure 4 The experimental results are presented on the CIFAR10 and MNIST datasets, respectively. It can be observed that even with only a few malicious actors participating in training, the global model's test performance significantly decreases. Specifically, in federated systems with 2, 3, and 4 training actors, when only one malicious actor participates, the global model's test performance decreases by 78.23%, 77.48%, and 65.54% on the CIFAR10 dataset, respectively, and by 67.33%, 61.15%, and 48.56% on the MNIST dataset, respectively. Notably, on the CIFAR10 dataset, when all actors are malicious, the global model's accuracy is lower than that of random guessing.

[0016] It can be observed that, compared to the MNIST dataset, the attack methods and strategies of this invention pose a greater threat to federated systems trained on the CIFAR10 dataset. For example, in a federated system with four training participants, when there is only one malicious participant, the test performance of the global model trained on the CIFAR10 dataset decreases by 16.98% more than that of the global model trained on the MNIST dataset.

[0017] Analysis shows that the success rate of attacks against federal systems depends on: (1) Test performance of the local model of the malicious party. Using targeted adversarial poisoning attacks, the test performance of the malicious party on the CIFAR10 dataset is as low as 6%, while on the MNIST dataset, the test performance of the malicious party is around 25%.

[0018] (2) Training characteristics of the model itself. We hope to accelerate the generation of "poison" by malicious training participants through learning rate amplification, so that the global model relies on the local model of the malicious participants "earlier" and "more". However, under the same settings, the test performance of normal training participants in the CIFAR10 dataset is 69.47% within 5 epochs, while in the MNIST dataset, the test performance of normal training participants in the MNIST dataset has reached 97.89% within 5 epochs. This also means that during aggregation, compared with the CIFAR10 dataset, the federated learning system trained on the MNIST dataset has a greater impact on the performance of the global model from non-malicious training participants.

[0019] Figure 5 and Figure 6 The image shows the experimental results of a malicious actor using a non-targeted attack method to generate adversarial examples to poison a federated learning system in a federated IID scenario. (Comparison) Figure 3 and Figure 4It can be seen that its attack performance is weaker than that of attacks using targeted attack methods to generate adversarial examples. However, it still significantly degrades the performance of federated systems. In multi-party federated scenarios where only one attacker participates in training, the test performance of the global model on the CIFAR10 dataset decreased by 48.19%, 38.88%, and 26.14%, respectively, and on the MNIST dataset, it decreased by 38.04%, 41.52%, and 17.10%, respectively.

[0020] In the federated learning Non-IID scenario shown in Table 1, we used targeted attack generation adversarial attacks to poison the federated learning system, and the results are as follows. Figure 7 and Figure 8 As shown, for the CIFAR10 dataset, when only one malicious actor participates in training, the global model's test accuracy drops to near random guessing. For the MNIST dataset, the global model's test performance also significantly decreases.

[0021] Table 2 shows the degree of decrease in global model test accuracy under different attack methods when there is only one malicious party. In the same federated learning scenario, we compared the attack method of this invention with the Fed-DeepConfuse method. In federated systems with 2, 3, and 4 participants, considering the decrease in global model test accuracy when there is only one malicious participant, the results shown in Table 2 demonstrate that the attack method in this paper is far more harmful to the federated learning system than Fed-DeepConfuse. For example, in a federated learning system with two participants, for the CIAFR0 dataset, the global model test performance decreased by 18.99% under the Fed-DeepConfuse attack, while under the targeted attack in this paper, the global model test accuracy decreased by 72.43%, and under the untargeted attack, the global model test accuracy decreased by 60.23%. Attacks using untargeted attacks to generate adversarial examples are unstable. Under the same settings, the attack performance in the IID scenario is weaker than that in the Non-IID scenario, which is related to the attacked model and the initialization of the "poison" during the attack.

[0022] Furthermore, it is worth noting that, compared to FedDeepConfuse, the method of this invention can generate "toxic" adversarial examples more quickly. In the process of generating adversarial examples, this paper uses a fixed pre-trained network, which only takes 1-2 hours to train. Fed-DeepConfuse trains an autoencoder as a noise generator, which takes 5-7 days on simple datasets.

[0023] (II) The impact of learning rate on the attack effect of the attack method of the present invention, specifically discussing the attack effect of attackers using different learning rate amplification factors and the impact of the learning rate of non-malicious participants on attack performance. Experiments have shown that the learning rate is a crucial factor affecting the success rate of attacks. In this section, we investigate the performance of the federated learning system under the attack described in this paper and different combinations of learning rates.

[0024] We fixed the learning rates for non-malicious participants on the CIFAR10 and MNIST datasets to 0.01 and 0.001, respectively, while amplifying the local training learning rate for malicious participants to varying degrees. Experimental results are as follows... Figure 9 and Figure 10 As shown, as the local training learning rate of malicious participants increases, the attack success rate will continue to increase, and the attack success rate will tend to stabilize within a certain learning rate range.

[0025] Furthermore, the impact of the local training learning rate of non-malicious participants on attack success is considered. In a federated learning system with two participants, one malicious participant has a learning rate set to 0.3, while the learning rates of non-malicious participants are set to 0.1, 0.01, 0.001, and 0.0001, respectively. Experimental results on the MNIST and CIFAR10 datasets are presented. Figure 11 and Figure 12 As shown in the experimental results, when the local learning rate is large, the global model exhibits significant oscillations, and these oscillations are even greater on the MNIST dataset under the same learning rate setting. Our analysis reveals that after the t-th round of aggregation, when the global model's parameters differ more significantly from the optimal solution of the non-malicious participants, before the t+1-th round of aggregation, the training participants obtain the global model parameters from the t-th round and perform gradient descent locally. Because the optimal solution of the non-malicious participants differs more from the global model parameters, their local gradient descent is more pronounced. After the t+1-th round of aggregation, the global model parameters are more biased towards the non-malicious participants. However, before the next round of aggregation, the gradient descent of the malicious participants is more significant. Therefore, the global model will oscillate within a certain range.

[0026] (iii) Generalization performance of adversarial examples, specifically discussing whether adversarial examples generated for a particular model are equally effective in attacking other federated learning models; For a federated learning system using model A as the training participant, adversarial examples are generated. When the model structure used by the training participant changes, do these adversarial examples still have an offensive effect? ​​In a federated system with two participants, one of whom is malicious, we conducted experiments on the CIFAR10 dataset. We generated adversarial examples using a targeted attack based on ResNet18 and tested them on VGG19 and MobileNetV2. For the MNIST dataset, we generated adversarial examples using a targeted attack based on the CNN network used in the Non-IID scenario, doubling and halving the number of channels, respectively. , As shown in Table 3, the adversarial examples have good generalization performance.

[0027] Table 3 shows the test performance of federated learning models using locally trained models with different structures under the attack described in this paper. (iv) Under the poisoning attack of this invention, the robustness of federated learning systems and machine learning systems is studied, and a comparison of the robustness of federated learning systems and machine learning systems is made: Robustness of Federated Learning and Machine Learning When the Same Proportion of Training Samples is Poisoned. The experiment considered a federated learning system with 10 participants. To ensure a fair comparison, the same proportion of training data was "poisoned" for both federated learning and machine learning. The results are shown in Table 4. The experimental results show that the federated learning system is more vulnerable. When 20% of the training data consists of adversarial examples, the machine learning model achieves a test accuracy of 91.39% on the CIFAR10 dataset, while the federated learning global model only achieves 45.90%.

[0028] Table 4 compares the robustness of federated learning systems and machine learning systems. 0.2, 0.5, 0.8, and 0.9 represent the proportions of adversarial examples, respectively. This invention addresses adversarial poisoning attacks against federated learning systems. It demonstrates that adversarial examples can attack federated learning systems not only during the testing phase but also during the training phase. Furthermore, this invention reveals that the learning rate is a crucial factor influencing attack success rate. Experiments show that malicious actors using adversarial examples and a large learning rate for local training can effectively attack federated learning systems, significantly reducing the global model's test accuracy. Simultaneously, experimental results show that the attack method of this invention has excellent generalization performance, maintaining good attack effectiveness even when the training actor's model changes. Compared to machine learning, federated learning systems are more vulnerable to attacks using this invention.

[0029] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An adversarial example poisoning attack method for federated learning, applied in the field of federated learning security technology, characterized in that, Includes the following steps: S1. The attacker provides local private training samples. Adding adversarial perturbations that are imperceptible to the human eye To generate "toxic" adversarial examples and to perform local training based on these examples; S2. In order to dominate the training process of the global model, the attacker increases the training learning rate during local training to accelerate the generation of malicious model parameters. S3. The attacker uploads their local model parameters to the server to participate in aggregation in order to affect the global model; In step S1, the following scenario is defined: assuming there are m participants in the training, where m>=2, and assuming the kth participant is the attacker, in the federated learning system, the local training of each participant is regarded as a traditional machine learning model training process. The attacker obtains malicious local model parameters by solving the following two-layer optimization problem, thereby achieving a poisoning attack against the federated learning system: in, and They represent Each training sample and its corresponding label, = It is the set of all training samples. = It is a collection of tags. For the attacker, select an optional set of noise that meets the perturbation constraints. Each sample Added perturbation Limit disturbances , This represents the upper limit of the perturbation. The set of perturbations added to all training samples. For the local training learning rate of the malicious participant, For loss function, For model parameters, for each Each of these will have a corresponding optimal model parameter obtained by training under "toxic" adversarial examples. The attacker's goal is to [address the issue of] the collection. Find an optimal set of perturbations. Make its corresponding optimal classifier exist Spatial distribution of samples The generalization error on the surface is not less than the preset value; In step S1, the adversarial poisoning attack against federated learning is achieved through the following process: First, the model is trained using original clean samples. This model is then used to generate adversarial examples, and the model parameters remain constant during the adversarial example generation process. Based on the aforementioned fixed, pre-trained model, adversarial examples are generated as "toxic" training samples. The classic adversarial example generation method, PGD, is then used to generate these "toxic" adversarial examples. Initialization noise is introduced before generating the adversarial examples to disrupt the distribution of the original image. Then, multi-step gradient updates are performed to generate the adversarial examples. The following optimizations generate adversarial examples using both untargeted and targeted attack methods, respectively. in, Indicates the process Samples generated after iterative attack Indicates the degree of single-step perturbation. Indicates in - The projection onto the sphere, meaning that when the disturbance amplitude is too large, it will be confined within the sphere. in, It is the target tag generation function. The choice is not fixed and varies depending on the distribution of samples, models, and federated learning data. , Malicious actors use adversarial examples for local training, obtaining malicious local model parameters. They then upload these parameters to the server to participate in aggregation, thereby launching an attack on the global model. in, For adversarial sample sets; In step S2, the idea of ​​learning rate amplification is adopted to maliciously increase the learning rate of local training during local training to accelerate gradient descent and promote the rapid generation of local malicious model parameters. This allows for the submission of stronger "poison" to the server in each round of aggregation, making the global model more dependent on the local model of the malicious participant in each round of aggregation. in, This is a factor that amplifies the local training learning rate for malicious participants.

Images