Privacy-aware incremental poisoning attack defense method and system

Abstract

The invention provides a privacy-aware incremental poisoning attack defense method and system, and the method comprises the steps: carrying out the calculation of an offset accumulation amount of an uploaded model parameter according to a clustering result and a density distance of the uploaded model parameter; based on the offset accumulation amount, dividing a trust domain by using a threshold value, calculating to obtain a trust degree score, and aggregating model parameters according to the trust degree score; and according to the credibility scores of the participants, adopting a differentiated differential privacy mechanism, adding differentiated Gaussian noise to the aggregated model parameters, and considering privacy protection and model precision. According to the privacy-aware incremental poisoning attack defense method provided by the invention, hidden poisoning attacks based on the generative adversarial network can be effectively resisted, the model precision can be kept as far as possible on the premise of protecting privacy, and the security and privacy in federated learning are greatly improved.

Description

Technical Field

[0001] The present invention relates to a privacy-aware incremental poisoning attack defense method and system, and belongs to the technical field of federated learning and network security. Background Art

[0002] Traditional centralized distributed machine learning is difficult to develop for private data of institutions in fields such as finance and healthcare due to the sensitivity of personal privacy. Federated learning is developed on the basis of distributed machine learning and can effectively solve some problems faced by traditional distributed machine learning. Federated learning allows participants to jointly train a global model without sharing training samples. Each participant uses his own data to train a local model locally without directly sharing data, and uses parameter servers to achieve the purpose of collaborative learning and training with other participants.

[0003] Although federated learning has made great progress, there are still a series of hidden dangers: poisoning attackers in federated learning upload malicious parameters (updates or gradients) to mislead the global model, intending to damage the model availability. Poisoning attacks include model poisoning attacks, data poisoning attacks, and poisoning attacks based on generative adversarial networks (GANs). One of the most severe poisoning attacks is the GAN-based poisoning attack. In the GAN-based poisoning attack, the cunning attacker only uploads "slightly poisoned" model parameters at the beginning of the attack to escape the basic defense measures of the server, and gradually increases the attack intensity in subsequent attacks to achieve the poisoning goal.

[0004] There are many studies on effective defense measures against GAN-based covert poisoning attacks. They are mainly divided into three categories: (1) Defense methods based on security model aggregation based on statistical features (average, median, similarity, etc.); (2) Defense methods based on verification data sets; (3) Defense methods based on unsupervised learning. Defense methods based on statistical features use the obvious abnormal features of malicious users to quickly identify malicious users. Defense methods based on verification data sets use a set of verification data to assist in detecting malicious users. Methods based on unsupervised learning design defense methods that can directly detect attackers based on unsupervised learning.

[0005] However, the above methods all have shortcomings. First, the defense method based on statistical features only focuses on using the obvious abnormal features of malicious users to quickly identify malicious users. However, if the attacker launches a continuous and covert attack, it is easy to escape these detection methods and achieve the poisoning goal. Even if the malicious user is identified after some iterations, the attacker has achieved the poisoning goal and used GAN to generate a large amount of forged data close to the training set information of other participants. Secondly, the defense method based on the verification data set requires a set of verification data to assist in detection, and the effectiveness of detection depends on the verification data set. The effectiveness of defending against poisoning attacks depends on the quality of the verification data set, but in reality, it is difficult for the server to have a data set that is almost close to the data distribution of all participants. In addition, most defense methods do not consider privacy protection. Some researchers add noise to the model parameters before uploading them to the server to protect privacy. However, if many participants have privacy protection requirements and add noise to their local model parameters, the global model aggregated by the server will have a low-precision model, which is very unfriendly to those benign participants. Summary of the invention

[0006] The purpose of the present invention is to provide a privacy-aware incremental poisoning attack defense method and system to solve the problems of insufficient privacy protection and insufficient model accuracy in the prior art.

[0007] In order to achieve the above object, the technical solution of the present invention is to disclose a privacy-aware incremental poisoning attack defense method, which is characterized by comprising the following steps:

[0008] According to the clustering results and density distance of the model parameters uploaded by the participants, the accumulated offset of the uploaded model parameters is calculated;

[0009] Based on the accumulated offset, the trust domain is divided using a threshold, the trust scores of the participants are calculated and the model parameters are aggregated accordingly;

[0010] According to the trust scores of the participants, a differentiated differential privacy mechanism is adopted to add differentiated Gaussian noise to the aggregated model parameters, taking into account both privacy protection and model accuracy.

[0011] Preferably, the offset accumulation amount is calculated by the following steps:

[0012] According to the clustering results obtained by the neighbor propagation clustering algorithm, the density distance of the model parameter point p(i,k) in the kth iteration is calculated:

[0013] nd(i,k)=dis(i,k)-Cir(i,k)

[0014] Where dis(i,k) is the normalized form of the Euclidean distance between the model parameter p(i,k) and its cluster center at the kth iteration; Cir(i,k) is the normalized form of the number of points in the circle with radius r and center p(i,k) in the kth iteration;

[0015] According to the changes in density distance of the model parameters provided by the participants in all current iterations, the time offset accumulation TOA(i,k) is calculated:

[0016]

[0017] Where Δnd(i,k) is the change in the density distance between the model parameter p(i,k) at the (k-1)th and kth times, k>1; K is the number of all iterations so far.

[0018] Preferably, the step of obtaining the clustering results of all uploaded model parameters according to the nearest neighbor propagation algorithm comprises the following steps:

[0019] The similarity s(i,j) between any two data points i and j is calculated according to the following formula:

[0020]

[0021] Where Ref is the value of the diagonal element of the similarity matrix S, and all diagonal elements have the same value; p(i, k) and p(j, k) are the model parameters uploaded by the i-th and j-th participants in the k-th global model training period, respectively;

[0022] Update the attraction matrix and the attribution matrix according to the initialized information matrix, where the attraction matrix r is updated in the t+1th iteration. t+1 (i,k) is:

[0023]

[0024] Update the attribution matrix a in the t+1th iteration t+1 (i,j):

[0025]

[0026] Among them, the attribution matrix A, the similarity matrix S and the attraction matrix R are initialized as zero matrices; the attribution a(i, j) in the attribution matrix A represents the degree to which data point i selects data point j as its cluster center; the similarity s(i, j) in the similarity matrix S represents the negative of the Euclidean distance between data points i and j; the attraction r(i, j) in the attraction matrix R represents the degree to which data point j is suitable as the cluster center of data point i; t represents the number of iterations run so far in the AP clustering algorithm;

[0027] According to the introduced attenuation coefficient λ, the attraction matrix and the belonging matrix are updated:

[0028] r t+1 (i,j)=(λ-1)r t+1 (i,j)+λr t (i,j)

[0029] a t+1 (i,j)=(λ-1)a t+1 (i,j)+λa t (i,j)

[0030] When the information matrix is ​​stable or reaches the maximum number of iterations, according to the sum of the attraction information and the belonging information, the data point with the information sum greater than 0 is selected as the cluster center, that is, the zth cluster center O z The choices are:

[0031] O z ={p(i,k)|r(i,i)+a(i,i)>0}

[0032] Among them, the model parameter point p(i,k) belongs to the cluster with the greatest similarity with the model parameter p(i,k) in the cluster center.

[0033] Preferably, the method of dividing the trust domain based on the accumulated offset using a threshold value, calculating the trust scores of the participants and aggregating the model parameters accordingly comprises the following steps:

[0034] According to the accumulated time offset, in each cluster, a completely credible threshold Thr is used. full and the semi-trust threshold Thr semi Divide into three trust domains:

[0035]

[0036] Among them, Dom ft 、Dom st and Dom ut are the fully trusted domain, semi-trusted domain, and untrusted domain respectively; TOA(i,k) is the normalized form of the accumulated time offset of the model parameter p(i,k) in the kth iteration;

[0037] According to the three divided trust domains, the trust scores of the model parameters are calculated:

[0038]

[0039] Where C(i,k) is the confidence score of the model parameter p(i,k) at the kth iteration; TOA(i,k) is the normalized form of the accumulated time offset of the model parameter p(i,k) at the kth iteration; Domft 、Dom st and Dom ut They are fully trusted domain, semi-trusted domain and untrusted domain respectively;

[0040] According to the trust score C(i,k) of the model parameters, different weights are assigned to each participant to calculate the global model parameters:

[0041]

[0042] Where w(i,k) is the model parameter provided by participant i to the server in the kth iteration; M is the number of participants; W k is the global model parameter at the kth iteration.

[0043] Preferably, the normalized form of the time offset accumulation of the model parameter p(i,k) in the kth iteration is defined as:

[0044]

[0045] Among them, (TOA(k)) max and(TOA(k)) min are the maximum and minimum TOA of all participants in a cluster respectively; TOA(i,k) is the accumulated time offset of participant i in the kth iteration.

[0046] Preferably, the differential privacy mechanism is defined as:

[0047]

[0048] Where N is the distribution of Gaussian noise; σ i is the standard deviation of the Gaussian noise N of participant i; is the local dataset D i Aggregation process on the server side.

[0049] Preferably, the standard deviation σ(i,k) of the Gaussian noise added to the global model parameters of the participant i at the kth iteration is defined as:

[0050]

[0051] in, is the global sensitivity; δ is the relaxation term; ∈(i,k) is the privacy budget of participant i in the kth iteration.

[0052] Preferably, the privacy budget of participant i in the k-th iteration is defined as:

[0053] ∈(i,k)=C(i,k)∈

[0054] Where C(i,k) is the trust score of the model parameters uploaded by participant i in the kth iteration; ∈ is the global privacy budget, expressed as ∈=max(∈(i,k)).

[0055] Preferably, the step of adding differentiated Gaussian noise to the aggregated model parameters by using a differentiated differential privacy mechanism according to the trust scores of the participants comprises the following steps:

[0056] According to the maximum distance between the trust score and the model parameters, the maximum distance between participant i in D after k iterations of aggregation operations is calculated. i Local sensitivity of model parameters on :

[0057]

[0058] Among them, C(i,k) is the confidence score of the model parameter; C sum Dist is the sum of the trust scores of all participants in one iteration; max is the maximum distance between two local model parameters;

[0059] Based on the local sensitivity, the global sensitivity of the model parameters in k iterations is calculated:

[0060]

[0061] in, For participant i in the local dataset D i The local sensitivity of the model parameters on the y-axis; C(i,k) is the confidence score of the model parameters; C sum Dist is the sum of the trust scores of all participants in one iteration; max is the maximum distance between two local model parameters; m is the minimum size of all user data sets;

[0062] According to the global sensitivity and the privacy budget, the Gaussian noise variance of each participant is calculated, and Gaussian noise is added to protect privacy before the global model parameters are sent from the central server to each user.

[0063] Another aspect of the present invention is to disclose a privacy-aware incremental poisoning attack defense system, characterized in that the system comprises:

[0064] The offset accumulation calculation module is used to calculate the offset accumulation of the uploaded model parameters according to the clustering results and density distance of the uploaded model parameters;

[0065] A model aggregation module, configured to divide the trust domain using a threshold based on the accumulated offset, calculate a trust score and aggregate model parameters accordingly;

[0066] The privacy protection module is used to add differentiated Gaussian noise to the aggregated model parameters according to the trust scores of the participants by using a differentiated differential privacy mechanism, so as to balance privacy protection and model accuracy.

[0067] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0068] It can not only effectively resist hidden poisoning attacks based on GAN, but also maintain model accuracy as much as possible while protecting privacy, greatly improving the security and privacy in federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Shown is a schematic diagram of a federated learning architecture in one embodiment of the present invention;

[0070] Figure 2 Shown is a schematic diagram of a poisoning attack based on GAN in federated learning in one embodiment of the present invention;

[0071] Figure 3 Shown is a flowchart of a privacy-aware incremental poisoning attack defense method disclosed in an embodiment of the present invention;

[0072] Figure 4 Shown is a schematic diagram of the principle structure of a privacy-aware incremental poisoning attack defense method disclosed in an embodiment of the present invention;

[0073] Figure 5 Shown is a schematic diagram of a progressive poisoning attack process in two-dimensional space in one embodiment of the present invention;

[0074] Figure 6 Shown is a schematic diagram of the principle structure of the privacy-aware incremental poisoning attack defense system of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0076] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0077] like Figure 1 As shown, Figure 1 Schematic diagram of a distributed federated learning model. In one embodiment of the present invention, the designed defense method is deployed on the server side of federated learning. In federated learning, there are usually multiple users and a central server. Each user has his or her local data set, and all users share the same model structure and reach a consistent model training goal. In each iteration, the user trains the local model on the local training set and uploads the model parameters (updates or gradients) to the central server. The central server aggregates the model parameters (updates or gradients) of all users and updates the global model. Subsequently, the updated global model is downloaded from the server side to each user side, and each user side uses the global model parameters as the new local model parameters and continues local training. This process continues to iterate until the model converges.

[0078] Although federated learning has solved the data island problem to a certain extent at this stage, it still faces multiple security issues and is vulnerable to different forms of attacks. Among them, poisoning attacks based on generative adversarial networks are common attacks in federated learning. Generative adversarial networks are a deep learning model composed of two neural networks, the generator and the discriminator. Its essence is a zero-sum game problem. The generator aims to input random noise to generate forged data that conforms to the distribution of real samples, while the discriminator is responsible for judging whether the generated data is consistent with the real samples. The goal of the generator is to generate forged samples that are as close to the real data as possible, and the goal of the discriminator is to distinguish the generated samples from the real samples as accurately as possible. When the data generated by the generator reaches the standard of being sufficiently real, the game between the generator and the discriminator reaches a balance.

[0079] In a GAN-based poisoning attack, the attacker disguises himself as a participant in federated learning and generates fake data by training GAN locally to implement a poisoning attack. Although the attacker did not upload his own training data, the model parameters he uploaded still contain the attacker's training data information. The attacker uses the global model parameters to replace the network parameters of the discriminator in GAN, which is equivalent to training the discriminator directly on the target data, so that the generator can generate fake samples similar to the target data. Then, the attacker poisons these fake samples, such as flipping labels or adding triggers, and uses these poisoned data to train the local model. Finally, the attacker uploads the poisoned model parameters to the central server. After aggregating the model parameters uploaded by all participants (including the poisoned model parameters), the central server generates an updated global model and sends it to all users. In the next iteration, the local models of other users are gradually affected by the poisoned global model, so that their models are gradually poisoned.

[0080] Figure 2This is a poisoning attack process based on GAN. After the victim trains the data with the correct label locally, he uploads the updated model parameters to the server. At the same time, the attacker uses GAN to imitate and generate fake data similar to the victim's data, and labels these fake data with wrong labels to train his local model. The attacker then updates and uploads the poisoned model parameters.

[0081] In the early stages of the attack, since the data generated by GAN has not yet fully matched the data of other users, the poisoning operation has little impact on the main task of the global model, and is therefore able to evade the basic defense mechanism of the server. Some attackers take advantage of this feature and only add a small amount of poisoned data in the early stages of global model training to generate "slightly poisoned" model parameters to evade detection. In subsequent iterations, the attacker gradually increases the intensity of poisoning, eventually causing the global model to be completely poisoned.

[0082] like Figure 3 As shown, the present invention provides a privacy-aware incremental poisoning attack defense method, comprising the steps of:

[0083] S1: Calculate the accumulated offset of the uploaded model parameters according to the clustering results and density distance of the uploaded model parameters;

[0084] S2: Based on the accumulated offset, a threshold is used to divide the trust domain, a trust score is calculated, and model parameters are aggregated accordingly.

[0085] S3: Based on the trust scores of the participants, a differentiated differential privacy mechanism is adopted to add differentiated Gaussian noise to the aggregated model parameters, taking into account both privacy protection and model accuracy.

[0086] The privacy-aware incremental poisoning attack defense method provided by the present invention is as follows: Figure 4 As shown, the user uploads the model parameters to the server in each round. The server first uses the Affinity Propagation (AP) clustering algorithm to determine the cluster center of the model parameters and divide them into multiple clusters. Then the present invention defines a density distance to obtain the time-accumulated offset of each model parameter, which can reduce the possibility of the cluster center being controlled by the poisoned model parameters. Then, combined with the time-accumulated offset, the trust domain is divided using a threshold, and the highest trust score is assigned to the participants in the completely trusted domain, while the trust score assigned to the completely untrusted domain is 0, which can timely detect malicious participants and extract them from the global model parameters. Finally, the present invention adds Gaussian noise to the global model based on differential privacy to protect the privacy of the participants' training data sets. This method can not only effectively resist the hidden poisoning attack based on GAN, but also maintain the model accuracy as much as possible under the premise of protecting privacy, greatly improving the security and privacy in federated learning.

[0087] A preferred implementation is that step S1 further includes the following steps:

[0088] S11: Calculate the density distance of the model parameter point p(i,k) at the kth iteration based on the clustering results obtained by the neighbor propagation clustering algorithm:

[0089] nd(i,k)=dis(i,k)-Cir(i,k) (1)

[0090] Wherein, dis(i,k) is the normalized form of the Euclidean distance between the model parameter p(i,k) and its cluster center at the k-th iteration; Cir(i,k) is the normalized form of the number of points in the circle with radius r and center p(i,k) in the k-th iteration. The present invention uses the number of model parameter points in the unit circle to reflect the degree of aggregation of the model parameter points. In this embodiment, all radii r are set to 0.5.

[0091] S12: Calculate the time offset accumulation TOA(i,k) based on the changes in density distance of the model parameters provided by the participants in all current iterations:

[0092]

[0093] Where Δnd(i,k) is the change in the density distance between the model parameter p(i,k) at the (k-1)th and kth times (k>1); K is the number of all iterations so far.

[0094] To facilitate subsequent calculations, the normalized form of the time offset accumulation of the model parameter p(i,k) in the kth iteration is calculated as TOA(i,k):

[0095]

[0096] Among them, (TOA(k)) max and(TOA(k)) min are the maximum and minimum TOA of all participants in a cluster respectively; TOA(i,k) is the accumulated time offset of participant i in the kth iteration.

[0097] Furthermore, the clustering results of all uploaded model parameters obtained according to the neighbor propagation algorithm include the following steps:

[0098] The AP algorithm selects cluster centers by transferring information between different points, which mainly includes three information matrices: the attribution matrix A, the similarity matrix S and the attraction matrix R.

[0099] The similarity s(i,j) between any two data points i and j is calculated according to the following formula:

[0100]

[0101] Wherein, Ref is the value of the diagonal element of the similarity matrix S (all diagonal elements have the same value). In this embodiment, the value of Ref is set to the minimum value of all non-diagonal elements. During the k-time global model training period, the model parameter set is defined as P = {p(1, k), p(2, k), ...}. The model parameter set P is the data point set of the AP clustering algorithm.

[0102] Update the attraction matrix and the attribution matrix according to the initialized information matrix. In the t+1th iteration, the attraction matrix r is updated. t+1 (i,k) is:

[0103] r t+1 (i,k)=s(i,j)-max k′≠k {a t (i,j ′ )+s(i,j′)} (5)

[0104] Update the attribution matrix a in the t+1th iteration t+1 (i,j):

[0105]

[0106] Among them, the attribution matrix A, the similarity matrix S and the attraction matrix R are initialized as zero matrices; the attribution degree a(i, j) in the attribution matrix A represents the degree to which data point i selects data point j as its cluster center; the similarity s(i, j) in the similarity matrix S represents the negative of the Euclidean distance between data points i and j; the attraction r(i, j) in the attraction matrix R represents the degree to which data point j is suitable as the cluster center of data point i; t represents the number of iterations run so far in the AP clustering algorithm.

[0107] In each iteration, if the update value is not controlled, the value in the information matrix will change dramatically, making it difficult to converge. Therefore, in order to control the pace of the update and reduce the oscillation phenomenon, it is necessary to introduce an attenuation coefficient λ when updating the information matrix. Therefore, the attraction matrix and the attribution matrix have the following update formulas:

[0108] r t+1 (i,j)=(λ-1)r t+1 (i,j)+λr t (i,j) (7)

[0109] a t+1 (i,j)=(λ-1)a t+1 (i,j)+λa t (i,j) (8)

[0110] When the information matrix is ​​stable or reaches the maximum number of iterations, the data point with the sum of the attraction information and the belonging information greater than 0 is selected as the cluster center. That is, the zth cluster center O z The choices are:

[0111] O z ={p(i,k)|r(i,i)+a(i,i)>0} (9)

[0112] The cluster center is obtained by the above AP clustering algorithm. Among them, the model parameter point p(i,k) belongs to the cluster with the greatest similarity with the model parameter p(i,k) in the cluster center.

[0113] Further, in step S12, the normalized form of the Euclidean distance between the model parameter p(i,k) and its cluster center at the kth iteration is defined as:

[0114]

[0115] The normalized form of the number of points in a circle with radius r and center p(i,k) in the kth iteration is defined as:

[0116]

[0117] Among them, Cir(max,k) and Cir(min,k) are the maximum and minimum values ​​of the number of model parameter points clustered around the model parameter point; dis(max,k) and dis(min,k) are the maximum and minimum values ​​of the Euclidean distances of all model parameter points relative to their cluster centers at the kth iteration.

[0118] In one embodiment of the present invention, the cluster center of the model parameters is obtained by the neighbor propagation clustering algorithm, based on whether the current model parameters obtain support from more model parameters rather than their statistical characteristics, thereby effectively reducing the risk of the cluster center being controlled by poisoned model parameters.

[0119] In view of the distribution diversity of the participant training set data and the uneven density of the model parameter clustering clusters, simply relying on the Euclidean distance between the model parameters and their cluster centers cannot fully reflect the degree of deviation of the model parameters. Therefore, the present invention also comprehensively considers the distribution density around the model parameters, combines the density aggregation degree and the Euclidean distance, and calculates the density distance nd(i,k). In order to make the density and the Euclidean distance have a unified dimension, the two are normalized separately so that the normalized Euclidean distance is in the range of 0 to 1.

[0120] Obviously, the number of parameter points around a model parameter point reflects the degree of clustering of the point, which is inversely proportional to the degree of its deviation from the cluster center. The fewer the number of parameter points and the greater the Euclidean distance, the higher the probability that the model parameter is abnormal. This combination of density and distance calculation can more accurately reflect the degree of deviation of the model parameters, thereby further improving the accuracy of detecting malicious actors.

[0121] In addition, in real applications, benign participants usually account for the majority, and the proportion of attackers is relatively low, not exceeding one-third of all participants. Since benign participants in the same group have consistent training goals, the change trends of their model parameters tend to be consistent. Based on this, by accumulating the changes in density distance in all iterations, the present invention calculates the time accumulation amount (TOA). This time offset accumulation amount avoids the influence of random offset of model parameters in a single iteration, can more accurately reflect the overall trend of model parameters deviating from the cluster center, effectively capture the abnormal behavior of malicious participants, and thus enhance the defense capability against poisoning attacks.

[0122] A preferred implementation is that step S2 further includes the following steps:

[0123] S21: According to the accumulated time offset, two thresholds Thr are used in each cluster. full (Fully Trusted Threshold) and Thr semi (Semi-trust threshold) is divided into three trust domains:

[0124]

[0125] Among them, Dom ft 、Dom st and Dom ut are the fully trusted domain, semi-trusted domain, and untrusted domain respectively; TOA(i,k) is the normalized form of the accumulated time offset of the model parameter p(i,k) in the kth iteration; the fully trusted threshold is set to: Thr full =0.9, and the semi-trust domain value is set to: Thr semi =0.4.

[0126] S22: Calculate the trust scores of the model parameters based on the three divided trust domains:

[0127]

[0128] Where C(i,k) is the confidence score of the model parameter p(i,k) at the kth iteration; TOA(i,k) is the normalized form of the accumulated time offset of the model parameter p(i,k) at the kth iteration; Dom ft 、Dom st and Domut They are fully trusted domain, semi-trusted domain and untrusted domain.

[0129] S23: Assign different weights to each participant according to the trust score C(i,k) of the model parameters, thereby calculating the global model parameters:

[0130]

[0131] Where w(i,k) is the model parameter provided by participant i to the server in the kth iteration; M is the number of participants; W k is the global model parameter at the kth iteration.

[0132] In one embodiment of the present invention, during the global model training process, the model parameters uploaded by the participants will change with iterations. The model parameters of benign participants tend to gradually approach the cluster center, while potential malicious participants try to deviate from the correct direction through poisoning operations. Figure 5 The process of a progressive poisoning attack in two-dimensional space is shown, and the dotted line trajectory represents the impact of the poisoning attack on the model. The attacker gradually injects malicious data with bias to cause the model to deviate in the wrong direction. Over time, the attacker gradually manipulates the model training process to gradually deviate from the real target to the poisoned target. This attack method does not inject a large amount of malicious data at one time, but slowly changes the model's learning path through small, continuous interference. Because the model changes slowly and insignificantly, the attack effect is not easy to detect.

[0133] To deal with such attacks, the present invention divides each cluster into three trust domains by setting two thresholds: the fully trusted domain Dom ft , Semi-trusted Domain Dom st and untrusted domain Dom ut The standard for fully trusted participants is relatively strict, and potential malicious participants are excluded as much as possible, so we set a high standard for the fully trusted threshold: Thr full = 0.9, and set a higher semi-trust threshold value to distinguish potential attackers: Thr semi =0.4. To avoid the influence of random factors, the present invention adopts the statistical variable Thr count To count the number of times the model parameters are divided into the third domain (untrusted domain). count =2, that is, when Thr count When ρ>2, the present invention completely deletes the participant's model parameters from the global training.

[0134] In addition, in the fully trusted domain, participants are considered fully trusted and are assigned the highest trust score of 1; in the semi-trusted domain, some participants may be benign but have limited resources, while others are malicious participants who are still in the early stages of launching attacks; in the untrusted domain, participants are judged to be attackers, and their uploaded model parameters will be completely deleted, and the trust score is set to 0. Next, the trust score C(i,k)=TOA(i,k) of the participants in the semi-trusted domain is calculated based on the time offset accumulation of the model parameters. Model parameters with higher trust scores are more trustworthy, so the present invention performs differentiated aggregation for model parameters based on the trust score. Specifically, participants with higher trust scores are assigned higher aggregation weights, and participants with lower trust scores are assigned lower aggregation weights.

[0135] Finally, the central server aggregates the model parameters of each participant according to the proportion of the participant's trust score in the sum of all participants' trust scores: Get the global model parameter W k The present invention can amplify the optimization effect of benign participant model parameters, quickly detect malicious participants, and resist the poisoning of the global model by undetected malicious model parameters through the trust score calculation mechanism based on the time offset accumulation and the differentiated aggregation method.

[0136] A preferred implementation is that step S3 further includes the following steps:

[0137] S31: Based on the maximum distance between the trust score and the model parameters, the distance between participant i in the local dataset D after k iterations of aggregation operations is calculated. i Local sensitivity of model parameters on :

[0138]

[0139] Among them, C(i,k) is the confidence score of the model parameter; C sum Dist is the sum of the trust scores of all participants in one iteration; max is the maximum distance between two local model parameters.

[0140] Calculate the global sensitivity of the model parameters in k iterations based on the local sensitivity:

[0141]

[0142] in, For participant i in the local dataset D i The local sensitivity of the model parameters on the y-axis; C(i,k) is the confidence score of the model parameters; C sum Dist is the sum of the trust scores of all participants in one iteration;max is the maximum distance between two local model parameters; m is the minimum size of all user data sets; in this embodiment, maxC(i,k)=1.

[0143] S32: The present invention adopts a differentiated differential privacy mechanism: according to the trust scores of the participants, different degrees of noise are added to the aggregated global model parameters. Local dataset D i The differential privacy mechanism on is:

[0144]

[0145] Where N is the distribution of Gaussian noise; σ i is the standard deviation of the Gaussian noise N of participant i; is the local dataset D i Aggregation process on the server side.

[0146] Further, in step S32, according to the Gaussian mechanism, the standard deviation σ(i,k) of the Gaussian noise added by participant i to the global model parameters of the kth iteration is defined as:

[0147]

[0148] in, is the global sensitivity; δ is the relaxation term, which means that the present invention can tolerate a small error; ∈(i,k) is the privacy budget of participant i in the kth iteration.

[0149] Furthermore, the privacy budget of participant i in the kth iteration is defined as:

[0150] ∈(i,k)=C(i,k)∈(19)

[0151] Where C(i,k) is the trust score of the model parameters uploaded by participant i in the kth iteration; ∈ is the global privacy budget, expressed as ∈=max(∈(i,k)).

[0152] In one embodiment of the present invention, the local data sets of all participants are disjoint, and each participant uses the local data set for training. i The local training process on As shown below:

[0153]

[0154] Where w(i,k) is the model parameter of participant i in the kth iteration; D is the training dataset of all participants; D i is the training dataset of participant i; F i(·) is the data set D i The loss function of the local model of participant i on the dataset D i The jth sample in D i,j express.

[0155] In each iteration, all participants upload their local model parameters to the central server. The central server calculates the maximum distance Dist between two local model parameters based on the model parameters of all participants. max , thus ensuring that any two model parameters w(i,k) and w(j,k) satisfy ‖w(i,k)-w(j,k)‖≤Dist max This process ensures reasonable consistency of model parameters across participants, helps prevent model drift, and reduces the risk of model poisoning.

[0156] Furthermore, before the central server sends the global model parameters to each participant, Gaussian noise is used to perturb the global model parameters. Local sensitivity dynamically adjusts the noise size based on the training results of each participant on the local data set; while global sensitivity is based on the model parameters uploaded by all participants to ensure that the added noise is sufficient to achieve privacy protection. For participants with smaller differences, smaller noise is added to maintain model accuracy; for participants with larger differences, larger noise is added to enhance privacy protection. Through this adaptive noise adjustment, a balance can be achieved between the models uploaded by different participants, so that the risk of privacy leakage is reduced while the model accuracy will not be too negatively affected by the indiscriminate addition of noise.

[0157] In addition, a privacy budget allocation mechanism based on trust is used in the present invention. Participants with low trust scores may be at higher risk of being attackers, and thus are allocated less privacy budget. For such participants, the central server adds more Gaussian noise before sending the global model parameters to prevent them from obtaining accurate global model parameters. The trust score is proportional to the privacy budget. The same global privacy budget value is set in each iteration, and differentiated Gaussian noise is added to the global model parameters received by each participant according to their trust.

[0158] In response to GAN-based poisoning attacks, this paper adopts the differential privacy (DP) mechanism to resist the risk of attackers stealing global model information through GAN. The DP mechanism dynamically adjusts the noise size and injects noise into the global model parameters to ensure that the data uploaded by the participants cannot be directly restored, thereby effectively preventing attackers from obtaining the private information of the participants through the global model. At the same time, we consider the impact of model accuracy in the noise adjustment process. By fine-tuning the noise size, we minimize the damage to the model accuracy while ensuring privacy. Finally, the noise-processed global model parameters are securely distributed to each participant.

[0159] Through this mechanism, the present invention protects the privacy and security of participants' data in the federated learning process while resisting GAN poisoning attacks, achieves a balance between privacy protection and model accuracy, and significantly improves the robustness and security of the system.

[0160] The steps of the above method are divided only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the protection scope of the present invention. Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of the invention.

[0161] like Figure 6 As shown, the present invention also provides a privacy-aware incremental poisoning attack defense system, including: an offset accumulation calculation module, a model aggregation module and a privacy protection module. Among them, according to the clustering results and density distance of the uploaded model parameters, the offset accumulation of the uploaded model parameters is calculated; the model aggregation module is used to divide the trust domain based on the offset accumulation using a threshold, calculate the trust score and aggregate the model parameters accordingly; the privacy protection module is used to add differentiated Gaussian noise to the aggregated model parameters according to the trust scores of the participants, using a differentiated differential privacy mechanism, taking into account both privacy protection and model accuracy.

[0162] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0163] In addition, those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation; for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0164] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0167] As described above, the present invention provides a privacy-aware incremental poisoning attack defense method. It can effectively resist the hidden poisoning attack based on GAN, and maintain the model accuracy as much as possible under the premise of protecting privacy, which greatly improves the security and privacy in federated learning. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0168] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A privacy-aware incremental poisoning attack defense method, characterized in that: The following steps are involved: According to the clustering results and density distance of the model parameters uploaded by the participants, the accumulated offset of the uploaded model parameters is calculated; Based on the accumulated offset, the trust domain is divided using a threshold, the trust scores of the participants are calculated and the model parameters are aggregated accordingly; According to the trust scores of the participants, a differentiated differential privacy mechanism is adopted to add differentiated Gaussian noise to the aggregated model parameters, taking into account both privacy protection and model accuracy.

2. A privacy-aware incremental poisoning attack defense method as claimed in claim 1, characterized in that: The offset accumulation amount is calculated by the following steps: According to the clustering results obtained by the neighbor propagation clustering algorithm, the density distance of the model parameter point p(i,k) in the kth iteration is calculated: nd(i,k)=dis(i,k)-Cir(i,k) Where dis(i,k) is the normalized form of the Euclidean distance between the model parameter p(i,k) and its cluster center at the kth iteration; Cir(i,k) is the normalized form of the number of points in the circle with radius r and center p(i,k) in the kth iteration; According to the changes in density distance of the model parameters provided by the participants in all current iterations, the time offset accumulation TOA(i,k) is calculated: Where Δnd(i,k) is the change in the density distance between the model parameter p(i,k) at the (k-1)th and kth times, k>1; K is the number of all iterations so far.

3. A privacy-aware incremental poisoning attack defense method as claimed in claim 1, characterized in that: The clustering results of all uploaded model parameters obtained according to the nearest neighbor propagation algorithm include the following steps: The similarity s(i,j) between any two data points i and j is calculated according to the following formula: Where Ref is the value of the diagonal element of the similarity matrix S, and all diagonal elements have the same value; p(i, k) and p(j, k) are the model parameters uploaded by the i-th and j-th participants in the k-th global model training period, respectively; Update the attraction matrix and the attribution matrix according to the initialized information matrix, where the attraction matrix r is updated in the t+1th iteration. t+1 (i,k) is: Update the attribution matrix a in the t+1th iteration t+1 (i,j): Among them, the attribution matrix A, the similarity matrix S and the attraction matrix R are initialized as zero matrices; the attribution a(i, j) in the attribution matrix A represents the degree to which data point i selects data point k as its cluster center; the similarity s(i, j) in the similarity matrix S represents the negative of the Euclidean distance between data points i and j; the attraction r(i, j) in the attraction matrix R represents the degree to which data point j is suitable as the cluster center of data point i; t represents the number of iterations run so far in the AP clustering algorithm; According to the introduced attenuation coefficient λ, the attraction matrix and the belonging matrix are updated: r t+1 (i,j)=(λ-1)r t+1 (i,j)+λr t (i, j) a t+1 (i,j)=(λ-1)a t+1 (i,j)+λa t (i,j) When the information matrix is ​​stable or reaches the maximum number of iterations, according to the sum of the attraction information and the belonging information, the data point with the information sum greater than 0 is selected as the cluster center, that is, the zth cluster center O z The choices are: O z ={p(i,k)|r(i,i)+a(i,i)>0} Among them, the model parameter point p(i,k) belongs to the cluster with the greatest similarity with the model parameter p(i,k) in the cluster center.

4. A privacy-aware incremental poisoning attack defense method as claimed in claim 1, characterized in that: The method of dividing the trust domain based on the accumulated offset using a threshold, calculating the trust scores of the participants and aggregating the model parameters accordingly comprises the following steps: According to the accumulated time offset, in each cluster, a completely credible threshold Thr is used. full and the semi-trust threshold Thr semi Divide into three trust domains: Among them, Dom ft 、Dom st and Dom ut are the fully trusted domain, semi-trusted domain, and untrusted domain respectively; TOA(i,k) is the normalized form of the accumulated time offset of the model parameter p(i,k) in the kth iteration; According to the three divided trust domains, the trust scores of the model parameters are calculated: Where C(i,k) is the confidence score of the model parameter p(i,k) at the kth iteration; TOA(i,k) is the normalized form of the accumulated time offset of the model parameter p(i,k) at the kth iteration; Dom ft 、Dom st and Dom ut They are fully trusted domain, semi-trusted domain and untrusted domain respectively; According to the trust score C(i,k) of the model parameters, different weights are assigned to each participant to calculate the global model parameters: Where w(i,k) is the model parameter provided by participant i to the server in the kth iteration; M is the number of participants; W k is the global model parameter at the kth iteration.

5. A privacy-aware incremental poisoning attack defense method as claimed in claim 4, characterized in that: The normalized form of the time offset accumulation of the model parameter p(i,k) in the kth iteration is defined as: Among them, (TOA(k)) max and(TOA(k)) min are the maximum and minimum TOA of all participants in a cluster respectively; TOA(i,k) is the accumulated time offset of participant i in the kth iteration.

6. A privacy-aware incremental poisoning attack defense method as claimed in claim 1, characterized in that: The differential privacy mechanism is defined as: Where N is the distribution of Gaussian noise; σ i is the standard deviation of the Gaussian noise N of participant i; is the local dataset D i Aggregation process on the server side.

7. A privacy-aware incremental poisoning attack defense method as claimed in claim 6, characterized in that: The standard deviation σ(i,k) of the Gaussian noise added by participant i to the global model parameters at the kth iteration is defined as: in, is the global sensitivity; δ is the relaxation term; ∈(i,k) is the privacy budget of participant i in the kth iteration.

8. A privacy-aware incremental poisoning attack defense method as claimed in claim 7, characterized in that: The privacy budget of participant i in the kth iteration is defined as: ∈(i,k)=C(i,k)∈ Where C(i,k) is the trust score of the model parameters uploaded by participant i in the kth iteration; ò is the global privacy budget, expressed as ∈=max(∈(i,k)).

9. A privacy-aware incremental poisoning attack defense method as claimed in claim 1, characterized in that: The step of adding differentiated Gaussian noise to the aggregated model parameters by using a differentiated differential privacy mechanism according to the trust scores of the participants includes the following steps: According to the maximum distance between the trust score and the model parameters, the maximum distance between participant i in D after k iterations of aggregation operations is calculated. i Local sensitivity of model parameters on : Among them, C(i,k) is the confidence score of the model parameter; C sum Dist is the sum of the trust scores of all participants in one iteration; max is the maximum distance between two local model parameters; Based on the local sensitivity, the global sensitivity of the model parameters in k iterations is calculated: in, For participant i in the local dataset D i The local sensitivity of the model parameters on the y-axis; C(i,k) is the confidence score of the model parameters; C sum Dist is the sum of the trust scores of all participants in one iteration; max is the maximum distance between two local model parameters; m is the minimum size of all user data sets; According to the global sensitivity and the privacy budget, the Gaussian noise variance of each participant is calculated, and Gaussian noise is added to protect privacy before the global model parameters are sent from the central server to each user.

10. A privacy-aware incremental poisoning attack defense system, characterized in that: The system comprises: The offset accumulation calculation module is used to calculate the offset accumulation of the uploaded model parameters according to the clustering results and density distance of the uploaded model parameters; A model aggregation module, configured to divide the trust domain using a threshold based on the accumulated offset, calculate a trust score and aggregate model parameters accordingly; The privacy protection module is used to add differentiated Gaussian noise to the aggregated model parameters according to the trust scores of the participants by using a differentiated differential privacy mechanism, so as to balance privacy protection and model accuracy.

Images