Fair regression model construction method and system based on differential privacy federated learning

Abstract

The invention belongs to the technical field of federated learning, and provides a fair regression algorithm based on differential privacy federated learning, and the technical scheme for realizing the fair regression algorithm mainly comprises three modules: firstly, training a global regression model by using a federated learning algorithm; secondly, centralized differential privacy is introduced, and the variance of discrete Gaussian distribution is determined according to a given differential privacy parameter; and finally, under the framework of federated learning, a frequency distribution histogram is calculated by using a differential privacy technology based on a discrete Gaussian mechanism, so that a distribution function and a quantile function are estimated, and fairness post-processing is performed on the regression model. In the post-processing process, the information exchanged between the client and the central server is only the frequency distribution histogram of the regression prediction value and the corresponding sensitive attribute value, and the communication cost is low. Experiments show that the method can greatly improve fairness of the regression model while providing relatively strong privacy protection, and ensures that the accuracy of the model is maintained at a relatively high level.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and in particular to a method and system for constructing a fair regression model by post-processing a regression model using federated learning based on differential privacy. Background Art

[0002] In the context of big data, machine learning technology driven by big data is developing rapidly. Algorithmic fairness is an important goal of using algorithms for decision-making. It means that the algorithm can treat data fairly and without bias, and does not produce discriminatory results. However, the actual training data set itself is often biased. In this case, the machine learning algorithm is likely to learn these biases from the data and give discriminatory results. Group equality is an important indicator for evaluating algorithm fairness. Post-processing the algorithm results is one of the ways to achieve algorithm fairness.

[0003] In the data-driven era, data privacy protection issues are also receiving more and more attention. Federated learning is a distributed machine learning paradigm that allows clients to collaboratively train global models without sharing original data, solving the "data island" problem. Federated learning does not require data to be stored centrally and naturally has privacy protection characteristics, but its privacy protection is not enough to completely resist external privacy attacks, and there is still a risk of privacy leakage. Differential privacy is a data privacy protection technology that perturbs data by adding appropriate noise, while protecting data privacy and still ensuring a certain degree of data availability. Using differential privacy technology for federated learning can enhance the effect of data privacy protection in federated learning.

[0004] At present, there is still a lack of technology based on the federated learning framework to achieve algorithmic fairness under privacy protection conditions. How to effectively train fair models while preventing privacy leakage needs to be urgently addressed. Summary of the invention

[0005] This paper provides an algorithm for implementing fair regression using post-processing methods under the framework of differential privacy federated learning, taking group equality as the evaluation index of algorithm fairness. The regression model considered is Where S represents the sensitive attribute, X represents the explanatory variables other than the sensitive attribute, and Y represents the response variable.

[0006] The technical solution to implement the present invention mainly includes three modules: global regression model training, determining the noise size according to the privacy protection strength, and global regression model fairness post-processing. The specific implementation method is as follows:

[0007] Global regression model training: The global regression model is trained using the FedAvg algorithm. The steps are as follows:

[0008] S1: The central server sends the initial regression model to the client to complete model initialization;

[0009] S2: The client trains the regression model using the local training data set and updates the local regression model;

[0010] S3: The client uploads its local regression model parameters and training data set size to the central server;

[0011] S4: The central server aggregates the model parameters in a weighted average manner according to the size of the training dataset of each client and updates the global regression model;

[0012] S5: The central server sends the global regression model to the client;

[0013] S6: Repeat steps S2-S5 until model training is completed;

[0014] In the present invention, the purpose of the global regression model training part is to obtain a global regression model under the federated learning framework. There is no restriction on the specific method used. The above method uses the federated average algorithm. In practical applications, other federated learning algorithms such as FedProx, SCAFFOLD, and FedNova can also be used.

[0015] In addition, the present invention focuses on solving the problem of post-processing the regression model to make it fair under the condition of privacy protection in the federated learning framework, so the above-mentioned global regression model training part does not include additional privacy protection technology. In practical applications, the global regression model training part can also use privacy protection technologies such as differential privacy, homomorphic encryption, and secure multi-party computing to provide privacy protection. It should be pointed out that although differential privacy has post-processing invariance, even if differential privacy technology is used in the stage of training the global regression model, privacy protection technology must be used in the post-processing stage of the regression model fairness to provide privacy protection for this stage. This is because the post-processing invariance of differential privacy requires that the post-processing part does not use private data, while the client's original data will still be used when the regression model is post-processed to be fair.

[0016] Global regression model fairness post-processing: The present invention performs post-processing on the regression model f(x, s) to make it fair based on the following formula:

[0017]

[0018] in, F f|s is the distribution function of f(X,S) given S=s, Q f|s is the quantile function of f(X,S) given S=s, and g(x,s) is the fair regression model.

[0019] From the above formula, it can be seen that fair post-processing requires knowing the distribution function of sensitive attributes and the distribution function of the predicted value under the given sensitive attributes (and the corresponding conditional quantile function). This paper uses histograms and differential privacy technology based on discrete Gaussian mechanisms to estimate the required distribution function under privacy protection conditions, thereby obtaining a fair regression model. The specific steps are as follows:

[0020] S1: The central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters;

[0021] S2: The central server sends the regression model, discrete Gaussian distribution variance, interval division points, sensitive attribute set, noise addition amount, and scaling factor to the client;

[0022] S3: The client calculates the local frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute;

[0023] S4: The client adds a certain amount of discrete Gaussian noise to each local frequency distribution histogram, and uploads the disturbed local frequency distribution histogram and its corresponding sensitive attribute value to the central server;

[0024] S5: The central server performs aggregation operations to obtain the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute, and sends the overall frequency distribution histogram and its corresponding sensitive attribute values ​​to the client;

[0025] S6: client estimates the distribution of sensitive attributes and the conditional distribution function of the predicted value;

[0026] S7: Client-side estimated fair regression model.

[0027] In the global regression model fairness post-processing module, step S6 uses the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value, combined with the linear interpolation method, to estimate the distribution function of the predicted value under the condition of known sensitive attributes; further, the sum of the frequencies of all grouping intervals of the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value is calculated, and it is used as the frequency estimate of the corresponding sensitive attribute value, and then the probability distribution of the sensitive attribute is estimated from the frequency distribution of the sensitive attribute.

[0028] In the global regression model fairness post-processing module, step S7 performs The post-fairness processing is as follows:

[0029]

[0030] Among them, ξ~U([-v,v]) is a smooth random variable, v is a very small number (such as 10 -6 ) to ensure that it does not change The statistical properties of is the sensitive attribute distribution estimate; is the conditional distribution function estimate of the predicted value; It is the conditional quantile function estimate of the predicted value. Specifically, the conditional quantile function corresponding to the probability p is: the histogram interval division points l0, l1, ..., l b The partition point closest to p in the corresponding conditional distribution function value.

[0031] Determine the noise size according to the privacy protection strength: In step S1 of the above fair post-processing process, the specific method for the central server to calculate the variance of the discrete Gaussian distribution according to the given differential privacy parameters is as follows:

[0032] S1: According to the following formula, the centralized differential privacy parameters ε and δ are approximated to solve

[0033]

[0034] S2: According to the following formula, the centralized differential privacy parameter Approximate solution to the variance of discrete Gaussian distribution

[0035]

[0036] Among them, Δ p For the algorithm sensitivity, d is the dimension of the added discrete Gaussian noise, n is the number of added discrete Gaussian noise,

[0037] In step S1 of the module for determining the noise size according to the privacy protection strength, an explicit expression for calculating the differential privacy parameter δ from the centralized differential privacy parameter ρ and the privacy budget ε is given. Specifically, let

[0038] g(α)=(α-1)(αρ-ε)+(α-1)·log(1-1 / α)-log(α)

[0039] but It can be proved that g(α) is a smooth convex function on α∈(1,∞) and has a unique minimum point. Furthermore, when 0<ρ<ε, g(α) has a unique minimum point on α∈(1,∞) If the centralized differential privacy parameter ρ and privacy budget ε (0<ρ<ε) are given, the dichotomy method can be used to approximately solve the differential privacy parameter δ.

[0040] The centralized differential privacy parameter ρ is obtained from the differential privacy parameters ε and δ, and an inverse problem needs to be solved. Given ε, δ is strictly monotonically increasing with respect to ρ. Considering 0<ρ<ε, using the dichotomy method, if and ε calculated If the difference between the given δ and the given precision is less than the given precision, then This is the desired centralized differential privacy parameter.

[0041] In step S2 of the noise size module that determines the strength of privacy protection, since the fairness post-processing algorithm adds discrete Gaussian noise to the frequency distribution histogram of the client's prediction value, the dimension of the added discrete Gaussian noise is the number of grouping intervals of the histogram. Further, the central server aggregates the overall frequency distribution histogram of the prediction value corresponding to each sensitive attribute value, so the amount of added discrete Gaussian noise is the product of the number of clients and the amount of noise added to each histogram. In addition, since the unique hot encoding of each sample prediction value has only one element that is 1 and the rest of the elements are 0, the algorithm Sensitivity is Δ p =c, where c is the scaling factor.

[0042] Step S2 gives the discrete Gaussian distribution variance σ 2 Calculate the explicit expression of the concentrated differential privacy parameter ρ. It can be seen that the concentrated differential privacy parameter ρ increases with the variance of the discrete Gaussian distribution σ. 2 If the differential privacy parameter ρ is given, it can be gradually increased by Until Calculated differential privacy parameters Less than or equal to the given ρ, then This is the variance of the desired discrete Gaussian distribution.

[0043] The present invention also provides a fair regression system based on differential privacy federated learning, which includes the following modules:

[0044] Global regression model training module: trains global regression models under the federated learning framework, and supports the use of federated learning algorithms such as FedAvg and FedProx. According to privacy protection requirements, it can combine privacy protection technologies such as differential privacy, homomorphic encryption, and secure multi-party computing to provide privacy protection for the training process.

[0045] Noise calculation module: Based on the given differential privacy parameters, the discrete Gaussian distribution variance is calculated through the formula; according to the privacy protection strength, the noise size is determined and the noise factor is generated;

[0046] Fairness post-processing module: Using the global regression model, combined with the sensitive attribute distribution and the conditional distribution of the predicted value, the model is post-processed for fairness, including local frequency distribution calculation, discrete Gaussian noise addition, distribution estimation and regression model adjustment.

[0047] Furthermore, the specific steps of the noise calculation module include:

[0048] The centralized differential privacy parameters are approximated by differential privacy parameters;

[0049] The variance of the discrete Gaussian distribution is approximated using concentrated differential privacy parameters.

[0050] Determine the number of grouping intervals and noise dimension of the histogram corresponding to the sensitive attribute, and generate a noise factor for each client based on the given discrete Gaussian noise addition amount, which is used to perturb the local frequency distribution histogram.

[0051] Furthermore, the fairness post-processing module performs the following steps:

[0052] The client calculates the local frequency distribution histogram of the predicted value corresponding to each sensitive attribute, adds discrete Gaussian noise and uploads it to the central server;

[0053] The central server aggregates the local frequency distribution histogram after disturbance and generates the overall frequency distribution histogram of the corresponding predicted value of each sensitive attribute;

[0054] The client estimates the probability distribution of sensitive attributes and the conditional distribution function of the predicted value based on the overall frequency distribution histogram;

[0055] The client uses the probability distribution of sensitive attributes and the conditional quantile function of the predicted value to perform fairness post-processing on the regression model to generate a fair regression model, ensuring that the regression results are not unfairly affected by sensitive attributes.

[0056] Another object of the present invention is to provide a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the fair regression algorithm applicable to differential privacy federated learning.

[0057] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the fair regression algorithm based on differential privacy federated learning.

[0058] Another object of the present invention is to provide an information data processing terminal, which is used to implement the fair regression algorithm based on differential privacy federated learning.

[0059] The present invention provides a fair regression algorithm based on differential privacy federated learning. In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by the present invention are as follows:

[0060] Providing privacy protection: The global regression model fairness post-processing module of the present invention uses differential privacy technology based on discrete Gaussian mechanism to provide privacy protection by perturbing data. Experiments show that the present invention can significantly improve the fairness of the regression model through fairness post-processing while providing strong privacy protection, while the model still has high accuracy. In addition, the global regression model training module of the present invention can also use privacy protection technology in this module according to the privacy protection requirements to improve privacy protection for the training process.

[0061] High flexibility: The global regression model training module of the present invention does not restrict the specific regression model and the federated learning algorithm, and the global regression model fairness post-processing module does not restrict the distribution of regression prediction values, and has high flexibility.

[0062] Higher performance of the regression model: Compared with clients using their own local data sets to train regression models, the present invention uses a federated learning algorithm to train a global regression model. By aggregating model parameters from multiple clients, it can learn a wider range of data distribution characteristics, improve the generalization ability of the regression model, and effectively prevent problems such as overfitting and insufficient training that may be caused by the small local data set of the client.

[0063] The accuracy and applicability of fair post-processing are high: compared with the client using their own local data sets for fair post-processing, the present invention performs fair post-processing based on federated learning. By aggregating the frequency distribution of each client's data, on the one hand, the amount of data used to estimate the distribution is increased, which is conducive to improving the accuracy of distribution estimation; on the other hand, the applicability of fair post-processing is improved. Specifically, according to the formula for fair post-processing, when the client's local post-processing set lacks a certain sensitive attribute value, it is impossible to estimate the conditional distribution of the predicted value under the condition of the sensitive attribute value, so it is impossible to perform fair post-processing on the samples of the sensitive attribute value. Under the framework of federated learning, the conditional distribution of the predicted value is estimated using the data of all clients. As long as the local post-processing set of any client contains a certain sensitive attribute value, the samples of the sensitive attribute value can be fair post-processed.

[0064] The communication cost of fair post-processing is low: in the fair post-processing module of the present invention, the information exchanged between each client and the central server is only the frequency distribution histogram of the predicted value and its corresponding sensitive attribute value, and only one round of communication is required, so the communication cost is low.

[0065] The effect of fair post-processing is consistent with centralized learning: When privacy protection is not required, the effect of the fair post-processing module of the present invention is consistent with centralized learning. That is, given a regression model, the fair post-processing of the present invention (the variance of the discrete Gaussian distribution is set to 0) is equivalent to collecting the local post-processing sets of each client on the central server and then performing fair post-processing (still using histograms to estimate the distribution function and quantile function of the predicted value). BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of the present invention using the federated average algorithm to train a global regression model;

[0067] Figure 2 A schematic diagram of a process for determining the variance of a discrete Gaussian distribution according to differential privacy parameters in the present invention;

[0068] Figure 3 It is a flowchart of the fair post-processing of the regression model based on differential privacy federated learning in the present invention;

[0069] Figure 4 This is a flowchart of the entire process of fair regression based on differential privacy federated learning in the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0071] Group fairness: The sensitive attribute set is Call an algorithm is fair if it satisfies: for any have

[0072]

[0073] Group fairness requires that for any The Kolmogorov-Smirnov distance (KS distance for short) between the distributions of g given S=s and given S=s' is 0.

[0074] Federated learning: Federated learning is an advanced distributed machine learning method that allows multiple devices or computing nodes (usually called clients) to collaboratively train a global model without sharing local raw data. Federated learning does not require data to be stored centrally, but only requires the client to upload model parameters or update gradients to the central server for aggregation, which is conducive to protecting data privacy. However, the privacy protection of federated learning is not enough to completely resist external privacy attacks, and there is still a risk of privacy leakage. Differential privacy, homomorphic encryption, secure multi-party computing and other technologies can be used to provide stronger privacy protection.

[0075] Discrete Gaussian distribution: mean μ, variance σ 2 The discrete Gaussian distribution of Its probability at non-integer points is 0, and its probability at integer points is:

[0076]

[0077] Centralized Differential Privacy: Calling a Randomized Algorithm Satisfies ρ-centralized differential privacy if it satisfies: for any two adjacent data sets And for any α∈(1,∞), we have

[0078]

[0079] in represents the Rényi divergence of order α between the distributions P and Q.

[0080] Adjacent data sets: If two data sets D and D' differ by only one piece of data, that is, data set D can be obtained by deleting or adding one piece of data from another data set D', then D and D' are called adjacent data sets.

[0081] (ε,δ)-Differential Privacy: A randomized algorithm Satisfies (ε,δ)-differential privacy if it satisfies: for any two adjacent data sets and any have

[0082]

[0083] Among them, ε ≥ 0 is called the privacy budget, and δ ≥ 0 represents the upper limit of the probability of violating ε-differential privacy. In particular, when δ = 0, (ε, 0)-differential privacy is also called ε-differential privacy. In addition, the case of δ = 0 is called pure differential privacy, and the case of δ > 0 is called approximate differential privacy.

[0084] Differential privacy requires that for any two adjacent data sets, the output results obtained by the randomization algorithm should be probabilistically indistinguishable. The smaller the privacy budget ε, the stronger the privacy protection. In differential privacy, sensitivity is the key to determining the amount of noise added. The sensitivity of an algorithm refers to the maximum difference between its output results on adjacent data sets. The higher the sensitivity, the more noise needs to be added to ensure differential privacy.

[0085] Sensitivity: Mapping of Sensitivity is defined as:

[0086]

[0087] Among them, X, X' are adjacent data sets.

[0088] Post-processing invariance of (ε,δ)-differential privacy: randomized algorithms Satisfying (ε,δ)-differential privacy, let is any mapping, then Satisfies (ε,δ)-differential privacy.

[0089] Post-processing invariance of (ε,δ)-differential privacy for any deterministic or randomized mapping All are established, but require Only use The output result of is used as input without access to the original private data.

[0090] Parallel Combination Theorem for Pure Differential Privacy: Let X be a dataset, k be a positive integer, are mutually disjoint sub-datasets, that is, for any i≠j, X i is the input, satisfying ε i -Differentially private randomization algorithm, then the sequence The combination satisfies -Differential privacy.

[0091] In fact, in the above parallel combination theorem, the sequence Each element in can also take the output of the algorithm preceding it as input, that is, Can As input. It should be noted that the parallel combination theorem does not hold for approximate differential privacy. In order to obtain similar conclusions in the case of approximate differential privacy, the concept of f-differential privacy is introduced below.

[0092] f-differential privacy is an extension of (ε,δ)-differential privacy, and its definition is based on hypothesis testing. Specifically, the attacker aims to The output distinguishes two adjacent data sets X and X', that is, consider the hypothesis testing problem:

[0093] H0: The dataset is X vs H1: The dataset is X'

[0094] make Respectively Given a rejection rule φ for the above hypothesis test, the probability of making a Type I error is The probability of making a type II error is

[0095] Trade-off function: For any two probability distributions in the same space, define a trade-off function for:

[0096]

[0097] The infimum is taken to be all (measurable) rejection rules.

[0098] A function f:[0,1]→[0,1] is a trade-off function if and only if it is convex, continuous, non-increasing, and for any x∈[0,1], f(x)≤1-x.

[0099] Let f1,…,f k is a trade-off function, f1,…,f k The convex lower hull of k}, which is defined as:

[0100] lce{f1,…,f k}=sup{f(x)|f is a convex function and f≤min{f1,…,f k}}.

[0101] The trade-off function f1,…,f k The convex lower hull lce{f1,…,f k} is still a trade-off function. If f1,…,f k satisfy Then lce{f1,…,f k}=f1.

[0102] f-Differential Privacy: f is a trade-off function, called a randomization algorithm satisfies f-differential privacy if it satisfies: for any two adjacent data sets Both in Respectively The probability distribution of .

[0103] Parallel Combination Theorem of f-Differential Privacy: Let X be a dataset, k be a positive integer, are mutually disjoint sub-datasets, that is, for any i≠j, is to satisfy f i -Randomization algorithm for differential privacy, remember

[0104]

[0105] but Satisfies lce{f1,…,f k}-Differential privacy.

[0106] The relationship between f-differential privacy and (ε,δ)-differential privacy is: a randomized algorithm satisfies (ε,δ)-differential privacy if and only if it satisfies f ε,δ - Differential privacy, where

[0107] f ε,δ (α)=max{0,1-δ-e ε α,e -ε (1-δ-α)},α∈[0,1].

[0108] From the above, we can see that: Let X be a data set, k be a positive integer, are mutually disjoint sub-datasets, that is, for any i≠j, like are all randomized algorithms that satisfy (ε,δ)-differential privacy, then All satisfy f ε,δ - Differential privacy, where f ε,δ (α)=max{0,1-δ-e ε α,e -ε (1-δ-α)},α∈[0,1]. Furthermore, according to the parallel combination theorem of f-differential privacy, let:

[0109]

[0110] but Satisfy lce{f ε,δ ,…,f ε,δ}-Differential privacy, that is, f ε,δ -differential privacy, and thus also (ε,δ)-differential privacy. Thus, we get the parallel combination property of approximate differential privacy.

[0111] The present invention provides an algorithm for implementing fair regression using a post-processing method under a differential privacy federated learning framework, taking group equality as an evaluation index of algorithm fairness. The technical solution for implementing the present invention mainly includes three modules: global regression model training, determining the noise size according to the privacy protection strength, and global regression model fairness post-processing. The specific implementation method is as follows:

[0112] Let S represent the sensitive attribute, X represent the explanatory variables other than the sensitive attribute, and Y represent the response variable. The regression model considered is Assume there are N clients, and the local data set of the client is The local post-processing dataset is Where i = 1, 2, ..., N. It is worth mentioning that since the training regression model is supervised learning, the training data set needs to contain the values ​​of the response variable, but the post-processing data set only needs to contain the values ​​of the explanatory variables and does not need to contain the values ​​of the response variable.

[0113] In the present invention, the global regression model is trained using the federated average algorithm. Figure 1 , the algorithm steps include:

[0114]

[0115] It should be noted that the purpose of the global regression model training module of the present invention is to obtain a global regression model under the federated learning framework. There is no restriction on the specific method used. The federated average algorithm is used above. In practical applications, other federated learning algorithms such as FedProx, SCAFFOLD, and FedNova can also be used. In addition, the present invention focuses on solving the problem of post-processing the regression model to make it fair under the condition of privacy protection under the federated learning framework. Therefore, the algorithm 1 for training the global regression model above does not include additional privacy protection technology. In practical applications, the global regression model training part can also use privacy protection technologies such as differential privacy, homomorphic encryption, and secure multi-party computing to provide privacy protection. It should be pointed out that although differential privacy has post-processing invariance, even if differential privacy technology is used in the stage of training the global regression model, privacy protection technology must be used in the post-processing stage of the regression model fairness to provide privacy protection for this stage. This is because the post-processing invariance of differential privacy requires that the post-processing part does not use private data, and the client's original data will still be used to calculate the predicted value when the regression model is faired.

[0116] After the global regression model is trained, the present invention performs post-processing on it based on the following conclusion 1 to make it fair:

[0117] Conclusion 1 For regression model where f *is the regression function that minimizes the squared error, is a random variable with an expected value of 0. Then the optimal fair regression function is:

[0118]

[0119] in F f|s is the distribution function of f(X,S) given S=s, Q f|s It is the quantile function of f(X,S) given S=s.

[0120] From Conclusion 1, we can know that for the regression function f * To perform post-processing to make it fair, we need to know the distribution of sensitive attribute S and f given the sensitive attribute S. * The distribution function of (X, S) (and the corresponding conditional quantile function). In the framework of federated learning, histograms and differential privacy technology based on discrete Gaussian mechanisms can be used to estimate p under privacy protection conditions. s , F f|s and Q f|s Specifically, the client can calculate the one-hot encoding corresponding to each sample regression prediction value and sensitive attribute, add discrete Gaussian noise to it, and then calculate f * The local perturbation frequency distribution histogram of (X, S)|S=s and the local perturbation frequency distribution of the sensitive attribute S are uploaded to the central server for aggregation to obtain the overall frequency distribution, thereby estimating p s , F f|s and Q f|s .

[0121] To achieve privacy protection using differential privacy technology based on discrete Gaussian mechanism, it is necessary to determine the amount of noise added according to the privacy protection strength, that is, to determine the variance of discrete Gaussian distribution. The following gives the parameters ε and δ based on differential privacy to determine the variance σ of discrete Gaussian distribution. 2 Principles and methods:

[0122] Conclusion 2: If the randomized algorithm Satisfies ρ-centralized differential privacy (ρ ≥ 0), then for any ε ≥ 0, and

[0123]

[0124] Satisfies (ε,δ)-differential privacy.

[0125] From Conclusion 2, we can see that given the centralized differential privacy parameter ρ and the privacy budget ε, the parameter δ can be determined by solving the infimum. In fact, an efficient numerical solution is as follows:

[0126] Let g(α)=(α-1)(αρ-ε)+(α-1)·log(1-1 / α)-log(α), then Taking the derivative of g(α), we get:

[0127] g′(α)=(2α-1)ρ-ε+log(1-1 / α), When α∈(1,∞), g″(α)>0, so g′(α) is strictly monotonically increasing on α∈(1,∞).

[0128] In addition, g′(α) is a continuous function on α∈(1,∞) and satisfies:

[0129]

[0130] Therefore, g(α) is a smooth convex function on α∈(1,∞).

[0131] In addition, when ε>ρ>0, we have:

[0132]

[0133] Then, g(α) has a unique minimum point on α∈(1,∞) Therefore, to solve equation (1), we only need to determine So that g'(α*)=0.

[0134] The following are the algorithm steps for solving equation (2) using the bisection method given the differential privacy parameter ε and the concentrated differential privacy parameter ρ (ε>ρ>0):

[0135]

[0136] In addition, according to Conclusion 2, given the differential privacy parameters ε and δ, the corresponding centralized differential privacy parameters ρ can be inversely solved as follows:

[0137] Let h(ρ)=(α-1)(αρ-ε)+(α-1)·log(1-1 / α)-log(α), where α∈(1,∞).

[0138] The derivative is: h'(ρ) = (α-1)α>0, then h(ρ) is strictly monotonically increasing with respect to ρ, and thus δ is strictly monotonically increasing with respect to ρ.

[0139] The estimated value of the parameter ρ of centralized differential privacy can be obtained by using the bisection method Set the termination condition to: Substituting ε into formula (2), we can get The difference between the given value δ is less than the given precision. The algorithm steps are as follows:

[0140]

[0141]

[0142] Conclusion 3: A mapping is known That Sensitivity is Δ p , define a randomization algorithm as in when hour, Satisfy ρ - Centralized differential privacy, where:

[0143]

[0144] in

[0145] From formula (3), we can see that the centralized differential privacy parameter ρ increases with the variance σ of the discrete Gaussian distribution. 2 Assuming that mapping A and randomization algorithm As stated in Conclusion 3, if If ρ-centralized differential privacy is satisfied, the variance σ of the corresponding discrete Gaussian distribution can be approximated by equation (3): 2 , the algorithm steps are as follows:

[0146]

[0147] Based on the above analysis, we know a mapping That Sensitivity is Δ p , given the differential privacy parameters ε, δ, let a randomized algorithm be in To make Satisfies (ε,δ)-differential privacy, corresponding to discrete Gaussian distribution variance σ 2 The calculation method is as follows Figure 2 As shown, the steps include:

[0148]

[0149]

[0150] According to the attached Figure 3 The algorithm steps of the present invention for fair post-processing the global regression model are as follows:

[0151]

[0152]

[0153] In Algorithm 6 above, the differential privacy parameter δ is often taken as a very small number. A rule of thumb is to take in is the sum of the client post-processing dataset sizes.

[0154] Algorithm 6 Step S1 uses Algorithm 5 to determine the variance σ of the discrete Gaussian distribution 2 In the process, remember yes The set of standard bases of Selected Mapping Specifically, let the dataset where β ij is the regression prediction value Corresponding one-hot encoding. Divide X into k disjoint sub-datasets X1,…,X k , where X r (r=1,…,k) only contains sensitive attributes with values ​​of S r A sample, namely X r ={β ij :β ij ∈X and s ij =S r}. Let the mapping That

[0155] where r = 1,…,k, assuming X r With X r is an adjacent data set, then the mapping A r of The sensitivity is:

[0156]

[0157] The last equality is true because is a b-dimensional vector with one element being 1 and the rest being 0. In addition, according to the noise addition method of Algorithm 6, for r = 1, ..., k, A r The corresponding randomization algorithm is in Right now The amount of noise added is N·size.

[0158] If for r=1,…,k, They all satisfy (ε,δ)-differential privacy. From the post-processing invariance of differential privacy and the parallel combination property of approximate differential privacy, we can see that Algorithm 6 satisfies (ε,δ)-differential privacy.

[0159] In step S3 of Algorithm 6, if the local post-processing set of client i lacks a sensitive attribute value s lack For a sample of is a zero vector; but after the noise addition in step S4, is a nonzero vector.

[0160] In steps S4 and S5 of Algorithm 6, the information exchanged between the client and the central server should include the sensitive attribute values ​​corresponding to each histogram in addition to the histogram, because it is necessary to estimate the predicted conditional distribution under each sensitive attribute value. On the one hand, when the central server aggregates the histograms corresponding to each sensitive attribute, it needs the sensitive attribute values ​​corresponding to the histograms; on the other hand, the client When performing fair processing, it is also necessary to clarify the histogram corresponding to s.

[0161] In step S6 of Algorithm 6, the overall frequency distribution histogram of the predicted values ​​corresponding to the values ​​of each sensitive attribute is used Combined with the linear interpolation method, the distribution function of the predicted value under the condition of known sensitive attributes is estimated; further, the sum of the frequencies of all grouping intervals of the overall frequency distribution histogram of the predicted value corresponding to each sensitive attribute value is calculated. And take it as the corresponding sensitive attribute value S r The frequency distribution of sensitive attributes is estimated by the frequency distribution of sensitive attributes, and the probability distribution of sensitive attributes is estimated by the frequency distribution of sensitive attributes.

[0162] In step S7 of Algorithm 6, ξ~U([-v,v]) is a smooth random variable, and v is a very small number (e.g., 10 -6 ) to ensure that it does not change When estimating the conditional quantile function, the conditional quantile function corresponding to the probability p is: the interval division points l0, l1,…, l b The partition point closest to p in the corresponding conditional distribution function value.

[0163] The fairness post-processing process shown in Algorithm 6 has several advantages:

[0164] (1) Algorithm 6 uses histograms to estimate distribution functions and quantile functions. It does not depend on the distribution of specific regression prediction values ​​and has high flexibility.

[0165] (2) In Algorithm 6, the information exchanged between each client and the central server is only the k frequency distribution histograms (b-dimensional vectors) of the predicted values ​​and their corresponding sensitive attribute values, and only one round of communication is required, so the communication cost is relatively low.

[0166] (3) If Algorithm 6 does not add noise (i.e., the variance of the discrete Gaussian distribution is 0), the effect of fair post-processing using Algorithm 6 is consistent with centralized learning. In other words, given a regression model, using Algorithm 6 (the variance of the discrete Gaussian distribution is set to 0) for fair post-processing is equivalent to collecting the local post-processing sets of each client on the central server and then performing fair post-processing (still using histograms to estimate the distribution function and quantile function of the predicted value).

[0167] (4) Compared with the fairness post-processing performed by the clients using their own local data sets, Algorithm 6 performs fairness post-processing based on federated learning. By aggregating the local frequency distribution histograms of each client, on the one hand, the amount of data used to estimate the distribution is increased, which is conducive to improving the accuracy of distribution estimation; on the other hand, the applicability of fairness post-processing is improved. Specifically, according to the fairness post-processing formula (1), if the client's local post-processing set lacks a sensitive attribute value s lack If the sample is not large, it is impossible to estimate the conditional distribution of the predicted value under the value of the sensitive attribute. Therefore, it is impossible to set the sensitive attribute value to s in the test set. lack In the framework of federated learning, the conditional distribution of the predicted value is estimated using the data of all clients. As long as the local post-processing set of any client contains a sensitive attribute value, the samples of the sensitive attribute value can be post-processed fairly.

[0168] 1. Specific application fields or related products of the present invention

[0169] The embodiment of the present invention provides a fair regression algorithm based on differential privacy federated learning as described in claims 1-7, characterized in that the regression model considered is in the form of Where S represents the sensitive attribute, X represents the explanatory variable other than the sensitive attribute, and Y represents the response variable; the algorithm includes three modules:

[0170] Global regression model training module: The embodiment of the present invention uses a federated average algorithm to train a global regression model;

[0171] Global regression model fairness post-processing module: The embodiment of the present invention performs post-processing on the regression model f(x, s) to make it fair based on the following formula:

[0172]

[0173] in, F f|s is the distribution function of f(X,S) given S=s, Q f|sis the quantile function of f(X,S) given S=s, and g(x,s) is the fair regression model. The steps include:

[0174] S1: The central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters;

[0175] S2: The central server sends the regression model, discrete Gaussian distribution variance, interval division points, sensitive attribute set, noise addition amount, and scaling factor to the client;

[0176] S3: The client calculates the local frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute;

[0177] S4: The client adds a certain amount of discrete Gaussian noise to each local frequency distribution histogram, and uploads the disturbed local frequency distribution histogram and its corresponding sensitive attribute value to the central server;

[0178] S5: The central server performs aggregation operations to obtain the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute, and sends the overall frequency distribution histogram and its corresponding sensitive attribute values ​​to the client;

[0179] S6: client estimates the distribution of sensitive attributes and the conditional distribution function of the predicted value;

[0180] S7: Client-side estimated fair regression model.

[0181] Module for determining the noise size according to the privacy protection strength: Step S1 in the global regression model fairness post-processing module is a module for determining the noise size according to the privacy protection strength. The specific steps of calculating the variance of the discrete Gaussian distribution based on the given differential privacy parameters include:

[0182] S1: According to the following formula, the centralized differential privacy parameters ε and δ are approximated to solve

[0183]

[0184] S2: According to the following formula, the centralized differential privacy parameter Approximate solution to the variance of discrete Gaussian distribution

[0185]

[0186] Among them, Δ p For the algorithm sensitivity, d is the dimension of the added discrete Gaussian noise, n is the number of added discrete Gaussian noise,

[0187] An embodiment of the present invention provides a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the fair regression algorithm applicable to differential privacy federated learning.

[0188] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the fair regression algorithm based on differential privacy federated learning.

[0189] An embodiment of the present invention provides an information data processing terminal, which is used to implement the fair regression algorithm based on differential privacy federated learning.

[0190] II. Evidence related to the technical effects obtained by the embodiments of the present invention

[0191] Example 1: Law Dataset

[0192] The LAW dataset comes from the R package fairml and contains survey data on American law school students in 1991. The dataset size is 20,800 and contains 10 explanatory variables and 1 response variable. The goal of this dataset is to predict the GPA of students.

[0193] This example uses age as a protected attribute and constructs a three-category sensitive attribute s, where s=0 represents students aged 18 ≤ < 20, with a sample size of 17,493; s=1 represents students aged 20 ≤ < 22, with a sample size of 1,201; s=2 represents students of other age groups, with a sample size of 2,106. Next, the categorical variables in the data set are converted into dummy variables, and the continuous variables are normalized. Finally, the data set samples are described by 15 attributes, and the sample label is the student's GPA.

[0194] In this example, 20% of the data in the data set is randomly selected as the test set, and the test set size is 4160. The number of clients is set to 100, and the remaining data is randomly distributed to the clients using Dirichlet distribution (parameter set to 1). The minimum size of each client's local data set is 10 and the maximum size is 715. Each client divides the local data set into a training set and a post-processing set in a ratio of 6:4, which are used for training the regression model and post-processing of the regression model fairness, respectively.

[0195] Model and parameter settings: The regression models are selected as neural network (Network) and linear regression model (solved by recursive least squares (RLS)), where the loss function of the neural network is the mean square error function; in the post-processing process, the scaling factor is selected as c = 10, and the discretization method of the response variable value range [0,1] is divided into 64 intervals. If discrete Gaussian noise is added, the noise addition amount is set to size = 5, and each group of experiments is repeated 100 times. The same below.

[0196] In the federated learning framework, the experimental results obtained in this embodiment are as follows:

[0197] Table 1 Experimental results of federated learning on LAW dataset

[0198]

[0199] In Table 4, mse represents the mean square error on the test set, which is used to measure the accuracy of the regression model; ks represents the Kolmogorov-Smirnov distance (KS distance for short) on the test set, which is used to measure the fairness of the regression model; the values ​​outside the brackets are the means, and the values ​​inside the brackets are the standard deviations. The same applies below.

[0200] Fed represents the regression model obtained by federated learning training, that is, the result obtained by Algorithm 1; Fed_fair represents the fair post-processing of the regression model obtained by the Fed method, and the post-processing process does not consider privacy protection, that is, Algorithm 6 does not add noise; Fed_DP_fair represents the fair post-processing of the regression model obtained by the Fed method, and the differential privacy technology is used to protect data privacy in the post-processing process, that is, the result obtained by Algorithm 6, ε represents the privacy budget, and the differential privacy parameters δ are all set to where n i Indicates the size of the client's local post-processing set. Same below.

[0201] Comparing the results of Fed and Fed_fair, we can see that, under the federated learning framework, after the regression model is faired, the KS distance has dropped significantly, indicating that the fairness of the model has been greatly improved; at the same time, the mean square error has increased slightly, and the accuracy of the model has decreased slightly. Specifically, when the regression model is Network, compared with the results of Fed, the mean square error of Fed_fair increased by 0.0006, a relative increase of 6.39%, and the KS distance decreased by 0.5093, a relative decrease of 75.93%; when the regression model is RLS, compared with the results of Fed, the mean square error of Fed_fair increased by 0.0006, a relative increase of 7.24%, and the KS distance decreased by 0.6274, a relative decrease of 77.82%; overall, the increase in model fairness is much higher than the decrease in accuracy.

[0202] In this example, Fed_DP_fair sets the privacy budget to ε = 0.2, 0.5, 1.0, 1.5, and the differential privacy parameter is The corresponding added discrete Gaussian noise variances are 45.5453, 8.9044, 2.5600, and 1.2416. From the results of Fed_DP_fair, it can be seen that even if noise is added in the post-processing process to provide privacy protection, the KS distance still decreases significantly, while the mean square error increases slightly. In contrast, when ε=0.2, 0.5, 1.0, and 1.5, the KS distance and mean square error of Fed_DP_fair are higher than those of Fed_fair, and both increase as ε decreases. This is because the noise added by Fed_DP_fair in the post-processing process will cause the model performance to degrade, and the smaller ε is, the stronger the privacy protection provided, and the more noise is added, the greater the performance degradation of the model. Specifically, when the regression model is Network, when ε=0.2, 0.5, 1.0, 1.5, compared with the results of Fed, the relative increase of the mean square error of Fed_DP_fair is 40.49%, 21.31%, 12.90%, 10.41% respectively; the relative decrease of KS distance is 49.42%, 69.73%, 74.17%, 75.91% respectively; when the regression model is RLS, when ε=0.2, 0.5, 1.0, 1.5, compared with the results of Fed, The relative increases in the mean square error of Fed_DP_fair are 40.23%, 18.88%, 14.39%, and 11.56%, respectively; the relative decreases in the KS distance are 64.98%, 75.27%, 77.47%, and 77.72%, respectively. This shows that when the privacy budget is set to be greater than or equal to 0.2 (or even smaller), the increase in model fairness is higher than the decrease in accuracy, that is, the fair post-processing process can provide strong privacy protection while still ensuring high model accuracy and fairness.

[0203] To highlight the advantages of the present invention, this embodiment allows the clients to use their own local data sets for training and fairness post-processing, and then use the obtained fair regression model for the test set (size is 4160). The experiments on each client are repeated 100 times.

[0204] It should be noted that the clients use their own local post-processing sets for fair post-processing. According to formula (1), if the client's local post-processing set lacks a sensitive attribute value s lack , then it is impossible to estimate the conditional distribution of the predicted value under the sensitive attribute value. Therefore, it is impossible to set the sensitive attribute value to s in the test set. lack In this embodiment, a total of 25 clients cannot use the local post-processing set for post-processing, accounting for 25.00%. The results of the remaining 75 clients are as follows:

[0205] Table 2 Experimental results of non-federated learning on LAW dataset

[0206]

[0207] Table 3. Client local training and post-processing results of LAW dataset

[0208]

[0209] In Table 2, client_mean and client_median represent the mean and median of 75 clients after local training, respectively; client_fair_mean and client_fair_median represent the mean and median of 75 clients after local training and fairness post-processing, respectively, the same below.

[0210] Table 2 shows that compared with federated learning, the mean square error of the regression model obtained by local client training is larger, and after local fairness post-processing, the mean square error of the client increases relatively large, while the KS distance decreases relatively small. In fact, it can be seen from Table 3 that after local fairness post-processing, not all clients' KS distances are reduced. Specifically, when the regression model is Network, 85.00% of the clients' KS distances decrease, with an average decrease of 0.2126; 15.00% of the clients' KS distances increase, with an average increase of 0.1203; when the regression model is RLS, 64.00% of the clients' KS distances decrease, with an average decrease of 0.1956; 36.00% of the clients' KS distances increase, with an average increase of 0.1909.

[0211] Analysis of the experimental results on the client side: The local training effect of the client regression model is not good. The possible reason is that the local training data set of the client has a small amount of data, which is prone to overfitting, resulting in weak generalization ability of the trained regression model. In addition, for the RLS method, too little data will lead to insufficient iterations, resulting in insufficient model training. The local fairness post-processing effect of the client regression model is not good. On the one hand, it is affected by the error of the predicted value of the regression model; on the other hand, the local post-processing set of the client has a small amount of data, or there is a large gap between the distribution of the local post-processing set and the test set distribution, which makes the distribution function and quantile function estimation have large errors.

[0212] Example 2: CRIME dataset

[0213] The CRIME dataset comes from UCI and contains data on the age composition of the population in American communities. The dataset size is 1994, and contains 122 predictive attributes, 5 non-predictive attributes, and 1 target attribute. The goal of this dataset is to predict the incidence of social disorder (per 105 To prevent discrimination, the law requires that the predictions be fair and not treat different communities unfairly based on their age composition.

[0214] First, the dataset is preprocessed by deleting attributes with null values; and age is taken as a protected attribute. According to the proportion of the youth population (aged between 15 and 24), a binary sensitive attribute s is constructed, where s = 1 means that the majority of the community is youth population, the number of samples is 970, and the average incidence of social disorder is 0.35; s = 0 means that the majority of the community is non-youth population, the number of samples is 1024, and the average incidence of social disorder is 0.13. Finally, the dataset samples are described by 99 attributes, and the sample label is the incidence of social disorder.

[0215] In addition, since the code of the state to which each sample belongs is given in the non-predictive attributes of the dataset, when the dataset is used for federated learning, the number of clients and the client's local dataset can be determined based on the information of the state to which each sample belongs. That is, samples belonging to the same state constitute a client's local dataset. This dataset division method can better simulate real-life scenarios.

[0216] In this example, 20% of the data in the dataset is randomly selected as the test set, and the test set size is 399. The remaining data constitutes the local dataset of the client according to the state to which it belongs, and finally 46 clients are obtained (note that the number of clients may change with the setting of the random seed). It should be noted that during the global regression model training process, if the local training dataset of the client is too small, it may lead to insufficient model training and poor model performance. Therefore, the global regression model training process needs to limit the size of the client's local training dataset. In the fair post-processing process, the client calculates the perturbation frequency distribution and uploads it to the central server, and then adds and aggregates it. Overall, the estimated effect of the distribution function is only related to the sum of the sizes of the local post-processing sets of each client. The size of the client's local post-processing dataset will not affect the estimated effect of the distribution function. Therefore, the fair post-processing process has no requirements for the size of the client's local post-processing dataset.

[0217] Based on the above analysis, the clients are divided into two categories: if the size of the client's local dataset is above a given threshold (set to 10), the client divides the local dataset into a training set and a post-processing set in a ratio of 6:4, and participates in the global regression model training process and the fairness post-processing process; otherwise, the client only participates in the fairness post-processing process, and the local dataset is the post-processing set. Finally, 33 clients participated in the training of the global regression model, and 46 clients participated in the fairness post-processing of the regression model.

[0218] In the federated learning framework, the experimental results obtained in this embodiment are as follows:

[0219] Table 4. Experimental results of federated learning on CRIME dataset

[0220]

[0221] Comparing the results of Fed and Fed_fair, we can see that in the federated learning framework, after the regression model is faired, the KS distance decreases and the mean square error increases, indicating that the fairness of the model increases and the accuracy decreases. Specifically, when the regression model is Network, compared with the results of Fed, the mean square error of Fed_fair increases by 0.0125, a relative increase of 56.01%, and the KS distance decreases by 0.4280, a relative decrease of 83.99%; when the regression model is RLS, compared with the results of Fed, the mean square error of Fed_fair increases by 0.0187, a relative increase of 50.43%, and the KS distance decreases by 0.3604, a relative decrease of 77.69%; overall, the increase in model fairness is higher than the decrease in accuracy.

[0222] In this example, Fed_DP_fair sets the privacy budget to ε = 0.2, 0.5, 1.0, 1.5, and the differential privacy parameter The corresponding discrete Gaussian noise variances added are 62.7065, 12.9049, 3.8884, and 1.9337. From the results of Fed_DP_fair, we can see that when ε=0.2, 0.5, 1.0, and 1.5, the KS distance and mean square error of Fed_DP_fair are higher than those of Fed_fair, and both increase as ε decreases. This is because the noise added by Fed_DP_fair in the post-processing process will cause the model performance to degrade, and the smaller ε is, the stronger the privacy protection provided, and the more noise is added, the greater the performance degradation of the model. Specifically, when the regression model is Network, when ε=0.2, 0.5, 1.0, 1.5, compared with the results of Fed, the relative increase of the mean square error of Fed_DP_fair is 91.79%, 67.24%, 57.20%, 56.49% respectively; the relative decrease of KS distance is 46.88%, 69.20%, 76.08%, 79.88% respectively; when the regression model is RLS, when ε=0.2, 0.5, 1.0, 1.5, compared with the results of Fed, Fed_ The relative increases in the mean square error of DP_fair are 81.59%, 57.31%, 55.77%, and 53.47%, respectively; the relative decreases in the KS distance are 36.98%, 63.49%, 72.27%, and 75.48%, respectively. This shows that when the privacy budget is set to be greater than or equal to 0.5 (or even smaller), the increase in model fairness is greater than the decrease in accuracy, that is, the fairness post-processing process can improve the fairness of the regression model and ensure a higher model accuracy while providing stronger privacy protection.

[0223] In this embodiment, the clients use their own local data sets for training and fair post-processing, and then use the obtained fair regression model for the test set (size 399). The experiment on each client was repeated 100 times. In this embodiment, a total of 11 clients were unable to use the local post-processing set for post-processing, accounting for 33.33%. The results of the remaining 22 clients are as follows:

[0224] Table 5 Experimental results of non-federated learning on CRIME dataset

[0225]

[0226] Table 6 CRIME dataset client local training and post-processing results

[0227]

[0228] As can be seen from Table 5, compared with federated learning, the mean square error of the regression model obtained by local client training is larger, and after local fairness post-processing, the client's KS distance decreases slightly or even increases. In fact, as can be seen from Table 6, after local fairness post-processing, not all clients' KS distances are reduced. Specifically, when the regression model is Network, 68.18% of the clients' KS distances decrease, with an average decrease of 0.1127; 31.82% of the clients' KS distances increase, with an average increase of 0.4090; when the regression model is RLS, 45.45% of the clients' KS distances decrease, with an average decrease of 0.1652; 54.55% of the clients' KS distances increase, with an average increase of 0.1102.

[0229] The comparison of the experimental results of non-federated learning and federated learning in Example 1 and Example 2 highlights the effectiveness and superiority of the fair regression model based on differential privacy federated learning proposed in the present invention: First, the present invention trains a global regression model based on federated learning. By aggregating model parameters from multiple clients, it can learn a wider range of data distribution characteristics, improve the generalization ability of the regression model, and effectively prevent problems such as overfitting and insufficient training that may be caused by the small local data set of the client. Secondly, the present invention performs fair post-processing based on federated learning. By aggregating the frequency distribution of each client's data, on the one hand, it increases the amount of data used to estimate the distribution, which is conducive to improving the accuracy of distribution estimation; on the other hand, it improves the applicability of fair post-processing. Specifically, according to the formula of fair post-processing, when the local post-processing set of the client lacks a certain sensitive attribute value, it is impossible to estimate the conditional distribution of the predicted value under the condition of the sensitive attribute value, so it is impossible to perform fair post-processing on the sample of the sensitive attribute value. In the framework of federated learning, the conditional distribution of the predicted value is estimated using the data of all clients. As long as the local post-processing set of any client contains a certain sensitive attribute value, the sample of the sensitive attribute value can be fair post-processed.

[0230] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A fair regression system based on differential privacy federated learning, characterized in that: The system includes the following modules: Global regression model training module: trains global regression models under the federated learning framework, supports the use of FedAvg and FedProx federated learning algorithms; provides privacy protection for the training process by combining differential privacy, homomorphic encryption, and secure multi-party computing privacy protection technologies according to privacy protection requirements; Noise calculation module: Based on the given differential privacy parameters, the variance of the discrete Gaussian distribution is calculated through the formula; According to the privacy protection strength, determine the noise size and generate the noise factor; Fairness post-processing module: Using the global regression model, combined with the sensitive attribute distribution and the conditional distribution of the predicted value, the model is post-processed for fairness, including local frequency distribution calculation, discrete Gaussian noise addition, distribution estimation and regression model adjustment.

2. The fair regression system based on differential privacy federated learning according to claim 1, characterized in that: The specific steps of the noise calculation module include: The centralized differential privacy parameters are approximated by differential privacy parameters; The variance of the discrete Gaussian distribution is approximated using concentrated differential privacy parameters. Determine the number of grouping intervals and noise dimension of the histogram corresponding to the sensitive attribute, and generate a noise factor for each client based on the given discrete Gaussian noise addition amount, which is used to perturb the local frequency distribution histogram.

3. The fair regression system based on differential privacy federated learning according to claim 1, characterized in that: The fair post-processing module performs the following steps: The client calculates the local frequency distribution histogram of the predicted value corresponding to each sensitive attribute, adds discrete Gaussian noise and uploads it to the central server; The central server aggregates the local frequency distribution histogram after disturbance and generates the overall frequency distribution histogram of the corresponding predicted value of each sensitive attribute; The client estimates the probability distribution of sensitive attributes and the conditional distribution function of the predicted value based on the overall frequency distribution histogram; The client uses the probability distribution of sensitive attributes and the conditional quantile function of the predicted value to perform fairness post-processing on the regression model to generate a fair regression model, ensuring that the regression results are not unfairly affected by sensitive attributes.

4. A fair regression algorithm based on differential privacy federated learning, characterized in that: The regression model considered is of the form Where S represents the sensitive attribute, and X represents the explanatory variables other than the sensitive attribute. Y Represents the response variable; the algorithm consists of three modules: global regression model training, determining the noise size according to the privacy protection strength, and global regression model fairness post-processing.

5. The fair regression algorithm based on differential privacy federated learning according to claim 4 is characterized in that: The purpose of the global regression model training module is to obtain a global regression model under the federated learning framework. There is no restriction on the specific method used, and federated learning algorithms such as FedAvg and FedProx can be used. If there is a need for privacy protection, privacy protection technologies such as differential privacy, homomorphic encryption, and secure multi-party computing can be used in the global regression model training module to provide privacy protection for the training process.

6. The fair regression algorithm based on differential privacy federated learning according to claim 4, characterized in that: The global regression model fairness post-processing module post-processes the regression model f(x, s) to make it fair based on the following formula: in, F f|s is the distribution function of f(X,S) given S=s, Q f|s is the quantile function of f(X,S) given S=s, g(x,s) is the fair regression model; the global regression model fairness post-processing module includes the following steps: S1: The central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters; S2: The central server sends the regression model, discrete Gaussian distribution variance, interval division points, sensitive attribute set, noise addition amount, and scaling factor to the client; S3: The client calculates the local frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute; S4: The client adds a certain amount of discrete Gaussian noise to each local frequency distribution histogram, and uploads the disturbed local frequency distribution histogram and its corresponding sensitive attribute value to the central server; S5: The central server performs aggregation operations to obtain the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute, and sends the overall frequency distribution histogram and its corresponding sensitive attribute values ​​to the client; S6: client estimates the distribution of sensitive attributes and the conditional distribution function of the predicted value; S7: Client-side estimated fair regression model.

7. The fair regression algorithm based on differential privacy federated learning according to claim 4, characterized in that: Step S1 in the fair post-processing module of the global regression model is a module for determining the noise size according to the privacy protection strength. The specific steps of calculating the variance of the discrete Gaussian distribution based on the given differential privacy parameters include: S1: According to the following formula, the centralized differential privacy parameters ε and δ are approximated to solve S2: According to the following formula, the centralized differential privacy parameter Approximate solution to the variance of discrete Gaussian distribution Among them, Δ p For the algorithm sensitivity, d is the dimension of the added discrete Gaussian noise, n is the number of added discrete Gaussian noise, 8. The fair regression algorithm based on differential privacy federated learning according to claim 4, characterized in that: In the fair post-processing module of the global regression model, steps S3-S4 are the process of the client calculating the local disturbance frequency histogram of the predicted value. Discrete Gaussian noise is added to the predicted value frequency distribution histogram corresponding to each sensitive attribute value of the client. The dimension of the added discrete Gaussian noise is the number of grouping intervals of the histogram. In step S5, the central server aggregates and obtains the overall frequency distribution histogram of the predicted value corresponding to each sensitive attribute value. Therefore, the number of added discrete Gaussian noise is the product of the number of clients and the amount of noise added to each histogram. Furthermore, since the unique hot encoding of each sample predicted value has only one element that is 1 and the rest of the elements are 0, the algorithm Sensitivity is Δ p =c, where c is the scaling factor.

9. The fair regression algorithm based on differential privacy federated learning according to claim 4, characterized in that: In the global regression model fairness post-processing module, step S6 uses the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value, combined with the linear interpolation method, to estimate the distribution function of the predicted value under the condition of known sensitive attributes; further, the sum of the frequencies of all grouping intervals of the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value is calculated, and it is used as the frequency estimate of the corresponding sensitive attribute value, and then the probability distribution of the sensitive attribute is estimated from the frequency distribution of the sensitive attribute.

10. The fair regression algorithm based on differential privacy federated learning according to claim 4, characterized in that: In the global regression model fairness post-processing module, step S7 performs The post-fairness processing is as follows: Among them, ξ~U([-v,v]) is a smooth random variable, v is a very small number (such as 10 -6 ) to ensure that it does not change The statistical properties of is the sensitive attribute distribution estimate; is the conditional distribution function estimate of the predicted value; It is the conditional quantile function estimate of the predicted value. Specifically, the conditional quantile function corresponding to the probability p is: the histogram interval division points l0, l1, ..., l b The partition point closest to p in the corresponding conditional distribution function value.

Images