Fair regression model construction method and system based on differential privacy federated learning

Abstract

This invention belongs to the field of federated learning technology and provides a fair regression algorithm based on differentially private federated learning. The technical solution for implementing this invention primarily comprises three modules: first, a global regression model is trained using a federated learning algorithm; second, centralized differential privacy is introduced to determine the variance of a discrete Gaussian distribution based on given differential privacy parameters; finally, within the framework of federated learning, differential privacy techniques based on a discrete Gaussian mechanism are used to calculate the frequency distribution histogram, thereby estimating the distribution function and quantile function, thereby performing fair post-processing on the regression model. During this post-processing process, the only information exchanged between the client and the central server is the frequency distribution histogram of the regression prediction value and its corresponding sensitive attribute values, resulting in minimal communication overhead. Experiments have demonstrated that this invention significantly improves the fairness of the regression model while providing strong privacy protection and maintaining a high level of model accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning technology, and in particular relates to a method and system for constructing a fair regression model by post-processing a regression model using federated learning based on differential privacy. Background Art

[0002] In the context of big data, big-data-driven machine learning technologies are rapidly developing. Algorithmic fairness is a key goal of algorithmic decision-making. It requires that algorithms treat data fairly and without bias, avoiding discriminatory results. However, real-world training datasets often contain biases. In such cases, machine learning algorithms are likely to learn these biases from the data and produce discriminatory results. Group equality is a key metric for evaluating algorithmic fairness. Post-processing algorithm results is one method for achieving algorithmic fairness.

[0003] In the data-driven era, data privacy protection is gaining increasing attention. Federated learning is a distributed machine learning paradigm that allows clients to collaboratively train global models without sharing raw data, addressing the "data silo" problem. Federated learning does not require centralized data storage and inherently offers privacy-preserving properties. However, its level of privacy protection is insufficient to fully protect against external privacy attacks, and the risk of privacy breaches persists. Differential privacy is a data privacy protection technique that perturbs data by adding appropriate noise, preserving data privacy while maintaining a certain level of data availability. Applying differential privacy to federated learning can enhance the effectiveness of data privacy protection in federated learning.

[0004] At present, there is still a lack of technology based on the federated learning framework to achieve algorithmic fairness under privacy protection conditions. How to effectively train fair models while preventing privacy leaks needs to be urgently addressed. Summary of the Invention

[0005] This paper provides an algorithm for achieving fair regression using post-processing methods under the framework of differential privacy federated learning, taking group equality as the evaluation index of algorithm fairness. The regression model considered is Where S represents the sensitive attribute, X represents the explanatory variables other than the sensitive attribute, and Y represents the response variable.

[0006] The technical solution to implement this invention mainly includes three modules: global regression model training, noise level determination based on privacy protection strength, and global regression model fairness post-processing. The specific implementation method is as follows:

[0007] Global regression model training: The FedAvg algorithm is used to train the global regression model. The steps are as follows:

[0008] S1: The central server sends the initial regression model to the client to complete model initialization;

[0009] S2: The client uses the local training dataset to train the regression model and updates the local regression model.

[0010] S3: The client uploads its local regression model parameters and training dataset size to the central server;

[0011] S4: The central server aggregates model parameters in a weighted average manner based on the size of each client's training dataset and updates the global regression model;

[0012] S5: The central server sends the global regression model to the client;

[0013] S6: Repeat steps S2-S5 until model training is completed;

[0014] In the present invention, the purpose of the global regression model training part is to obtain a global regression model under the federated learning framework. There is no restriction on the specific method used. The federated averaging algorithm is used above. In practical applications, other federated learning algorithms such as FedProx, SCAFFOLD, and FedNova can also be used.

[0015] In addition, the present invention focuses on solving the problem of post-processing the regression model to make it fair under the condition of privacy protection in the federated learning framework. Therefore, the above-mentioned global regression model training part does not include additional privacy protection technology. In practical applications, the global regression model training part can also use privacy protection technologies such as differential privacy, homomorphic encryption, and secure multi-party computing to provide privacy protection. It should be pointed out that although differential privacy has post-processing invariance, even if differential privacy technology is used in the stage of training the global regression model, privacy protection technology must be used in the post-processing stage of regression model fairness to provide privacy protection for this stage. This is because the post-processing invariance of differential privacy requires that the post-processing part does not use private data, while the client's original data will still be used when the regression model is faired post-processed.

[0016] Global regression model fairness post-processing: The present invention performs post-processing on the regression model f(x, s) to make it fair based on the following formula:

[0017]

[0018] in, F f|s is the distribution function of f(X,S) given S=s, Q f|s is the quantile function of f(X,S) given S=s, and g(x,s) is the fair regression model.

[0019] As can be seen from the above formula, fair post-processing requires knowing the distribution function of the sensitive attribute and the distribution function of the predicted value under the given sensitive attribute (and the corresponding conditional quantile function). This paper uses histograms and differential privacy technology based on discrete Gaussian mechanisms to estimate the required distribution function under privacy protection conditions, thereby obtaining a fair regression model. The specific steps are as follows:

[0020] S1: The central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters;

[0021] S2: The central server sends the regression model, discrete Gaussian distribution variance, interval division points, sensitive attribute set, noise addition amount, and scaling factor to the client;

[0022] S3: The client calculates the local frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute;

[0023] S4: The client adds a certain amount of discrete Gaussian noise to each local frequency distribution histogram, and uploads the disturbed local frequency distribution histogram and its corresponding sensitive attribute value to the central server;

[0024] S5: The central server performs aggregation operations to obtain the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute, and sends the overall frequency distribution histogram and its corresponding sensitive attribute values ​​to the client;

[0025] S6: client-estimated distribution of sensitive attributes and conditional distribution function of predicted values;

[0026] S7: Client-side estimated fairness regression model.

[0027] In the global regression model fairness post-processing module, step S6 uses the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value, combined with the linear interpolation method, to estimate the distribution function of the predicted value under the condition of known sensitive attributes; further, the sum of the frequencies of all grouping intervals of the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value is calculated and used as the frequency estimate of the corresponding sensitive attribute value, and then the probability distribution of the sensitive attribute is estimated based on the frequency distribution of the sensitive attribute.

[0028] In the global regression model fairness post-processing module, step S7 is to The fair post-processing is as follows:

[0029]

[0030] Among them, ξ~U([-v,v]) is a smooth random variable, v is a very small number (such as 10 -6 ) to ensure that it does not change The statistical properties of is the sensitive attribute distribution estimate; is the conditional distribution function estimate of the predicted value; It is the conditional quantile function estimate of the predicted value. Specifically, the conditional quantile function corresponding to the probability p is: the histogram interval division points l0, l1, ..., l b The partition point closest to p in the corresponding conditional distribution function value.

[0031] Determine the noise level based on the privacy protection strength: In step S1 of the fair post-processing process, the central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters as follows:

[0032] S1: According to the following formula, the centralized differential privacy parameters ε and δ are approximated to solve the centralized differential privacy parameters

[0033]

[0034] S2: According to the following formula, the centralized differential privacy parameter Approximately solve the variance of discrete Gaussian distribution

[0035]

[0036] Among them, Δ p For the algorithm sensitivity, d is the dimension of the added discrete Gaussian noise, n is the number of added discrete Gaussian noise,

[0037] In step S1 of the module that determines the noise size according to the privacy protection strength, an explicit expression for calculating the differential privacy parameter δ from the centralized differential privacy parameter ρ and the privacy budget ε is given. Specifically, let

[0038] g(α)=(α-1)(αρ-ε)+(α-1)·log(1-1 / α)-log(α)

[0039] but It can be proved that g(α) is a smooth convex function on α∈(1,∞) and has a unique minimum point. Furthermore, when 0<ρ<ε, g(α) has a unique minimum point on α∈(1,∞) If the centralized differential privacy parameter ρ and privacy budget ε (0<ρ<ε) are given, the dichotomy method can be used to approximately solve the differential privacy parameter δ.

[0040] The centralized differential privacy parameter ρ is obtained from the differential privacy parameters ε and δ, and an inverse problem needs to be solved. Given ε, δ is strictly monotonically increasing with respect to ρ. Considering 0<ρ<ε, using the bisection method, if and ε calculated If the difference between the given δ and the given precision is less than the given precision, then This is the desired centralized differential privacy parameter.

[0041] In step S2 of the noise size module that determines the strength of privacy protection, since the fairness post-processing algorithm adds discrete Gaussian noise to the frequency distribution histogram of the client's predicted value, the dimension of the added discrete Gaussian noise is the number of grouping intervals of the histogram. Furthermore, the central server aggregates and obtains the overall frequency distribution histogram of the predicted value corresponding to each sensitive attribute value, so the amount of added discrete Gaussian noise is the product of the number of clients and the amount of noise added to each histogram. In addition, since the unique hot encoding of each sample prediction value has only one element that is 1 and the rest are 0, the algorithm Sensitivity is Δ p =c, where c is the scaling factor.

[0042] Step S2 gives the discrete Gaussian distribution variance σ 2 Calculating the explicit expression of the centralized differential privacy parameter ρ, it can be seen that the centralized differential privacy parameter ρ increases with the variance of the discrete Gaussian distribution σ 2 If the differential privacy parameter ρ is given, it can be reduced by gradually increasing Until Calculated differential privacy parameters Less than or equal to the given ρ, then This is the variance of the desired discrete Gaussian distribution.

[0043] The present invention also provides a fair regression system based on differential privacy federated learning, which includes the following modules:

[0044] Global regression model training module: This module trains global regression models within the federated learning framework, supporting federated learning algorithms such as FedAvg and FedProx. Based on privacy protection requirements, it can combine privacy protection technologies such as differential privacy, homomorphic encryption, and secure multi-party computation to provide privacy protection for the training process.

[0045] Noise calculation module: Based on the given differential privacy parameters, the variance of the discrete Gaussian distribution is calculated using a formula; based on the privacy protection strength, the noise level is determined and a noise factor is generated;

[0046] Fairness post-processing module: Utilizes the global regression model, combined with the distribution of sensitive attributes and the conditional distribution of predicted values, to perform fairness post-processing on the model, including local frequency distribution calculation, discrete Gaussian noise addition, distribution estimation, and regression model adjustment.

[0047] Furthermore, the specific steps of the noise calculation module include:

[0048] The centralized differential privacy parameters are approximated by differential privacy parameters;

[0049] The variance of the discrete Gaussian distribution is approximated using concentrated differential privacy parameters.

[0050] Determine the number of grouping intervals and noise dimension of the histogram corresponding to the sensitive attribute, and generate a noise factor for each client based on the given discrete Gaussian noise addition amount, which is used to perturb the local frequency distribution histogram.

[0051] Furthermore, the fairness post-processing module performs the following steps:

[0052] The client calculates the local frequency distribution histogram of the predicted value of each sensitive attribute, adds discrete Gaussian noise, and uploads it to the central server;

[0053] The central server aggregates the local frequency distribution histogram after disturbance and generates the overall frequency distribution histogram of the corresponding predicted value of each sensitive attribute;

[0054] The client estimates the probability distribution of sensitive attributes and the conditional distribution function of the predicted value based on the overall frequency distribution histogram;

[0055] The client uses the probability distribution of sensitive attributes and the conditional quantile function of the predicted value to perform fairness post-processing on the regression model to generate a fair regression model, ensuring that the regression results are not unfairly affected by sensitive attributes.

[0056] Another object of the present invention is to provide a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the fair regression algorithm applicable to differential privacy federated learning.

[0057] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to execute the steps of the fair regression algorithm based on differential privacy federated learning.

[0058] Another object of the present invention is to provide an information data processing terminal, which is used to implement the fair regression algorithm based on differential privacy federated learning.

[0059] The present invention provides a fair regression algorithm based on differentially private federated learning. In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by the present invention are as follows:

[0060] Providing Privacy Protection: The global regression model fairness post-processing module of the present invention utilizes differential privacy technology based on the discrete Gaussian mechanism to provide privacy protection by perturbing data. Experiments have shown that, while providing strong privacy protection, the fairness of the regression model can be significantly improved through fair post-processing, while maintaining high model accuracy. Furthermore, the global regression model training module of the present invention can also utilize privacy protection technology within the module, depending on privacy requirements, to enhance privacy protection during the training process.

[0061] High flexibility: The global regression model training module of the present invention does not restrict the specific regression model and federated learning algorithm, and the global regression model fairness post-processing module does not restrict the distribution of regression prediction values, and has high flexibility.

[0062] Higher regression model performance: Compared with clients using their own local data sets to train regression models, the present invention uses a federated learning algorithm to train a global regression model. By aggregating model parameters from multiple clients, it can learn a wider range of data distribution characteristics, improve the generalization ability of the regression model, and effectively prevent problems such as overfitting and insufficient training that may occur due to the small local data set of the client.

[0063] The accuracy and applicability of fair post-processing are high: Compared with the clients using their own local data sets to perform fair post-processing, the present invention performs fair post-processing based on federated learning. By aggregating the frequency distribution of each client's data, on the one hand, the amount of data used to estimate the distribution is increased, which is conducive to improving the accuracy of distribution estimation; on the other hand, the applicability of fair post-processing is improved. Specifically, according to the formula for fair post-processing, when the client's local post-processing set lacks a certain sensitive attribute value, it is impossible to estimate the conditional distribution of the predicted value under the condition of the sensitive attribute value, and therefore it is impossible to perform fair post-processing on the samples with the sensitive attribute value. Under the framework of federated learning, the conditional distribution of the predicted value is estimated using the data of all clients. As long as the local post-processing set of any client contains a certain sensitive attribute value, fair post-processing can be performed on the samples with the sensitive attribute value.

[0064] The communication cost of fair post-processing is low: in the fair post-processing module of the present invention, the information exchanged between each client and the central server is only the frequency distribution histogram of the predicted value and its corresponding sensitive attribute value, and only one round of communication is required, so the communication cost is low.

[0065] The effect of fair post-processing is consistent with centralized learning: When privacy protection is not required, the fair post-processing module of the present invention achieves the same effect as centralized learning. Specifically, given a regression model, the fair post-processing of the present invention (setting the variance of the discrete Gaussian distribution to 0) is equivalent to collecting the local post-processing data from each client on a central server and then performing fair post-processing (still using histograms to estimate the distribution function and quantile function of the predicted values). BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of the present invention's use of the federated averaging algorithm to train a global regression model;

[0067] Figure 2 This is a flow chart of determining the variance of a discrete Gaussian distribution based on differential privacy parameters in the present invention;

[0068] Figure 3 This is a flowchart of the fairness post-processing of the regression model based on differential privacy federated learning in the present invention;

[0069] Figure 4 This is a flowchart of the entire process of fair regression based on differentially private federated learning in the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0071] Group fairness: The sensitive attribute set is Call an algorithm is fair if it satisfies: for any have

[0072]

[0073] Group fairness requires that for any The Kolmogorov-Smirnov distance (KS distance for short) between the distributions of g given S=s and given S=s' is 0.

[0074] Federated Learning: Federated learning is an advanced distributed machine learning method that allows multiple devices or computing nodes (often referred to as clients) to collaboratively train a global model without sharing local raw data. Federated learning does not require centralized data storage; it only requires clients to upload model parameters or updated gradients to a central server for aggregation, thus helping to protect data privacy. However, the privacy protection provided by federated learning is insufficient to fully defend against external privacy attacks, and the risk of privacy leaks still exists. Technologies such as differential privacy, homomorphic encryption, and secure multi-party computation can be used to provide stronger privacy protection.

[0075] Discrete Gaussian distribution: mean μ, variance σ 2 The discrete Gaussian distribution of Its probability at non-integer points is 0, and its probability at integer points is:

[0076]

[0077] Centralized differential privacy: a randomized algorithm Satisfies ρ-centralized differential privacy if it satisfies: for any two adjacent data sets And for any α∈(1,∞), we have

[0078]

[0079] in represents the Rényi divergence of order α between distributions P and Q.

[0080] Adjacent datasets: If two datasets D and D' differ by only one piece of data, that is, dataset D can be obtained by deleting or adding a piece of data from another dataset D', then D and D' are called adjacent datasets.

[0081] (ε,δ)-Differential Privacy: A randomized algorithm Satisfies (ε,δ)-differential privacy if it satisfies: for any two adjacent data sets and any have

[0082]

[0083] Here, ε ≥ 0 is called the privacy budget, and δ ≥ 0 represents the upper bound on the probability of violating ε-differential privacy. In particular, when δ = 0, (ε, 0)-differential privacy is also called ε-differential privacy. Furthermore, the case where δ = 0 is called pure differential privacy, and the case where δ > 0 is called approximate differential privacy.

[0084] Differential privacy requires that for any two adjacent datasets, the outputs obtained by a randomized algorithm should be probabilistically indistinguishable. A smaller privacy budget, ε, indicates stronger privacy protection. In differential privacy, sensitivity is key to determining the amount of noise to add. The sensitivity of an algorithm refers to the maximum difference between its outputs on adjacent datasets. The higher the sensitivity, the more noise needs to be added to ensure differential privacy.

[0085] Sensitivity: Mapping of Sensitivity is defined as:

[0086]

[0087] Among them, X, X' are adjacent data sets.

[0088] Post-processing invariance of (ε,δ)-differential privacy: randomized algorithms Satisfying (ε,δ)-differential privacy, let is any mapping, then Satisfies (ε,δ)-differential privacy.

[0089] Post-processing invariance of (ε,δ)-differential privacy for any deterministic or randomized mapping Both are established, but require Only use The output result of is used as input without access to the original private data.

[0090] Parallel Combination Theorem for Pure Differential Privacy: Let X be a dataset, k be a positive integer, are mutually disjoint sub-datasets, that is, for any i≠j, So X i is the input, satisfying ε i -Randomization algorithm for differential privacy, then the sequence The combination of -Differential privacy.

[0091] In fact, in the above parallel combination theorem, the sequence Each element in can also take the output of the algorithm preceding it as input, i.e. You can As input. It should be noted that the parallel combination theorem does not hold for approximate differential privacy. To obtain similar conclusions in the case of approximate differential privacy, the concept of f-differential privacy is introduced below.

[0092] f-differential privacy is an extension of (ε,δ)-differential privacy, and its definition is based on hypothesis testing. Specifically, the attacker aims to The output distinguishes two adjacent data sets X and X', that is, consider the hypothesis testing problem:

[0093] H0: The dataset is X vs H1: The dataset is X'

[0094] make Respectively Given a rejection rule φ for the hypothesis test above, the probability of making a Type I error is The probability of making a Type II error is

[0095] Trade-off function: For any two probability distributions in the same space, define a trade-off function for:

[0096]

[0097] The infimum is taken to be all (measurable) rejection rules.

[0098] A function f:[0,1]→[0,1] is a trade-off function if and only if it is convex, continuous, non-increasing, and f(x) ≤ 1-x for any x∈[0,1].

[0099] Let f1,…,f k is a trade-off function, f1,…,f k The convex lower hull is denoted as lce{f1,…,f k}, which is defined as:

[0100] lce{f1,…,f k}=sup{f(x)|f is a convex function and f≤min{f1,…,f k}}.

[0101] Trade-off functions f1,…,f k The convex lower hull lce{f1,…,f k} is still a trade-off function. If f1,…,f k satisfy Then lce{f1,…,f k}=f1.

[0102] f-differential privacy: f is a trade-off function, called a randomized algorithm satisfies f-differential privacy if it satisfies: for any two adjacent data sets Both in Respectively The probability distribution of .

[0103] Parallel Combination Theorem of f-Differential Privacy: Let X be a dataset, k be a positive integer, are mutually disjoint sub-datasets, that is, for any i≠j, is to satisfy f i -Randomization algorithm for differential privacy, remember

[0104]

[0105] but Satisfy lce{f1,…,f k}-Differential privacy.

[0106] The relationship between f-differential privacy and (ε,δ)-differential privacy is: a randomized algorithm satisfies (ε,δ)-differential privacy if and only if it satisfies f ε,δ - Differential privacy, where

[0107] f ε,δ (α)=max{0,1-δ-e ε α,e -ε (1-δ-α)},α∈[0,1].

[0108] From the above, we can see that: Let X be a data set, k be a positive integer, are mutually disjoint sub-datasets, that is, for any i≠j, like are all randomized algorithms that satisfy (ε,δ)-differential privacy, then All satisfy f ε,δ - Differential privacy, where f ε,δ (α)=max{0,1-δ-e ε α,e -ε (1-δ-α)},α∈[0,1]. Furthermore, according to the parallel combination theorem of f-differential privacy, let:

[0109]

[0110] but Satisfy lce{f ε,δ ,…,f ε,δ}-Differential privacy, that is, f ε,δ -differential privacy, thus also satisfying (ε,δ)-differential privacy. Thus, we obtain the parallel combination property of approximate differential privacy.

[0111] This invention provides an algorithm for achieving fair regression using post-processing methods within a differentially private federated learning framework, using group equality as an evaluation metric for algorithmic fairness. The technical solution for implementing this invention primarily comprises three modules: global regression model training, noise level determination based on privacy protection strength, and global regression model fairness post-processing. The specific implementation is as follows:

[0112] Let S represent the sensitive attribute, X represent the explanatory variables other than the sensitive attribute, and Y represent the response variable. The regression model considered is Assume there are N clients and the local dataset of the client is The local post-processing dataset is Where i = 1, 2, ..., N. It is worth mentioning that since training the regression model is supervised learning, the training dataset needs to contain the values ​​of the response variable, but the post-processing dataset only needs to contain the values ​​of the explanatory variables and does not need to contain the values ​​of the response variable.

[0113] In the present invention, the global regression model is trained using the federated average algorithm. Figure 1 , the algorithm steps include:

[0114]

[0115] It should be noted that the purpose of the global regression model training module of the present invention is to obtain a global regression model within a federated learning framework. The specific method used is not limited. While the federated averaging algorithm is used above, other federated learning algorithms such as FedProx, SCAFFOLD, and FedNova can also be used in practical applications. Furthermore, the present invention focuses on solving the problem of post-processing the regression model to make it fair while preserving privacy within a federated learning framework. Therefore, Algorithm 1 for training the global regression model does not include additional privacy-preserving techniques. In practical applications, the global regression model training phase can also utilize privacy-preserving techniques such as differential privacy, homomorphic encryption, and secure multi-party computation to provide privacy protection. It is important to note that although differential privacy is post-processing invariant, even if differential privacy is utilized during the global regression model training phase, additional privacy-preserving techniques must be employed during the post-processing phase to ensure privacy protection for the regression model. This is because the post-processing invariance of differential privacy requires that the post-processing phase does not utilize private data, while the client's original data is still used to calculate the predicted value during the fair post-processing of the regression model.

[0116] After the global regression model is trained, the present invention performs post-processing on it based on the following conclusion 1 to make it fair:

[0117] Conclusion 1 For the regression model where f *is the regression function that minimizes the squared error, is a random variable with an expected value of 0. Then the optimal fair regression function is:

[0118]

[0119] in F f|s is the distribution function of f(X,S) given S=s, Q f|s It is the quantile function of f(X,S) given S=s.

[0120] From Conclusion 1, we can see that for the regression function f * To perform post-processing to make it fair, we need to know the distribution of sensitive attribute S and the condition f given sensitive attribute S. * The distribution function of (X, S) (and the corresponding conditional quantile function). In the framework of federated learning, histograms and differential privacy technology based on discrete Gaussian mechanisms can be used to estimate p under privacy protection conditions. s , F f|s and Q f|s Specifically, the client can calculate the one-hot encoding corresponding to each sample regression prediction value and sensitive attribute, add discrete Gaussian noise to it, and then calculate f * The local perturbation frequency distribution histogram of (X, S)|S=s and the local perturbation frequency distribution of sensitive attribute S are uploaded to the central server for aggregation to obtain the overall frequency distribution, thereby estimating p s , F f|s and Q f|s .

[0121] To achieve privacy protection using differential privacy technology based on discrete Gaussian mechanism, it is necessary to determine the amount of noise added according to the strength of privacy protection, that is, to determine the variance of discrete Gaussian distribution. The following gives the parameters ε and δ based on differential privacy to determine the variance σ of discrete Gaussian distribution. 2 Principles and methods:

[0122] Conclusion 2: If the randomized algorithm Satisfying ρ-centralized differential privacy (ρ ≥ 0), then for any ε ≥ 0, and

[0123]

[0124] Satisfies (ε,δ)-differential privacy.

[0125] From Conclusion 2, we can see that given the centralized differential privacy parameter ρ and the privacy budget ε, the parameter δ can be determined by solving the infimum. In fact, an efficient numerical solution is as follows:

[0126] Let g(α)=(α-1)(αρ-ε)+(α-1)·log(1-1 / α)-log(α), then Taking the derivative of g(α), we get:

[0127] g′(α)=(2α-1)ρ-ε+log(1-1 / α), When α∈(1,∞), g″(α)>0, so g′(α) is strictly monotonically increasing on α∈(1,∞).

[0128] In addition, g′(α) is a continuous function on α∈(1,∞) and satisfies:

[0129]

[0130] Therefore, g(α) is a smooth convex function on α∈(1,∞).

[0131] In addition, when ε>ρ>0, we have:

[0132]

[0133] Then, g(α) has a unique minimum point on α∈(1,∞) Therefore, to solve equation (1), we only need to determine Make g'(α*)=0.

[0134] The following are the algorithm steps for solving Equation (2) using the bisection method given the differential privacy parameter ε and the centralized differential privacy parameter ρ (ε>ρ>0):

[0135]

[0136] In addition, according to Conclusion 2, given the differential privacy parameters ε and δ, the corresponding centralized differential privacy parameters ρ can be inversely solved as follows:

[0137] Let h(ρ)=(α-1)(αρ-ε)+(α-1)·log(1-1 / α)-log(α), where α∈(1,∞).

[0138] The derivative is: h'(ρ) = (α-1)α>0, then h(ρ) is strictly monotonically increasing with respect to ρ, and thus δ is strictly monotonically increasing with respect to ρ.

[0139] The estimated value of the parameter ρ of centralized differential privacy can be obtained using the bisection method Set the termination condition to: Substituting ε into formula (2), we can get The difference between the given value δ and the given precision is less than the given accuracy. The algorithm steps are as follows:

[0140]

[0141]

[0142] Conclusion 3: A mapping is known That Sensitivity is Δ p , define a randomization algorithm as in when hour, Satisfy ρ - Centralized differential privacy, where:

[0143]

[0144] in

[0145] From formula (3), we can see that the centralized differential privacy parameter ρ increases with the variance σ of the discrete Gaussian distribution. 2 Assume that mapping A and randomization algorithm As stated in Conclusion 3, if If ρ-centralized differential privacy is satisfied, the variance σ of the corresponding discrete Gaussian distribution can be approximated by formula (3): 2 , the algorithm steps are as follows:

[0146]

[0147] Based on the above analysis, we know a mapping That Sensitivity is Δ p , given the differential privacy parameters ε, δ, let a randomized algorithm be in To make Satisfies (ε,δ)-differential privacy, corresponding to discrete Gaussian distribution variance σ 2 The calculation method is as follows Figure 2 As shown, the steps include:

[0148]

[0149]

[0150] According to the attached Figure 3 The algorithm steps for the fairness post-processing of the global regression model in the present invention are as follows:

[0151]

[0152]

[0153] In the above algorithm 6, the parameter δ of differential privacy is often taken as a very small number. A rule of thumb is to take in is the sum of the client post-processing dataset sizes.

[0154] Algorithm 6 Step S1 uses Algorithm 5 to determine the variance σ of the discrete Gaussian distribution 2 In the process, remember yes The set of standard bases of Selected Mapping Specifically, let the dataset where β ij is the regression prediction value Corresponding one-hot encoding. Divide X into k mutually disjoint sub-datasets X1,…,X k , where X r (r=1,…,k) only contains sensitive attributes with the value of S r The sample, that is, X r ={β ij :β ij ∈X and s ij =S r}. Let the mapping That

[0155] where r=1,…,k, assuming X r With X r are adjacent data sets, then mapping A r of The sensitivity is:

[0156]

[0157] The last equal sign is true because is a b-dimensional vector with one element being 1 and the rest being 0. In addition, according to the noise addition method of Algorithm 6, for r=1,…,k, A r The corresponding randomization algorithm is in Right now The amount of noise added is N·size.

[0158] If for r=1,…,k, Both satisfy (ε, δ)-differential privacy. From the aforementioned post-processing invariance of differential privacy and the parallel combination property of approximate differential privacy, it can be seen that Algorithm 6 satisfies (ε, δ)-differential privacy.

[0159] In step S3 of Algorithm 6, if the local post-processing set of client i lacks a sensitive attribute value s lack For a sample of is a zero vector; but after the noise addition in step S4, is a nonzero vector.

[0160] In steps S4 and S5 of Algorithm 6, the information exchanged between the client and the central server should include not only the histogram but also the sensitive attribute value corresponding to each histogram. This is because it is necessary to estimate the predicted conditional distribution under each sensitive attribute value. On the one hand, when the central server aggregates the histograms corresponding to each sensitive attribute, it needs the sensitive attribute value corresponding to the histogram; on the other hand, the client When performing fair processing, it is also necessary to clarify the histogram corresponding to s.

[0161] In step S6 of algorithm 6, the overall frequency distribution histogram of the predicted values ​​corresponding to the values ​​of each sensitive attribute is used Combined with the linear interpolation method, the distribution function of the predicted value under the condition of known sensitive attributes is estimated; further, the sum of the frequencies of all grouping intervals of the overall frequency distribution histogram of the predicted value corresponding to each sensitive attribute value is calculated. And take it as the corresponding sensitive attribute value S r The frequency distribution of sensitive attributes is estimated by the frequency distribution of sensitive attributes, and the probability distribution of sensitive attributes is estimated by the frequency distribution of sensitive attributes.

[0162] In step S7 of Algorithm 6, ξ~U([-v,v]) is a smooth random variable, and v is a very small number (such as 10 -6 ) to ensure that it does not change When estimating the conditional quantile function, the conditional quantile function corresponding to the probability p is: the interval division points l0, l1, ..., l b The partition point closest to p in the corresponding conditional distribution function value.

[0163] The fairness post-processing process shown in Algorithm 6 has several advantages:

[0164] (1) Algorithm 6 uses histograms to estimate the distribution function and quantile function, which does not depend on the distribution of specific regression prediction values ​​and has high flexibility.

[0165] (2) In Algorithm 6, the information exchanged between each client and the central server is only the k frequency distribution histograms (b-dimensional vectors) of the predicted values ​​and their corresponding sensitive attribute values, and only one round of communication is required, so the communication cost is relatively low.

[0166] (3) If Algorithm 6 does not add noise (i.e., the variance of the discrete Gaussian distribution is 0), then the effect of fair post-processing using Algorithm 6 is consistent with centralized learning. In other words, given a regression model, using Algorithm 6 (with the variance of the discrete Gaussian distribution set to 0) for fair post-processing is equivalent to collecting the local post-processing sets of each client on the central server and then performing fair post-processing (still using histograms to estimate the distribution function and quantile function of the predicted value).

[0167] (4) Compared with the fairness post-processing performed by the clients using their own local data sets, Algorithm 6 performs fairness post-processing based on federated learning. By aggregating the local frequency distribution histograms of each client, on the one hand, the amount of data used to estimate the distribution is increased, which is conducive to improving the accuracy of distribution estimation; on the other hand, the applicability of fairness post-processing is improved. Specifically, according to the fairness post-processing formula (1), if the client's local post-processing set lacks a sensitive attribute value s lack If the sample is not large, it is impossible to estimate the conditional distribution of the predicted value under the condition of the sensitive attribute value. Therefore, it is impossible to set the value of sensitive attributes in the test set to s lack In the framework of federated learning, the conditional distribution of the predicted value is estimated using the data of all clients. As long as the local post-processing set of any client contains a sensitive attribute value, fair post-processing can be performed on samples with that sensitive attribute value.

[0168] 1. Specific application fields or related products of the present invention

[0169] The embodiment of the present invention provides a fair regression algorithm based on differential privacy federated learning as described in claims 1-7, characterized in that the regression model considered is Where S represents the sensitive attribute, X represents the explanatory variable other than the sensitive attribute, and Y represents the response variable. The algorithm consists of three modules:

[0170] Global regression model training module: This embodiment of the present invention uses a federated averaging algorithm to train a global regression model;

[0171] Global regression model fairness post-processing module: This embodiment of the present invention performs post-processing on the regression model f(x, s) to make it fair based on the following formula:

[0172]

[0173] in, F f|s is the distribution function of f(X,S) given S=s, Q f|sis the quantile function of f(X,S) given S=s, and g(x,s) is the fair regression model. The steps include:

[0174] S1: The central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters;

[0175] S2: The central server sends the regression model, discrete Gaussian distribution variance, interval division points, sensitive attribute set, noise addition amount, and scaling factor to the client;

[0176] S3: The client calculates the local frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute;

[0177] S4: The client adds a certain amount of discrete Gaussian noise to each local frequency distribution histogram, and uploads the disturbed local frequency distribution histogram and its corresponding sensitive attribute value to the central server;

[0178] S5: The central server performs aggregation operations to obtain the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute, and sends the overall frequency distribution histogram and its corresponding sensitive attribute values ​​to the client;

[0179] S6: client-estimated distribution of sensitive attributes and conditional distribution function of predicted values;

[0180] S7: Client-side estimated fairness regression model.

[0181] Module for determining noise size based on privacy protection strength: Step S1 in the global regression model fairness post-processing module is a module for determining noise size based on privacy protection strength. The specific steps for calculating the variance of the discrete Gaussian distribution based on given differential privacy parameters include:

[0182] S1: According to the following formula, the centralized differential privacy parameters ε and δ are approximated to solve the centralized differential privacy parameters

[0183]

[0184] S2: According to the following formula, the centralized differential privacy parameter Approximately solve the variance of discrete Gaussian distribution

[0185]

[0186] Among them, Δ p For the algorithm sensitivity, d is the dimension of the added discrete Gaussian noise, n is the number of added discrete Gaussian noise,

[0187] An embodiment of the present invention provides a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the fair regression algorithm applicable to differential privacy federated learning.

[0188] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the fair regression algorithm based on differential privacy federated learning.

[0189] An embodiment of the present invention provides an information data processing terminal, which is used to implement the fair regression algorithm based on differential privacy federated learning.

[0190] II. Evidence Related to the Technical Effects Obtained by the Embodiments of the Present Invention

[0191] Example 1: Law Dataset

[0192] The LAW dataset comes from the R package fairml and contains survey data on American law school students in 1991. The dataset size is 20,800 and contains 10 explanatory variables and 1 response variable. The goal of this dataset is to predict students' GPA.

[0193] This example uses age as a protected attribute and constructs a three-category sensitive attribute s. Here, s = 0 represents students aged 18 ≤ < 20, with a sample size of 17,493; s = 1 represents students aged 20 ≤ < 22, with a sample size of 1,201; and s = 2 represents students in other age groups, with a sample size of 2,106. Next, categorical variables in the dataset are converted into dummy variables, and continuous variables are normalized. Finally, the dataset samples are described by 15 attributes, with the sample label being the student's GPA.

[0194] In this example, 20% of the data set is randomly selected as the test set, resulting in a test set size of 4160. The number of clients is set to 100, and the remaining data is randomly distributed to the clients using a Dirichlet distribution (with a parameter set to 1). The minimum size of each client's local dataset is 10 and the maximum size is 715. Each client divides the local dataset into a training set and a post-processing set in a 6:4 ratio, which are used for training the regression model and for post-processing the regression model to achieve fairness.

[0195] Model and parameter settings: A neural network and a linear regression model (solved using recursive least squares (RLS)) were selected as the regression model. The loss function for the neural network was the mean squared error function. During post-processing, a scaling factor of c = 10 was selected. The response variable range [0, 1] was discretized into 64 equal intervals. If discrete Gaussian noise was added, the noise addition amount was set to size = 5. Each experiment was repeated 100 times. The same applies below.

[0196] In the federated learning framework, the experimental results obtained in this embodiment are as follows:

[0197] Table 1. Experimental results of federated learning on the LAW dataset

[0198]

[0199] In Table 4, mse represents the mean squared error on the test set, which is used to measure the accuracy of the regression model; ks represents the Kolmogorov-Smirnov distance (KS distance) on the test set, which is used to measure the fairness of the regression model; the values ​​outside the brackets are the means, and the values ​​inside the brackets are the standard deviations. The same applies below.

[0200] Fed represents the regression model obtained by federated learning training, that is, the result obtained by Algorithm 1; Fed_fair represents the fair post-processing of the regression model obtained by the Fed method, and the post-processing process does not consider privacy protection, that is, Algorithm 6 does not add noise; Fed_DP_fair represents the fair post-processing of the regression model obtained by the Fed method, and the differential privacy technology is used to protect data privacy in the post-processing process, that is, the result obtained by Algorithm 6, ε represents the privacy budget, and the differential privacy parameter δ is set to where n i Indicates the size of the client's local post-processing set. Same below.

[0201] Comparing the results of Fed and Fed_fair, we can see that, under the federated learning framework, after fairness post-processing of the regression model, the KS distance decreases significantly, indicating a significant improvement in model fairness. At the same time, the mean squared error increases slightly, while the model accuracy decreases slightly. Specifically, when the regression model is Network, compared to the Fed result, the mean squared error of Fed_fair increases by 0.0006, a relative increase of 6.39%, and the KS distance decreases by 0.5093, a relative decrease of 75.93%. When the regression model is RLS, compared to the Fed result, the mean squared error of Fed_fair increases by 0.0006, a relative increase of 7.24%, and the KS distance decreases by 0.6274, a relative decrease of 77.82%. Overall, the improvement in model fairness is much greater than the decrease in accuracy.

[0202] In this example, Fed_DP_fair sets the privacy budget to ε = 0.2, 0.5, 1.0, 1.5, and the differential privacy parameter is The corresponding discrete Gaussian noise variances added are 45.5453, 8.9044, 2.5600, and 1.2416. The results for Fed_DP_fair show that even with the addition of noise during post-processing to provide privacy protection, the KS distance still decreases significantly, while the mean squared error increases slightly. In contrast, when ε = 0.2, 0.5, 1.0, and 1.5, the KS distance and mean squared error of Fed_DP_fair are both higher than those of Fed_fair, and both increase as ε decreases. This is because the noise added during post-processing by Fed_DP_fair leads to a decrease in model performance. A smaller ε provides stronger privacy protection, while the more noise added, the greater the performance degradation. Specifically, when the regression model is Network, when ε=0.2, 0.5, 1.0, and 1.5, the mean square error of Fed_DP_fair increases by 40.49%, 21.31%, 12.90%, and 10.41% respectively compared with the results of Fed; the relative decreases in KS distance are 49.42%, 69.73%, 74.17%, and 75.91% respectively; when the regression model is RLS, when ε=0.2, 0.5, 1.0, and 1.5, compared with the results of Fed, The relative increases in mean squared error of Fed_DP_fair are 40.23%, 18.88%, 14.39%, and 11.56%, respectively; the relative decreases in KS distance are 64.98%, 75.27%, 77.47%, and 77.72%, respectively. This shows that when the privacy budget is set to be greater than or equal to 0.2 (or even smaller), the increase in model fairness is greater than the decrease in accuracy. In other words, the fair post-processing process can provide strong privacy protection while still ensuring high model accuracy and fairness.

[0203] To highlight the advantages of the present invention, this embodiment allows the clients to use their own local datasets for training and fairness post-processing, and then use the obtained fair regression model on the test set (size 4160). The experiment on each client is repeated 100 times.

[0204] It should be noted that the clients use their own local post-processing sets to perform fair post-processing. According to formula (1), if the client's local post-processing set lacks a sensitive attribute value s lack , then it is impossible to estimate the conditional distribution of the predicted value under the condition of the sensitive attribute value Therefore, it is impossible to set the value of sensitive attributes in the test set to s lack In this example, 25 clients were unable to use the local post-processing set for post-processing, accounting for 25.00%. The results for the remaining 75 clients are as follows:

[0205] Table 2 Experimental results of non-federated learning on the LAW dataset

[0206]

[0207] Table 3. Client local training and post-processing results of LAW dataset

[0208]

[0209] In Table 2, client_mean and client_median represent the mean and median of the 75 clients after local training, respectively. client_fair_mean and client_fair_median represent the mean and median of the 75 clients after local training and fairness post-processing, respectively. The same applies below.

[0210] Table 2 shows that compared to federated learning, the regression model trained locally on the client achieves a higher mean squared error (MSE). Furthermore, after local fairness post-processing, the client's MSE increases significantly, while the KS distance decreases slightly. In fact, Table 3 shows that not all clients experience a reduction in KS distance after local fairness post-processing. Specifically, when the regression model is Network, 85.00% of clients experience a decrease in KS distance, with an average decrease of 0.2126; 15.00% of clients experience an increase in KS distance, with an average increase of 0.1203; when the regression model is RLS, 64.00% of clients experience a decrease in KS distance, with an average decrease of 0.1956; and 36.00% of clients experience an increase in KS distance, with an average increase of 0.1909.

[0211] Analysis of the client-side experimental results revealed that the local training of the client-side regression model was not effective. This may be due to the small size of the local training dataset, which is prone to overfitting and results in weak generalization of the trained regression model. Furthermore, for the RLS method, too little data leads to insufficient iterations, resulting in an inadequately trained model. The local fairness post-processing of the client-side regression model was not effective, partly due to errors in the regression model's predicted values. Furthermore, the small size of the local post-processing dataset, or the significant disparity between the distribution of the local post-processing dataset and the test dataset, resulted in significant errors in the estimated distribution and quantile functions.

[0212] Example 2: CRIME dataset

[0213] The CRIME dataset comes from UCI and contains data on the age composition of the population in American communities. The dataset size is 1994, and contains 122 predictive attributes, 5 non-predictive attributes, and 1 target attribute. The goal of this dataset is to predict the incidence of social disorder (per 105 To prevent discrimination, the law requires that the predictions be fair and not treat different communities unfairly based on their age composition.

[0214] First, the dataset was preprocessed by removing attributes with null values. Age was added as a protected attribute, and a binary sensitive attribute s was constructed based on the proportion of youth (aged between 15 and 24). Where s = 1 indicates a community with a majority of youth, with a sample size of 970 and an average incidence of social disorder of 0.35; s = 0 indicates a community with a majority of non-youth, with a sample size of 1024 and an average incidence of social disorder of 0.13. Finally, the dataset samples were described by 99 attributes, with the sample label being the incidence of social disorder.

[0215] In addition, since the non-predictive attributes of the dataset provide the code of the state to which each sample belongs, when using this dataset for federated learning, the number of clients and the client's local dataset can be determined based on the information of the state to which each sample belongs. That is, samples belonging to the same state constitute the local dataset of a client. This dataset division method can better simulate real-world scenarios.

[0216] In this example, 20% of the data in the dataset is randomly selected as the test set, resulting in a test set size of 399. The remaining data constitutes the client's local dataset based on the state to which it belongs, and finally 46 clients are obtained (note that the number of clients may vary with the setting of the random seed). It should be noted that during the global regression model training process, if the client's local training dataset is too small, it may lead to insufficient model training and poor model performance. Therefore, the global regression model training process needs to limit the size of the client's local training dataset. During the fair post-processing process, the client calculates the perturbation frequency distribution and uploads it to the central server, and then performs addition aggregation. Overall, the estimated effect of the distribution function is only related to the sum of the sizes of the local post-processing sets of each client. The size of the client's local post-processing dataset will not affect the estimated effect of the distribution function. Therefore, the fair post-processing process has no requirements for the size of the client's local post-processing dataset.

[0217] Based on the above analysis, clients were divided into two categories: if the size of a client's local dataset was greater than a given threshold (set to 10), the client would split its local dataset into a training set and a post-processing set in a 6:4 ratio and participate in both global regression model training and fairness post-processing. Otherwise, the client would only participate in fairness post-processing, with its local dataset serving as the post-processing set. Finally, 33 clients participated in global regression model training, and 46 clients participated in both regression model fairness post-processing.

[0218] In the federated learning framework, the experimental results obtained in this embodiment are as follows:

[0219] Table 4. Experimental results of federated learning on the CRIME dataset

[0220]

[0221] Comparing the results of Fed and Fed_fair, we can see that, in the federated learning framework, after fairness post-processing of the regression model, the KS distance decreases and the mean squared error increases, indicating that the model's fairness improves while its accuracy decreases. Specifically, when the regression model is Network, compared to the Fed result, the mean squared error of Fed_fair increases by 0.0125, a relative increase of 56.01%, and the KS distance decreases by 0.4280, a relative decrease of 83.99%. When the regression model is RLS, compared to the Fed result, the mean squared error of Fed_fair increases by 0.0187, a relative increase of 50.43%, and the KS distance decreases by 0.3604, a relative decrease of 77.69%. Overall, the improvement in model fairness is greater than the decrease in accuracy.

[0222] In this example, Fed_DP_fair sets the privacy budget to ε = 0.2, 0.5, 1.0, 1.5, and the differential privacy parameter The corresponding added discrete Gaussian noise variances are 62.7065, 12.9049, 3.8884, and 1.9337. The results for Fed_DP_fair show that when ε = 0.2, 0.5, 1.0, and 1.5, the KS distance and mean squared error of Fed_DP_fair are all higher than those of Fed_fair, and both increase as ε decreases. This is because the noise added by Fed_DP_fair during post-processing can lead to a decline in model performance. Furthermore, smaller ε provides stronger privacy protection, while the more noise added, the greater the decline in model performance. Specifically, when the regression model is Network, when ε = 0.2, 0.5, 1.0, 1.5, compared with the results of Fed, the relative increase in the mean square error of Fed_DP_fair is 91.79%, 67.24%, 57.20%, 56.49% respectively; the relative decrease in KS distance is 46.88%, 69.20%, 76.08%, 79.88% respectively; when the regression model is RLS, when ε = 0.2, 0.5, 1.0, 1.5, compared with the results of Fed, Fed_ The relative increases in the mean squared error of DP_fair are 81.59%, 57.31%, 55.77%, and 53.47%, respectively; the relative decreases in the KS distance are 36.98%, 63.49%, 72.27%, and 75.48%, respectively. This shows that when the privacy budget is set to be greater than or equal to 0.5 (or even smaller), the increase in model fairness is greater than the decrease in accuracy. That is, the fair post-processing process can improve the fairness of the regression model while providing strong privacy protection and ensuring high model accuracy.

[0223] In this example, the clients were trained and fair post-processed using their own local datasets, and the resulting fair regression model was then applied to a test set (size 399). The experiment was repeated 100 times on each client. In this example, 11 clients were unable to perform post-processing using their local post-processed datasets, accounting for 33.33%. The results for the remaining 22 clients are as follows:

[0224] Table 5 Experimental results of non-federated learning on the CRIME dataset

[0225]

[0226] Table 6 CRIME dataset client local training and post-processing results

[0227]

[0228] Table 5 shows that compared to federated learning, the regression model trained locally on the client achieves a higher mean squared error. Furthermore, after local fairness post-processing, the client's KS distance decreases slightly or even increases. In fact, Table 6 shows that not all clients' KS distances decrease after local fairness post-processing. Specifically, when the regression model is Network, the KS distance decreases for 68.18% of clients, with an average decrease of 0.1127; the KS distance increases for 31.82% of clients, with an average increase of 0.4090; when the regression model is RLS, the KS distance decreases for 45.45% of clients, with an average decrease of 0.1652; and the KS distance increases for 54.55% of clients, with an average increase of 0.1102.

[0229] Comparing the experimental results of non-federated learning and federated learning in Examples 1 and 2 highlights the effectiveness and superiority of the fair regression model based on differentially private federated learning proposed in this invention. First, the present invention trains a global regression model based on federated learning. By aggregating model parameters from multiple clients, it can learn a wider range of data distribution characteristics, improve the generalization ability of the regression model, and effectively prevent problems such as overfitting and insufficient training that may occur due to small local client datasets. Second, the present invention performs fair post-processing based on federated learning. By aggregating the frequency distribution of each client's data, it not only increases the amount of data used to estimate the distribution, which is conducive to improving the accuracy of the distribution estimation, but also improves the applicability of fair post-processing. Specifically, according to the fair post-processing formula, when a client's local post-processing set lacks a certain sensitive attribute value, it is impossible to estimate the conditional distribution of the predicted value under the sensitive attribute value, and therefore, fair post-processing cannot be performed on samples with this sensitive attribute value. In the federated learning framework, however, the conditional distribution of the predicted value is estimated using data from all clients. As long as any client's local post-processing set contains a certain sensitive attribute value, fair post-processing can be performed on samples with this sensitive attribute value.

[0230] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A fair regression system based on differentially private federated learning, characterized by: The system includes the following modules: Global regression model training module: This module trains global regression models under the federated learning framework, supports the use of FedAvg and FedProx federated learning algorithms, and combines differential privacy technology to provide privacy protection for the training process. Noise calculation module: Based on the given differential privacy parameters, the variance of the discrete Gaussian distribution is calculated using the formula; According to the privacy protection strength, the noise size is determined and the noise factor is generated; Fairness post-processing module: This module uses a global regression model, combined with the distribution of sensitive attributes and the conditional distribution of predicted values, to perform fairness post-processing on the model, including local frequency distribution calculation, discrete Gaussian noise addition, distribution estimation, and regression model adjustment. The specific steps of the noise calculation module include: The centralized differential privacy parameters are approximated by differential privacy parameters; The variance of the discrete Gaussian distribution is approximated using concentrated differential privacy parameters. Determine the number of grouping intervals and noise dimension of the histogram corresponding to the sensitive attribute, and generate a noise factor for each client based on the given discrete Gaussian noise addition amount, which is used to perturb the local frequency distribution histogram; The fair post-processing module performs the following steps: The client calculates the local frequency distribution histogram of the predicted value of each sensitive attribute, adds discrete Gaussian noise, and uploads it to the central server; The central server aggregates the local frequency distribution histogram after disturbance and generates the overall frequency distribution histogram of the corresponding predicted value of each sensitive attribute; The client estimates the probability distribution of sensitive attributes and the conditional distribution function of the predicted value based on the overall frequency distribution histogram; The client uses the probability distribution of sensitive attributes and the conditional quantile function of the predicted value to perform fairness post-processing on the regression model to generate a fair regression model, ensuring that the regression results are not unfairly affected by sensitive attributes; For regression models The fair post-processing is as follows: Among them, ξ~U([-v,v]) is a smooth random variable, v is 10 -6 , to ensure that it does not change The statistical properties of is the sensitive attribute distribution estimate; is the conditional distribution function estimate of the predicted value; is the conditional quantile function estimate of the predicted value.

2. A fair regression algorithm based on differentially private federated learning, characterized by: The algorithm consists of three modules: global regression model training, noise level determination based on privacy protection strength, and global regression model fairness post-processing. The global regression model training module aims to obtain a global regression model under the federated learning framework. It supports the use of FedAvg and FedProx federated learning algorithms. In the global regression model training module, differential privacy technology is used to provide privacy protection for the training process. The global regression model fairness post-processing module includes the following steps: S1: The central server calculates the variance of the discrete Gaussian distribution based on the given differential privacy parameters; S2: The central server sends the regression model, discrete Gaussian distribution variance, interval division points, sensitive attribute set, noise addition amount and scaling factor to the client; S3: The client calculates the local frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute; S4: The client adds a certain amount of discrete Gaussian noise to each local frequency distribution histogram, and uploads the disturbed local frequency distribution histogram and its corresponding sensitive attribute value to the central server; S5: The central server performs aggregation operations to obtain the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute, and sends the overall frequency distribution histogram and its corresponding sensitive attribute values ​​to the client; S6: client-estimated distribution of sensitive attributes and conditional distribution function of predicted values; S7: Client-side estimated fairness regression model; For regression models The fair post-processing is as follows: Among them, ξ~U([-v,v]) is a smooth random variable, v is 10 -6 , to ensure that it does not change The statistical properties of is the sensitive attribute distribution estimate; is the conditional distribution function estimate of the predicted value; is the conditional quantile function estimate of the predicted value.

3. The fair regression algorithm based on differential privacy federated learning according to claim 2 is characterized in that Step S1 in the global regression model fairness post-processing module determines the noise level based on the privacy protection strength. The specific steps for calculating the variance of the discrete Gaussian distribution using given differential privacy parameters include: S1: According to the following formula, the centralized differential privacy parameters ε and δ are approximated to solve the centralized differential privacy parameters : Consider 0<ρ<ε, use the bisection method, if and ε calculated If the difference between the given δ and the given precision is less than the given precision, then This is the desired centralized differential privacy parameter; S2: According to the following formula, the centralized differential privacy parameter Approximately solve the variance of discrete Gaussian distribution : Given a differential privacy parameter ρ, by gradually increasing Until Calculated differential privacy parameters Less than or equal to the given ρ, then That is the variance of the discrete Gaussian distribution; Among them, Δ p is the algorithm's l p sensitivity, d is the dimension of the added discrete Gaussian noise, n is the number of added discrete Gaussian noise, 4. The fair regression algorithm based on differentially private federated learning according to claim 2, characterized in that: In the global regression model fairness post-processing module, steps S3-S4 are the process of the client calculating the local perturbation frequency histogram of the predicted value. Discrete Gaussian noise is added to the predicted value frequency distribution histogram corresponding to each sensitive attribute value of the client. The dimension of the added discrete Gaussian noise is the number of grouping intervals of the histogram. In step S5, the central server aggregates and obtains the overall frequency distribution histogram of the predicted value corresponding to each sensitive attribute value. Therefore, the amount of added discrete Gaussian noise is the product of the number of clients and the amount of noise added to each histogram. Since the unique hot encoding of each sample predicted value has only one element that is 1 and the rest are 0, the algorithm l p Sensitivity is Δ p =c, where c is the scaling factor.

5. The fair regression algorithm based on differential privacy federated learning according to claim 2 is characterized in that In the global regression model fairness post-processing module, step S6 uses the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value, combined with the linear interpolation method, to estimate the distribution function of the predicted value under the condition of known sensitive attributes; calculates the sum of the frequencies of all grouping intervals of the overall frequency distribution histogram of the predicted values ​​corresponding to each sensitive attribute value, and uses it as the frequency estimate of the corresponding sensitive attribute value, and then estimates the probability distribution of the sensitive attribute from the frequency distribution of the sensitive attribute.

Images