A longitudinal federated logistic regression algorithm

Abstract

This invention relates to the field of information security technology, specifically disclosing a vertical federated logistic regression algorithm. In this invention, the host generates the public and private keys of party A, and generates a random vector r. A After decryption and noise removal, the Guest side generates its own gradient term and generates the B side's public and private keys. A random vector r is then generated. B After decryption and noise removal, a gradient term is generated and added to the model. The host obtains the gradient, and the guest obtains the gradient. The guest's public key is used to encrypt the gradient and send it to the host. The host obtains the gradient and calculates the Newton direction using the Hessian matrix training protocol. Finally, the parameters are updated by calculating the Newton direction. In this way, the guest and host only obtain their respective gradients and Newton directions, and do not obtain other valid data. This reduces the probability of data leakage to a certain extent and improves the security of model training.

Description

Technical Field

[0001] This invention belongs to the field of information security technology, specifically relating to a vertical federated logistic regression algorithm. Background Technology

[0002] Federated learning is a distributed machine learning technique. Its core idea is to train models in a distributed manner across multiple data sources that have local data. Without exchanging local individual or sample data, it constructs a global model based on virtual fused data by exchanging model parameters or intermediate results, thereby achieving a balance between data privacy protection and data sharing computation.

[0003] Logistic regression is a widely used machine learning model. In longitudinal federated learning, in order to train a federated logistic regression model, we need to protect the original data. This data is stored by each party and is prohibited from being transmitted to each other. It is also expected that this data will not be learned by others from any intermediate data transmitted during model training.

[0004] The main steps in the Federated Logistic Regression training process are as follows:

[0005] Step 1: Create public and private keys, and send the public key to the Host;

[0006] Step 2, Initialization ,calculate ;

[0007] Step 3, Calculation , , ,send Give to Host;

[0008] Step 4, Calculation , ;

[0009] Step 5, Decrypt ,send Give to Host;

[0010] Step 6, Update .

[0011] However, when training a model on a dataset using this step, steps 2 through 6 are typically iterated multiple times until the maximum number of iterations is reached or certain convergence conditions are met. In steps 2, 3, and 4, when the number of samples in the batch is less than the number of features, the encrypted data can be derived, potentially leading to data leakage during intermediate data exchange and posing a security risk. Therefore, we need to propose a longitudinal federated logistic regression algorithm to address these issues. Summary of the Invention

[0012] The purpose of this invention is to provide a longitudinal federated logistic regression algorithm. In this way, the Guest and Host sides only receive their respective gradients and Newton directions, and do not receive any other valid data. This reduces the probability of data leakage to a certain extent, thereby improving the security of model training and solving the problems mentioned in the background art.

[0013] To achieve the above objectives, the present invention adopts the following technical solution:

[0014] A longitudinal federated logistic regression algorithm includes the following steps:

[0015] Host:

[0016] A1. Generate A's public and private keys;

[0017] A2, Calculate [[1 / 4] X A W A ]] A ;

[0018] A3, Calculation ▽ W A ]] B =X A T [[1 / 4 X B W B -0.5 y B ]] B ;

[0019] A4. Generate a random vector r A ,calculate[[ ▽ W A ]] B +[[r A ]] B ;

[0020] A5, decrypted ▽ W B +r B ;

[0021] A6. Obtained by eliminating noise ▽ W B Add its own gradient term;

[0022] Guest side:

[0023] B1. Generate B's public and private keys;

[0024] B2, Calculate [[1 / 4] X BW B -0.5 y B ]] B ;

[0025] B3. Calculation ▽ W B ]] A =X B T [[1 / 4 X A W A ]] A ;

[0026] B4. Generate a random vector r B ,calculate[[ ▽ W B ]] A +[[r B ]] A ;

[0027] B5, decrypted ▽ W A +r A ;

[0028] B6. Noise removal results in ▽ W A Add its own gradient term;

[0029] In step A2, the Host calculates [[1 / 4] X A W A ]] A The value is transferred to Guest, and in step B2, Guest calculates [[1 / 4]]. X B W B -0.5 y B ]] B The value is transferred to the Host;

[0030] In step A4, the Host calculates [[ ▽ W A ]] B +[[r A ]] B The value is transferred to Guest, and the [[ calculated by Guest in step B4]] ▽ W B ]] A +[[r B ]] A The value is transferred to the Host;

[0031] The Host calculation in step A5▽ W B +r B The value is transferred to the Guest, and the Guest calculates it in step B5. ▽ W A +r A The value is transferred to the Host.

[0032] Preferably, the Host is set as Party A, the Guest is set as Party B, and the raw data of Party A includes sample data X. A Weight vector W A The original data from Party B includes sample data x. B y B Weight vector W B .

[0033] Preferably, before calculation, given that the exponential operation in logistic regression is performed under encrypted operation, the exponential part is expanded:

[0034]

[0035] ,

[0036] in, The items that require assistance in calculation are: , The items that require assistance in calculation are: .

[0037] Preferably, the Guest creates a public key and a private key, sends the public key to the Host, and the Guest calculates... And send it to the Host, the Host calculates... Send it to the Guest, who decrypts it to obtain the predicted label.

[0038] The preferred option also undergoes second-order optimization, with the specific steps as follows:

[0039] S1, Training the Host to obtain Guest received Encrypt using Guest's public key Send it to the Host, and the Host receives the gradient. ;

[0040] S2, Calculate using the Hessian matrix training protocol ;

[0041] S3. Calculate the Newton direction and update the parameters.

[0042] Preferably, the training protocol for calculating the Hessian matrix is... The specific training process is as follows:

[0043] 3.1 The Guest creates a public key and a private key, and sends the public key to the Host;

[0044] 3.2 Host Calculation , , Guest calculation ,send , Give to Host;

[0045] 3.3 Host Calculation ,get .

[0046] Preferably, the specific steps for updating the parameters in step S3 are as follows:

[0047] 4.1 The Guest and Host generate public and private keys. The Guest sends the public key to the Host, and they jointly compute... And send it to the Host;

[0048] 4.2. Add an invertible noise matrix to the Host. and noise vector Calculate using Guest's public key and And send it to the Guest;

[0049] 4.3 Guest Decryption and To obtain noisy and noise vector Inverse calculus yields ,calculate After encryption, it is sent to the Host;

[0050] 4.4, Host obtains Noise matrix is ​​eliminated using homomorphic scalar multiplication. and noise vector Specifically To obtain Newton's direction Noise is added to its own Newton direction and then given to Guest;

[0051] 4.5. The Guest decrypts its own Newton direction and updates its own parameters, then sends back the Newton direction of the noisy Host.

[0052] 4.6. The Host removes noise to obtain its own Newton direction and updates its own parameters.

[0053] Preferably, in step 4.1, the Host uses the Guest's public key to encrypt... get ,but , That is .

[0054] Preferably, step 4.3 requires... Finding the inverse, that is, first finding the generated ones. The noise matrix is ​​an invertible matrix.

[0055] The longitudinal federated logistic regression algorithm proposed in this invention has the following advantages compared with existing technologies:

[0056] 1. This invention mainly involves the Host generating the public and private keys of Party A, and generating a random vector r. A ,calculate[[ ▽ W A ]] B +[[r A ]] B After decryption, noise is eliminated and generated. ▽ W B By adding its own gradient term, the Guest side generates the public and private keys of the B side, and generates a random vector r. B ,calculate[[ ▽ W B ]] A +[[r B ]] A After decryption, noise is eliminated and generated. ▽ W A Add its own gradient term and train the Host to obtain Guest received Encrypt using Guest's public key Send it to the Host, and calculate it using the Hessian matrix training protocol. Finally, the parameters are updated. This method ensures that the Guest and Host sides only receive their respective gradients and Newton directions, without any other valid data. This reduces the chance of data leakage to some extent and improves the security of model training. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the calculation of Host and Guest in this invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] This invention provides, for example Figure 1 The longitudinal federated logistic regression algorithm shown includes the following steps:

[0060] Host:

[0061] A1. Generate A's public and private keys;

[0062] A2, Calculate [[1 / 4] X A W A ]] A ;

[0063] A3, Calculation ▽ W A ]] B =X A T [[1 / 4 X B W B -0.5 y B ]] B ;

[0064] A4. Generate a random vector r A ,calculate[[ ▽ W A ]] B +[[r A ]] B ;

[0065] A5, decrypted ▽ W B +r B ;

[0066] A6. Obtained by eliminating noise ▽ W B Add its own gradient term;

[0067] Guest side:

[0068] B1. Generate B's public and private keys;

[0069] B2, Calculate [[1 / 4] XB W B -0.5 y B ]] B ;

[0070] B3. Calculation ▽ W B ]] A =X B T [[1 / 4 X A W A ]] A ;

[0071] B4. Generate a random vector r B ,calculate[[ ▽ W B ]] A +[[r B ]] A ;

[0072] B5, decrypted ▽ W A +r A ;

[0073] B6. Noise removal results in ▽ W A Add its own gradient term;

[0074] In step A2, the Host calculates [[1 / 4] X A W A ]] A The value is transferred to Guest, and in step B2, Guest calculates [[1 / 4]]. X B W B -0.5 y B ]] B The value is transferred to the Host; the value calculated by the Host in step A4 is... ▽ W A ]] B +[[r A ]] B The value is transferred to Guest, and the [[ calculated by Guest in step B4]] ▽ W B ]] A +[[r B ]] A The value is transferred to the Host; the value calculated by the Host in step A5. ▽ W B +r BThe value is transferred to the Guest, and the Guest calculates it in step B5. ▽ W A +r A The value is transferred to the Host.

[0075] The above method can be used to perform first-order optimization of the federated logistic regression algorithm, increasing its communication volume and computational complexity in subsequent comparisons, without requiring a trusted third party.

[0076] Based on the first-order optimization described above, this application also provides a second-order optimization, the specific steps of which are as follows:

[0077] S1, Training the Host to obtain Guest received Encrypt using Guest's public key Send it to the Host, and the Host receives the gradient. ;

[0078] S2, Calculate using the Hessian matrix training protocol The specific training process is as follows:

[0079] 3.1 The Guest creates a public key and a private key, and sends the public key to the Host;

[0080] 3.2 Host Calculation , , ,calculate ,get ;

[0081] S3. Calculate the Newton direction and update the parameters. The specific steps are as follows:

[0082] 4.1 The Guest and Host generate public and private keys. The Guest sends the public key to the Host, and they jointly compute... And send it to the Host; the Host encrypts it using the Guest's public key. get ,but , That is

[0083] 4.2. Add an invertible noise matrix to the Host. and noise vector Calculate using Guest's public key and And send it to the Guest;

[0084] 4.3 Guest Decryption and To obtain noisy and noise vector Inverse calculus yields ,calculate After encryption, it is sent to the Host; among other things, it is necessary to... To find the inverse, we must ensure that it is reversible. First, we need to require that the generated [property] is [invertible]. The noise matrix is ​​invertible. Secondly, it is necessary to ensure... It is reversible, and the specific method is as follows: Since the logistic regression loss function is a convex function, then Therefore, the Levenberg-Marquardt algorithm is used to add a very small identity matrix. ,Right now = Then it can make ,thereby Reversible. Among them... It is the identity matrix. For smaller real numbers, such as wait.

[0085] 4.4, Host obtains Noise matrix is ​​eliminated using homomorphic scalar multiplication. and noise vector Specifically To obtain Newton's direction Noise is added to its own Newton direction and then given to Guest;

[0086] 4.5. The Guest decrypts its own Newton direction and updates its own parameters, then sends back the Newton direction of the noisy Host.

[0087] 4.6. The Host removes noise to obtain its own Newton direction and updates its own parameters;

[0088] Wherein, the Host side is designated as side A, the Guest side as side B, and the original data of side A includes sample data X. A Weight vector W A The original data from Party B includes sample data x. B y B Weight vector W B In this agreement, no trusted third party is required.

[0089] Before calculation, given that the exponential operation in logistic regression is performed using encrypted computation, the exponential part is expanded:

[0090]

[0091] ,

[0092] in, The items that require assistance in calculation are: , The items that require assistance in calculation are: .

[0093] To evaluate the accuracy of the current trained model's calculations, this application also provides a method for evaluating model predictions, the specific steps of which are as follows:

[0094] The Guest creates a public and private key, sends the public key to the Host, and the Guest calculates... And send it to the Host, the Host calculates... Send to Guest, Guest decrypts to obtain the predicted label; Acc is calculated, and the prediction formula is used. The prediction results depend on The symbol is , where Guest contains its own true label. After decryption, the predicted label needs to be compared with the true label to obtain the accuracy of the model prediction.

[0095] The traditional federated logistic regression algorithm is described below:

[0096] Logistic regression is a widely used machine learning model, and its expression is as follows:

[0097]

[0098] In vertical federated learning, let A and B be the two participants, and the dataset be... , Model parameters and Corresponding to the feature space and The objective function for training is as follows:

[0099]

[0100] The gradient calculation formula is as follows:

[0101] ;

[0102] For two-party vertical federated learning, among which It can be expressed as follows:

[0103] .

[0104] To train a federated logistic regression model, we need to protect the original data, such as... , , and model parameters , The summary is shown in Table 1 below:

[0105]

[0106] This data is stored by each party and is prohibited from being shared with each other. It is also expected that this data will not be learned by others from any intermediate data transmitted during the model training process.

[0107] In practice, when training a model on a dataset, first-order optimization typically iterates through steps 2 to 6 repeatedly, while second-order optimization iterates through steps 2 to 6 multiple times until the maximum number of iterations is reached or certain convergence conditions are met. Note that some intermediate data is exchanged during model training, but the data listed in Table 1 is not transferred between the two sides.

[0108]

[0109] The main steps involved in intermediate data exchange that could potentially lead to data leakage are as follows:

[0110] In Step 2, B sends the message in plaintext to each instance. Give it to A.

[0111] Step 3, A sends Give it to B.

[0112] Step 4, B sends Given A, where To use a randomized encryption mask to prevent A from learning the gradient of B.

[0113] The complexity of this application is compared with the aforementioned traditional algorithms, and the results are as follows:

[0114]

[0115] in, Indicates the number of samples. Represents the characteristic number.

[0116] Guest iterations exceeded In this way, the original data of the Host can be derived.

[0117] In summary, this application mainly involves the Host generating the public and private keys of Party A, and generating a random vector r. A ,calculate[[ ▽ W A ]] B +[[r A ]] B After decryption, noise is eliminated and generated. ▽ W B By adding its own gradient term, the Guest side generates the public and private keys of the B side, and generates a random vector r. B ,calculate[[▽ W B ]] A +[[r B ]] A After decryption, noise is eliminated and generated. ▽ W A Add its own gradient term and train the Host to obtain Guest received Encrypt using Guest's public key Send it to the Host, and calculate it using the Hessian matrix training protocol. Finally, the parameters are updated. This method ensures that the Guest and Host sides only receive their respective gradients and Newton directions, without any other valid data. This reduces the chance of data leakage to some extent and improves the security of model training.

[0118] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A longitudinal federated logistic regression algorithm, characterized in that: Includes the following steps: Host: A1. Generate A's public and private keys; A2, Calculate [[1 / 4] X A W A ]] A ; A3, Calculation ▽ W A ]] B =X A T [[1 / 4 X B W B -0.5 y B B; A4. Generate a random vector r A ,calculate[[ ▽ W A ]] B +[[r A ]] B ; A5, decrypted ▽ W B +r B ; A6. Obtained by eliminating noise ▽ W B Add its own gradient term; Guest side: B1. Generate B's public and private keys; B2, Calculate [[1 / 4] X B W B -0.5 y B ]] B ; B3. Calculation ▽ W B ]] A =X B T [[1 / 4 X A W A ]] A ; B4. Generate a random vector r B ,calculate[[ ▽ W B ]] A +[[r B ]] A ; B5, decrypted ▽ W A +r A ; B6. Noise removal results in ▽ W A Add its own gradient term; In step A2, the Host calculates [[1 / 4] X A W A ]] A The value is transferred to Guest, and in step B2, Guest calculates [[1 / 4]]. X B W B -0.5 y B ]] B The value is transferred to the Host; In step A4, the Host calculates [[ ▽ W A ]] B +[[r A ]] B The value is transferred to Guest, and the [[ calculated by Guest in step B4]] ▽ W B ]] A +[[r B ]] A The value is transferred to the Host; The Host calculation in step A5 ▽ W B +r B The value is transferred to the Guest, and the Guest calculates it in step B5. ▽ W A +r A The value is transferred to the Host; Second-order optimization is also performed, with the specific steps as follows: S1, Training the Host to obtain Guest received Encrypt using Guest's public key Send it to the Host, and the Host receives the gradient. ; S2, Calculate using the Hessian matrix training protocol ; S3. Calculate the Newton direction and update the parameters; The training protocol for calculating the Hessian matrix The specific training process is as follows: 3.1 The Guest creates a public key and a private key, and sends the public key to the Host; 3.2 Host Calculation , , Guest calculation ,send , Give to Host; 3.3 Host Calculation ,get .

2. The longitudinal federated logistic regression algorithm according to claim 1, characterized in that: The Host side is designated as side A, and the Guest side is designated as side B. The original data of side A includes sample data X. A Weight vector W A The original data from Party B includes sample data x. B y B Weight vector W B .

3. The longitudinal federated logistic regression algorithm according to claim 2, characterized in that: Before calculation, given that the exponential operation in logistic regression is performed using encrypted computation, the exponential part is expanded: ； ， in, The items that require assistance in calculation are: , The items that require assistance in calculation are: .

4. The longitudinal federated logistic regression algorithm according to claim 3, characterized in that: The Guest creates a public and private key, sends the public key to the Host, and the Guest calculates... And send it to the Host, the Host calculates... Send it to the Guest, who decrypts it to obtain the predicted label.

5. The longitudinal federated logistic regression algorithm according to claim 4, characterized in that: The specific steps for updating the parameters in step S3 are as follows: 4.1 The Guest and Host generate public and private keys. The Guest sends the public key to the Host, and they jointly compute... And send it to the Host; 4.

2. Add an invertible noise matrix to the Host. and noise vector Calculate using Guest's public key and And send it to the Guest; 4.3 Guest Decryption and To obtain noisy and noise vector Inverse calculus yields ,calculate After encryption, it is sent to the Host; 4.4, Host obtains Noise matrix is ​​eliminated using homomorphic scalar multiplication. and noise vector Specifically To obtain Newton's direction Noise is added to its own Newton direction and then given to Guest; 4.

5. The Guest decrypts its own Newton direction and updates its own parameters, then sends back the Newton direction of the noisy Host. 4.

6. The Host removes noise to obtain its own Newton direction and updates its own parameters.

6. The longitudinal federated logistic regression algorithm according to claim 5, characterized in that: In step 4.1, the Host uses the Guest's public key to encrypt... get ,but , That is .

7. A longitudinal federated logistic regression algorithm according to claim 6, characterized in that: In step 4.3, it is necessary to... Finding the inverse, that is, first finding the generated ones. The noise matrix is ​​an invertible matrix.

Images