Model training method and device based on vertical federated learning

Abstract

The present disclosure relates to a model training method and device based on longitudinal federated learning. In this method, when multiple nodes jointly train a global logistic regression model through longitudinal federated learning, the node with the sample label generates differential privacy noise through local differential privacy technology, and then adds the differential privacy noise to the model residual (i.e., the first residual) of each round of training, so as to protect the model residual from being reversed and broken by other nodes participating in the federated learning, and further protect the labels corresponding to the training samples from being obtained by other nodes, thereby improving the security of the training samples.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a model training method and device based on vertical federated learning. Background Art

[0002] Federated learning is one of the hottest machine learning technologies currently. It can solve the problem of how to jointly train a global model on virtual "aggregated" data while protecting the data security of each data center when multiple data centers are involved. Federated learning can be further divided into: horizontal federation, vertical federation, and federated migration. Among them, in vertical federation, the training samples in the data sets owned by multiple data centers overlap with each other, but the sample features are complementary, which can be applied to scenarios where multiple features serve the same business label. Therefore, vertical federated learning is widely used.

[0003] At present, in the logistic regression training protocol adopted by vertical federated learning, the data center uses homomorphic encryption to transmit the model residual information to another data center. The data center that receives the model residual information completes the gradient calculation in the ciphertext space, and then the data center that sends the model residual information assists in decryption to complete the model training.

[0004] When this method is used for federated learning, the security of training samples is low. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a model training method and device based on vertical federated learning.

[0006] In a first aspect, the present disclosure provides a model training method based on vertical federated learning, comprising:

[0007] The first information is calculated based on the first logistic regression model and the first characteristic value of the common training sample participating in the longitudinal federated learning, and the second information is received by other nodes participating in the longitudinal federated learning based on the second logistic regression model and the second characteristic value of the training sample;

[0008] Performing linear fusion and category prediction on the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculating, for the training sample, a difference between the probability value and a label corresponding to the training sample as a first residual;

[0009] Generate differential privacy noise according to a differential privacy algorithm, and use the differential privacy noise to perturb the first residual to obtain a differential privacy residual;

[0010] The first logistic regression model is updated according to the differential privacy residual, and the differential privacy residual is sent to the other federated learning nodes to update the second logistic regression model.

[0011] In some embodiments, the perturbation processing of the first residual using the differential privacy noise to obtain the differential privacy residual includes:

[0012] Mapping the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the inverse of the preset residual lower bound;

[0013] Multiplying the differential privacy noise by the second residual to obtain an intermediate calculation result;

[0014] The intermediate calculation result is mapped to the differential privacy residual according to a preset privacy upper bound, wherein the differential privacy residual is smaller than the preset privacy upper bound or larger than the inverse of the preset privacy upper bound.

[0015] In some embodiments, using the differential privacy noise to perturb the first residual to obtain a differential privacy residual includes:

[0016] The differential privacy noise is added to the first residual to obtain the differential privacy residual.

[0017] In some embodiments, generating differential privacy noise according to the differential privacy algorithm includes:

[0018] Determine a Laplace distribution that satisfies local differential privacy according to the differential privacy algorithm;

[0019] A random number is extracted from the Laplace distribution as the differential privacy noise.

[0020] In some embodiments, determining a Laplace distribution satisfying local differential privacy according to the differential privacy algorithm includes:

[0021] Mapping the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the opposite number of the preset residual lower bound;

[0022] Calculate the reciprocal difference of the second residuals corresponding to any two of the training samples, and determine the reciprocal difference with the largest value as the target reciprocal difference;

[0023] Obtaining the Laplace variance according to the relationship between the preset differential privacy budget, the preset differential privacy upper bound, and the target reciprocal difference;

[0024] The Laplace distribution is determined according to the Laplace variance and a first preset mean.

[0025] In some embodiments, determining a Laplace distribution satisfying local differential privacy according to the differential privacy algorithm includes:

[0026] Calculate the difference between the first residuals corresponding to any two of the training samples, and take the maximum difference as the target difference;

[0027] Determining the Laplace variance according to an inverse relationship between a preset differential privacy budget and the target difference;

[0028] The Laplace distribution is determined according to the Laplace variance and a second preset mean.

[0029] In a second aspect, the present disclosure provides a model training device based on vertical federated learning, comprising:

[0030] A processing module, configured to calculate first information based on a first logistic regression model and a first characteristic value of a common training sample participating in longitudinal federated learning;

[0031] A receiving module, used for receiving second information calculated by other nodes participating in the longitudinal federated learning according to the second logistic regression model and the second eigenvalue of the training sample;

[0032] The processing module is further used to perform linear fusion and category prediction on the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculate, for the training sample, a difference between the probability value and a label corresponding to the training sample as a first residual;

[0033] The processing module is further configured to generate differential privacy noise according to a differential privacy algorithm, perform perturbation processing on the first residual using the differential privacy noise to obtain a differential privacy residual; and update the first logistic regression model according to the differential privacy residual;

[0034] A sending module is used to send the differential privacy residual to the other federated learning nodes to update the second logistic regression model.

[0035] In a third aspect, the present disclosure provides an electronic device, including: a memory and a processor;

[0036] The memory is configured to store computer program instructions;

[0037] The processor is configured to execute the computer program instructions so that the electronic device implements the model training method based on vertical federated learning as described in the first aspect and any one of the first aspects.

[0038] In a fourth aspect, an embodiment of the present disclosure further provides a readable storage medium, including: computer program instructions;

[0039] When the computer program instructions are executed by at least one processor of an electronic device, the electronic device implements the model training method based on vertical federated learning as described in the first aspect and any one of the first aspects.

[0040] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product. When the computer program product is executed by an electronic device, the electronic device implements the model training method based on vertical federated learning as described in the first aspect and any one of the first aspects.

[0041] The disclosed embodiments provide a model training method and device based on vertical federated learning, wherein, when multiple nodes in the method jointly train a global model through vertical federated learning, the nodes with sample labels generate differential privacy noise through local differential privacy technology, and then add the differential privacy noise to the model residual (i.e., the first residual) of each round of training, so as to protect the model residual from being reversed and cracked by other nodes participating in the federated learning, and further protect the labels corresponding to the training samples from being obtained by other nodes, thereby improving the security of the training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A system framework diagram for longitudinal federated learning model training based on local differential privacy provided in one embodiment of the present disclosure;

[0045] Figure 2 A schematic diagram of multiple nodes provided by the present disclosure each having training data;

[0046] Figure 3 A flowchart of a model training method based on vertical federated learning provided in one embodiment of the present disclosure;

[0047] Figure 4 A flowchart of a model training method based on vertical federated learning provided in another embodiment of the present disclosure;

[0048] Figure 5 A flowchart of a model training method based on vertical federated learning provided in another embodiment of the present disclosure;

[0049] Figure 6 A schematic diagram of the structure of a model training device based on vertical federated learning provided in another embodiment of the present disclosure;

[0050] Figure 7 A schematic structural diagram of an electronic device provided in another embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0053] At present, when multiple nodes, also called data centers, jointly train a global model based on vertical federated learning technology, the data center with labels corresponding to the training samples uses homomorphic encryption to encrypt the relevant calculation results obtained during the training process to obtain ciphertext calculation results and then transmit them to another data center. The data center that receives the ciphertext calculation results can complete gradient calculations in the ciphertext space, and then the data center that sends the ciphertext calculation results can assist it in decryption, thereby helping the data center complete model training.

[0054] When this method is used for vertical federated learning, if the batch size │B│ of the training samples is not appropriately selected (for example, the batch size │B│ is too small), the data center that receives the ciphertext calculation results can reversely infer the actual calculation results through linear equations whose number is greater than the number of training samples, and further obtain the labels of the training samples based on the actual calculation results. It can be seen that the traditional method will result in lower security of the training samples.

[0055] The present disclosure provides a model training method and device based on vertical federated learning. The method adopts a federated logistic regression training protocol based on differential privacy residuals to implement vertical federated learning for multiple nodes, which is beneficial to improving the security of common training samples participating in vertical federated learning. Among them, the model training method based on vertical federated learning provided by the present disclosure can be executed by the model training device based on vertical federated learning provided by the present disclosure, and the device can be implemented in any software and / or hardware manner. Exemplarily, the device can be: a server, a cloud server, a server cluster, a service platform, a desktop computer, a laptop computer, etc.

[0056] Among them, the method provided in the present disclosure adopts differential privacy technology to perturb the model residual, which is specifically achieved through local differential privacy technology. In order to make this solution clearer, the local differential privacy technology is first introduced here.

[0057] Figure 1 A system framework diagram for training a longitudinal federated learning model with local differential privacy provided by an embodiment of the present disclosure. Figure 1 As shown in the figure, assume that there are N data centers (i.e., N nodes participating in vertical federated learning), namely data center 1 to data center N, with a total of K training samples, and each data center has m corresponding training samples. i Without loss of generality, assume that data center 1 has sample labels corresponding to training samples. The data distribution corresponding to training samples can be referred to Figure 2 shown.

[0058] See also Figure 1 As shown, when the model is trained by local differential privacy technology, data center 1 to data center N first need to perform model initialization respectively. After that, data center 2 to data center N respectively fuse the model initialization weight information and the features of the training samples they have to obtain the results and send the results to data center 1 which has the sample labels. Data center 1 fuses the received information and predicts the category, and obtains the prediction result corresponding to the training sample. The prediction result can be the probability value that the training sample belongs to the specified category. After that, data center 1 calculates the model residual according to the prediction result and the label of the training sample, and perturbs the model residual based on the differential privacy algorithm to obtain the differential privacy residual. Data center 1 then sends the differential privacy residual to data center 2 to data center N, so that data center 2 to data center N respectively update the model according to the received differential privacy residual.

[0059] First, the perturbation of the model residual through differential privacy is random, and this randomness further protects the security of the labels of the training samples. In addition, with the method disclosed in the present invention, there is no need for a trusted third party to collect the features and sample labels of the training samples owned by each data center, which is conducive to protecting the security of the training samples.

[0060] It should be noted that in the embodiments of the present disclosure, the logistic regression model trained by vertical federated learning can be any type of model, for example, a deep learning network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a decision tree, etc. The present disclosure does not limit the network structure of the logistic regression model. The logistic regression model is used to perform regression tasks corresponding to business needs. The business needs can be any business needs, and the business needs may be different based on different application scenarios. The present disclosure does not limit the regression tasks corresponding to the business needs.

[0061] The method provided by the present disclosure is described in detail below through several specific embodiments. In the following embodiments, the first node has a common first characteristic value and label of the training sample, and the second node has a second characteristic value of the training sample.

[0062] Figure 3 This is a flow chart of a model training method based on vertical federated learning provided in an embodiment of the present disclosure. The method of this embodiment is executed by the first node. Figure 3 As shown, the method of this embodiment includes:

[0063] S301, first information calculated based on a first logistic regression model and a first eigenvalue of a common training sample participating in longitudinal federated learning, and second information calculated based on a second logistic regression model and a second eigenvalue of the training sample by other nodes participating in longitudinal federated learning.

[0064] In this embodiment, the number of the second nodes may be one or more. The embodiment of the present disclosure does not limit the number of the second nodes. This embodiment takes one second node as an example.

[0065] For a training sample X = (x, y x ), x represents the sample characteristics, y x Represents the label corresponding to the training sample X, where the first node has the feature x1∈R m1 , label y∈{0,1}, the second node has the sample feature x2∈R of the training sample X m2 ; and satisfy x=[x1 T x2 T ] T, that is, the sample feature x of the training sample X can be obtained by transposing the sample feature x1 and the sample feature x2 respectively, then concatenating them, and then transposing the concatenated feature vector.

[0066] The first node stores the network structure and parameters of the first logistic regression model, and the second node stores the network structure and parameters of the second logistic regression model, wherein the first logistic regression model and the second logistic regression model can be two models with consistent network structures and parameters, or, it can also be understood that before model training, the models stored in different nodes are the same model.

[0067] As a possible implementation, the first node and the second node respectively perform model initialization to obtain the initialization model weight. Assume that the initialization model weight obtained by the first node is w1, and the initialization model weight obtained by the second node is w2. And respectively perform the process of multiplying the initialization model weight with the sample feature, that is, for each training sample, the first node obtains the first information w1 corresponding to the training sample T x1, the second node obtains the second information w2 corresponding to the training sample T x2. Afterwards, the second node sends the second information w2 corresponding to the training sample T x2 is sent to the first node.

[0068] In each round of model training (i.e., in each epoch), the training sample set usually includes multiple training samples. The first node and the second node respectively perform the above process for the multiple training samples, and the second node converts the w2 corresponding to each training sample into T x2 is sent to the first node.

[0069] S302, linearly fuse and class predict the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculate, for the training sample, a difference between the probability value and a label corresponding to the training sample as a first residual.

[0070] In some embodiments, the first node can linearly fuse the first information and the second information in a weighted manner, and can fuse them through a linear predictor. In the weighted calculation, the weight coefficients corresponding to the first information and the second information can be pre-set by the user according to the needs. For example, it is assumed that the weight coefficient corresponding to the first information and the weight coefficient corresponding to the second information are both 1, that is, the linear predictor is 1. x Indicates that it satisfies the formula: x =w1 T x1+w2 T x2.

[0071] According to the fusion result obtained by the linear predictor, a prediction result for the training sample can be obtained in a linear or nonlinear manner, and the prediction result is used to indicate the probability value of the training sample belonging to a specified category.

[0072] For example, the formula: Calculate and obtain prediction results

[0073] Afterwards, the first residual (i.e., model residual) is obtained by calculating the difference between the prediction result corresponding to the training sample and the label of the training sample. For example, the first residual (i.e., model residual) can be obtained according to the formula: Get the first residual r x .

[0074] S303: Generate differential privacy noise according to a differential privacy algorithm, and use the differential privacy noise to perturb the first residual to obtain a differential privacy residual.

[0075] In some embodiments, a probability distribution function that meets the differential privacy requirements can be determined by a differential privacy algorithm, and a random number is extracted from the corresponding probability distribution function as differential privacy noise, and the first residual is perturbed based on the random number to obtain a differential privacy residual. The perturbation process can be a multiplicative perturbation process or an additive perturbation process, which is not limited in this disclosure.

[0076] Among them, the probability distribution function that meets the differential privacy requirements can be, but is not limited to, Laplace distribution, Gaussian distribution, etc. The determination method of the probability distribution function can be related to the first residual corresponding to the training sample, can be pre-set by the user, or can be dynamically generated in real time through certain algorithms. The present disclosure does not limit the specific implementation method for determining the probability distribution function.

[0077] The random number is the noise used to perturb the first residual. Since the noise is random, it can protect the model residual from being reversed or cracked by other nodes except the first node, thereby protecting the security of the training sample label.

[0078] S304: Update the first logistic regression model according to the differential privacy residual and send the differential privacy residual to the other federated learning nodes to update the second logistic regression model.

[0079] The first node can obtain updated weights according to the differential privacy residual and the initialization weights of the first logistic regression model, in combination with a pre-set learning rate, and according to the relationship between the three, and configure the updated weights as the weights of the first logistic regression model.

[0080] The first node sends the differential privacy residual to the second node. The second node obtains the updated weights based on the differential privacy residual, the initialization weights of the second logistic regression model, and a pre-set learning rate according to the relationship between the three, and configures the updated weights as the weights of the second logistic regression model.

[0081] As described above, model training may include multiple epochs, and each epoch may be understood as a round of training. By executing S301 to S304 for each epoch, the first logistic regression model and the second logistic regression model are continuously converged until the convergence conditions are met to complete the longitudinal federated learning and obtain the global model.

[0082] In the method of this embodiment, when the first node and the second node jointly train the global model through vertical federated learning, the first node generates differential privacy noise through local differential privacy technology, and then adds the differential privacy noise to the model residual (i.e., the first residual) of each round of training, so as to protect the model residual from being reversed and cracked by the second node, and further protect the labels of the training samples from being obtained by the second node, thereby improving the security of the training sample data.

[0083] As mentioned above, the perturbation processing of the model residual based on differential privacy noise can be but is not limited to multiplicative perturbation processing and additive perturbation processing. Figure 4 and Figure 5 The illustrated embodiments respectively and exemplarily illustrate how to achieve differential privacy of model residuals through the above two different disturbance processing methods.

[0084] Figure 4 A flowchart of a model training method based on federated learning provided in one embodiment of the present disclosure. Figure 4 In the illustrated embodiment, the first node processes the first residual by multiplicative perturbation to obtain a differential privacy residual. Figure 4 As shown, the method of this embodiment includes:

[0085] S401, first information calculated based on a first logistic regression model and a first eigenvalue of a common training sample participating in longitudinal federated learning, and second information calculated based on a second logistic regression model and a second eigenvalue of the training sample by other nodes participating in longitudinal federated learning.

[0086] S402, linearly fuse and class predict the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculate, for the training sample, a difference between the probability value and a label corresponding to the training sample as a first residual.

[0087] In this embodiment, steps S401 and S402 are respectively Figure 3 In the embodiment shown, steps S301 and S302 are similar, and can be referred to as above. Figure 3 For the sake of brevity, the detailed description of the illustrated embodiment will not be repeated here.

[0088] S403: Map the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the opposite of the preset residual lower bound.

[0089] The default lower bound of the residual is a positive number greater than 0. Assume that the default lower bound of the residual is r * , then the purpose of this step is to x Mapping is performed so that the second residual after mapping (denoted as ) is located at (-∞, -r * ]∪[r * ,∞), so the reciprocal of the second residual is It is in a bounded interval, thus ensuring that the reciprocal difference of any two second residuals is less than the preset residual lower bound.

[0090] As a possible implementation, the formula can be: Get the second residual in,

[0091] S404: Calculate the reciprocal difference of the second residuals corresponding to any two of the training samples, and determine the reciprocal difference with the largest value as the target reciprocal difference.

[0092] A training sample set used in a training period includes multiple training samples, each of which corresponds to a second residual. The difference between the reciprocal values ​​of the second residuals corresponding to any two training samples in the training sample set (i.e., the reciprocal difference) is obtained, and the reciprocal difference with the largest value is used as a parameter for determining the Laplace variance.

[0093] This step can be expressed by the formula as follows: Among them, x and x′ represent any two training samples in the training sample set. is the second residual corresponding to the training samples x and x′ respectively.

[0094] S405. Obtain the Laplace variance according to the relationship between the preset differential privacy budget, the preset differential privacy upper bound, and the target reciprocal difference.

[0095] The upper bound of differential privacy is preset as a constant greater than 0. The differential privacy budget is preset as a constant greater than 0. The purpose of this step is to find the Laplace variance that satisfies local differential privacy.

[0096] For example, the formula: Get the Laplace variance. In this formula, ε represents the preset differential privacy budget, b represents the Laplace variance, and z * represents the preset upper bound of differential privacy. Substituting the corresponding parameters into the formula, we can get the Laplace variance b.

[0097] S406: Determine a Laplace distribution according to the Laplace variance and the first preset mean, and extract a random number from the Laplace distribution as differential privacy noise.

[0098] Optionally, the first preset mean is 1. According to the mean being 1 and the standard deviation being The Laplace distribution can be determined.

[0099] As a possible implementation method, for each training sample in the training sample set, the mean is 1 and the standard deviation is Draw a random number (denoted as α) from the Laplace distribution x ) is used as the differential privacy noise for multiplicatively perturbing the model residuals corresponding to the training samples.

[0100] This step can be expressed by the formula:

[0101] Another possible implementation can be to start with a mean of 1 and a standard deviation of Multiple random numbers are drawn from the Laplace distribution, and the distance between the multiple random numbers is less than the preset distance. The training sample set is then divided into multiple subsets, and each subset corresponds to a random number that is differentially privately processed by multiplicative perturbation.

[0102] S407: multiply the differential privacy noise and the second residual to obtain an intermediate calculation result.

[0103] For each training sample in the training sample set B, a random number α drawn from the Laplace distribution is x The second residual corresponding to the training sample Multiply to get the intermediate calculation result z x This step can be expressed by the formula:

[0104] S408: Map the intermediate calculation result to the differential privacy residual according to a preset privacy upper bound, wherein the differential privacy residual is smaller than the preset privacy upper bound or larger than the opposite of the preset privacy upper bound.

[0105] In some possible cases, the intermediate calculation result obtained in step S407 may not meet the preset differential privacy upper bound. For example, the intermediate calculation result is greater than the preset differential privacy upper bound. Therefore, it is necessary to further trim the intermediate calculation result to ensure that the differential privacy residual sent to the second node meets the local differential privacy requirements.

[0106] As a possible implementation, the clipping can be performed by the formula Get the differential privacy residual The sgn function is shown in step S403.

[0107] S409: Update the first logistic regression model according to the differential privacy residual and send the differential privacy residual to the other federated learning nodes to update the second logistic regression model.

[0108] As a possible implementation, the first node may be based on the formula: The updated weight w1′ of the first logistic regression model is obtained, and the weight w1′ is configured as the latest weight of the first machine model. Wherein η represents a preset learning rate.

[0109] In addition, the first node can differentially privatize the residual The second node is sent to the second node, so that the second node updates the second logistic regression model. Exemplarily, the second node can update the second logistic regression model according to the formula: The updated weight w2′ of the second logistic regression model is obtained, and the weight w2′ is configured as the latest weight of the second logistic regression model. Wherein, η represents a preset learning rate.

[0110] The preset learning rates corresponding to the first node and the second node may be the same.

[0111] As described above, model training may include multiple epochs, and each epoch may be understood as a round of training. By executing S401 to S409 for each epoch, the first logistic regression model and the second logistic regression model are continuously converged until the convergence conditions are met to complete the longitudinal federated learning and obtain the global model.

[0112] In this embodiment, by obtaining a Laplace distribution that satisfies local differential privacy, and extracting random numbers from the Laplace distribution as differential privacy noise to perturb the model residual, the differential privacy noise is added to the model residual (i.e., the first residual) of each round of training by multiplication, thereby protecting the model residual from being reversed and cracked by the second node, and further protecting the training sample data label from being obtained by the second node, thereby improving the security of the training sample data.

[0113] In a specific embodiment, there are four data sets, which are used as training sample sets for longitudinal federated learning, respectively using homomorphic encryption and Figure 4 The federated learning is performed in the manner of the embodiment shown, and the model indicators Acc and Auc are used to compare the model effects obtained in the two methods. Acc represents accuracy, and Auc represents the area under the ROC curve.

[0114] Specifically, the four data sets are federated learning in two ways, and the model indicators Acc and Auc are shown in Table 1:

[0115] Table 1

[0116]

[0117] According to the model indicators shown in Table 1 above, Figure 4 When the method provided in the embodiment shown in the figure performs vertical federated learning, an appropriate r is selected. * 、z * In this case, a nearly lossless model effect can be obtained.

[0118] Figure 5 A flowchart of a model training method based on federated learning provided in one embodiment of the present disclosure. In this embodiment, the first node processes the first residual by additive perturbation to obtain a differential privacy residual. Figure 5 As shown, the method of this embodiment includes:

[0119] S501, first information calculated based on a first logistic regression model and a first eigenvalue of a common training sample participating in longitudinal federated learning, and second information calculated based on a second logistic regression model and a second eigenvalue of the training sample by other nodes participating in longitudinal federated learning.

[0120] S502: linearly fuse and class predict the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified class, and calculate the difference between the probability value and the label corresponding to the training sample as a first residual for the training sample.

[0121] In this embodiment, steps S501 and S502 are respectively Figure 3 In the embodiment shown, steps S301 and S302 are similar, and can be referred to as above. Figure 3 For the sake of brevity, the detailed description of the illustrated embodiment will not be repeated here.

[0122] S503: Calculate the difference between the first residuals corresponding to any two of the training samples, and obtain the maximum difference as the target difference.

[0123] This step can be expressed by the formula: Wherein, Θ represents the maximum value of the difference between the first residuals corresponding to the training sample x and the training sample x′, that is, the target difference; x and x′ represent any two training samples in the training sample set B.

[0124] S504. Determine the Laplace variance according to the inverse relationship between the preset differential privacy budget and the target difference.

[0125] The purpose of this step is to find the Laplace variance that satisfies local differential privacy.

[0126] As a possible implementation method, the formula: Get the Laplace variance. In this formula, ε represents the preset differential privacy budget and b represents the Laplace variance.

[0127] S505 . Determine the Laplace distribution according to the Laplace variance and a second preset mean, and extract a random number from the Laplace distribution as differential privacy noise.

[0128] Optionally, the second preset mean is 0. The Laplace distribution can be determined.

[0129] As a possible implementation method, for each training sample in the training sample set, the mean is 0 and the standard deviation is Draw a random number (denoted as β) from the Laplace distribution x ) is used as the differential privacy noise for additively perturbing the model residuals corresponding to the training samples.

[0130] The random number extraction can be expressed by the formula:

[0131] Another possible implementation method can be to start with a mean of 0 and a standard deviation of Multiple random numbers are drawn from the Laplace distribution, and the distance between the multiple random numbers is less than the preset distance. The training sample set is then divided into multiple subsets, and each subset corresponds to a random number that is differentially privately processed by additive perturbation.

[0132] S506: Add the differential privacy noise to the first residual to obtain a differential privacy residual.

[0133] For each training sample in the training sample set B, a random number β drawn from the Laplace distribution is x and the corresponding first residual r x Add.

[0134] This step can be expressed by the formula: x =β x +r x , where p x represents the differentially private residual obtained by additive perturbation processing.

[0135] S507: Update the first logistic regression model according to the differential privacy residual and send the differential privacy residual to the other federated learning nodes to update the second logistic regression model.

[0136] As a possible implementation, the first node may be based on the formula: Obtain the updated weight w1′ of the first logistic regression model and configure the weight w1′ as the latest weight of the first logistic regression model. The first node sends the differential privacy residual to the second node, and the second node can use the formula: Obtain the updated weight w2′ of the second logistic regression model, and configure the weight w2′ as the latest weight of the second logistic regression model. Wherein, η represents a preset learning rate. The preset learning rates corresponding to the first node and the second node may be the same.

[0137] In this embodiment, by obtaining a Laplace distribution that satisfies local differential privacy, and extracting random numbers from the Laplace distribution as differential privacy noise to perturb the model residual, the differential privacy noise is added to the model residual (i.e., the first residual) of each round of training by superposition, thereby protecting the model residual from being reversed and cracked by the second node, and then protecting the label of the training sample from being obtained by the second node, thereby improving the security of the training sample data.

[0138] In a specific embodiment, there are four data sets, which are used as training sample sets for longitudinal federated learning, respectively using homomorphic encryption and Figure 5 The federated learning is performed in the manner of the embodiment shown, and the model indicators Acc and Auc are used to compare the model effects obtained by the two methods. Specifically, the four data sets are federated learning using two methods respectively, and the model indicators Acc and Auc are shown in Table 2 below:

[0139] Table 2

[0140]

[0141] According to the model indicators shown in Table 2 above, Figure 5 When the method provided in the embodiment shown in the figure performs federated learning, as the value of the preset privacy budget ε increases, Figure 5 The model effect obtained by the method shown is getting closer and closer to the ideal state.

[0142] It should be noted that based on Figure 4 as well as Figure 5 As shown, it is possible to combine, for example, Figure 4 The Laplace distribution is determined by the embodiment to extract random numbers, and then Figure 5 The additive perturbation shown in the embodiment adds differential privacy noise, or it can also be done by Figure 5 The Laplace distribution is used to extract random numbers, and then Figure 4 The multiplicative perturbation shown in the embodiment adds differential privacy noise.

[0143] Exemplarily, the present disclosure also provides a model training device based on vertical federated learning.

[0144] Figure 6 This is a schematic diagram of the structure of a model training device based on vertical federated learning provided in one embodiment of the present disclosure. Figure 6 As shown, the device 600 provided in this embodiment includes:

[0145] The processing module 601 is used to calculate the first information according to the first logistic regression model and the first characteristic value of the common training samples participating in the longitudinal federated learning.

[0146] The receiving module 602 is used to receive second information calculated by other nodes participating in the longitudinal federated learning according to the second logistic regression model and the second eigenvalue of the training sample.

[0147] The processing module 601 is also used to perform linear fusion and category prediction on the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculate the difference between the probability value and the label corresponding to the training sample as a first residual for the training sample; generate differential privacy noise according to a differential privacy algorithm, use the differential privacy noise to perturb the first residual to obtain a differential privacy residual; and update the first logistic regression model according to the differential privacy residual.

[0148] The sending module 603 is used to send the differential privacy residual to the other federated learning nodes to update the second logistic regression model.

[0149] In some embodiments, the processing module 601 is specifically used to map the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the inverse of the preset residual lower bound; multiply the differential privacy noise by the second residual to obtain an intermediate calculation result; and map the intermediate calculation result to the differential privacy residual according to a preset privacy upper bound, wherein the differential privacy residual is less than the preset privacy upper bound or greater than the inverse of the preset privacy upper bound.

[0150] In some embodiments, the processing module 601 is specifically configured to add the differential privacy noise to the first residual to obtain the differential privacy residual.

[0151] In some embodiments, the processing module 601 is specifically configured to determine a Laplace distribution that satisfies local differential privacy according to the differential privacy algorithm; and extract a random number from the Laplace distribution as the differential privacy noise.

[0152] In some embodiments, the processing module 601 is specifically used to map the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the opposite of the preset residual lower bound; calculate the reciprocal difference of the second residuals corresponding to any two of the training samples, and determine the reciprocal difference with the largest value as the target reciprocal difference; obtain the Laplace variance according to the relationship between the preset differential privacy budget, the preset differential privacy upper bound and the target reciprocal difference; determine the Laplace distribution according to the Laplace variance and the first preset mean.

[0153] In some embodiments, the processing module 601 is specifically used to calculate the difference between the first residuals corresponding to any two of the training samples, and obtain the maximum difference as the target difference; determine the Laplace variance according to the inverse relationship between the preset differential privacy budget and the target difference; determine the Laplace distribution according to the Laplace variance and the second preset mean.

[0154] The device provided in this embodiment can be used to execute the technical solution of any of the aforementioned method embodiments. Its implementation method and technical principle are similar. Please refer to the detailed description of the aforementioned method embodiments. For the sake of brevity, they will not be repeated here.

[0155] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device 700 provided in this embodiment includes: a memory 701 and a processor 702 .

[0156] The memory 701 may be an independent physical unit, and may be connected to the processor 702 via a bus 703. The memory 701 and the processor 702 may also be integrated together and implemented via hardware.

[0157] The memory 701 is used to store program instructions, and the processor 702 calls the program instructions to execute the model training method based on vertical federated learning provided in any of the above method embodiments.

[0158] Optionally, when part or all of the methods of the above embodiments are implemented by software, the above electronic device 700 may also only include a processor 702. The memory 701 for storing programs is located outside the electronic device 700, and the processor 702 is connected to the memory through circuits / wires to read and execute the programs stored in the memory.

[0159] The processor 702 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0160] The processor 702 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0161] The memory 701 may include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0162] The present disclosure also provides a readable storage medium, including: computer program instructions, which, when executed by at least one processor of an electronic device, enable the electronic device to implement a model training method based on vertical federated learning provided in any of the above method embodiments.

[0163] The present disclosure also provides a computer program product. When the computer program product is run on a computer, the computer implements the model training method based on vertical federated learning provided in any of the above method embodiments.

[0164] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0165] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method based on vertical federated learning, characterized in that: Applied to the first node, including: The first information is calculated based on the first logistic regression model and the first characteristic value of the common training sample participating in the longitudinal federated learning, and the second information is received by other nodes participating in the longitudinal federated learning based on the second logistic regression model and the second characteristic value of the training sample; Performing linear fusion and category prediction on the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculating, for the training sample, a difference between the probability value and a label corresponding to the training sample as a first residual; Generate differential privacy noise according to a differential privacy algorithm, and use the differential privacy noise to perturb the first residual to obtain a differential privacy residual; The first logistic regression model is updated according to the differential privacy residual, and the differential privacy residual is sent to other federated learning nodes to update the second logistic regression model.

2. The method according to claim 1, characterized in that The using the differential privacy noise to perform perturbation processing on the first residual to obtain the differential privacy residual includes: Mapping the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the inverse of the preset residual lower bound; multiplying the differential privacy noise by the second residual to obtain an intermediate calculation result; The intermediate calculation result is mapped to the differential privacy residual according to a preset privacy upper bound, wherein the differential privacy residual is smaller than the preset privacy upper bound or larger than the inverse of the preset privacy upper bound.

3. The method according to claim 1, characterized in that The using the differential privacy noise to perform perturbation processing on the first residual to obtain the differential privacy residual includes: The differential privacy noise is added to the first residual to obtain the differential privacy residual.

4. The method according to claim 1, characterized in that: Generating differential privacy noise according to the differential privacy algorithm includes: Determine a Laplace distribution that satisfies local differential privacy according to the differential privacy algorithm; A random number is extracted from the Laplace distribution as the differential privacy noise.

5. The method according to claim 4, characterized in that The determining, according to the differential privacy algorithm, a Laplace distribution satisfying local differential privacy includes: Mapping the first residual to a second residual according to a preset residual lower bound, wherein the second residual is greater than the preset residual lower bound or less than the opposite number of the preset residual lower bound; Calculate the reciprocal difference of the second residuals corresponding to any two of the training samples, and determine the reciprocal difference with the largest value as the target reciprocal difference; Obtaining the Laplace variance according to the relationship between the preset differential privacy budget, the preset differential privacy upper bound, and the target reciprocal difference; The Laplace distribution is determined according to the Laplace variance and a first preset mean.

6. The method according to claim 4, characterized in that The step of determining a Laplace distribution satisfying local differential privacy according to the differential privacy algorithm includes: Calculate the difference between the first residuals corresponding to any two of the training samples, and take the maximum difference as the target difference; Determining the Laplace variance according to an inverse relationship between a preset differential privacy budget and the target difference; The Laplace distribution is determined according to the Laplace variance and a second preset mean.

7. A model training device based on vertical federated learning, characterized in that: Applied to the first node, including: A processing module, configured to calculate first information based on a first logistic regression model and a first characteristic value of a common training sample participating in longitudinal federated learning; A receiving module, used for receiving second information calculated by other nodes participating in the longitudinal federated learning according to the second logistic regression model and the second eigenvalue of the training sample; The processing module is further used to perform linear fusion and category prediction on the first information and the second information to obtain a probability value for determining that the training sample belongs to a specified category, and calculate, for the training sample, a difference between the probability value and a label corresponding to the training sample as a first residual; The processing module is further configured to generate differential privacy noise according to a differential privacy algorithm, perform perturbation processing on the first residual using the differential privacy noise to obtain a differential privacy residual; and update the first logistic regression model according to the differential privacy residual; A sending module is used to send the differential privacy residual to other federated learning nodes to update the second logistic regression model.

8. An electronic device, characterized in that: include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions so that the electronic device implements the model training method based on vertical federated learning as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: include: Computer program instructions; When the computer program instructions are executed by at least one processor of an electronic device, the electronic device implements the model training method based on vertical federated learning as described in any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product is executed by an electronic device, the electronic device implements the model training method based on vertical federated learning as described in any one of claims 1 to 6.

Images