Vertical federated learning training and prediction method and device based on tree structure
By adopting a tree-based structure method in vertical federated learning, combining blinded value and public key encryption technology, the problem of model leakage and communication burden is solved, and a safer and more efficient vertical federated learning training and prediction process is achieved.
Patent Information
- Application Number
- CN202111609468.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2025-04-11
- Estimated Expiration
- 2041-12-24
AI Technical Summary
The existing vertical federated learning model has the risk of model leakage in the training stage, and the communication burden in the prediction stage is relatively large, which affects the model performance.
Using a vertical federated learning method based on tree structure, the initiator creates a decision tree and collaborates with partners to establish multiple decision trees. By setting blinded values and public key encryption, the model is prevented from leaking, and the prediction results are obtained through one communication during the prediction stage.
In the training stage, the risk of model leakage is reduced and the possibility of partners obtaining model information is reduced. In the prediction stage, the communication burden between the initiator and the partner is reduced and the prediction performance of the model is improved.
Smart Images
Figure CN114372517B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a vertical federated learning training and prediction method and device based on a tree structure. Background Art
[0002] In real-world scenarios, there is a lot of useful data distributed across different devices or organizations, and this data is usually sensitive. Therefore, how to design a machine learning framework that allows users' data to be efficiently shared while ensuring data privacy and security has become an urgent problem to be solved, and federated learning has emerged. When the data user IDs owned by different organizations have a large overlap but the features overlap less, it is suitable for vertical federated learning scenarios. In the model, only one party can have a label, which we call the initiator, and we call the data provider the partner. The initiator and the partner are collectively referred to as participants.
[0003] At present, the most representative, advanced and widely used vertical federated learning framework Scureboost is a tree-based model framework that integrates the XGBoost algorithm in machine learning algorithms into the vertical federated learning scenario. It requires the participation of multiple parties when building a decision tree. First, the initiator with the label calculates the gradient of the loss function, encrypts the gradient and shares it with each partner; then, the partner partitions the user according to its own characteristics and aggregates the encrypted gradient and sends the aggregated result to the initiator; the initiator decrypts the aggregated gradient and calculates the optimal segmentation point, and then notifies the owner of the optimal segmentation point to segment the sample space; the partner notifies the initiator of the latest space division result, and the initiator synchronizes the latest node division result to other partners. By iterating the above node segmentation process, a decision tree is gradually built. Finally, multiple decision trees are iteratively constructed by the gradient boosting method to finally complete the training and obtain the federated learning model. The initiator has a complete model, but because the initiator and the partner share the node space information of the tree during the training phase, the partner has the structure of the tree and some node information, so there may be a risk of model leakage.
[0004] In the model prediction stage, the initiator uses the trained model to ask the node feature owner about the next search direction based on the relevant information of the tree node; after receiving the message, the node feature owner calls the feature information obtained in the model training stage to determine the next search direction and sends it to the initiator; after receiving the search direction from the partner, the initiator goes to the corresponding child node; the above process is iterated until the sample reaches the leaf node of the tree to obtain the sample classification label.
[0005] The existing model has the following defects:
[0006] 1. During the model training phase, each time a node is split, the initiator will always share the classification of the node with all partners, which will cause the partners to obtain information about the training model (decision tree structure and some node information). Especially when the leaf nodes of the tree are generated by the partners, there is even a risk of sample label leakage.
[0007] 2. In the model prediction stage, processing a sample to be predicted requires multiple rounds of communication between the initiator and the partner. Especially when the number of samples to be predicted is large or the decision tree depth is large, the number of communications between the initiator and the partner will increase sharply. However, in practice, multiple communications will bring serious communication burden and affect the model prediction performance. Summary of the invention
[0008] In order to solve at least some of the defects in the prior art, the present invention provides a vertical federated learning training and prediction method and device based on a tree structure.
[0009] The present invention is achieved in that:
[0010] In a first aspect, the present invention provides a vertical federated learning training and prediction method based on a tree structure, which is applied to an initiator and multiple partners. The method comprises the following steps:
[0011] The initiator uses the feature data it holds to create the first decision tree;
[0012] The initiator and the partner collaborate to establish the mth decision tree in turn, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains a vertical federated learning model based on a tree structure; in the process of establishing the mth decision tree, for each node sample space, the initiator calculates the optimal feature and feature threshold, sets the blinding value, and notifies the partner holding the corresponding feature. The feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the result;
[0013] The longitudinal federated learning model obtained by the above training is used to predict the label of the object to be predicted.
[0014] Furthermore, the initiator has a public key PK and a private key SK, and shares the public key PK with the partner.
[0015] Furthermore, the creation process of the initiator and the partner collaborating to establish the mth decision tree is as follows:
[0016] Step 1: The initiator selects the best feature from the features it holds as the root node of the tree and divides the sub-node sample space; uses the public key PK to encrypt the training sample loss function gradient, and sends the sample loss function gradient ciphertext and sub-node sample space to each partner;
[0017] Step 2: The partner partitions the current node sample space according to its own characteristics, aggregates the gradient ciphertexts of samples in the same partition and sends each aggregation result to the initiator;
[0018] Step 3: For each node sample space, the initiator calculates the optimal feature and feature threshold, and arbitrarily selects a feature and feature threshold as the blinding value, and sends the above optimal feature and feature threshold as well as the feature and feature threshold as the blinding value to the partner holding the corresponding feature;
[0019] Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and stores the features and feature thresholds locally;
[0020] Step 5: The initiator updates the model using the real node sample space partition results and sends all the partition results to all partners;
[0021] Step 6: For all received node sample spaces, the partner aggregates the gradient ciphertext according to the method in step 2 and sends it to the initiator;
[0022] Step 7: For the real node sample space, the initiator calculates the optimal feature and feature threshold; for the node sample space used as the blind value, the initiator arbitrarily selects a feature and feature threshold, and sends the above features and feature threshold to the corresponding feature holder;
[0023] Repeat steps 4 to 7 until the preset stopping condition is reached and the creation of the mth tree is completed.
[0024] Furthermore, the method of using the longitudinal federated learning model obtained by the above training to predict the label of the object to be predicted specifically includes:
[0025] Step 1: For the object to be predicted, the partner uses the features and feature thresholds selected by the partner during the model training phase to compare with the corresponding feature values of the object to be predicted, and sends all comparison results to the initiator;
[0026] Step 2: The initiator performs deblinding on the comparison result sets from each partner to obtain the comparison results of the real features and feature thresholds;
[0027] Step 3: The initiator uses the model obtained in the training phase and the actual comparison results to predict the object label.
[0028] In a second aspect, the present invention provides a vertical federated learning training and prediction device based on a tree structure, which is applied to an initiator and multiple partners, and the device includes:
[0029] The first decision tree building module is used by the initiator to create the first decision tree using the feature data it holds;
[0030] The m-th decision tree establishment module is used for the initiator and the partner to collaborate and sequentially establish the m-th decision tree, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains a vertical federated learning model based on a tree structure; in the process of establishing the m-th decision tree, for each node sample space, the initiator calculates the optimal feature and feature threshold, sets the blinding value, and notifies the partner holding the corresponding feature. The feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the result;
[0031] The label prediction module is used to predict the label of the object to be predicted using the longitudinal federated learning model obtained through the above training.
[0032] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.
[0033] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The tree-structured vertical federated learning training and prediction method and device provided by the present invention, during the model training stage, the collaborating party cannot obtain any node-related information, thereby reducing the risk of model leakage; during the model prediction stage, the initiator and the collaborating party only need to communicate once to obtain the prediction result, thereby reducing the communication burden and improving the model prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A flowchart of a tree-structured vertical federated learning training and prediction method provided in an embodiment of the present invention;
[0037] Figure 2 A flowchart of establishing the mth decision tree provided by an embodiment of the present invention;
[0038] Figure 3 A schematic diagram of a decision tree obtained by an initiator provided in an embodiment of the present invention;
[0039] Figure 4 A model prediction flow chart provided for an embodiment of the present invention;
[0040] Figure 5A block diagram of a tree-structured vertical federated learning training and prediction device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] like Figure 1 As shown, the embodiment of the present invention provides a vertical federated learning training and prediction method based on a tree structure, with C+1 participants, including 1 initiator A, C partners B1,...B C , where only the initiator has the sample label information, and the partner B i Possess p i features, the initiator A has p0 features, then p0+p1+…+p c = n. Input before training: training data set T = {(x1, y1), (x2, y2), ..., (x N ,y N )}, Loss function L(y,f(x)), training sample ID set G = {1,2,...,N}.
[0043] The method comprises the following steps:
[0044] S101, the initiator A uses the p0 feature data it holds to create the first decision tree;
[0045] S102, the initiator and the partner collaborate to establish the mth decision tree in sequence, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains a vertical federated learning model based on a tree structure; in the process of establishing the mth decision tree, for each node sample space, the initiator calculates the optimal feature and feature threshold, sets the blinding value, and notifies the partner holding the corresponding feature. The feature holder divides the current node sample space according to the received feature and feature threshold, and notifies the initiator of the result;
[0046] To refine the above steps, the initiator has the public key PK and the private key SK, and shares the public key PK with the partner.
[0047] like Figure 2 As shown, the creation process of the initiator and the partner collaborating to establish the mth decision tree is as follows:
[0048] Step 1: The initiator A selects the best feature from the p0 features it holds as the root node of the mth decision tree, and divides the N samples into two sub-nodes to form two sub-node sample spaces G1 and G2, satisfying G1∪G2=G; uses the public key PK to encrypt the gradient of the training sample loss function [[g i ]],[[h i ]],i=1,2,...,N.(g i ,h i They represent the first-order partial derivative and the second-order partial derivative of the loss function L with respect to f(x), respectively, and [[]] represents homomorphic encryption). The sample loss function gradient ciphertext and the sub-node sample space are sent to each partner;
[0049] Step 2: The partner partitions the current node sample space according to its own characteristics, aggregates the gradient ciphertexts of samples in the same partition (feature k, feature value v) and converts each aggregation result [[G kv ]],[[H kv ]]Sent to the initiator;
[0050] Step 3: For each node sample space, the initiator calculates the optimal feature k opt and feature threshold v opt , and arbitrarily select a feature k' opt and feature threshold v' opt As the blinding value, the above optimal features and feature thresholds (k opt ,v opt ) and the features and feature thresholds (k') as blinding values opt ,v' opt ) is sent to the partner holding the corresponding feature, i.e. the feature holder;
[0051] Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and stores the features and feature thresholds locally, recording ID, feature attributes, and feature thresholds;
[0052] Step 5: The initiator updates the model using the real node sample space partition results and sends all the partition results to all partners;
[0053] Step 6: For all received node sample spaces, including real node sample spaces and node sample spaces used as blinded values, the partner aggregates the gradient ciphertext according to the method in step 2 and sends it to the initiator;
[0054] Step 7: For the real node sample space, the initiator calculates the optimal feature and feature threshold (k opt ,v opt); For the node sample space as the blind value, that is, the false node space, the initiator arbitrarily selects a feature and feature threshold (k' opt ,v' opt ), the above features and feature thresholds (k opt ,v opt ),(k' opt ,v' opt ) is sent to the corresponding feature holder;
[0055] Repeat steps 4 to 7 until the preset stop condition is reached, and the creation of the mth tree is completed. The preset stop condition is the training stop condition set according to the initiator's needs before model training, such as the depth of the tree, convergence value, training time, etc. In the model training phase of this method, the initiator adds false features and feature thresholds as blinding values to prevent the partner from obtaining the true classification information of the sample during the training phase, making it impossible for the partner to obtain the true model, thereby preventing model leakage.
[0056] In the above method, the decision tree obtained by the initiator is as follows Figure 3 shown.
[0057] The initiator provides 2 sets of features and feature thresholds [A, 1], [A, 2] to divide the node sample space, the partner B1 provides 3 sets of features and feature thresholds [B1, 1], [B1, 2], [B1, 3] to divide the node sample space, and the partner B2 provides 3 sets of features and feature thresholds [B2, 1], [B2, 2], [B2, 3] to divide the node sample space. At the same time, the 3 sets of features and feature thresholds [B1, 4], [B1, 5], [B1, 6] of partner B1 and the 3 sets of features and feature thresholds [B2, 4], [B2, 5], [B2, 6] of partner B2 are used as blinding values.
[0058] Partner B1 obtains the following table of features and feature threshold information:
[0059]
[0060] S103. Use the longitudinal federated learning model obtained through the above training to predict the label of the object to be predicted.
[0061] Specifically, Figure 4 As shown, the use of the longitudinal federated learning model obtained by the above training to predict the label of the object to be predicted specifically includes:
[0062] Step 1: For the object to be predicted, the partner uses the features and feature thresholds selected by the partner during the model training phase to compare with the corresponding feature values of the object to be predicted, and sends all comparison results to the initiator;
[0063] Step 2: The initiator performs deblinding on the comparison result sets from each partner to obtain the real comparison results from the real features and feature thresholds; the comparison results from each partner include the real comparison results and the false comparison results used for blinding;
[0064] Step 3: The initiator uses the model obtained in the training phase and the actual comparison results to predict the object label.
[0065] In the model prediction stage, this method only communicates once between each partner and the initiator during the entire prediction process, and because of the addition of blinding values about features and feature thresholds, it is impossible to infer the label prediction results of the samples.
[0066] The comparison result table of partner B1 for sample X is as follows:
[0067]
[0068] The deblinding comparison result table of initiator A for B1 is as follows:
[0069]
[0070] like Figure 5 As shown, an embodiment of the present invention further provides a vertical federated learning training and prediction device based on a tree structure, which is applied to an initiator and multiple partners, and the device includes:
[0071] The first decision tree building module 11 is used for the initiator to create the first decision tree using the feature data it holds;
[0072] The m-th decision tree establishment module 12 is used for the initiator and the partner to collaborate and sequentially establish the m-th decision tree, m=2,3,...,M, where M is the total number of decision trees trained, and the training obtains a vertical federated learning model based on a tree structure; in the process of establishing the m-th decision tree, for each node sample space, the initiator calculates the optimal feature and feature threshold, sets the blinding value, and notifies the partner holding the corresponding feature, and the feature holder divides the current node sample space according to the received feature and feature threshold, and notifies the initiator of the result;
[0073] The label prediction module 13 is used to predict the label of the object to be predicted by using the longitudinal federated learning model obtained through the above training.
[0074] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method embodiment when executing the computer program.
[0075] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method embodiment are implemented.
[0076] Since the principles of the technology solved by the above-mentioned device, electronic device and computer-readable storage medium embodiments are similar to those of the above-mentioned method embodiments, the implementation of the device, electronic device and computer-readable storage medium can refer to the above-mentioned method embodiments, and the repeated parts will not be repeated.
[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A vertical federated learning training and prediction method based on a tree structure, applied to an initiator and multiple partners, characterized in that: The method comprises the following steps: The initiator uses the feature data it holds to create the first decision tree; The initiator and the partner collaborate to establish the mth decision tree in turn, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains a vertical federated learning model based on a tree structure; in the process of establishing the mth decision tree, for each node sample space, the initiator calculates the optimal feature and feature threshold, sets the blinding value, and notifies the partner holding the corresponding feature. The feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the result; Use the longitudinal federated learning model obtained through the above training to predict the label of the object to be predicted; Among them, the initiator has the public key PK and the private key SK, and shares the public key PK with the partner; The creation process of the mth decision tree established by the initiator and the partner is as follows: Step 1: The initiator selects the best feature from the features it holds as the root node of the tree and divides the sub-node sample space; uses the public key PK to encrypt the gradient of the training sample loss function, and sends the sample loss function gradient ciphertext and the sub-node sample space to each partner; Step 2: The partner partitions the current node sample space based on its own features, aggregates the gradient ciphertext of the samples in the same partition and sends each aggregation result to the initiator; Step 3: For each node sample space, the initiator calculates the optimal feature and feature threshold, and arbitrarily selects a feature and feature threshold as the blinding value, and sends the above optimal feature and feature threshold as well as the feature and feature threshold as the blinding value to the partner holding the corresponding feature; Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and locally stores the features and feature thresholds; Step 5: The initiator updates the model using the real node sample space division results, and sends all division results to all partners; Step 6: For all received node sample spaces, the partners aggregate the gradient ciphertext according to the method in step 2 and send it to the initiator; Step 7: For the real node sample space, the initiator calculates the optimal features and feature thresholds; For the node sample space used as the blind value, the initiator arbitrarily selects a feature and feature threshold, and sends the above features and feature thresholds to the corresponding feature holder; Repeat steps 4-7 until the preset stop condition is reached, and the creation of the mth tree is completed.
2. The tree-structured vertical federated learning training and prediction method according to claim 1, characterized in that: The use of the longitudinal federated learning model obtained by the above training to predict the label of the object to be predicted specifically includes: Step 1: For the object to be predicted, the partner uses the features and feature thresholds selected by the partner during the model training phase to compare with the corresponding feature values of the object to be predicted, and sends all comparison results to the initiator; Step 2: The initiator performs deblinding on the comparison result sets from each partner to obtain the comparison results of the real features and feature thresholds; Step 3: The initiator uses the model obtained in the training phase and the actual comparison results to predict the object label.
3. A vertical federated learning training and prediction device based on a tree structure, applied to an initiator and multiple partners, characterized in that: The device includes: The first decision tree building module is used by the initiator to create the first decision tree using the feature data it holds; The m-th decision tree establishment module is used for the initiator and the partner to collaborate and sequentially establish the m-th decision tree, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains a vertical federated learning model based on a tree structure; in the process of establishing the m-th decision tree, for each node sample space, the initiator calculates the optimal feature and feature threshold, sets the blinding value, and notifies the partner holding the corresponding feature. The feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the result; A label prediction module, used to predict the label of the object to be predicted using the longitudinal federated learning model obtained through the above training; Among them, the initiator has the public key PK and the private key SK, and shares the public key PK with the partner; The creation process of the mth decision tree established by the initiator and the partner is as follows: Step 1: The initiator selects the best feature from the features it holds as the root node of the tree and divides the sub-node sample space; uses the public key PK to encrypt the gradient of the training sample loss function, and sends the sample loss function gradient ciphertext and the sub-node sample space to each partner; Step 2: The partner partitions the current node sample space based on its own features, aggregates the gradient ciphertext of the samples in the same partition and sends each aggregation result to the initiator; Step 3: For each node sample space, the initiator calculates the optimal feature and feature threshold, and arbitrarily selects a feature and feature threshold as the blinding value, and sends the above optimal feature and feature threshold as well as the feature and feature threshold as the blinding value to the partner holding the corresponding feature; Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and locally stores the features and feature thresholds; Step 5: The initiator updates the model using the real node sample space division results, and sends all division results to all partners; Step 6: For all received node sample spaces, the partners aggregate the gradient ciphertext according to the method in step 2 and send it to the initiator; Step 7: For the real node sample space, the initiator calculates the optimal features and feature thresholds; For the node sample space used as the blind value, the initiator arbitrarily selects a feature and feature threshold, and sends the above features and feature thresholds to the corresponding feature holder; Repeat steps 4-7 until the preset stop condition is reached, and the creation of the mth tree is completed.
4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 1 or 2 are implemented.
5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 or 2 are implemented.
Citation Information
Patent Citations
Citizen credit score evaluation method applying Xgboost modeling
CN111724174A
Chi-square box separation method based on secure multi-party calculation
CN113688354A