Federated learning training and prediction method and device based on XGBoost

Abstract

The present invention provides a federated learning training and prediction method and device based on XGBoost, which is applied to an initiator and multiple partners. The method includes the following steps: the initiator uses the feature data held by itself to create the first decision tree; the initiator and the partner cooperate to sequentially establish the mth decision tree, m=2,3,...,M, where M is the total number of decision trees trained, and the training obtains a federated learning model based on XGBoost; in the process of establishing the mth decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and simultaneously matches an additional node sample space as a false node sample space for each group of optimal features and feature thresholds, and sends it to the corresponding feature holder, the feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the result; the federated learning model obtained by the above training is used to predict the label of the object to be predicted. The present invention reduces the risk of model leakage and has a fast model prediction speed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a federated learning training and prediction method and device based on XGBoost. Background Art

[0002] With the continuous development of big data and artificial intelligence, people's demand for data privacy protection is increasing. In traditional scenarios, the entire data set is stored in the same organization or device, and the results are obtained through centralized training. However, in real scenarios, a large amount of useful data is distributed in different organizations or devices. If sensitive data from different devices or organizations can be gathered under the premise of meeting data privacy and security, and a global model can be trained, the "data island" problem can be solved and a better experience can be obtained. Federated learning is mainly divided into three categories: horizontal federation, vertical federation, and federated migration. When different institutions or organizations have less overlap in data features but more overlap in user IDs, vertical federation is used for learning.

[0003] In the process of vertical federated modeling, only the initiator can have a global model and labels, while the party that only uses its own data for training and cannot have a global model and labels is called a partner. Vertical federated learning is often used in risk control and marketing. In traditional centralized learning, machine learning algorithms are widely used for their good accuracy and efficiency. Among them, decision trees are a classic and effective method for classification and regression. The XGBoost algorithm is a gradient boosting algorithm with decision trees as the basis function. It is widely used due to its good performance. The XGBoost algorithm, a machine learning algorithm for centralized learning, is integrated into the federated learning scenario. Through distributed learning, a global model can be trained under the premise of protecting data privacy and security. Scureboost is an advanced solution that integrates the XGBoost algorithm into the federated learning scenario.

[0004] In the Scureboost model training phase, the initiator calculates the current loss function gradient and sends the gradient ciphertext to all partners. The partners aggregate node sample gradient ciphertexts based on their own features. The initiator decrypts the aggregated gradient ciphertext and calculates the optimal segmentation of the node sample space and notifies the optimal segmentation feature holder to divide the node sample space. The selected partner performs the segmentation task and sends the segmentation result to the initiator, who synchronizes the node sample space segmentation result to other partners. Iterate the above node sample space segmentation process to build a single decision tree, and train the model through the gradient boosting method.

[0005] In actual production, not only the accuracy and security of the model need to be considered, but how to optimize the model reasoning performance is also a major difficulty in model implementation and engineering. In the current tree-structured model prediction method, the initiator obtains the predicted object label based on the training model through multiple rounds of communication with the partner. This is a classic method, but in practice, multiple rounds of communication will bring a serious communication burden. In addition, the participants can first use the local model for reasoning and then submit it to the initiator for intersection to obtain the object label. Although this method only requires one round of communication between the initiator and the partner, the communication volume and the partner's calculation amount will increase significantly.

[0006] The existing models have the following main defects:

[0007] 1. During the model training phase, the initiator shares the node sample space with the partner, which will cause model information leakage.

[0008] 2. In the model prediction stage, the model prediction performance is poor. If the label is obtained through multiple rounds of communication between the initiator and the partner, it will bring the burden of communication times. If the label is obtained through a single communication intersection, it will bring the burden of calculation and communication volume. Summary of the invention

[0009] In order to solve at least some of the defects in the prior art, the present invention provides a federated learning training and prediction method and device based on XGBoost.

[0010] The present invention is achieved in that:

[0011] In a first aspect, the present invention provides a federated learning training and prediction method based on XGBoost, which is applied to an initiator and multiple partners. The method comprises the following steps:

[0012] The initiator uses the feature data it holds to create the first decision tree;

[0013] The initiator and the partner collaborate to establish the mth decision tree in turn, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains the federated learning model based on XGBoost; in the process of establishing the mth decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and at the same time matches an additional node sample space as a false node sample space for each set of optimal features and feature thresholds, and sends each set of optimal features and feature thresholds and their corresponding two node sample spaces to the corresponding feature holder. The feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the results;

[0014] The federated learning model obtained through the above training is used to predict the label of the object to be predicted.

[0015] Furthermore, the initiator has a public key PK and a private key SK, and shares the public key PK with the partner.

[0016] Furthermore, the creation process of the initiator and the partner collaborating to establish the mth decision tree is as follows:

[0017] Step 1: The initiator selects the best feature from the features it holds as the root node of the tree and divides the sub-node sample space; uses the public key PK to encrypt the training sample loss function gradient, and sends the sample loss function gradient ciphertext and sub-node sample space to each partner;

[0018] Step 2: The partner partitions the current node sample space according to its own characteristics, aggregates the gradient ciphertexts of samples in the same partition and sends each aggregation result to the initiator;

[0019] Step 3: The initiator calculates the optimal features and feature thresholds of each real node sample space, and matches an additional node sample space as a fake node sample space for each set of optimal features and feature thresholds, and sends each set of optimal features and feature thresholds and their corresponding two node sample spaces to the corresponding feature holder;

[0020] Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and stores the features and feature thresholds locally;

[0021] Step 5: The initiator updates the model using the actual segmentation results and sends all segmentation results to all partners;

[0022] Repeat steps 2 to 5 until the preset stopping condition is reached and the creation of the mth tree is completed.

[0023] Furthermore, when the initiator randomly matches the node sample space with the optimal feature and feature threshold, the false node sample space is preferentially selected.

[0024] Furthermore, the method of using the federated learning model obtained through the above training to predict the label of the object to be predicted specifically includes:

[0025] Step 1: The partner obtains the ID of the object to be predicted, uses the optimal features and feature thresholds obtained through training to compare with the corresponding feature values ​​of the object, and sends the results to the initiator;

[0026] Step 2: The initiator uses the comparison result set from the partner and the trained federated learning model to predict the object label.

[0027] In a second aspect, the present invention provides a federated learning training and prediction device based on XGBoost, which is applied to an initiator and multiple partners, and the device includes:

[0028] The first decision tree building module is used by the initiator to create the first decision tree using the feature data it holds;

[0029] The m-th decision tree establishment module is used for the initiator and the partner to collaborate and sequentially establish the m-th decision tree, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the federated learning model based on XGBoost is obtained through training; in the process of establishing the m-th decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and at the same time matches an additional node sample space as a false node sample space for each set of optimal features and feature thresholds, and sends each set of optimal features and feature thresholds and their corresponding two node sample spaces to the corresponding feature holder, and the feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the results;

[0030] The label prediction module is used to predict the label of the object to be predicted using the federated learning model obtained through the above training.

[0031] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The XGBoost-based federated learning training and prediction method and device provided by the present invention, in the model training stage, the partners obtain a blinded node sample space, reducing the risk of model leakage; in the model prediction stage, a fast prediction solution is provided, that is, the initiator only needs to communicate with each partner once to obtain the label of the object to be predicted without increasing the additional computing power and communication volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a federated learning training and prediction method based on XGBoost provided in an embodiment of the present invention;

[0036] Figure 2 A flowchart of establishing the mth decision tree provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a decision tree obtained by an initiator provided in an embodiment of the present invention;

[0038] Figure 4 A model prediction flow chart provided for an embodiment of the present invention;

[0039] Figure 5 A block diagram of a federated learning training and prediction device based on XGBoost provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the embodiment of the present invention provides a federated learning training and prediction method based on XGBoost, with C+1 participants, including 1 initiator A, C partners B1,...B C , where only the initiator has the sample label information, and the partner B i Possess p i features, the initiator A has p0 features, then p0+p1+…+p c = n. Input before training: training data set T = {(x1, y1), (x2, y2), ..., (x N ,y N )}, Loss function L(y,f(x)), training sample ID set G = {1,2,...,N}.

[0042] The method comprises the following steps:

[0043] S101, the initiator A uses the p0 feature data it holds to create the first decision tree;

[0044] S102, the initiator and the partner collaborate to establish the mth decision tree in turn, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the federated learning model based on XGBoost is obtained through training; in the process of establishing the mth decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and at the same time matches an additional node sample space as a pseudo node sample space for each set of optimal features and feature thresholds, and the pseudo node sample space is a blinding item, and each set of optimal features and feature thresholds and their corresponding two node sample spaces are sent to the corresponding feature holder, and the feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the result;

[0045] To refine the above steps, the initiator has the public key PK and the private key SK, and shares the public key PK with the partner.

[0046] like Figure 2 As shown, the creation process of the initiator and the partner collaborating to establish the mth decision tree is as follows:

[0047] Step 1: The initiator A selects the best feature from the p0 features it holds as the root node of the mth decision tree, and divides the N samples into two sub-nodes to form two sub-node sample spaces G1 and G2, satisfying G1∪G2=G; uses the public key PK to encrypt the gradient of the training sample loss function [[g i ]],[[h i ]],i=1,2,...,N.(g i ,h i They represent the first-order partial derivative and the second-order partial derivative of the loss function L with respect to f(x), respectively, and [[]] represents homomorphic encryption). The sample loss function gradient ciphertext and the sub-node sample space are sent to each partner;

[0048] Step 2: The partner partitions the current node sample space according to its own characteristics, aggregates the gradient ciphertexts of samples in the same partition (feature k, feature value v) and converts each aggregation result [[G kv ]],[[H kv ]]Sent to the initiator;

[0049] Step 3: The initiator calculates the sample space G of each real node i The optimal feature k opt and feature threshold v opt , and is the optimal feature and feature threshold for each group (k opt ,v opt ) additionally matches a node sample space G i 'As a pseudo node sample space, each set of optimal features and feature thresholds and their corresponding two node sample spaces (k opt ,v opt ,G i ,G i ')Send to the holder of feature k opt The partner is the corresponding feature holder;

[0050] Step 4: The feature holder receives opt ,v opt ,G i ,G i '), using features and feature thresholds (k opt ,v opt ) divide the current node sample space G i ,Gi ', send the division result to the initiator, store the features and feature thresholds locally, and record ID, feature attributes, and feature thresholds;

[0051] Step 5: The initiator updates the model using the actual segmentation results and sends all segmentation results to all partners;

[0052] Repeat steps 2 to 5 until the preset stop condition is reached to complete the creation of the mth tree. The preset stop condition can be the depth of the tree, the expected convergence value, the training time, etc. All decision trees are created in sequence according to the XGboost algorithm.

[0053] Furthermore, when the initiator randomly matches the node sample space with the optimal feature and feature threshold, the false node sample space is preferentially selected.

[0054] In the model training phase, the initiator matches the false node sample space with the optimal features and feature thresholds, making it impossible for the partner to obtain the real node sample space during the training phase, thereby reducing the risk of model leakage.

[0055] In the above method, the decision tree obtained by the initiator is as follows Figure 3 shown.

[0056] The initiator provides 2 sets of features and feature thresholds [A, 1], [A, 2] to divide the node sample space, the partner B1 provides 3 sets of features and feature thresholds [B1, 1], [B1, 2], [B1, 3] to divide the node sample space, and the partner B2 provides 3 sets of features and feature thresholds [B2, 1], [B2, 2], [B2, 3] to divide the node sample space.

[0057] The feature and feature threshold information table obtained by partner B1 is as follows:

[0058]

[0059] S103: Use the federated learning model obtained through the above training to predict the label of the object to be predicted.

[0060] Specifically, Figure 4 As shown, the method of using the federated learning model obtained by the above training to predict the label of the object to be predicted specifically includes:

[0061] Step 1: The partner obtains the ID of the object to be predicted, uses the optimal features and feature thresholds obtained through training to compare with the corresponding feature values ​​of the object, and sends the results to the initiator;

[0062] Step 2: The initiator uses the comparison result set from the partner and the trained federated learning model to predict the object label.

[0063] In the model prediction stage, since the features and feature thresholds selected by each partner are real, the calculations made by the partner are effective and necessary, and no redundant computing and communication tasks are generated. The efficiency of the partners in the prediction stage is maximized. Because the initiator only needs to communicate with each partner in one round to obtain the predicted object label, this prediction method has good performance.

[0064] The comparison result table of partner B1 for sample X is as follows:

[0065]

[0066]

[0067] like Figure 5 As shown, an embodiment of the present invention further provides a federated learning training and prediction device based on XGBoost, which is applied to an initiator and multiple partners, and the device includes:

[0068] The first decision tree building module 11 is used for the initiator to create the first decision tree using the feature data it holds;

[0069] The m-th decision tree establishment module 12 is used for the initiator and the partner to collaborate to sequentially establish the m-th decision tree, m=2,3,...,M, where M is the total number of decision trees trained, and the federated learning model based on XGBoost is obtained through training; in the process of establishing the m-th decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and at the same time matches an additional node sample space as a false node sample space for each group of optimal features and feature thresholds, and sends each group of optimal features and feature thresholds and their corresponding two node sample spaces to the corresponding feature holder, and the feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the results;

[0070] The label prediction module 13 is used to predict the label of the object to be predicted by using the federated learning model obtained through the above training.

[0071] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method embodiment when executing the computer program.

[0072] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method embodiment are implemented.

[0073] Since the principles of the technology solved by the above-mentioned device, electronic device and computer-readable storage medium embodiments are similar to those of the above-mentioned method embodiments, the implementation of the device, electronic device and computer-readable storage medium can refer to the above-mentioned method embodiments, and the repeated parts will not be repeated.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A federated learning training and prediction method based on XGBoost, applied to an initiator and multiple partners, characterized in that: The method comprises the following steps: The initiator uses the feature data it holds to create the first decision tree; The initiator and the partner collaborate to establish the mth decision tree in turn, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains the federated learning model based on XGBoost; in the process of establishing the mth decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and at the same time matches an additional node sample space as a false node sample space for each set of optimal features and feature thresholds, and sends each set of optimal features and feature thresholds and their corresponding two node sample spaces to the corresponding feature holder. The feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the results; the feature holder is the partner; The federated learning model obtained by the above training is used to predict the label of the object to be predicted. The initiator has the public key PK and the private key SK, and shares the public key PK with the partner. The creation process of the initiator and the partner collaborating to establish the mth decision tree is as follows: Step 1: The initiator selects the best feature from the features it holds as the root node of the tree and divides the sub-node sample space; uses the public key PK to encrypt the gradient of the training sample loss function, and sends the sample loss function gradient ciphertext and the sub-node sample space to each partner; Step 2: The partner partitions the current node sample space based on its own features, aggregates the gradient ciphertext of the samples in the same partition and sends each aggregation result to the initiator; Step 3: The initiator calculates the optimal features and feature thresholds of each real node sample space, At the same time, an additional node sample space is matched for each set of optimal features and feature thresholds as a false node sample space, and each set of optimal features and feature thresholds and their corresponding two node sample spaces are sent to the corresponding feature holders; Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and stores the features and feature thresholds locally; Step 5: The initiator updates the model using the real division results, and sends all division results to all partners; Repeat steps 2 to 5 until the preset stop condition is reached, and the creation of the mth tree is completed; The specific method of using the federated learning model obtained through the above training to predict the label of the object to be predicted includes: Step 1: The partner obtains the ID of the object to be predicted, uses the optimal features and feature thresholds obtained through its own training to compare with the corresponding feature values ​​of the object, and sends the results to the initiator; Step 2: The initiator uses the comparison result set from the partner and the trained federated learning model to predict the object label.

2. The XGBoost-based federated learning training and prediction method according to claim 1, characterized in that: When the initiator randomly matches the node sample space with the optimal feature and feature threshold, the false node sample space is preferred.

3. A federated learning training and prediction device based on XGBoost, applied to an initiator and multiple partners, characterized in that: The device includes: The first decision tree building module is used by the initiator to create the first decision tree using the feature data it holds; The m-th decision tree establishment module is used for the initiator and the partner to collaborate and sequentially establish the m-th decision tree, m = 2, 3, ..., M, where M is the total number of decision trees trained, and the training obtains the XGBoost-based federated learning model; in the process of establishing the m-th decision tree, the initiator calculates the optimal features and feature thresholds of each real node sample space, and at the same time matches an additional node sample space as a false node sample space for each set of optimal features and feature thresholds, and sends each set of optimal features and feature thresholds and their corresponding two node sample spaces to the corresponding feature holder, and the feature holder divides the current node sample space according to the received features and feature thresholds, and notifies the initiator of the results; the feature holder is the partner; A label prediction module is used to predict the label of the object to be predicted using the federated learning model obtained through the above training; Among them, the initiator has the public key PK and the private key SK, and shares the public key PK with the partner; The creation process of the initiator and the partner collaborating to establish the mth decision tree is as follows: Step 1: The initiator selects the best feature from the features it holds as the root node of the tree and divides the sub-node sample space; uses the public key PK to encrypt the gradient of the training sample loss function, and sends the sample loss function gradient ciphertext and the sub-node sample space to each partner; Step 2: The partner partitions the current node sample space based on its own features, aggregates the gradient ciphertext of the samples in the same partition and sends each aggregation result to the initiator; Step 3: The initiator calculates the optimal features and feature thresholds of each real node sample space, At the same time, an additional node sample space is matched for each set of optimal features and feature thresholds as a false node sample space, and each set of optimal features and feature thresholds and their corresponding two node sample spaces are sent to the corresponding feature holders; Step 4: The feature holder divides the current node sample space according to the received features and feature thresholds, sends the division results to the initiator, and stores the features and feature thresholds locally; Step 5: The initiator updates the model using the real division results, and sends all division results to all partners; Repeat steps 2 to 5 until the preset stop condition is reached, and the creation of the mth tree is completed; The specific method of using the federated learning model obtained through the above training to predict the label of the object to be predicted includes: Step 1: The partner obtains the ID of the object to be predicted, uses the optimal features and feature thresholds obtained through its own training to compare with the corresponding feature values ​​of the object, and sends the results to the initiator; Step 2: The initiator uses the comparison result set from the partner and the trained federated learning model to predict the object label.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 1 or 2 are implemented.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 or 2 are implemented.

Images