Efficient mask aggregation learning method for federal gradient boosting tree

Abstract

The invention provides a federal gradient boosting tree-oriented efficient mask aggregation learning method, which adopts a mode of combining local data preprocessing and centralized training, participants send data to a server after executing a secret extreme value bucket construction algorithm and an erasable label mask generation algorithm, and the server carries out centralized training, so that the data are acquired; only one round of communication exists between the participants and the server; encrypting by adding masks, wherein the masks counteract each other in a model training stage; the secret extreme value bucket construction algorithm is used for constructing a global sub-bucket structure and confusing feature values; the erasable label mask generation algorithm is used for adding a mask to a sample label.

Description

Technical Field

[0001] The present invention belongs to the technical field of information security, and in particular relates to an efficient mask aggregation learning method for a federated gradient boosting tree. Background Art

[0002] Gradient Boosting Decision Trees (GBDT) is an ensemble learning algorithm that minimizes the loss function by iteratively building decision trees, where each tree attempts to correct the errors of the previous tree, thereby gradually improving the performance of the model. This method performs well in various machine learning tasks, especially in classification and regression problems, and is favored for its strong predictive power and ability to handle complex data. High-quality data is critical to the performance of gradient boosting trees. In reality, high-quality datasets usually come from multiple data sources. Traditional gradient boosting tree implementations rely on centralized data processing, which, however, becomes undesirable when dealing with multiple data sources due to the release of multiple privacy protection regulations.

[0003] To solve this problem, the existing technology proposes the Federated Gradient Boosting Tree method (Federated GBDT), which allows them to collaboratively train the gradient boosting tree without disclosing or collecting the original data of each party. However, the existing Federated Gradient Boosting Tree framework requires a lot of communication to create each subtree because each participant trains the data locally. Moreover, these methods usually involve complex secure multi-party computing or homomorphic encryption technology, which makes the training inefficient and leads to reduced model accuracy when the data is unevenly distributed. Summary of the invention

[0004] Federated gradient boosting tree is a distributed machine learning technology that allows multiple participants to jointly train gradient boosting tree models without sharing original data, thereby protecting data privacy. However, current federated gradient boosting tree schemes require multiple rounds of communication to obtain the final model. In order to reduce the communication overhead of federated gradient boosting trees, the present invention proposes a masked aggregation learning (MAL) scheme for federated gradient boosting trees. The scheme combines distributed data preprocessing and centralized training to ensure the security, accuracy and efficiency of the scheme, and is not affected by data heterogeneity. It has the characteristics of low computational complexity, low communication overhead and high model accuracy.

[0005] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0006] An efficient masked aggregation learning method for federated gradient boosting trees: a mode combining local data preprocessing and centralized training is adopted. After the participants execute the secret extreme value bucket construction algorithm and the erasable label mask generation algorithm, the data is sent to the server, which is centrally trained by the server. There is only one round of communication between the participants and the server. Encryption is performed by adding masks, and the masks cancel each other out during the model training stage. The secret extreme value bucket construction algorithm is used to construct a global bucket structure and confuse eigenvalues. The erasable label mask generation algorithm is used to add masks to sample labels.

[0007] Furthermore, after the participants have successively executed the secret extreme value bucket construction algorithm and the erasable label mask generation algorithm, they send all obfuscated feature values ​​and masked label values ​​to the server; the server trains the gradient boosting tree model and sends it to all participants.

[0008] Furthermore, in the secret extreme value bucket construction algorithm, suppose that participant P j Hold n j samples, each sample has d eigenvalues ​​X = {x 1 ,x 2 ,…,x d}; The participants negotiate a unified order-preserving encryption key, including the master key msk and the comparison key ck; the comparator only holds the comparison key ck; the comparator and the participants respectively perform the following steps to construct a global bucket structure and obfuscate the eigenvalues:

[0009] (1) Participant P j Select the local maximum of all features of local data and minimum value And send it to the comparator after encrypting it with msk;

[0010] (2) The comparator uses ck to compare the encrypted local feature maximum of all participants and minimum value Get the encrypted global maximum value c(max i ) and the minimum value c(min i ) and returns it to all participants;

[0011] (3) Participant P j Receive c(max i ) and c(min i ), use msk to decrypt and obtain the global maximum value of the feature max i and minimum value;

[0012] (4) Participant P j Calculate the bucket quantiles for each feature, where the kth quantile is q is the number of buckets for each feature;

[0013] (5) Participant P j Confusion is performed on all feature values ​​of the held samples: for each sample, if the i-th feature value x i Belong to the bucket Let

[0014] Furthermore, in the erasable label mask generation algorithm, let participant P j Hold n j samples, each with a label y; after executing the secret extreme bucket construction algorithm, each feature has q buckets, and d features have a total of Q = q × d buckets; each participant P j The following steps are performed to add a mask to the sample label:

[0015] (1) Participant P j Set an unknown vector X = [x1, x2, ..., x Q+1 ];

[0016] (2) Participant P j For each sample Label y i Generate a random vector y i The mask is mask i =X·A i ;

[0017] (3) Participant P j For each bucket B k Set a mask aggregation vector A k =[a1,a2,…,a Q+1 ] T , where a1, a2, …, aQ+1 are initially set to 0;

[0018] (4) Participant P j Check each sample Is it B? k , if it belongs to, the mask aggregation vector is incremented according to the following rules: A k =A k +A i ;

[0019] (5) Participant P j Solve a homogeneous system of linear equations: Each homogeneous equation is expanded into a1·x1+a2·x2+…+a Q+1 ·x Q+1 = 0, and we get X = [x1, x2, …, x Q+1];

[0020] (6) Participant P j For each sample Label y i Add mask: y′ i =y i +X·A i .

[0021] And, an efficient mask aggregation learning system for federated gradient boosting trees: comprising participants and servers; the participants send data to the server after executing a secret extreme value bucket construction module and an erasable label mask generation module, and the server performs centralized training through a centralized training module, with only one round of communication between the participants and the server; the secret extreme value bucket construction module is used to construct a global bucket structure and confuse eigenvalues; the erasable label mask generation module is used to add masks to sample labels, and the masks offset each other during the model training phase.

[0022] Furthermore, after the participants have successively executed the secret extreme value bucket construction algorithm and the erasable label mask generation algorithm, all obfuscated feature values ​​and masked label values ​​are sent to the server; the centralized training module trains the gradient boosting tree model and sends it to all participants.

[0023] Furthermore, in the secret extreme value bucket construction module, let participant P j Hold n j samples, each sample has d eigenvalues ​​X = {x 1 ,x 2 ,…,x d}; The participants negotiate a unified order-preserving encryption key, including the master key msk and the comparison key ck; the comparator only holds the comparison key ck; the comparator and the participants respectively perform the following steps to construct a global bucket structure and obfuscate the eigenvalues:

[0024] (1) Participant P j Select the local maximum of all features of local data and minimum value And send it to the comparator after encrypting it with msk;

[0025] (2) The comparator uses ck to compare the encrypted local feature maximum of all participants and minimum value Get the encrypted global maximum value c(max i ) and the minimum value c(min i ) and returns it to all participants;

[0026] (3) Participant P j Receive c(maxi ) and c(min i ), use msk to decrypt and obtain the global maximum value of the feature max i and minimum value;

[0027] (4) Participant P j Calculate the bucket quantiles for each feature, where the kth quantile is q is the number of buckets for each feature;

[0028] (5) Participant P j Confusion is performed on all feature values ​​of the held samples: for each sample, if the i-th feature value x i Belong to the bucket Let

[0029] Furthermore, in the erasable label mask generation module, it is assumed that the participant P j Hold n j samples, each with a label y; after executing the secret extreme bucket construction algorithm, each feature has q buckets, and d features have a total of Q = q × d buckets; each participant P j The following steps are performed to add a mask to the sample label:

[0030] (1) Participant P j Set an unknown vector X = [x1, x2, ..., x Q+1 ];

[0031] (2) Participant P j For each sample Label y i Generate a random vector y i The mask is mask i =X·A i ;

[0032] (3) Participant P j For each bucket B k Set a mask aggregation vector A k =[a1,a2,…,a Q+1 ] T , where a1, a2, …, a Q+1 The initial value is 0;

[0033] (4) Participant P j Check each sample Is it B? k , if it belongs to, the mask aggregation vector is incremented according to the following rules: A k =A k +Ai ;

[0034] (5) Participant P j Solve a homogeneous system of linear equations: Each homogeneous equation is expanded into a1·x1+a2·x2+…+a Q+1 ·x Q+1 = 0, and we get X = [x1, x2, …, x Q+1 ];

[0035] (6) Participant P j For each sample Label y i Add mask: y' i =y i +X·A i .

[0036] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of an efficient mask aggregation learning method for a federated gradient boosting tree as described above are implemented.

[0037] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an efficient mask aggregation learning method for a federated gradient boosting tree as described above.

[0038] Compared with the prior art, the present invention and its preferred embodiments have at least the following advantages:

[0039] First, the existing federated learning solutions, such as the Chinese patent "Federated Learning Gradient Attack Defense Method, System, Device and Medium" (CN 115222057A), "Dynamic Semi-decentralized Federated Learning Privacy Protection Security Aggregation Method and System" (CN 118413383A), etc., require multiple rounds of communication between participants and the server, especially when facing non-independent and identically distributed data, which may require hundreds of communications. The present invention adopts a mode combining local data preprocessing and centralized training, that is, participants execute the secret extreme value bucket construction algorithm and the erasable label mask generation algorithm and then send the data to the server, which is trained centrally by the server. Only one round of communication is required between participants and the server, which reduces communication overhead and improves the efficiency of model training.

[0040] Secondly, in order to further protect data privacy, federated learning solutions are usually combined with homomorphic encryption and secure multi-party computing technology, resulting in huge communication and computing overheads, which makes it difficult to meet the high-performance, low-power application requirements in fields such as cloud computing and big data, especially in the application scenarios involving lightweight IoT devices or mobile terminal devices. In addition, there are some solutions that use the method of adding noise (such as the invention of "federated learning gradient attack defense method, system, device and medium") to protect data privacy, but the added noise will reduce the accuracy of the model. The present invention only uses a lightweight encryption algorithm to reduce computing overhead; it uses the method of adding masks to protect data privacy, and the masks offset each other during the model training stage, which does not affect the accuracy of the model.

[0041] Finally, the model performance of most solutions will be affected by data heterogeneity, resulting in reduced accuracy. Since the model of the present invention is centrally trained by the server, its model performance is not affected by data heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] Figure 1 This is a flow chart of a dense extreme value bucket construction algorithm according to an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of a label mask generation algorithm according to an embodiment of the present invention;

[0045] Figure 3 The figure is a flow chart of a centralized training algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0047] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] In order to facilitate the detailed description of the present invention, the relevant basic concepts are first described in a unified manner.

[0050] Symbols and definitions

[0051] P j : The jth participant.

[0052] d: The number of features.

[0053] f i : The ith feature.

[0054] The local maximum of the i-th feature on the j-th participant.

[0055] The local minimum of the i-th feature on the j-th participant.

[0056] max i : The global maximum of the ith feature.

[0057] min i : The global minimum of the ith feature.

[0058] q: The number of buckets for each feature.

[0059] The k-th bucket of the ith feature.

[0060] Q: The total number of buckets.

[0061] x i : the ith eigenvalue.

[0062] B k : The kth bucket.

[0063] The number of samples from the jth participant that belong to the kth bucket.

[0064] y i : The label value of the i-th sample.

[0065] The purpose of the present invention is to achieve an efficient federated gradient boosting tree while ensuring user privacy and model accuracy. In view of the purpose of the present invention, this embodiment proposes a masked aggregation learning scheme for a federated gradient boosting tree, which is described in detail below.

[0066] The implementation of this scheme includes three algorithms: secret extreme value bucket construction algorithm, erasable label mask generation algorithm and centralized training algorithm.

[0067] 1. Secret extreme value bucket construction algorithm

[0068] Participant P j Hold n j samples, each sample has d eigenvalues ​​X = {x 1,x 2 ,…,x d}. The participants negotiate a unified order-preserving encryption key, including the master key msk and the comparison key ck. The comparator only holds the comparison key ck. The comparator and the participants respectively perform the following steps to construct a global bucket structure and obfuscate the eigenvalues:

[0069] (1)P j Select the local maximum of all features of local data and minimum value And send it to the comparator after encrypting it with msk;

[0070] (2) The comparator uses ck to compare the encrypted local feature maximum of all participants and minimum value Get the encrypted global maximum value c(max i ) and the minimum value c(min i ) and returns it to all participants;

[0071] (3)P j Receive c(max i ) and c(min i ), use msk to decrypt and obtain the global maximum and minimum values ​​of the feature max i and min i ;

[0072] (4)P j Calculate the bucket quantiles for each feature, where the kth quantile is

[0073] (5)P j Confusion is performed on all feature values ​​of the held samples: for each sample, if the i-th feature value x i Belong to the bucket Let

[0074] 2. Erasable label mask generation algorithm

[0075] Participant P j Hold n j samples, each with a label y. After executing the secret extreme bucket construction algorithm, each feature has q buckets, and d features have a total of Q = q × d buckets. Each participant P j The following steps are performed to add a mask to the sample label:

[0076] (1)P j Set an unknown vector X = [x1, x2, ..., x Q+1 ];

[0077] (2)Pj For each sample Label y i Generate a random vector y i The mask is mask i =X·A i ;

[0078] (3)P j For each bucket B k Set a mask aggregation vector A k =[a1,a2,…,a Q+1 ] T , where a1, a2, …, a Q+1 The initial value is 0;

[0079] (4)P j Check each sample Is it B? k If it belongs to, then calculate A k =A k +A i ;

[0080] (5)P j Solve a homogeneous system of linear equations: Each homogeneous equation is expanded into a1·x1+a2·x2+…+a Q+1 ·x Q+1 = 0, and we get X = [x1, x2, …, x Q+1 ];

[0081] (6)P j For each sample Label y i Add mask and calculate y' i =y i +X·A i .

[0082] 3. Centralized training algorithm

[0083] After the participants have executed the secret extreme bucket construction algorithm and the erasable label mask generation algorithm, they send all obfuscated feature values ​​and masked label values ​​to the server. The server performs the following operations to train the gradient boosting tree model:

[0084] (1) The server collects data from all participants;

[0085] (2) The server uses this data to train the gradient boosting tree model;

[0086] (3) The server sends the trained model to all participants.

[0087] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0088] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0089] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0090] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

[0091] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of an efficient mask aggregation learning method for federated gradient boosting trees under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. An efficient mask aggregation learning method for federated gradient boosting trees, characterized by: A mode combining local data preprocessing and centralized training is adopted. Participants execute the secret extreme value bucket construction algorithm and the erasable label mask generation algorithm and then send the data to the server for centralized training. There is only one round of communication between participants and the server. Encryption is performed by adding masks, and the masks cancel each other out during the model training phase; the secret extreme bucket construction algorithm is used to construct a global bucket structure and confuse eigenvalues; the erasable label mask generation algorithm is used to add masks to sample labels.

2. The efficient mask aggregation learning method for federated gradient boosting trees according to claim 1, characterized in that: After the participants have executed the secret extreme bucket construction algorithm and the erasable label mask generation algorithm, they send all obfuscated feature values ​​and masked label values ​​to the server; the server trains the gradient boosting tree model and sends it to all participants.

3. The efficient mask aggregation learning method for federated gradient boosting trees according to claim 2, characterized in that: In the secret extreme value bucket construction algorithm, suppose that participant P j Hold n j samples, each sample has d eigenvalues ​​X = {x 1 , x 2 , …, x d }; The participants negotiate a unified order-preserving encryption key, including the master key msk and the comparison key ck; the comparator only holds the comparison key ck; the comparator and the participants respectively perform the following steps to construct a global bucket structure and obfuscate the eigenvalues: (1) Participant P j Select the local maximum of all features of local data and minimum value And send it to the comparator after encrypting it with msk; (2) The comparator uses ck to compare the encrypted local feature maximum of all participants and minimum value Get the encrypted global maximum value c(max i ) and the minimum value c(min i ) and returns it to all participants; (3) Participant P j Receive c(max i ) and c(min i ), use msk to decrypt and obtain the global maximum value of the feature max i and minimum value; (4) Participant P j Calculate the bucket quantiles for each feature, where the kth quantile is q is the number of buckets for each feature; (5) Participant P j Confusion is performed on all feature values ​​of the held samples: for each sample, if the i-th feature value x i Belong to the bucket Let 4. The efficient mask aggregation learning method for federated gradient boosting trees according to claim 3, characterized in that: In the erasable label mask generation algorithm, let participant P j Hold n j samples, each with a label y; after executing the secret extreme bucket construction algorithm, each feature has q buckets, and d features have a total of Q = q × d buckets; each participant P j The following steps are performed to add a mask to the sample label: (1) Participant P j Set an unknown vector X = [x1, x2, ..., x Q+1 ]; (2) Participant P j For each sample Label y i Generate a random vector y i The mask is mask i =X·A i ; (3) Participant P j For each bucket B k Set a mask aggregation vector A k =[a1, a2, …, a Q+1 ] T , where a1, a2, …, a Q+1 The initial value is 0; (4) Participant P j Check each sample Is it B? k , if it belongs to, then the mask aggregation vector is incremented according to the following rules: A k =A k +A i ; (5) Participant P j Solve a homogeneous system of linear equations: Each homogeneous equation has the expanded form of a1·x1+a2·x2+…+a Q+1 ·x Q+1 = 0, and we get X = [x1, x2, …, x Q+1 ]; (6) Participant P j For each sample Label y i Add mask: y′ i =y i +X·A i .

5. An efficient mask aggregation learning system for federated gradient boosting trees, characterized by: Includes participants and servers; The participant sends the data to the server after executing the secret extreme value bucket construction module and the erasable label mask generation module, and the server conducts centralized training through the centralized training module. There is only one round of communication between the participant and the server; the secret extreme value bucket construction module is used to construct a global bucket structure and confuse the eigenvalues; the erasable label mask generation module is used to add masks to sample labels, and the masks offset each other during the model training stage.

6. The efficient mask aggregation learning system for federated gradient boosting trees according to claim 5, characterized in that: After the participants have successively executed the secret extreme value bucket construction algorithm and the erasable label mask generation algorithm, all obfuscated feature values ​​and masked label values ​​are sent to the server; the centralized training module trains the gradient boosting tree model and sends it to all participants.

7. The efficient mask aggregation learning system for federated gradient boosting trees according to claim 6, characterized in that: In the secret extreme value bucket construction module, let participant P j Hold n j samples, each sample has d eigenvalues ​​X = {x 1 , x 2 , …, x d }; The participants negotiate a unified order-preserving encryption key, including the master key msk and the comparison key ck; the comparator only holds the comparison key ck; the comparator and the participants respectively perform the following steps to construct a global bucket structure and obfuscate the eigenvalues: (1) Participant P j Select the local maximum of all features of local data and minimum value And send it to the comparator after encrypting it with msk; (2) The comparator uses ck to compare the encrypted local feature maximum of all participants and minimum value Get the encrypted global maximum value c(max i ) and the minimum value c(min i ) and returns it to all participants; (3) Participant P j Receive c(max i ) and c(min i ), use msk to decrypt and obtain the global maximum value of the feature max i and minimum value; (4) Participant P j Calculate the bucket quantiles for each feature, where the kth quantile is q is the number of buckets for each feature; (5) Participant P j Confusion is performed on all feature values ​​of the held samples: for each sample, if the i-th feature value x i Belong to the bucket Let 8. The efficient mask aggregation learning system for federated gradient boosting trees according to claim 7, characterized in that: In the erasable label mask generation module, let participant P j Hold n j samples, each with a label y; after executing the secret extreme bucket construction algorithm, each feature has q buckets, and d features have a total of Q = q × d buckets; each participant P j The following steps are performed to add a mask to the sample label: (1) Participant P j Set an unknown vector X = [x1, x2, ..., x Q+1 ]; (2) Participant P j For each sample Generate a random vector with the label yi y i The mask is mask i =X·A i ; (3) Participant P j For each bucket B k Set a mask aggregation vector A k =[a1, a2, …, a Q+1 ] T , where a1, a2, …, a Q+1 The initial value is 0; (4) Participant P j Check each sample Is it B? k , if it belongs to, then the mask aggregation vector is incremented according to the following rules: A k =A k +A i ; (5) Participant P j Solve a homogeneous system of linear equations: Each homogeneous equation has the expanded form of a1·x1+a2·x2+…+a Q+1 ·x Q+1 = 0, and we get X = [x1, x2, …, x Q+1 ]; (6) Participant P j For each sample Label y i Add mask: y′ i =y i +X·A i .

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of an efficient mask aggregation learning method for a federated gradient boosting tree as described in any one of claims 1-4 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of an efficient mask aggregation learning method for a federated gradient boosting tree as described in any one of claims 1 to 4 are implemented.

Images