Training method, device, electronic device and storage medium for federated learning model

Abstract

The present application proposes a training method, device, electronic device and storage medium for a federated learning model, wherein the training method includes: performing sample alignment with a data provider server; numbering the features of the business server and the data provider server according to the number of features of the business server and the data provider server respectively to generate a feature coding set, and sending the feature number and public key of the data provider server to the data provider server; obtaining the current sample set and training parameter set of the federated learning model; performing M iterations of training on the federated learning model according to the current sample set, training parameter set and feature coding set; and obtaining the target parameters of the federated learning model obtained by the Mth iteration training. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, thereby making the joint training between the business server and the data provider server more efficient, thereby improving the modeling efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a training method, device, electronic device and storage medium for a federated learning model. Background Art

[0002] With the development of machine learning, more and more machine learning technologies are being applied to various industries. The quantity and quality of data often determine the upper limit of the effect of machine learning models. However, as laws and regulations become more stringent, and people pay more and more attention to data security and privacy protection, data silos are formed. In such a scenario, federated learning came into being, which allows participants to jointly train without sharing data, solving the problem of data silos.

[0003] Among related technologies, federated learning is an encrypted distributed machine learning technology that integrates multiple technologies such as information encryption, distributed computing, and machine learning. Federated learning can be divided into horizontal federated learning, vertical federated learning, and federated transfer learning based on the characteristics of the data held by the participants. In risk control scenarios, vertical federated learning is more widely used. Summary of the invention

[0004] The first aspect of the present application proposes a method for training a federated learning model, which can reduce the complexity of modeling while ensuring the modeling effect, thereby making the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0005] The second embodiment of the present application proposes a training method for a federated learning model.

[0006] The third aspect of the present application provides a training device for a federated learning model.

[0007] The fourth embodiment of the present application proposes a training device for a federated learning model.

[0008] A fifth aspect of the present application provides an electronic device.

[0009] The sixth aspect of the present application provides a computer-readable storage medium.

[0010] The first embodiment of the present application proposes a method for training a federated learning model, including:

[0011] Perform sample alignment with the data provider's server;

[0012] Respectively obtaining the number of features of the business party server and the data provider server, and numbering the features of the business party server and the data provider server according to the number of features to generate a feature coding set, and sending the feature number and public key of the data provider server to the data provider server;

[0013] Get the current sample set and training parameter set of the federated learning model;

[0014] According to the current sample set, the training parameter set and the feature encoding set, the federated learning model is iterated M times, where M is a positive integer greater than 1; and

[0015] Obtain target parameters of the federated learning model obtained by the M-th iterative training.

[0016] According to the training method of the federated learning model of the embodiment of the present application, firstly, sample alignment is performed with the data provider server, and then the number of features of the business party server and the data provider server is obtained respectively, and the features of the business party server and the data provider server are numbered respectively according to the number of features to generate a feature coding set, and the feature number and public key of the data provider server are sent to the data provider server, and then the current sample set and training parameter set of the federated learning model are obtained, and according to the current sample set, training parameter set and feature coding set, the federated learning model is iteratively trained M times, and finally the target parameters of the federated learning model obtained by the Mth iterative training are obtained. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0017] In addition, the training method of the federated learning model according to the above embodiment of the present application may also have the following additional technical features:

[0018] In one embodiment of the present application, the training parameter set includes feature sampling rate, training sample upper limit value, training sample lower limit value, decision tree number upper limit value, decision tree number lower limit value, first parameter change speed and second parameter change speed.

[0019] In one embodiment of the present application, each iteration of training includes:

[0020] Using the current iteration training in the M iteration training as the Nth iteration training, wherein N is a positive integer less than M;

[0021] Generate a sample sampling rate according to the M, the N, the training sample upper limit, the training sample lower limit and the first parameter change speed;

[0022] Generate a target number of trees according to the M, the N, the upper limit of the number of decision trees, the lower limit of the number of decision trees, and the second parameter change speed;

[0023] Selecting samples of the sample sampling rate from the current sample set to generate a target training set;

[0024] Selecting the feature code of the feature sampling rate from the feature code set to generate a target feature code set;

[0025] Sending the number of each sample in the target training set and the target feature number of the data provider server in the target feature coding set to the data provider server;

[0026] Generate target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees;

[0027] Based on the gradient boosting algorithm and according to the target parameter and the federated learning model, an optimized label of the current sample is generated, wherein the optimized label is a training label of the current sample for the next round of iterative training.

[0028] In one embodiment of the present application, generating target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees includes:

[0029] Calculating the gradient information of the samples in the target training set, and sending the gradient information to the data provider server;

[0030] Receiving gradient return information provided by the data provider server;

[0031] Generate a target splitting point number according to the gradient return information and the target feature encoding set, generate a ciphertext based on a private key and the target splitting point number, and send the ciphertext to the data provider server;

[0032] Receiving the decryption operation value sent by the data provider server, and performing node splitting according to the decryption operation value;

[0033] Repeat the above steps until the model converges to establish the target number of decision trees, complete the training of the federated learning model, and obtain the target parameters through the trained federated learning model.

[0034] In one embodiment of the present application, the calculating the gradient information of the samples in the target training set includes:

[0035] Generate first-order gradient values ​​and second-order gradient values ​​of samples in the target training set;

[0036] The first-order gradient value and the second-order gradient value are homomorphically encrypted to generate the gradient information.

[0037] In one embodiment of the present application, the gradient return information is multiple, and each of the gradient return information corresponds to a corresponding number, wherein the generating the target splitting point number according to the gradient return information and the target feature encoding set includes:

[0038] Generate corresponding multiple information gains respectively according to the multiple gradient return information and the target feature encoding set;

[0039] A maximum information gain is selected from the multiple information gains, and a number corresponding to the maximum information gain is used as the target split point number.

[0040] In one embodiment of the present application, performing node splitting according to the decryption operation value includes:

[0041] Generate splitting space information according to the decryption operation value;

[0042] Node splitting is performed according to the samples in the target training set and the splitting space information.

[0043] The second aspect of the present application proposes a method for training a federated learning model, including:

[0044] Perform sample alignment with the business server;

[0045] Receiving the characteristic number and public key of the data provider server sent by the business party server;

[0046] Receive the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set;

[0047] Receive the gradient information of the currently trained sample sent by the business party server, and obtain gradient return information according to the gradient information;

[0048] Sending the gradient return information to the business party server;

[0049] Receiving a ciphertext generated based on a private key and a target splitting point number and sent by the business party server, wherein the target splitting point number is generated according to the gradient return information and the target feature code set; and

[0050] The ciphertext is decrypted based on the public key to obtain a decryption operation value, which is sent to the business party server.

[0051] According to the training method of the federated learning model of the embodiment of the present application, firstly, the sample is aligned with the business party server, and then the feature number and public key of the data provider server sent by the business party server are received, and the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set are received, and the gradient information of the currently trained sample sent by the business party server is received, and the gradient return information is obtained according to the gradient information, and then the gradient return information is sent to the business party server, and the ciphertext generated based on the private key and the target split point number sent by the business party server is received, and finally the ciphertext is decrypted based on the public key to obtain the decrypted operation value, and sent to the business party server. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0052] In addition, the training device of the federated learning model according to the above embodiment of the present application may also have the following additional technical features:

[0053] In one embodiment of the present application, the step of obtaining gradient return information according to the gradient information includes:

[0054] Determining a feature set according to the target feature number and the feature number of the data provider server;

[0055] Splitting the sample space according to the splitting threshold corresponding to each feature in the feature set to obtain the splitting space on the specified side;

[0056] According to the gradient information, obtain the gradient summation information of the split space on the specified side corresponding to each feature, and number the gradient summation information;

[0057] The gradient return information is generated by using the gradient summation information and the number of the gradient summation information.

[0058] In one embodiment of the present application, after numbering the gradient sum information, the method further includes:

[0059] Generate the number, and a mapping relationship between the feature corresponding to the number, the split threshold, and the gradient sum information corresponding to the number.

[0060] The third aspect of the present application provides a training device for a federated learning model, including:

[0061] An alignment module, used to align samples with the data provider server;

[0062] A sending module, used to obtain the number of features of the business party server and the data provider server respectively, and number the features of the business party server and the data provider server respectively according to the number of features to generate a feature coding set, and send the feature number and public key of the data provider server to the data provider server;

[0063] A first acquisition module is used to obtain a current sample set and a training parameter set of a federated learning model;

[0064] an iterative training module, configured to perform M iterative training on the federated learning model according to the current sample set, the training parameter set and the feature encoding set, wherein M is a positive integer greater than 1; and

[0065] The second acquisition module is used to obtain the target parameters of the federated learning model obtained by the M-th iterative training.

[0066] The training device of the federated learning model of the embodiment of the present application first performs sample alignment with the data provider server through the alignment module, then obtains the number of features of the business party server and the data provider server respectively through the sending module, and numbers the features of the business party server and the data provider server respectively according to the number of features to generate a feature coding set, and sends the feature number and public key of the data provider server to the data provider server, and then obtains the current sample set and training parameter set of the federated learning model through the first acquisition module, and performs M iterations of training on the federated learning model according to the current sample set, training parameter set and feature coding set through the iterative training module, and finally obtains the target parameters of the federated learning model obtained by the Mth iteration training through the second acquisition module. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0067] In addition, the training device of the federated learning model according to the above embodiment of the present application may also have the following additional technical features:

[0068] In one embodiment of the present application, the training parameter set includes feature sampling rate, training sample upper limit value, training sample lower limit value, decision tree number upper limit value, decision tree number lower limit value, first parameter change speed and second parameter change speed.

[0069] In one embodiment of the present application, the iterative training module includes:

[0070] A setting submodule, used for taking the current iteration training in the M iteration training as the Nth iteration training, wherein N is a positive integer less than M;

[0071] A first generating submodule, used for generating a sample sampling rate according to the M, the N, the upper limit value of the training sample, the lower limit value of the training sample and the first parameter change speed;

[0072] A second generation submodule is used to generate a target number of trees according to the M, the N, the upper limit of the number of decision trees, the lower limit of the number of decision trees and the second parameter change speed;

[0073] A third generating submodule, configured to select samples of the sample sampling rate from the current sample set to generate a target training set;

[0074] A fourth generating submodule, configured to select the feature code of the feature sampling rate from the feature code set to generate a target feature code set;

[0075] A sending submodule, used for sending the number of each sample in the target training set and the target feature number of the data provider server in the target feature coding set to the data provider server;

[0076] a fifth generating submodule, configured to generate target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees;

[0077] The sixth generation submodule is used to generate an optimized label of the current sample based on the gradient boosting algorithm and according to the target parameters and the federated learning model, wherein the optimized label is a training label of the current sample for the next round of iterative training.

[0078] In one embodiment of the present application, the fifth generation submodule includes:

[0079] A calculation unit, used to calculate the gradient information of the samples in the target training set and send the gradient information to the data provider server;

[0080] A receiving unit, configured to receive gradient return information provided by the data provider server;

[0081] A generating unit, configured to generate a target splitting point number according to the gradient return information and the target feature encoding set, generate a ciphertext based on a private key and the target splitting point number, and send the ciphertext to the data provider server;

[0082] A node splitting unit, used for receiving the decryption operation value sent by the data provider server, and performing node splitting according to the decryption operation value;

[0083] The acquisition unit is used to repeat the above steps until the model converges to establish the target number of decision trees, complete the training of the federated learning model, and obtain the target parameters through the trained federated learning model.

[0084] In one embodiment of the present application, the computing unit is specifically used to:

[0085] Generate first-order gradient values ​​and second-order gradient values ​​of samples in the target training set;

[0086] The first-order gradient value and the second-order gradient value are homomorphically encrypted to generate the gradient information.

[0087] In one embodiment of the present application, there are multiple pieces of gradient return information, and each piece of gradient return information has a corresponding number, wherein the generating unit is specifically used to:

[0088] Generate corresponding multiple information gains respectively according to the multiple gradient return information and the target feature encoding set;

[0089] A maximum information gain is selected from the multiple information gains, and a number corresponding to the maximum information gain is used as the target split point number.

[0090] In one embodiment of the present application, the node splitting unit is specifically used to:

[0091] Generate splitting space information according to the decryption operation value;

[0092] Node splitting is performed according to the samples in the target training set and the splitting space information.

[0093] The fourth aspect of the present application provides a training device for a federated learning model, including:

[0094] Alignment module, used to align samples with the business server;

[0095] A first receiving module, used for receiving the characteristic number and public key of the data provider server sent by the business party server;

[0096] A second receiving module is used to receive the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set;

[0097] A third receiving module is used to receive the gradient information of the currently trained sample sent by the business party server, and obtain gradient return information according to the gradient information;

[0098] A first sending module, configured to send the gradient return information to the business party server;

[0099] a fourth receiving module, configured to receive a ciphertext generated based on a private key and a target splitting point number and sent by the business party server, wherein the target splitting point number is generated according to the gradient return information and the target feature code set; and

[0100] The second sending module is used to decrypt the ciphertext based on the public key, obtain a decryption operation value, and send it to the business party server.

[0101] The training device of the federated learning model of the embodiment of the present application first performs sample alignment with the business party server through the alignment module, then receives the feature number and public key of the data provider server sent by the business party server through the first receiving module, and receives the number of each sample in the target training set sent by the business party server and the target feature number of the data provider server in the target feature coding set through the second receiving module, and receives the gradient information of the currently trained sample sent by the business party server through the third receiving module, and obtains the gradient return information according to the gradient information, and then sends the gradient return information to the business party server through the first sending module, and receives the ciphertext generated based on the private key and the target splitting point number sent by the business party server through the fourth receiving module, and finally decrypts the ciphertext based on the public key through the second sending module to obtain the decrypted operation value, and sends it to the business party server. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0102] In addition, the training device of the federated learning model according to the above embodiment of the present application may also have the following additional technical features:

[0103] In one embodiment of the present application, the third receiving module is specifically configured to:

[0104] Determining a feature set according to the target feature number and the feature number of the data provider server;

[0105] Splitting the sample space according to the splitting threshold corresponding to each feature in the feature set to obtain the splitting space on the specified side;

[0106] According to the gradient information, obtain the gradient summation information of the split space on the specified side corresponding to each feature, and number the gradient summation information;

[0107] The gradient return information is generated by using the gradient summation information and the number of the gradient summation information.

[0108] In one embodiment of the present application, the third receiving module is further used to:

[0109] Generate the number, and a mapping relationship between the feature corresponding to the number, the split threshold, and the gradient sum information corresponding to the number.

[0110] The fifth aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the training method of the federated learning model as described in the first aspect embodiment or the second aspect embodiment mentioned above.

[0111] The electronic device of the embodiment of the present application, by executing a computer program stored in a memory through a processor, can reduce the complexity of modeling while ensuring the modeling effect, thereby making the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0112] The sixth aspect embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the training method of the federated learning model as described in the first aspect embodiment or the second aspect embodiment mentioned above is implemented.

[0113] The computer-readable storage medium of the embodiment of the present application, by storing a computer program and being executed by a processor, can reduce the complexity of modeling while ensuring the modeling effect, thereby making the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0114] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0116] Figure 1 A flowchart of a method for training a federated learning model according to an embodiment of the present application;

[0117] Figure 2 A flowchart of a method for training a federated learning model according to another embodiment of the present application;

[0118] Figure 3 A flowchart of a method for training a federated learning model according to another embodiment of the present application;

[0119] Figure 4 A flowchart of a method for training a federated learning model according to another embodiment of the present application;

[0120] Figure 5 A schematic diagram of a training method for a federated learning model according to an embodiment of the present application;

[0121] Figure 6 A flowchart of a method for training a federated learning model according to another embodiment of the present application;

[0122] Figure 7 A schematic diagram of the structure of a training device for a federated learning model according to an embodiment of the present application;

[0123] Figure 8 A schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application; and

[0124] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0125] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0126] The following describes the training method, device, electronic device and storage medium of the federated learning model of the embodiment of the present application with reference to the accompanying drawings.

[0127] The training method of the federated learning model provided in the embodiment of the present application can be executed by an electronic device, which may be a PC (Personal Computer), a tablet computer or a server, etc., without any limitation here.

[0128] In the embodiment of the present application, a processing component, a storage component and a driving component may be provided in the electronic device. Optionally, the driving component and the processing component may be integrated, the storage component may store an operating system, an application program or other program modules, and the processing component implements the training method of the federated learning model provided in the embodiment of the present application by executing the application program stored in the storage component.

[0129] The training method of the federated learning model provided in the embodiment of the present application can be a training method of the federated learning model that integrates random forest based on bagging (guided aggregation algorithm) and GradientBoosting (gradient boosting), wherein each decision tree sub-model of the GBDT (machine learning algorithm) scheme can be replaced by a forest composed of multiple decision trees, and gradient boosting is performed on each layer of the forest. Using a forest to replace a single decision tree has the following advantages:

[0130] (1) Better robustness: Under the premise of parallelism, random forest selects part of the data and part of the features to establish multiple decision trees. Even if some individual decision trees may have poor model effects due to the influence of outliers, the final output of the forest is the comprehensive result of multiple decision trees (in the embodiment of the present application, the output of the random forest can be the average value of the output of each tree model rather than the vote of the classification results), which makes the model more robust to outliers;

[0131] (2) Alleviating the overfitting problem: Random forests can perform random sampling in both feature and sample dimensions. Breima pointed out in a related paper that the upper bound of the generalization error of random forests is in It can be the weighted correlation coefficient of the residual term of the decision tree fitting in the forest, PE * (tree) is the average generalization error of the decision trees in the forest. This formula indicates that a random forest with a small generalization error needs to ensure low correlation between decision tree residuals and low generalization error of a single decision tree. Therefore, from a theoretical analysis, we should pay attention to indicators such as sample sampling rate, sample characteristics, decision tree depth, and number of decision trees. Through these indicators, we can construct a rich and diverse decision tree, reduce the similarity between classifiers, and further control the overfitting problem.

[0132] In the embodiment of the present application, each layer of random forest needs to first determine the samples, features, number of decision trees, etc., and build a decision tree based on the determined results. In order to improve the modeling efficiency, the training method of the federated learning model provided in the embodiment of the present application mainly revolves around the two core parameters of the number of decision trees in each layer of the forest and the sample sampling rate of each layer of the forest, and uses the attenuation and increment strategies of cosine annealing to control the changes of the two core parameters. In theoretical analysis and actual modeling experiments, the cosine annealing strategy is better than the linear attenuation strategy and even better than the exponential attenuation strategy. The reason is that the objective function of the GBF (Gradient Boosting Forest) algorithm is:

[0133]

[0134] in Represents the output of the mth layer random forest, which is the average value of multiple decision trees. is the prediction result of the first t-1 layers of random forest, and in the tth round it is a fixed value, y i is the true label. In this application, the minimum mean square error (MSE) can be used as the loss function, that is:

[0135]

[0136] Since the fitting process Therefore, the loss function value of the algorithm GBF decreases as the number of modeling trees gradually increases, and the rate of decrease gradually slows down. Therefore, the strategy of cosine annealing with deceleration decay just fits the change trend of the algorithm itself, followed by linear decay with uniform decay, and exponential decay with accelerated decay (base greater than 1) is the worst. Therefore, this application can use the cosine function to control the number of trees in each layer of the forest (number of decision trees) and the change of sample sampling rate.

[0137] The following describes in detail the training method of the federated learning model in the embodiment of the present application with reference to the accompanying drawings:

[0138] Figure 1 The figure is a flowchart of a method for training a federated learning model according to an embodiment of the present application.

[0139] The training method of the federated learning model of the embodiment of the present application can also be executed by the training device of the federated learning model provided in the embodiment of the present application, which can be configured in an electronic device to achieve sample alignment with the data provider server, and then obtain the number of features of the business party server and the data provider server respectively, and number the features of the business party server and the data provider server respectively according to the number of features to generate a feature coding set, and send the feature number and public key of the data provider server to the data provider server, and then obtain the current sample set and training parameter set of the federated learning model, and according to the current sample set, training parameter set and feature coding set, perform M iterations of training on the federated learning model, and obtain the target parameters of the federated learning model obtained by the Mth iteration training, so as to make the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0140] As a possible scenario, the training method of the federated learning model of the embodiment of the present application can also be executed on the server side. The server can be a cloud server, and the training method of the federated learning model can be executed in the cloud.

[0141] like Figure 1 As shown, the training method of the federated learning model may include:

[0142] Step 101: Perform sample alignment with a data provider server, where there may be multiple data provider servers.

[0143] In the embodiment of the present application, the business party (i.e., the business party server) can perform sample alignment with the data provider server through a preset method. The preset method can be calibrated according to the actual situation. For example, since the user groups of the partners are not completely overlapped, the encryption-based user sample alignment technology (method) can be used to confirm the common users of both parties without disclosing their respective data, and without exposing non-overlapping users.

[0144] It should be noted that the sample alignment described in this embodiment may also refer to the alignment of sample positions between the business party server and the data provider server to facilitate accurate sample transmission. In addition, during the sample alignment process, a communication channel (channel) may be established between the business party and the data provider server, and the communication channel may be encrypted.

[0145] Step 102, respectively obtain the feature quantities of the business party server and the data provider server, and number the features of the business party server and the data provider server according to the feature quantities to generate a feature coding set, and send the feature number and public key of the data provider server to the data provider server.

[0146] In the embodiment of the present application, the data provider server can actively send its own local feature quantity to the business server. That is, the business server needs to know the number of local features of each data provider server, and the business server can directly obtain the feature quantity of the business server and the data provider server from its own storage space.

[0147] Specifically, after the data provider server and the business server complete sample alignment (for example, there are m samples), the business server can first generate a pair of keys locally, namely a public key and a private key, and synchronize the public key to each business server. The business server can generate a pair of public and private keys (i.e., a public key and a private key) according to a preset key generation algorithm, which can be numbered and recorded as {p , ,p'}, where p , is the public key, p' is the private key, and the preset key generation algorithm can be calibrated according to actual conditions.

[0148] The business party server can then directly obtain the total number of features F from its own storage space, and complete the numbering of the F features (the numbering rule can be determined by the business party (business party server) and is not limited to sequential numbering), and synchronize the numbering information to the corresponding data party (business party server) holding each feature, that is, each data party only has the numbering information of its own features, and the full feature numbering information is only held by the business party.

[0149] Step 103, obtaining the current sample set and training parameter set of the federated learning model. The training parameter set may include feature sampling rate, training sample upper limit, training sample lower limit, decision tree number upper limit, decision tree number lower limit, first parameter change speed, and second parameter change speed.

[0150] In the embodiment of the present application, it is assumed that after the Host (data party) (n≥1) and the Guest (service party) samples are aligned, there are m samples in total, and the Guest holds the label y and its own local data x Guest (characteristic data), the host only holds data that can be X 0 , X 1 , …, X n (Each host has its own local feature data.) It should be emphasized that during the modeling process, the guest and host share the above m samples and can put the m samples into the current sample set to construct the target training set of each layer of random forest.

[0151] Specifically, after sending the feature number and public key of the data provider server to the data provider server, the business server can obtain the current sample set of the federated learning model and the training parameter set of the federated learning model.

[0152] It should be noted that the training parameter set described in the above embodiment may be pre-generated and stored in the storage space of the service party server so as to be called when needed. The training parameter set described in the above embodiment may also include the decision tree depth.

[0153] Step 104, performing M iterative training on the federated learning model according to the current sample set, training parameter set and feature encoding set, where M is a positive integer greater than 1.

[0154] In order to clearly explain the above embodiment, in one embodiment of the present application, Figure 2 As shown, each iteration of training may include:

[0155] Step 201: Use the current iteration training in M ​​iteration trainings as the Nth iteration training, where N is a positive integer less than M.

[0156] Specifically, after the business server calls out (obtains) the training parameter set of the federated learning model from its own storage space, it can first use the current iteration training in the M iteration training as the Nth iteration training so that it can be used directly later.

[0157] Step 202: Generate a sample sampling rate according to M, N, an upper limit value of a training sample, a lower limit value of a training sample, and a first parameter change speed.

[0158] In the embodiment of the present application, the above sample sampling rate can be calculated by the following formula (1):

[0159]

[0160] Where f(X) is the sample rate, V min Can be the lower limit of the training sample, V max can be the upper limit of the training sample, b t can be the current round of iterative training (i.e., the above N), b T M may be the total number of iterative training rounds (ie, the above-mentioned M), k may be the first parameter change speed, and π may be the circumference of a circle.

[0161] It should be noted that the formula (1) described in this embodiment can be generated based on a preset parameter incremental change strategy, wherein the sample sampling rate f(X) can increase with the increase in the number of iterative training rounds, that is, the sample sampling rate used in each iterative training increases with the increase in the number of iterative training rounds. In addition, the formula (1) can be pre-stored in the storage space of the business party server so that it can be called when needed.

[0162] Step 203, generating a target number of decision trees according to M, N, an upper limit of the number of decision trees, a lower limit of the number of decision trees, and a change rate of the second parameter.

[0163] In the embodiment of the present application, the target number of trees can be calculated by the following formula (2):

[0164]

[0165] Where f(Y) is the target number of trees, W min Can be the lower limit of the number of decision trees, W max Can be the upper limit of the number of decision trees, d t It can be the current round of iterative training (i.e., the above N), d T M may be the total number of iterative training rounds (ie, the above-mentioned M), p may be the second parameter change speed, and π may be the circumference of a circle.

[0166] It should be noted that the formula (1) described in this embodiment can be generated based on a preset parameter attenuation change strategy, wherein the target number of trees f(Y) can decrease with the increase of the number of iterative training rounds, that is, the target number of trees used in each iterative training decreases with the increase of the number of iterative training rounds. In addition, the formula (2) can also be pre-stored in the storage space of the business party server so that it can be called when needed.

[0167] Specifically, after the business server uses the current iteration training in the M iteration training as the Nth iteration training, it can first call the above formula (1) and formula (2) from its own storage space, and can generate (calculate) the sample sampling rate based on the formula (1) and according to the above M, N, the upper limit value of the training sample, the lower limit value of the training sample and the first parameter change speed. Then, based on the formula (2) and according to the above M, N, the upper limit value of the number of decision trees, the lower limit value of the number of decision trees and the second parameter change speed, it can generate (calculate) the target number of trees.

[0168] It should be noted that the first parameter change speed and the second parameter change speed in formula (1) and formula (2) described in the above embodiments can respectively control the speed at which the sample sampling rate and the target number of trees change under a given (gradient) total number of iterative training rounds.

[0169] For example, taking the above target number of trees as an example, assuming that the target number of trees is calculated by the above formula (2) (i.e., the parameter change strategy is the attenuation strategy), the total number of iterative training (gradient boosting) rounds is 11 rounds, set in a layer of random forest (i.e., in one iterative training), the maximum number of decision trees allowed is 50 (i.e., the upper limit of the number of decision trees), and the minimum number of decision trees is 15 (i.e., the lower limit of the number of decision trees). If the number of trees change speed p = 1 (i.e., the second parameter change speed), this means that under the control of the above formula (2), from the 1st round of training (boosting) to the 11th round of training, the number of decision trees in each layer of random forest will eventually decrease from 50 to 15; if the number of trees change speed p = 0.5, this means that under the control of the above formula (2), from the 1st round of training to the 6th round of training, the number of decision trees in each layer of random forest will decrease from 50 to 15, and from the 7th round of training to the 11th round of training, the number of decision trees in each layer of random forest will remain unchanged at 15. The above formula (1) can be deduced by analogy and will not be elaborated here.

[0170] It should be further explained that in this application, only the random forest sample sampling rate and the number of random forest decision trees use the parameter change strategy, and the other parameters are fixed values. At the same time, in order to reduce the complexity of the algorithm and avoid the situation where a single-layer random forest has both a large number of trees and a large number of samples, the random forest sample sampling rate and the random forest decision tree number change strategy are exactly opposite, that is, the sample sampling rate selects an increasing strategy (i.e., the above formula (1)), and the number of trees selects a decay strategy (i.e., the above formula (2)), otherwise the sample sampling rate selects a decay strategy and the number of trees selects an increasing strategy.

[0171] To fully explain the meaning of the reference used in a layer of random forest (i.e., in one iteration of training), please see the following partial parameter definition table (Table 1):

[0172]

[0173]

[0174] Table 1

[0175] Step 204: Select samples of the sample sampling rate from the current sample set to generate a target training set.

[0176] Step 205 , selecting feature codes of feature sampling rates from the feature code set to generate a target feature code set.

[0177] Specifically, after generating the above-mentioned sample sampling rate and target number of trees, the business server may select samples of the sample sampling rate from the above-mentioned current sample set to generate a target training set, and select feature codes of the feature sampling rate from the above-mentioned feature code set to generate a target feature code set. For example, assuming that the sample sampling rate and the feature sampling rate are both 15%, 15% of the samples may be selected from the above-mentioned training set to generate a target training set, and 15% of the feature codes may be selected from the above-mentioned feature code set to generate a target feature code set.

[0178] Step 206 , sending the serial number of each sample in the target training set and the target feature serial number of the data provider server in the target feature coding set to the data provider server.

[0179] Step 207, generating target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees.

[0180] Specifically, after generating the target training set and the target feature coding set, the business server can send the number of each sample in the target training set and the target feature number of the data provider server in the target feature coding set to the data provider server. Then the business server can generate the target parameters of the federated learning model based on the target training set, the target feature coding set and the target number of trees.

[0181] Furthermore, the business server can generate a training parameter set in advance, and obtain the parameter values ​​of the random forest modeling of this layer (for example, sample sampling rate, feature sampling rate, number of decision trees, etc.) according to the above-mentioned parameter change strategy (for example, parameter increment change strategy and parameter decay change strategy), and complete random sampling locally based on the two parameter values ​​of sample sampling rate and feature sampling rate, based on the aligned sample number, so as to determine the sample number and feature number participating in this round of modeling, and synchronize the relevant information to the corresponding data provider server. For example, only 10% (sample sampling rate is 10%) of the samples participate in this modeling, then the Guest party (business server) needs to synchronize the IDs of the 10% samples participating in the modeling to each Host party (data provider server), and only sqrt (F) number of features participate in this modeling, wherein, after the Guest party randomly samples the features to obtain sqrt (F) selected features, it needs to synchronize the corresponding feature information to each corresponding Host party.

[0182] In order to clearly explain the above embodiment, in one embodiment of the present application, Figure 3 The target parameters of the federated learning model are generated according to the target training set, the target feature encoding set and the target number of trees, and may include:

[0183] Step 301, calculate the gradient information of the samples in the target training set, and send the gradient information to the data provider server.

[0184] In one embodiment of the present application, calculating the gradient information of samples in the target training set may include generating first-order gradient values ​​and second-order gradient values ​​of the samples in the target training set, and homomorphically encrypting the first-order gradient values ​​and the second-order gradient values ​​to generate gradient information.

[0185] Specifically, the business server can first generate the first-order gradient value g1 and the second-order gradient value h1 of the sample in the target training set (i.e., the aligned sample) according to the preset gradient generation algorithm, and homomorphically encrypt the first-order gradient value g1 and the second-order gradient value h1 to generate gradient information <g1> ， <h1>, and the gradient information <g1> ， <h1>Sent to the data provider server. The preset gradient generation algorithm can be calibrated according to the actual situation.

[0186] Furthermore, in the embodiment of the present application, there may be multiple samples in the target training set, and the business server may generate the first-order gradient value and the second-order gradient value (g1, h1), ..., (g n ,h n ), and then obtain ( <g1> , <h1>),...,( <g n >, <h n >) is sent to the data provider server, where n can be a positive integer.

[0187] In an embodiment of the present application, the data provider server can receive the feature number and public key of the data provider server sent by the business party server, and receive the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set, and receive the gradient information of the currently trained sample sent by the business party server, and obtain the gradient return information based on the gradient information, and then send the gradient return information to the business party server.

[0188] Wherein, obtaining the gradient return information according to the gradient information may include determining the feature set according to the target feature number and the feature number of the data provider server, and splitting the sample space according to the splitting threshold corresponding to each feature in the feature set to obtain the splitting space on the specified side, and then obtaining the gradient summation information of the splitting space on the specified side corresponding to each feature according to the gradient information, and numbering the gradient summation information, and generating the gradient return information using the gradient summation information and the number of the gradient summation information. Wherein, after numbering the gradient summation information, it may also include generating a number, and a mapping relationship between the feature corresponding to the number, the splitting threshold, and the gradient summation information corresponding to the number.

[0189] Specifically, after receiving the feature number of the data provider server sent by the business server, the data provider server can know the numbers of all its local features. It should be noted that after the business server and the data provider server perform sample alignment, both the business server and the data provider server can know the numbers of the aligned samples. Then, after the data provider server receives the number of each sample in the target training set sent by the business server and the target feature number of the data provider server in the target feature coding set, it can determine (acquire) the samples required for subsequent operations according to the above sample numbers, and determine (acquire) the features required for subsequent operations according to the above target feature numbers.

[0190] Furthermore, after receiving the gradient information of the currently trained sample sent by the business server, the data provider server may first determine the feature set according to the above-mentioned target feature number and the feature number of the data provider server, that is, select the feature corresponding to the target feature number from the features corresponding to the feature number of the data provider server according to the target feature number to form the feature set. Then the data provider server may split the sample space according to the splitting threshold corresponding to each feature in the feature set to obtain the split space on the specified side, and obtain the gradient summation information of the split space on the specified side corresponding to each feature according to the gradient information, that is, perform a binning operation, and obtain the gradient summation information of the split space on the specified side corresponding to each feature according to the gradient information, that is, calculate the gradient summation information of the samples in each bin, for example, calculate the gradient summation information in the split space on the left side (that is, the left space) by the following formulas (1) and (2):

[0191]

[0192]

[0193] in, <G L >It can be the first-order gradient summation information of the sample, <H L >It can be used to sum the second-order gradient information of the sample, <g i > can be the first-order gradient information of the sample, <h i > can be the first-order gradient information of the sample, i can be a positive integer less than or equal to n, I L It can be the split space on the left (ie, the space of i samples).

[0194] Then, the data provider server may number the gradient summation information, and generate gradient return information using the gradient summation information and the number of the gradient summation information.

[0195] Furthermore, after numbering the gradient sum information, the data provider server may also generate a mapping relationship between the number, the feature corresponding to the number, the splitting threshold, and the gradient sum information corresponding to the number, and generate a table thereof. For example, the mapping relationship in the following Table 2 (i.e., the number-feature-splitting threshold-gradient sum information table):

[0196]

[0197] Table 2

[0198] It should be noted that the gradient return information described in this embodiment may include number and gradient sum information.

[0199] Finally, the data provider server may send (synchronize) the gradient return information to the business server. The data provider server may encrypt the data sent (synchronized) to the business server.

[0200] Step 302: Receive gradient return information provided by a data provider server.

[0201] Step 303: Generate a target splitting point number according to the gradient return information and the target feature encoding set, generate a ciphertext based on the private key and the target splitting point number, and send the ciphertext to the data provider server.

[0202] In one embodiment of the present application, there may be multiple gradient return information, and each gradient return information corresponds to a corresponding number, wherein generating the target splitting point number according to the gradient return information and the target feature encoding set may include generating corresponding multiple information gains according to the multiple gradient return information and the target feature encoding set, and selecting the maximum information gain from the multiple information gains, and using the number corresponding to the maximum information gain as the target splitting point number.

[0203] Specifically, after receiving the above-mentioned gradient return information, the business server can generate corresponding multiple information gains according to multiple gradient return information and the target feature coding set, select the maximum information gain from the multiple information gains, and use the number corresponding to the maximum information gain as the target split point number.

[0204] For example, after receiving the above-mentioned gradient return information, the business server can parse the gradient return information to obtain the gradient summation information G of each feature and the corresponding bin combination. L , G R , H L , H R , where G L It can be the first-order gradient summation information of the samples in the left split space (ie, the left space), H L It can be the second-order gradient summation information of the samples in the left split space (ie, the left space), G R It can be the first-order gradient summation information of the samples in the right split space (ie, right space), H R It can be the second-order gradient summation information of the samples in the right split space (i.e., right space). The feature information gain of each decision tree node is calculated in this way. After the business server obtains the bin gradient summation result of the data provider server and combines it with the above target feature encoding set, the above multiple information gains can be calculated through relevant formulas.

[0205] Then, the business server can compare the information gain corresponding to each feature and select the maximum value, that is, select the maximum information gain from multiple information gains. It should be noted that the judgment can be made based on the target feature encoding set. If the feature ID and threshold ID involved in the above calculation are the features and thresholds of the data party, the feature number information needs to be sent to the data party, and the sample space is segmented using the features returned by the data party; if it is from the business party, it can be directly segmented.

[0206] Further, the business server finds the maximum information gain (i.e., the first-order gradient information gain and the second-order gradient information gain) among the above-mentioned multiple information gains and the number q (i.e., the target splitting point number) in the corresponding table 2. Then the business server can generate ciphertext based on the private key and the target splitting point number, and send the ciphertext to the data provider server. It should be noted that the ciphertext can be generated based on the private key and the target splitting point number based on the relevant technology, which will not be repeated here.

[0207] In an embodiment of the present application, the data provider server may receive a ciphertext generated based on a private key and a target splitting point number and sent by a business party server, wherein the target splitting point number may be generated based on the gradient return information and the target feature coding set, and decrypt the ciphertext based on the public key to obtain a decryption operation value, and send it to the business party server.

[0208] Specifically, after receiving the ciphertext sent by the business party server, the data provider server can use the public key to decrypt the ciphertext to obtain a decrypted operation value, and send the decrypted operation value to the business party server. It should be noted that the ciphertext can be decrypted using the public key based on relevant technologies to obtain a decrypted operation value, which will not be repeated here.

[0209] Step 304: Receive the decryption operation value sent by the data provider server, and perform node splitting according to the decryption operation value.

[0210] In one embodiment of the present application, Figure 4 The node splitting according to the decryption operation value may include:

[0211] Step 401, generating split space information according to the decryption operation value.

[0212] Step 402: Perform node splitting according to samples in the target training set and splitting space information.

[0213] Specifically, after receiving the decryption operation value sent by the data provider server, the business server can generate splitting space information based on the relevant formula and the decryption operation value, wherein the splitting space information can be the space information required by the business server. Then, the business server can perform a difference operation based on the sample information in the above model training set and the splitting space information on one side of the optimal splitting feature (i.e., the above splitting space information) to obtain the splitting space information on the other side of the optimal splitting feature, thereby completing the node splitting (i.e., the first node splitting).

[0214] Step 305, repeat the above steps until the model converges to establish a target number of decision trees, complete the training of the federated learning model, and obtain the target parameters through the trained federated learning model.

[0215] Specifically, the business server can repeat the above steps 301 to 304 until the model converges to establish the target number of decision trees and complete the training of the federated learning model. It should be noted that since the random forest can establish multiple decision trees in parallel, the random forest modeling can also be completed simultaneously according to the above method. Then the business server can obtain the target parameters by completing the federated learning model trained this time.

[0216] Therefore, the training method of the federated learning model provided in the embodiment of the present application can be based on Figure 3 The described method generates target parameters based on a target training set, a target feature encoding set and a target number of trees.

[0217] Step 208, based on the gradient boosting algorithm, and according to the target parameters and the federated learning model, an optimized label of the current sample is generated, wherein the optimized label is a training label of the current sample for the next round of iterative training.

[0218] Specifically, after the current round of iterative training is completed, the business server can generate an optimized label for the current sample based on the gradient boosting algorithm, according to the target parameters and the federated learning model, and use the optimized label as the training label for the current sample in the next round of iterative training, and then continue with the next round of iterative training.

[0219] Step 105, obtaining the target parameters of the federated learning model obtained by the Mth iterative training.

[0220] Specifically, after M rounds of iterative training are completed, the business server can obtain the target parameters of the federated learning model obtained by the Mth iterative training, that is, obtain the target parameters of the federated learning model that has completed all rounds of iterative training.

[0221] As a result, the joint training between the business server and the data provider server can be made more efficient, while the modeling effect is improved. The early stopping strategy is adopted to avoid overfitting of the model while reducing complexity.

[0222] In the examples of this application, see Figure 5 , which combines the federated modeling method of random forest (bagging) and gradient boosting (boosting) two integrated strategies, and dynamically optimizes the parameters of random forest (taking sample sampling rate and number of decision trees as examples) based on the parameter decay strategy of cosine function and the parameter increment strategy of cosine function, thereby reducing the complexity of modeling while ensuring the modeling effect. Specifically:

[0223] First, the model is gradient-boosted on the basis of random forest, which reduces the problem of relatively large deviation of random forest model. Moreover, since random forest can be built in parallel, it does not bring time inefficiency compared with models such as GBDT. And since the modeling effect of random forest based on bagging is better than that of a single decision tree sub-model, it can improve the modeling effect on the original boosting ensemble models such as GBDT.

[0224] Secondly, the present application dynamically optimizes the parameters of each layer of the forest. Taking the sample sampling rate and the number of decision trees as an example, the parameters of each layer of the forest in the entire model in the present application scheme are not consistent, but the number of decision trees is optimized layer by layer based on the parameter decay strategy of the cosine function, and the sample sampling rate is optimized layer by layer based on the parameter increment strategy of the cosine function. In the traditional parameter search scheme, it is necessary to run the entire model under the premise of each possible parameter combination to finally determine the optimal parameters. The traditional parameter search method brings a high time cost, and the present application can make the joint training between the business server and the data provider server more efficient, thereby improving the modeling efficiency and quality.

[0225] Furthermore, the training method of the federated learning model provided in the embodiment of the present application proposes the idea of ​​dynamically optimizing the two core parameters of the number of decision trees in each forest layer and the sample sampling rate of each forest layer to reduce the complexity, and after trying three strategies, namely linear, exponential, and cosine annealing, the cosine annealing strategy is selected as the optimal parameter dynamic control strategy. Among them, this idea and method reduces the complexity of the algorithm and ensures the modeling effect of the algorithm while trying that the algorithm only needs to set the change range and change speed of additional parameters and does not introduce a large number of hyperparameters.

[0226] In summary, according to the training method of the federated learning model of the embodiment of the present application, firstly, sample alignment is performed with the data provider server, and then the number of features of the business party server and the data provider server is obtained respectively, and the features of the business party server and the data provider server are numbered respectively according to the number of features to generate a feature coding set, and the feature number and public key of the data provider server are sent to the data provider server, and then the current sample set and training parameter set of the federated learning model are obtained, and according to the current sample set, training parameter set and feature coding set, the federated learning model is iteratively trained M times, and finally the target parameters of the federated learning model obtained by the Mth iterative training are obtained. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0227] Figure 6 A flowchart of a method for training a federated learning model according to another embodiment of the present application.

[0228] The training method of the federated learning model of the embodiment of the present application can also be executed by the training device of the federated learning model provided by the embodiment of the present application, which can be configured in an electronic device to realize sample alignment with the business party server, and receive the feature number, public key, number of each sample in the target training set, target feature number of the data provider server in the target feature coding set and gradient information of the currently trained sample sent by the business party server, and obtain gradient return information based on the gradient information, and then send the gradient return information to the business party server, and receive the ciphertext generated based on the private key and the target splitting point number sent by the business party server, and decrypt the ciphertext based on the public key to obtain the decryption operation value, and send it to the business party server, so as to make the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0229] As a possible scenario, the training method of the federated learning model of the embodiment of the present application can also be executed on the server side. The server can be a cloud server, and the training method of the federated learning model can be executed in the cloud.

[0230] like Figure 6 As shown, the training method of the federated learning model may include:

[0231] Step 601: Perform sample alignment with the business server.

[0232] In the embodiment of the present application, the data party (i.e., the data provider server) can perform sample alignment with the business party server through a preset method. The preset method can be calibrated according to the actual situation. For example, since the user groups of the partners are not completely overlapping, the encryption-based user sample alignment technology (method) can be used to confirm the common users of both parties without disclosing their respective data, and without exposing non-overlapping users.

[0233] It should be noted that the sample alignment described in this embodiment may also refer to the alignment of sample positions between the business party server and the data provider server to facilitate accurate sample transmission. In addition, during the sample alignment process, a communication channel (channel) may be established between the business party and the data provider server, and the communication channel may be encrypted.

[0234] Step 602: Receive the characteristic number and public key of the data provider server sent by the business party server.

[0235] Step 603: Receive the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set.

[0236] Step 604: Receive the gradient information of the currently trained sample sent by the business party server, and obtain gradient return information according to the gradient information.

[0237] Step 605: Send gradient return information to the business server.

[0238] Step 606: Receive the ciphertext generated based on the private key and the target splitting point number sent by the business party server, wherein the target splitting point number is generated according to the gradient return information and the target feature encoding set.

[0239] Step 607: decrypt the ciphertext based on the public key to obtain a decryption operation value, and send it to the business party server.

[0240] In one embodiment of the present application, gradient return information is obtained according to gradient information, including: determining a feature set according to a target feature number and a feature number of a data provider server; splitting a sample space according to a split threshold corresponding to each feature in the feature set to obtain a split space on a specified side; obtaining gradient summation information of the split space on a specified side corresponding to each feature according to the gradient information, and numbering the gradient summation information; and generating gradient return information using the gradient summation information and the number of the gradient summation information.

[0241] In one embodiment of the present application, after the gradient sum information is numbered, the method further includes: generating a number, and a mapping relationship between a feature corresponding to the number, a splitting threshold, and the gradient sum information corresponding to the number.

[0242] It should be noted that for details not disclosed in the training method of the federated learning model in the embodiment of the present application, please refer to the present application. Figures 1 to 5 The details disclosed in the training method of the federated learning model in the embodiment will not be repeated here.

[0243] In summary, according to the training method of the federated learning model of the embodiment of the present application, firstly, sample alignment is performed with the business party server, and then the feature number and public key of the data provider server sent by the business party server are received, and the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set are received, and the gradient information of the currently trained sample sent by the business party server is received, and the gradient return information is obtained according to the gradient information, and then the gradient return information is sent to the business party server, and the ciphertext generated based on the private key and the target splitting point number sent by the business party server is received, and finally the ciphertext is decrypted based on the public key to obtain the decrypted operation value, and sent to the business party server. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0244] Figure 7 A structural diagram of a training device for a federated learning model according to an embodiment of the present application.

[0245] The training device of the federated learning model of the embodiment of the present application can be configured in an electronic device to achieve sample alignment with the data provider server, and then obtain the number of features of the business server and the data provider server respectively, and number the features of the business server and the data provider server respectively according to the number of features to generate a feature coding set, and send the feature number and public key of the data provider server to the data provider server, and then obtain the current sample set and training parameter set of the federated learning model, and perform M iterations of training on the federated learning model according to the current sample set, training parameter set and feature coding set, and obtain the target parameters of the federated learning model obtained by the Mth iteration training, so as to make the joint training between the business server and the data provider server more efficient, thereby improving the modeling efficiency.

[0246] like Figure 7 As shown, the training device 700 of the federated learning model may include: an alignment module 710, a sending module 720, a first acquisition module 730, an iterative training module 740 and a second acquisition module 750.

[0247] The alignment module 710 is used to perform sample alignment with the data provider server.

[0248] The sending module 720 is used to obtain the feature quantity of the business party server and the data provider server respectively, and number the features of the business party server and the data provider server according to the feature quantity to generate a feature coding set, and send the feature number and public key of the data provider server to the data provider server.

[0249] The first acquisition module 730 is used to obtain the current sample set and training parameter set of the federated learning model.

[0250] The iterative training module 740 is used to perform M iterative training on the federated learning model according to the current sample set, training parameter set and feature encoding set, where M is a positive integer greater than 1.

[0251] The second acquisition module 750 is used to obtain the target parameters of the federated learning model obtained by the Mth iterative training.

[0252] In one embodiment of the present application, the training parameter set includes a feature sampling rate, an upper limit value of training samples, a lower limit value of training samples, an upper limit value of the number of decision trees, a lower limit value of the number of decision trees, a first parameter change speed, and a second parameter change speed.

[0253] In one embodiment of the present application, the iterative training module 740 may include: a setting submodule, a first generation submodule, a second generation submodule, a third generation submodule, a fourth generation submodule, a sending submodule, a fifth generation submodule and a sixth generation submodule.

[0254] Among them, the setting submodule is used to use the current iteration training in M ​​iteration trainings as the Nth iteration training, wherein N is a positive integer less than M.

[0255] The first generating submodule is used to generate a sample sampling rate according to M, N, an upper limit value of a training sample, a lower limit value of a training sample and a first parameter change speed.

[0256] The second generation submodule is used to generate the target number of decision trees according to M, N, the upper limit of the number of decision trees, the lower limit of the number of decision trees and the change speed of the second parameter.

[0257] The third generation submodule is used to select samples of the sample sampling rate from the current sample set to generate a target training set.

[0258] The fourth generating submodule is used to select feature codes of feature sampling rates from the feature code set to generate a target feature code set.

[0259] The sending submodule is used to send the number of each sample in the target training set and the target feature number of the data provider server in the target feature encoding set to the data provider server.

[0260] The fifth generation submodule is used to generate target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees.

[0261] The sixth generation submodule is used to generate an optimized label for the current sample based on the gradient boosting algorithm and according to the target parameters and the federated learning model, wherein the optimized label is a training label for the current sample for the next round of iterative training.

[0262] In one embodiment of the present application, the fifth generation submodule may include: a calculation unit, a receiving unit, a generation unit, a node splitting unit and an acquisition unit.

[0263] The calculation unit is used to calculate the gradient information of the samples in the target training set and send the gradient information to the data provider server.

[0264] The receiving unit is used to receive the gradient return information provided by the data provider server.

[0265] A generating unit is used to generate a target splitting point number according to the gradient return information and the target feature encoding set, generate a ciphertext based on the private key and the target splitting point number, and send the ciphertext to the data provider server.

[0266] The node splitting unit is used to receive the decryption operation value sent by the data provider server and perform node splitting according to the decryption operation value.

[0267] The acquisition unit is used to repeat the above steps until the model converges to establish a target number of decision trees, complete the training of the federated learning model, and obtain the target parameters through the trained federated learning model.

[0268] In one embodiment of the present application, the computing unit is specifically used to: generate first-order gradient values ​​and second-order gradient values ​​of samples in a target training set; and homomorphically encrypt the first-order gradient values ​​and second-order gradient values ​​to generate gradient information.

[0269] In one embodiment of the present application, there are multiple gradient return information, and each gradient return information corresponds to a corresponding number, wherein the generation unit is specifically used to: generate corresponding multiple information gains according to the multiple gradient return information and the target feature encoding set; select the maximum information gain from the multiple information gains, and use the number corresponding to the maximum information gain as the target split point number.

[0270] In one embodiment of the present application, the node splitting unit is specifically used to: generate splitting space information according to the decryption operation value; and perform node splitting according to the samples in the target training set and the splitting space information.

[0271] It should be noted that for details not disclosed in the training device of the federated learning model in the embodiment of the present application, please refer to the present application Figures 1 to 5 The details disclosed in the training method of the federated learning model in the embodiment will not be repeated here.

[0272] In summary, the training device of the federated learning model of the embodiment of the present application first performs sample alignment with the data provider server through the alignment module, and then obtains the number of features of the business party server and the data provider server respectively through the sending module, and numbers the features of the business party server and the data provider server respectively according to the number of features to generate a feature coding set, and sends the feature number and public key of the data provider server to the data provider server, and then obtains the current sample set and training parameter set of the federated learning model through the first acquisition module, and performs M iterations of training on the federated learning model according to the current sample set, training parameter set and feature coding set through the iterative training module, and finally obtains the target parameters of the federated learning model obtained by the Mth iteration training through the second acquisition module. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0273] Figure 8 It is a structural diagram of a training device for a federated learning model according to another embodiment of the present application.

[0274] The training device of the federated learning model of the embodiment of the present application can be configured in an electronic device to achieve sample alignment with the business party server, and receive the feature number, public key, number of each sample in the target training set, target feature number of the data provider server in the target feature coding set and gradient information of the currently trained sample sent by the business party server, and obtain gradient return information based on the gradient information, and then send the gradient return information to the business party server, and receive the ciphertext generated based on the private key and the target splitting point number sent by the business party server, and decrypt the ciphertext based on the public key to obtain the decryption operation value, and send it to the business party server, so as to make the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0275] like Figure 8 As shown, the training device 800 of the federated learning model may include: an alignment module 810, a first receiving module 820, a second receiving module 830, a third receiving module 840, a first sending module 850, a fourth receiving module 860 and a second sending module 870.

[0276] The alignment module 810 is used to perform sample alignment with the business server.

[0277] The first receiving module 820 is used to receive the characteristic number and public key of the data provider server sent by the business party server.

[0278] The second receiving module 830 is used to receive the number of each sample in the target training set sent by the business party server, and the target feature number of the data provider server in the target feature coding set.

[0279] The third receiving module 840 is used to receive the gradient information of the currently trained sample sent by the business party server, and obtain the gradient return information according to the gradient information.

[0280] The first sending module 850 is used to send gradient return information to the business party server.

[0281] The fourth receiving module 860 is used to receive the ciphertext generated based on the private key and the target splitting point number sent by the business party server, wherein the target splitting point number is generated according to the gradient return information and the target feature encoding set.

[0282] The second sending module 870 is used to decrypt the ciphertext based on the public key, obtain the decryption operation value, and send it to the business party server.

[0283] In one embodiment of the present application, the third receiving module 840 is specifically used to: determine a feature set according to the target feature number and the feature number of the data provider server; split the sample space according to the split threshold corresponding to each feature in the feature set to obtain the split space on the specified side; obtain the gradient summation information of the split space on the specified side corresponding to each feature according to the gradient information, and number the gradient summation information; generate gradient return information using the gradient summation information and the number of the gradient summation information.

[0284] In one embodiment of the present application, the third receiving module 840 is further used to: generate a number, and a mapping relationship between a feature corresponding to the number, a split threshold, and gradient sum information corresponding to the number.

[0285] It should be noted that for details not disclosed in the training device of the federated learning model in the embodiment of the present application, please refer to the present application Figures 1 to 5 The details disclosed in the training method of the federated learning model in the embodiment will not be repeated here.

[0286] In summary, the training device of the federated learning model of the embodiment of the present application first performs sample alignment with the business party server through the alignment module, then receives the feature number and public key of the data provider server sent by the business party server through the first receiving module, and receives the number of each sample in the target training set sent by the business party server and the target feature number of the data provider server in the target feature coding set through the second receiving module, and receives the gradient information of the currently trained sample sent by the business party server through the third receiving module, and obtains the gradient return information according to the gradient information, and then sends the gradient return information to the business party server through the first sending module, and receives the ciphertext generated based on the private key and the target splitting point number sent by the business party server through the fourth receiving module, and finally decrypts the ciphertext based on the public key through the second sending module to obtain the decrypted operation value, and sends it to the business party server. In this way, the complexity of modeling can be reduced while ensuring the modeling effect, so that the joint training between the business party server and the data provider server is more efficient, thereby improving the modeling efficiency.

[0287] In order to implement the above embodiment, Fig. 9 As shown, the present application also proposes an electronic device 900, including a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920, and the processor 920 executes the program to implement the training method of the federated learning model proposed in the aforementioned embodiment of the present application.

[0288] The electronic device of the embodiment of the present application, by executing a computer program stored in a memory through a processor, can reduce the complexity of modeling while ensuring the modeling effect, thereby making the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0289] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the training method of the federated learning model proposed in the aforementioned embodiments of the present application.

[0290] The computer-readable storage medium of the embodiment of the present application, by storing a computer program and being executed by a processor, can reduce the complexity of modeling while ensuring the modeling effect, thereby making the joint training between the business party server and the data provider server more efficient, thereby improving the modeling efficiency.

[0291] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0292] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0293] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.< / h1> < / g1> < / h1> < / g1> < / h1> < / g1>

Claims

1. A training method for a federated learning model, characterized in that: include: Perform sample alignment with the data provider's server; Respectively obtaining the number of features of the business party server and the data provider server, and numbering the features of the business party server and the data provider server according to the number of features to generate a feature coding set, and sending the feature number and public key of the data provider server to the data provider server; Get the current sample set and training parameter set of the federated learning model; According to the current sample set, the training parameter set and the feature encoding set, the federated learning model is iterated M times, where M is a positive integer greater than 1; and Obtaining target parameters of the federated learning model obtained by the M-th iterative training; The training parameter set includes a feature sampling rate, an upper limit of training samples, a lower limit of training samples, an upper limit of the number of decision trees, a lower limit of the number of decision trees, a first parameter change speed, and a second parameter change speed; Wherein, each iteration training includes: Using the current iteration training in the M iteration training as the Nth iteration training, wherein N is a positive integer less than M; Generate a sample sampling rate according to the M, the N, the training sample upper limit, the training sample lower limit and the first parameter change speed; Generate a target number of trees according to the M, the N, the upper limit of the number of decision trees, the lower limit of the number of decision trees, and the second parameter change speed; Selecting samples of the sample sampling rate from the current sample set to generate a target training set; Selecting the feature code of the feature sampling rate from the feature code set to generate a target feature code set; Sending the number of each sample in the target training set and the target feature number of the data provider server in the target feature coding set to the data provider server; Generate target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees; Based on the gradient boosting algorithm and according to the target parameter and the federated learning model, an optimized label of the current sample is generated, wherein the optimized label is a training label of the current sample for the next round of iterative training.

2. The method for training a federated learning model according to claim 1, wherein: The step of generating target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees includes: Calculating the gradient information of the samples in the target training set, and sending the gradient information to the data provider server; Receiving gradient return information provided by the data provider server; Generate a target splitting point number according to the gradient return information and the target feature encoding set, generate a ciphertext based on a private key and the target splitting point number, and send the ciphertext to the data provider server; Receiving the decryption operation value sent by the data provider server, and performing node splitting according to the decryption operation value; Repeat the above steps until the model converges to establish the target number of decision trees, complete the training of the federated learning model, and obtain the target parameters through the trained federated learning model.

3. The training method of the federated learning model according to claim 2, characterized in that: The calculating the gradient information of the samples in the target training set includes: Generate first-order gradient values ​​and second-order gradient values ​​of samples in the target training set; The first-order gradient value and the second-order gradient value are homomorphically encrypted to generate the gradient information.

4. The method for training a federated learning model according to claim 2, wherein: There are multiple gradient return information, and each of the gradient return information corresponds to a corresponding number, wherein the generating the target splitting point number according to the gradient return information and the target feature encoding set includes: Generate corresponding multiple information gains respectively according to the multiple gradient return information and the target feature encoding set; A maximum information gain is selected from the multiple information gains, and a number corresponding to the maximum information gain is used as the target split point number.

5. The method for training a federated learning model according to claim 2, wherein: The node splitting according to the decryption operation value includes: Generate splitting space information according to the decryption operation value; Node splitting is performed according to the samples in the target training set and the splitting space information.

6. A training device for a federated learning model, characterized in that: include: An alignment module, used to align samples with the data provider server; A sending module, used to obtain the number of features of the business party server and the data provider server respectively, and number the features of the business party server and the data provider server respectively according to the number of features to generate a feature coding set, and send the feature number and public key of the data provider server to the data provider server; A first acquisition module is used to obtain a current sample set and a training parameter set of a federated learning model; an iterative training module, configured to perform M iterative training on the federated learning model according to the current sample set, the training parameter set and the feature encoding set, wherein M is a positive integer greater than 1; and A second acquisition module is used to obtain the target parameters of the federated learning model obtained by the M-th iterative training; The training parameter set includes a feature sampling rate, an upper limit of training samples, a lower limit of training samples, an upper limit of the number of decision trees, a lower limit of the number of decision trees, a first parameter change speed, and a second parameter change speed; The iterative training module comprises: A setting submodule, used for taking the current iteration training in the M iteration training as the Nth iteration training, wherein N is a positive integer less than M; A first generating submodule, used for generating a sample sampling rate according to the M, the N, the upper limit value of the training sample, the lower limit value of the training sample and the first parameter change speed; A second generation submodule is used to generate a target number of trees according to the M, the N, the upper limit of the number of decision trees, the lower limit of the number of decision trees and the second parameter change speed; A third generating submodule, configured to select samples of the sample sampling rate from the current sample set to generate a target training set; A fourth generating submodule, configured to select the feature code of the feature sampling rate from the feature code set to generate a target feature code set; A sending submodule, used for sending the number of each sample in the target training set and the target feature number of the data provider server in the target feature coding set to the data provider server; a fifth generating submodule, configured to generate target parameters of the federated learning model according to the target training set, the target feature encoding set and the target number of trees; The sixth generation submodule is used to generate an optimized label of the current sample based on the gradient boosting algorithm and according to the target parameters and the federated learning model, wherein the optimized label is a training label of the current sample for the next round of iterative training.

7. The training device for the federated learning model according to claim 6, characterized in that: The fifth generation submodule comprises: A calculation unit, used to calculate the gradient information of the samples in the target training set and send the gradient information to the data provider server; A receiving unit, configured to receive gradient return information provided by the data provider server; A generating unit, configured to generate a target splitting point number according to the gradient return information and the target feature encoding set, generate a ciphertext based on a private key and the target splitting point number, and send the ciphertext to the data provider server; A node splitting unit, used for receiving the decryption operation value sent by the data provider server, and performing node splitting according to the decryption operation value; The acquisition unit is used to repeat the above steps until the model converges to establish the target number of decision trees, complete the training of the federated learning model, and obtain the target parameters through the trained federated learning model.

8. The training device for the federated learning model according to claim 7, characterized in that: The computing unit is specifically used for: Generate first-order gradient values ​​and second-order gradient values ​​of samples in the target training set; The first-order gradient value and the second-order gradient value are homomorphically encrypted to generate the gradient information.

9. The training device for the federated learning model according to claim 7, characterized in that: There are multiple pieces of gradient return information, and each piece of gradient return information has a corresponding number, wherein the generating unit is specifically used to: Generate corresponding multiple information gains respectively according to the multiple gradient return information and the target feature encoding set; A maximum information gain is selected from the multiple information gains, and a number corresponding to the maximum information gain is used as the target split point number.

10. The training device for a federated learning model according to claim 7, characterized in that: The node splitting unit is specifically used for: Generate splitting space information according to the decryption operation value; Node splitting is performed according to the samples in the target training set and the splitting space information.

11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the training method of the federated learning model as described in any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the training method of the federated learning model as described in any one of claims 1-5.

Images