An integrated learning model in a federated learning system and a construction method thereof

Abstract

The application provides an integrated learning model in a federal learning system and a construction method thereof. The method comprises the following steps: determining an optimization target and a constraint condition of the federal learning system; the system comprises a central server and a plurality of participants; setting an iteration number of the integrated learning model; in each iteration, the central server randomly selects a preset number of participants for parallel decision tree training; each selected participant trains a decision tree using a local data set; the trained decision tree is sent to the central server for aggregation, and the aggregation is a base learner for the next iteration, and the global integrated learning model is updated; the central server sends the global integrated learning model to each participant, each participant fine-tunes the global integrated learning model using a local data set, and finally obtains a personalized integrated learning model for prediction. Compared with the existing model, the integrated learning model of the application has higher prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning, and particularly relates to an integrated learning model in a federated learning system and a construction method thereof. BACKGROUND

[0002] Learning algorithms based on decision trees are an important branch of machine learning, which has achieved good results in various public data set competitions and is widely applied. Tree model has significant application value due to its insensitivity to data types and data distribution. TreeBoost refers to a class of ensemble learning algorithms, which iteratively trains decision trees based on gradient and error information and combines them into an ensemble learning model with better prediction performance, and achieves good results in applications such as recommendation, ranking, user behavior prediction, click rate prediction, product classification, etc. Although deep neural networks have received extensive attention in recent years, TreeBoost algorithms show advantages in limited data sets and lack of tuning information scenarios. Compared with deep neural networks, TreeBoost algorithms can analyze tabular data in a more interpretable way.

[0003] With the rise of the Internet of Things, big data is no longer dominated by a single entity, but is scattered in different fields, forming a large number of small data sets with different data distributions. For example, each hospital has its own limited patient records and can benefit from collaborative data analysis with other hospitals, because training machine learning models on integrated data sets can achieve higher prediction performance. However, with the awareness and demand for data protection security and more policies on data privacy, it will become increasingly difficult to collect and share data between different organizations. Federated learning (FL) is a widely accepted paradigm that enables multiple data providers to collaboratively train tree-based ensemble models without revealing user private data. Specifically, in a typical FL framework, each party (also known as a client) shares model parameters rather than raw data to collaboratively improve model performance. In addition, if the local data of the client is feature-aligned, the setting is considered to be a horizontal federated learning.

[0004] In recent years, researchers have conducted a number of research works on how to deploy TreeBoost in federated learning with horizontal partition. However, there is still room for improvement when applying existing solutions to real-world scenarios. First, in a federated learning system with a large number of participants, data heterogeneity can reduce prediction performance and training efficiency. Specifically, data heterogeneity refers to the setting where the data distribution of each client has arbitrary bias, also known as non-independent distribution (Non-IID). In general, personalized federated learning, which trains separate models for each client instead of sharing a model, has been legitimized as an effective means to address data heterogeneity. Second, the FL system does not have strict privacy protection. For gradient boosting tree algorithm, the data privacy leaked from the model mainly comes from two aspects: 1) local data information is leaked from the gradient; 2) the class distribution information of the data is leaked from the leaf nodes of the decision tree. However, existing cryptography-based solutions require a large amount of additional computation, while differential privacy-based solutions can cause training bias.

[0005] Therefore, how to improve the prediction accuracy of the ensemble learning model in the federated learning scenario while ensuring privacy is a technical problem that needs to be solved in the field. SUMMARY

[0006] The purpose of the present application is to provide an ensemble learning model in a federated learning system and a construction method thereof, so as to improve the prediction accuracy of the ensemble learning model in the federated learning scenario.

[0007] The first aspect of the present application provides a construction method of an ensemble learning model in a federated learning system, comprising:

[0008] determining the optimization target and the constraint condition of the federated learning system; the federated learning system comprises a center server and a plurality of participants, the plurality of participants participate in the training of the ensemble learning model while avoiding exposure of their local data; the federated learning system adopts a star-shaped network architecture, and the plurality of participants can simultaneously communicate with the center server;

[0009] setting the number of iterations of the ensemble learning model, in each iteration, the center server randomly selects a predetermined number of participants for parallel decision tree training, and each selected participant trains a decision tree using a local data set combined with a differential privacy mechanism;

[0010] Each selected participant sends the trained decision tree to the center server for aggregation, the center server aggregates the received decision trees into a base learner, and then sends the base learner to each participant for the next iteration, and updates the global ensemble learning model according to the base learner until the iteration is completed;

[0011] The central server sends a global ensemble learning model to each participant, each participant fine-tunes the global ensemble learning model using a local data set, and finally obtains an ensemble learning model for prediction for each participant.

[0012] In some embodiments, the fine-tuning of the global ensemble learning model using the local data set by each participant comprises:

[0013] Each participant fine-tunes the global ensemble learning model using a preset ensemble pruning algorithm based on the local data set.

[0014] In some embodiments, the fine-tuning of the global ensemble learning model using the local data set by each participant based on the preset ensemble pruning algorithm comprises:

[0015] Based on the local data set, the preset ensemble pruning algorithm is used to calculate the weight of each base learner in the ensemble learning model, and the weight is used to represent the fitness of the base learner for joining the ensemble learning model.

[0016] The base learners with weights greater than a preset threshold are retained in the ensemble learning model, and the remaining base learners are excluded from the ensemble learning model to obtain the fine-tuned ensemble learning model.

[0017] In some embodiments, the preset ensemble pruning algorithm comprises a genetic algorithm.

[0018] In some embodiments, the calculation of the weight of each base learner in the ensemble learning model using the preset ensemble pruning algorithm based on the local data set comprises:

[0019] The participant m randomly samples T ep data sets from the local data set

[0020] The participant m randomly generates P weight vectors of base learners as the initial population of the genetic algorithm;

[0021] For each weight vector in the population, a crossover probability p cross is performed to produce new individuals by pairwise crossing;

[0022] For each individual, a mutation probability p mut is performed to mutate any gene on the individual;

[0023] For each individual, the error rate of the fine-tuned ensemble learning model on the local data set is calculated to calculate the fitness where w represents the weight of the base learner, represents a preset threshold for fine-tuning the ensemble learning model;

[0024] selecting N individuals with the highest fitness as the next round of genetic algorithm iteration;

[0025] the iteration round reaches T ep selecting the individual with the highest fitness as the final weight of the base learner.

[0026] The second aspect of the application provides an ensemble learning model, which is created by the construction method of the ensemble learning model in the federated learning system.

[0027] The beneficial effects of the ensemble learning model in the federated learning system and the construction method thereof are as follows:

[0028] a) Realize the feasibility of concurrent training of decision trees

[0029] The construction method of the traditional ensemble learning model is to train the decision trees sequentially, which cannot train multiple decision trees at the same time. The federated learning system adopts a star-shaped network architecture, establishes the information flow of the federated learning system parallel transmission, designs the residual calculation method of parallel training of multiple decision trees, thereby improving the communication efficiency of the federated learning system, and realizes the feasibility of concurrent training of decision trees.

[0030] b) Realize the feasibility of performance improvement under privacy constraints

[0031] For the problem of non-independent and identically distributed data sets, the application uses an ensemble pruning algorithm to fine-tune the ensemble learning model on heterogeneous data sets, and uses the locally fine-tuned model to replace the global ensemble learning for prediction, so that the customized ensemble learning model has better adaptability to different data distributions, thereby improving its prediction performance. BRIEF DESCRIPTION OF DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for purposes of illustration only and are not considered a limitation of the application. Moreover, like reference numerals are used to designate identical components throughout the specification. In the drawings:

[0033] Figure 1 A flowchart of a construction method of an ensemble learning model in a federated learning system is shown;

[0034] Figure 2 A schematic diagram of a specific construction process of an ensemble learning model in a federated learning system is shown;

[0035] Figure 3 A system architecture schematic diagram of the federated learning system in the application is shown;

[0036] Figure 4 A schematic diagram of a gradient boosting tree training process in a federated learning system of the present application is shown. DETAILED DESCRIPTION

[0037] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is to be understood that the present application can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0038] It should be noted that the technical terms or scientific terms used in the present application should be understood in their ordinary meaning by those skilled in the art, unless otherwise specified.

[0039] In addition, the terms "first" and "second" and the like are used to distinguish different objects, rather than to describe a particular order. Furthermore, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include other steps or units not listed or can optionally include other steps or units inherent to such processes, methods, products or devices.

[0040] The embodiments of the present application provide an integrated learning model in a federated learning system and a construction method thereof, which can improve the prediction accuracy of the integrated learning model in the federated learning scenario. The following will be described with reference to the accompanying drawings.

[0041] Reference is made to Figure 1 which shows a flowchart of a construction method of an integrated learning model in a federated learning system provided by some embodiments of the present application, for the convenience of understanding, as shown in Figure 2 the embodiments of the present application also provide a schematic diagram of a specific construction process of an integrated learning model in a federated learning system.

[0042] As shown in Figure 1 the construction method of the integrated learning model in the federated learning system can include the following steps:

[0043] S101, determining the optimization target and constraint condition of the federated learning system; the federated learning system includes a central server and a plurality of participants, the plurality of participants participate in the training of the integrated learning model while avoiding exposure of their local data.

[0044] Figure 3 A schematic diagram of the system architecture of the federated learning system in the present application is shown, as Figure 3As shown, the federated learning system adopts a network architecture of star topology, the center server and each participant communicate point-to-point, and multiple participants can communicate with the center server at the same time.

[0045] The federated learning system of the present application is a federated learning system of data horizontal partitioning, please refer to Figure 3 In the federated learning system of data horizontal partitioning, M participants participate in the training of the integrated learning model while avoiding exposure of their local data. Each participant m has a private data set Wherein, n (m) is the total data amount of the data set D (m) , and and represent the i-th data sample in the data set D (m) . The data sets among the participants are not independent and identically distributed, that is, the data distribution of each participant is different, thus deviating from the global data distribution. The overall data set of the federated learning system is

[0046] Wherein, wherein the optimization target includes minimizing the weighted average loss function of all participants, improving the prediction accuracy, while taking into account the minimization of communication overhead. The constraint condition is to avoid leaking the local data information of the participants, and the focus is to avoid leaking the original data and the category distribution information.

[0047] The following introduces the establishment of privacy protection constraints in the federated learning scenario. The federated learning algorithm for gradient boosting tree aims to train an integrated learning model based on distributed and private data sets, thus providing a more practical implementation. Without loss of generality, the present application assumes that the communication security during the entire process of federated learning can be guaranteed by a secure channel with an authentication mechanism (irrelevant to the present application). All participants are semi-honest, which means that the participants will follow the protocol but may try to infer the local data information of other participants. The present application considers that collusion among participants can pose a potential security threat to the system, that is, a group of participants may collude to infer the local data information of other participants. In addition, the server may also try to obtain additional statistical information from the training results. Given the above security requirements, the purpose of the present application is to design a privacy-protected federated learning framework so that multiple participants can train an accurate integrated learning model while preventing collusion among semi-honest participants to ensure the data privacy of each participant.

[0048] The specific setting of the optimization target is introduced below.

[0049] Specifically, as Figure 2As shown, step S101 includes establishing an optimization target and establishing a system model. Establishing the optimization target can specifically be establishing a prediction performance target and a privacy protection constraint, and establishing the system model can specifically be establishing a network topology and information flow of the federated learning system.

[0050] S102, set the number of iterations of the integrated learning model, in each iteration, the center server randomly selects a preset number of participants for parallel decision tree training, and each selected participant trains a decision tree using a local data set combined with a differential privacy mechanism.

[0051] In this application, all M participants participate in the training phase and fine-tuning phase of the integrated learning model, and the goal is to efficiently obtain an integrated learning model with good prediction effect in a federated learning scenario. To achieve this goal, the federated learning system first performs T rounds of boosting iteration to generate base learners. When participant m trains a decision tree, in order to protect the data privacy of other participants, only the local data set F (m) of participant m is used for decision tree training. The trained decision tree will be shared with other participants to build a new decision tree.

[0052] S103, each selected participant sends the trained decision tree to the center server for aggregation, the center server aggregates the received decision trees into a base learner, and then sends the base learner to each participant for the next round of iteration, and updates the global integrated learning model according to the base learner until the iteration is completed.

[0053] This application uses H t to represent the base learner generated by the tth round of boosting iteration. Assuming that the user selection rate is κ (0 < κ ≤ 1), that is, in each round of iteration, κM parties are selected for parallel decision tree training, where represents the ceiling, so a base learner contains decision trees.

[0054] According to the update rule of gradient boosting tree, for regression task, the output of integrated learning model on data sample is:

[0055]

[0056] For binary classification task, the output of integrated learning model on sample is:

[0057]

[0058] where,

[0059]

[0060]

[0061] where, denotes an indicator function, which takes the value 1 if the logical argument in the brackets is true, and 0 otherwise.

[0062] The binary classification task output indicates that the prediction result is actually a probability distribution, but specifies the judgment way of the category.

[0063] For the multi-classification task, a decision tree is trained for each category, for example, there are Q categories, and Q decision trees are trained.

[0064] In step S101, the optimization objective of the federated learning system is to minimize the average loss function on all participant datasets. When all participants jointly train the ensemble learning model , the optimization objective is:

[0065]

[0066] For the binary classification task, the common loss function is the mean square error, that is, L(y, F(x)) = (y - F(x)) 2 / 2;

[0067] For the multi-classification task, the multi-classification logistic loss function is used, that is,

[0068]

[0069] For the regression task, the common loss function is the binomial logarithmic approximation, that is, L(y, F(x)) = log(1 + exp{-2F(x)}), where y ∈ {-1, 1}.

[0070] For the non-independent and identically distributed (Non-IID) data set, the optimization objective is set. The local data set of the participant is related to the data record of a specific institution, therefore, the data distribution difference between participants can be completely different, and also different from the global data distribution

[0071] In order to cope with the performance loss caused by the Non-IID data set, a fine-tuning step is introduced in this application, which aims to derive a set of ensemble learning models such that

[0072]

[0073] where, d m () denotes the model on the data set D (m)The predictive performance on the regression task is expressed as the misclassification rate for classification tasks and the mean squared error (MSE) for regression tasks.

[0074] S104. The central server sends the global ensemble learning model to each participant. Each participant uses its local dataset to fine-tune the global ensemble learning model, and finally obtains its own ensemble learning model for prediction.

[0075] Figure 4 A schematic diagram of the parallelized gradient boosting tree training process in the federated learning system of this application is shown. Figure 4 This includes the decision tree training phase and the integration and construction phase.

[0076] Please refer to Figure 2 and Figure 4 In the t-th iteration of boosting, the central server first randomly selects... There are 1 participating party. If t=1, then initialize the global ensemble learning model. The selected participant i then constructs a decision tree based on the local dataset. Then, each selected participant sends its trained decision tree to the central server for aggregation; finally, the central server aggregates the received decision trees into a base learner for the next iteration. Compared to the general sequential decision tree training method, this method can parallelize the construction of decision trees and simultaneously transmit the decision tree model, thereby improving the communication efficiency of gradient boosting tree training. In particular, the construction of each round of decision trees in gradient boosting trees depends on the negative gradients of the previous rounds of decision trees, thus creating a communication bottleneck. The method proposed in this application relaxes this constraint, making parallel construction of decision trees possible.

[0077] The main idea behind parallelizing the construction of gradient boosting trees in this application is to calculate the direction and step size of the steepest gradient descent based on all decision trees in the base learner, thereby inferring the expression for parameter updating based on multiple decision trees.

[0078] The detailed sub-steps of steps S102 and S103 are as follows:

[0079] 1. The central server is selected first. If there are 1 or more iteration rounds, the base learner H obtained from the previous training round is broadcast to the selected participants. t-1 Otherwise, it will not be sent;

[0080] 2. If the number of iterations is greater than or equal to 1, the selected participant m will be selected based on the base learner H sent by the central server. t-1 and local dataset D(u) , train a decision tree Otherwise, only based on the local dataset to build a decision tree. When combining differential privacy mechanism in FedEp framework, differential privacy noise will be injected when building decision tree to ensure privacy, that is, the leaf node value will be disturbed. Therefore, the present application first calculates the leaf node value of each decision tree in the base learner of the last round For binary classification and regression tasks, based on the local data and the partition nodes of the decision tree, the leaf node value and the area of each tree are recalculated:

[0081]

[0082] The calculation method of multi-classification task is as follows:

[0083]

[0084] Then, based on the recalculated leaf node value and area, the average value of the leaf node is calculated according to the designed formula Update the prediction value of the base learner Based on the evolution of the objective function in TreeBoost, γ j,t is the approximate value of the jth terminal node, which can also be regarded as the ideal update value based on the loss function L. Inspired by the gradient (parameter) aggregation method in FedSGD algorithm, the present application calculates

[0085]

[0086] Where, γ ju,t is the leaf node value of the uth decision tree t in the base learner H . q represents the category in multi-classification task.

[0087] Let the base learner sent to the participants in the tth round of boosting iteration participants be H t-1 . Therefore, for binary classification and regression tasks, the decision tree trained by the participant m needs to be updated:

[0088]

[0089] Where, indicates the indicator function, which is 1 when the logical argument in the square bracket is true, and 0 otherwise.

[0090] For multi-classification task, the update value is calculated as follows:

[0091]

[0092] Where, is the data point approximation on the left-hand side, is a decision tree the terminal region of the jth leaf node. The estimate of γ j,t is equivalent to minimizing the intercept between the loss function and disjoint regions:

[0093]

[0094] For binary classification tasks, we use Newton-Raphson equation in this paper for approximation:

[0095]

[0096] For multi-classification tasks, γ qj,t is calculated as follows:

[0097]

[0098] For regression tasks, γ j,t is calculated as follows:

[0099]

[0100] The main idea of parallelizing the construction of gradient boosting trees in this paper is based on the calculation of the direction and step size of the steepest gradient descent in all decision trees in machine learning, so as to deduce the expression of parameter update based on multiple decision trees.

[0101] Based on the evolution mode of the objective function in TreeBoost, γ j,t is the approximation value of the jth terminal node, which can also be regarded as the ideal update value based on the loss function L. Inspired by the gradient (parameter) aggregation method in FedSGD algorithm, we calculate γ

[0102]

[0103] For multi-classification tasks, the value of γ of class q is calculated as follows:

[0104]

[0105] From the calculation of the approximation value, it can be seen that the calculation of the general gradient boosting tree depends on the updated γ value of the decision tree in the last round. In this paper, since multiple decision trees are trained based on the same base learner in each iteration, it will increase the similarity between the decision trees, leading to the decline of the overall generalization performance of the model. Inspired by random forest, we use data sampling and feature sampling to increase the diversity between decision trees.

[0106] 3. For binary classification and regression tasks, for each sample The first and second order derivatives of the base learners at each local data point are calculated using the predicted values as the negative gradient to be fitted:

[0107]

[0108]

[0109] Then a training decision tree is obtained according to the calculation of the negative gradient The residual to be fitted is Then a decision tree is trained on the dataset

[0110] For multi-classification tasks, the residual is calculated as follows:

[0111]

[0112]

[0113] If t = 1, a new decision tree is trained according to the real label of the sampled data. The training of the decision tree is performed concurrently among the selected participants.

[0114] 4. The selected participants send the trained decision tree to the center server; the center server aggregates the received decision trees into a new base learner And update the global ensemble learning model with the base learner

[0115] 5. If the number of iteration rounds reaches the upper limit T, the center server sends the global ensemble learning model to each participant; otherwise, repeat steps 1 to 4.

[0116] The present application introduces a model fine-tuning step based on ensemble pruning to reduce the training error caused by Non-IID data. Ensemble pruning is a method of compressing the parameters of an ensemble learning model, which can effectively select part of the base learners from a set of base learners to form a new ensemble learning model.

[0117] In the federated learning system, since the decision tree is constructed only based on the local data set of a participant, and the base learner is composed of multiple (≤M) decision trees. Directly integrating all base learners may result in lower prediction accuracy. In order to better aggregate the base learners to obtain an ensemble learning model with better prediction performance, the ensemble pruning algorithm GASEN is improved in the present application as an example of using ensemble pruning method to improve performance. ​

[0118] GASEN stands for: A genetic algorithm based on the selective integration algorithm for neural networks.

[0119] A significant advantage of ensemble pruning algorithms is that they can produce ensemble learning models with lower generalization errors, allowing for the selection of more suitable base learners for each participant to form the ensemble learning model. Specifically, assuming the ensemble learning model... The weights of the j-th base learner are w j In this application, a weight vector w = (w1, w2, ..., w3) is used. T The weights are used to characterize the fitness of each base learner in the ensemble learning model, while weights greater than a preset threshold are considered. The base learner is retained in the ensemble learning model; otherwise, the base learner is excluded.

[0120] The weights of the base learners are determined by running an evolutionary algorithm, where the fitness is calculated as follows: in This indicates that weights w are assigned to the base learner and set at a preset threshold. The prediction error rate of the ensemble learning model is obtained by making a judgment.

[0121] In this application, each participant fine-tunes the global ensemble learning model based on its local dataset using a pre-defined ensemble pruning algorithm, specifically including:

[0122] Based on the local dataset, the weight of each base learner in the ensemble learning model is calculated using a preset ensemble pruning algorithm. The weight is used to characterize the fitness of the base learner to be added to the ensemble learning model.

[0123] Base learners with weights greater than a preset threshold are retained in the ensemble learning model, while the remaining base learners are excluded, resulting in a fine-tuned ensemble learning model.

[0124] In this application, the preset ensemble pruning algorithm includes a genetic algorithm; please refer to [reference needed]. Figure 2 The specific steps of the genetic algorithm in this application include:

[0125] 1. Each participant m randomly samples T from its local dataset. ep Datasets

[0126]

[0127] 2. Each participant m randomly generates a weight vector of P base learners as the initial population of the genetic algorithm. The weight vector w = (w1, w2,..., wP) is used to represent the fitness of each base learner to join the ensemble learning model, and the base learner with a weight value greater than a preset threshold is retained in the ensemble learning model, otherwise the base learner is excluded; T

[0128] 3. For each weight vector in the population, cross each other with a cross probability p cross to produce new individuals;

[0129] 4. For each individual, mutate any gene on the individual with a mutation probability p mut ;

[0130] 5. For each individual, calculate the fitness according to the error rate on the local data set ;

[0131] 6. Select the top N individuals with the highest fitness to enter the next round of genetic algorithm iteration;

[0132] 7. If the iteration round is less than T ep , repeat steps 1 to 6 above; otherwise, select the individual with the highest fitness as the final weight of the base learner. If the weight of the base learner is greater than a preset threshold , the corresponding base learner is retained, otherwise the base learner is excluded from the ensemble model.

[0133] Finally, a series of ensemble learning models will be obtained for the prediction of new data sets on participants 1, 2,..., M, respectively. For example, only for the prediction of the data set on participant m.

[0134] Compared with the prior art, the beneficial effects of the ensemble learning model and its construction method in the federated learning system of the present application are as follows:

[0135] a) Realize the feasibility of concurrent training of decision trees

[0136] The construction method of the traditional ensemble learning model is to train the decision trees sequentially, which cannot train multiple decision trees at the same time. The federated learning system of the present application adopts a star-shaped network architecture, establishes the information flow of the federated learning system in parallel transmission, designs the residual calculation method of parallel training of multiple decision trees, thereby improving the communication efficiency of the federated learning system, and realizing the feasibility of concurrent training of decision trees.

[0137] b) Realize the feasibility of performance improvement under privacy constraints

[0138] ​For the problem of non-independent and identically distributed data sets, the application adopts an integrated pruning algorithm to fine-tune the integrated learning model on heterogeneous data sets, and uses the locally fine-tuned model to replace the global integrated learning for prediction, so that the customized integrated learning model has better adaptability to different data distributions, thereby improving the prediction performance.

[0139] Another embodiment of the application also provides an integrated learning model created by the construction method of the integrated learning model in the federated learning system.

[0140] It should be noted that the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the system, method and computer program product according to the embodiments of the application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0142] In several embodiments provided by the application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0143] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0144] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0145] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various program code storage media.

[0146] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.

Claims

1. A method for constructing an integrated learning model in a federated learning system, the method comprising: The application relates to a method for constructing an ensemble learning model in a federated learning system. The method comprises the following steps: determining optimization objectives and constraint conditions of the federated learning system; the federated learning system comprises a center server and a plurality of participants, the plurality of participants participate in training of an ensemble learning model while avoiding exposure of local data, and the federated learning system adopts a star-shaped network architecture, and the plurality of participants can simultaneously communicate with the center server; setting an iteration number of the ensemble learning model, in each iteration, the center server randomly selects a preset number of participants for parallel decision tree training, each selected participant trains a decision tree by using a local data set and combining a differential privacy mechanism; each selected participant sends the trained decision tree to the center server for aggregation, the center server aggregates the received decision trees into a base learner, sends the base learner to each participant for next round iteration, and updates a global ensemble learning model according to the base learner until iteration is completed; the center server sends the global ensemble learning model to each participant, and each participant fine-tunes the global ensemble learning model by using a local data set to finally obtain an ensemble learning model used for prediction by each participant; the fine-tuning of the global ensemble learning model by each participant by using the local data set comprises the following steps: each participant fine-tunes the global ensemble learning model by using a preset ensemble pruning algorithm based on the local data set; the fine-tuning of the global ensemble learning model by each participant by using the local data set and the preset ensemble pruning algorithm comprises the following steps: based on the local data set, the preset ensemble pruning algorithm is used to calculate the weight of each base learner in the ensemble learning model, and the weight is used to represent the adaptability of the base learner to the ensemble learning model; base learners with weights greater than a preset threshold are retained in the ensemble learning model, and the remaining base learners are excluded from the ensemble learning model to obtain a fine-tuned ensemble learning model.

2. The method of claim 1, wherein, The preset ensemble pruning algorithm comprises a genetic algorithm.

3. The method of claim 2, wherein, the calculation of the weight of each base learner in the ensemble learning model by using the local data set and the preset ensemble pruning algorithm comprises the following steps: Parties m randomly sample from local data sets Individual data sets ​ a participant m randomly generates P weight vectors of base learners as initial populations of the genetic algorithm; For each weight vector in the population, with a crossover probability new individuals are generated by pairwise crossover; For each individual, at the mutation probability mutating any gene on the individual; for each individual, an error rate on the local dataset according to the fine-tuned ensemble learning model computing the fitness ; wherein, denotes the weight of the base learner, denotes a preset threshold value for fine-tuning the ensemble learning model; N individuals with the highest adaptability are selected to enter next round genetic algorithm iteration; The iteration round reaches The individual with the highest fitness is selected as the final weight of the base learner.

4. An ensemble learning model apparatus, comprising: The ensemble learning model device is created by using the method for constructing an ensemble learning model in the federated learning system according to any one of claims 1-3.

Images