Training methods, devices, and electronic devices of federated learning models

Abstract

The present application provides a method, apparatus, and electronic device for training a federated learning model. The training method of a federated learning model applied to a server comprises: a step of obtaining a target splitting method corresponding to a training node when the training node satisfies a preset splitting condition—wherein the training node is a node of one of a plurality of boosting trees—; a step of notifying a client to perform node splitting based on the target splitting method; a step of making the left subtree node created after the training node splits the training node again and performing the next round of training until the updated training node no longer satisfies the preset splitting condition; a step of making the other non-leaf node of one boosting tree the training node again and performing the next round of training; and a step of stopping training and creating a target federated learning model when the node data set of the plurality of boosting trees is all empty. Thus, by mixing transverse and longitudinal splitting methods in the present application, the tendency to automatically select a matching learning method improves the performance of the federated learning model without needing to be concerned with the data distribution method.

Description

Technology Field

[0001] (Cross-reference to related applications) The present application claims priority to the Chinese patent application number "202011621994.0" filed by JD Digital Technology Holding Co., Ltd. on December 31, 2020, with the invention title "Method, apparatus and electronic device for training a federated learning model" and Chinese patent application number "202011617342.X" filed by JD Digital Technology Holding Co., Ltd. on December 31, 2020, with the invention title "Method, apparatus and electronic device for training a federated learning model".

[0002] This application relates to the field of communication technology, and in particular to a training method, apparatus, and electronic device for a federated learning model. Background Technology

[0003] Federated learning is a new foundational technology for artificial intelligence. Its design objective is to ensure information security during big data exchange, protect the privacy of terminal and personal data, and deploy highly efficient machine learning among multiple participants or computational nodes under legal conditions. Here, the machine learning algorithms available through federated learning are not limited to neural networks but also include other important algorithms such as random forests. Federated learning will serve as the foundation for next-generation AI cooperative algorithms and collaborative networks.

[0004] Based on data characteristics, federated learning is primarily divided into lateral federated learning and longitudinal federated learning. Here, lateral federated learning requires homogeneous data, while longitudinal federated learning requires heterogeneous data. However, it is difficult to guarantee complete heterogeneity or homogeneity with multi-dimensional data used for joint federated learning. Therefore, when training a model based on federated learning, partially homogeneous or heterogeneous data must be discarded. If the amount of discarded data is relatively large, the performance of the model obtained by training based on federated learning deteriorates.

[0005] The present application aims to solve at least one technical problem of the related technology to some extent.

[0006] Thus, the first objective of the present application is to provide a training method for a federated learning model and to solve a technical problem in which training is not possible by fully utilizing all the data present in the process of a conventional federated learning model training method, resulting in poor training effects due to insufficient data utilization.

[0007] The second objective of the present invention is to provide a training method for other federated learning models.

[0008] The third objective of the present invention is to provide a training device for a federated learning model.

[0009] The fourth objective of the present invention is to provide a training device for other federated learning models.

[0010] The fifth objective of the present invention is to provide an electronic device.

[0011] The sixth object of the present invention provides a computer-readable storage medium.

[0012] To implement the above objective, an embodiment of the first aspect of the present application provides a method for training a federated learning model applied to a server, the method comprising: a step of obtaining a target splitting method corresponding to a training node when the training node satisfies a preset splitting condition, wherein the training node is a node of one of a plurality of boosting trees; a step of notifying a client to perform node splitting based on the target splitting method; a step of performing the next round of training by making the left subtree node created after the training node is split the training node again until the updated training node no longer satisfies the preset splitting condition; a step of performing the next round of training by making the other non-leaf node of the one boosting tree the training node again; and a step of stopping training and creating a target federated learning model when the node data set of the plurality of boosting trees is all empty.

[0013] In addition, the training method of the federated learning model of the above embodiment of the present application may further include the following additional techniques.

[0014] According to one embodiment of the present application, the step of obtaining a target splitting method corresponding to the training node comprises: a step of performing lateral federated learning in cooperation with the client based on a first training set and obtaining a first splitting value corresponding to the training node; a step of performing longitudinal federated learning in cooperation with the client based on a second training set and obtaining a second splitting value corresponding to the training node; and a step of determining a target splitting method corresponding to the training node according to the first splitting value and the second splitting value.

[0015] According to one embodiment of the present application, the step of determining a target splitting method corresponding to the training node according to the first splitting value and the second splitting value comprises: determining a relatively large value among the first splitting value and the second splitting value as the target splitting value corresponding to the training node; and determining a splitting method corresponding to the training node according to the target splitting value.

[0016] According to one embodiment of the present application, the step of obtaining a first split value corresponding to the training node by performing lateral federated learning in cooperation with the client based on the first training set comprises: generating a first feature subset available to the training node in the first training set and transmitting it to the client; receiving a feature value of each feature of the first feature subset transmitted from the client; determining each feature as a lateral split value corresponding to a split feature point according to the feature value of each feature of the first feature subset; and determining the first split value of the training node according to the lateral split value corresponding to each feature.

[0017] According to one embodiment of the present application, the step of determining each feature as a transverse splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset comprises: a step of determining a splitting threshold of any feature according to the feature value of any feature of the first feature subset; a step of obtaining a first data instance identifier set and a second data instance identifier set corresponding to the arbitrary feature according to the splitting threshold, wherein the first data instance identifier set includes a data instance identifier belonging to a first left subtree space and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space; and a step of determining the transverse splitting value corresponding to the arbitrary feature according to the first data instance identifier set and the second data instance identifier set.

[0018] According to one embodiment of the present application, the step of obtaining a first data instance identifier set and a second data instance identifier set corresponding to any feature according to the splitting threshold comprises: transmitting the splitting threshold to the client; receiving an initial data instance identifier set corresponding to the training node transmitted from the client, wherein the initial data instance identifier set is generated when the client splits the node for any feature according to the splitting threshold, and the initial data instance identifier set includes a data instance identifier belonging to the first left subtree space; and obtaining the first data instance identifier set and the second data instance identifier set based on the initial data instance identifier set and all data instance identifiers.

[0019] According to one embodiment of the present application, the step of obtaining the first data instance identifier set and the second data instance identifier set based on the first data instance identifier set comprises: comparing each data instance identifier of the first data instance identifier set with the data instance identifier of the client and obtaining an anomalous data instance identifier; performing preprocessing on the anomalous data instance identifier and obtaining the first data instance identifier set; and obtaining the second data instance identifier set based on all data instance identifiers and the first data instance identifier set.

[0020] According to one embodiment of the present application, the step of performing longitudinal federated learning in cooperation with the client based on the second training set and obtaining a second split value corresponding to the training node comprises: a step of notifying the client to perform longitudinal federated learning based on the second training set; a step of receiving first gradient information of at least one third data instance identifier set of each feature transmitted from the client - wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting one feature value among the features, and different feature values ​​correspond to different second left subtree spaces -; a step of determining a longitudinal split value of each feature according to the first gradient information of each feature and the total gradient information of the training node; and a step of determining the second split value of the training node according to the longitudinal split value corresponding to each feature.

[0021] According to one embodiment of the present application, the step of determining a longitudinal split value of each feature according to the first slope information of each feature and the total slope information of the training node comprises: for any feature, obtaining a second slope information corresponding to each first slope information according to the total slope information and each first slope information; for each first slope information, obtaining a candidate longitudinal split value of the arbitrary feature according to the first slope information and the second slope information corresponding to the first slope information; and selecting a maximum value among the candidate longitudinal split values ​​and setting it as the longitudinal split value of the arbitrary feature.

[0022] According to one embodiment of the present application, the first slope information includes a first slope sum of features corresponding to a data instance belonging to the second left subtree space and a second slope sum of features corresponding to a data instance belonging to the second left subtree space; and the second slope information includes a first slope sum of features corresponding to a data instance belonging to the second right subtree space and a second slope sum of features corresponding to a data instance belonging to the second right subtree space.

[0023] According to one embodiment of the present application, if the training node does not satisfy a preset splitting condition, the method further includes the steps of determining the training node as a leaf node and obtaining the weight of the leaf node; and transmitting the weight of the leaf node to the client.

[0024] According to one embodiment of the present application, the step of obtaining the weight of the leaf node comprises: obtaining a data instance belonging to the leaf node; obtaining first gradient information and second gradient information of the data instance belonging to the leaf node, and obtaining the weight of the leaf node according to the first gradient information and the second gradient information.

[0025] According to one embodiment of the present application, prior to the step of notifying the client to perform node splitting based on the target splitting method, the method further comprises the step of transmitting splitting information to the client, wherein the splitting information includes the target splitting method, the target splitting feature selected as a feature splitting point, and the target splitting value.

[0026] According to one embodiment of the present application, when the target splitting method is a longitudinal splitting method, prior to the step of notifying the client to perform node splitting based on the target splitting method, the method further comprises: the step of transmitting splitting information to a client having a label; the step of receiving a left subtree space set transmitted from the client having the label; the step of splitting the second training set according to the left subtree space set; and the step of associating the training node with the identifier of the client having the label.

[0027] According to one embodiment of the present application, prior to the step of obtaining a target splitting method corresponding to the training node when the training node satisfies a preset splitting condition, the method further comprises: receiving a data instance identifier transmitted from the client; determining a common data instance identifier between clients according to the data instance identifier - the common data instance identifier is used to instruct the client to determine the first training set and the second training set -;

[0028] According to one embodiment of the present application, after the step of notifying the client to perform node splitting based on the target splitting method, the method comprises: the step of obtaining the updated training node; the step of determining that the updated training node satisfies a training cessation condition, stopping the training, and generating a target federated learning model; and the step of obtaining a validation set and validating the target federated learning model in cooperation with a validation client - the validation client is one of the clients participating in the federated learning model training -;

[0029] According to one embodiment of the present application, the step of verifying the target model in cooperation with a verification client based on the verification set comprises: transmitting to the verification client a data instance identifier of one of the verification sets and splitting information of a verification node - said verification node is a node of one of a plurality of boosting trees -; receiving a node direction corresponding to said verification node transmitted from the verification client - said node direction is determined by the verification client according to the data instance identifier and splitting information -; entering the next node according to said node direction and making said next node the updated verification node; and if said updated verification node satisfies the preset node splitting condition, returning until all data instance identifiers of the verification set are verified and transmitting said data instance identifier and said splitting information to the verification client;

[0030] According to one embodiment of the present application, if the updated verification node does not satisfy the preset node splitting condition, the method further includes the step of determining the updated verification node as a leaf node and obtaining a model prediction value of a data instance represented by the data instance identifier.

[0031] According to one embodiment of the present application, the method further comprises: a step of transmitting a model prediction value of a data instance to a verification client when all data instance identifiers of the verification set are verified; a step of receiving verification instruction information transmitted from the verification client, wherein the verification instruction information is an instruction information obtained according to the model prediction value and indicating whether to suspend the model; and a step of determining whether to suspend or use the target federated learning model according to the verification instruction information, and transmitting the decision result to the client.

[0032] According to one embodiment of the present application, the step of obtaining a target splitting method corresponding to the training node comprises: a step of performing lateral federated learning in cooperation with the client based on a first training set and obtaining a first splitting value corresponding to the training node; a step of performing longitudinal federated learning in cooperation with the client based on a second training set and obtaining a second splitting value corresponding to the training node; and a step of determining a target splitting method corresponding to the training node according to the first splitting value and the second splitting value.

[0033] According to one embodiment of the present application, the step of determining a target splitting method corresponding to the training node according to the first splitting value and the second splitting value comprises: determining a relatively large value among the first splitting value and the second splitting value as the target splitting value corresponding to the training node; and determining a splitting method corresponding to the training node according to the target splitting value.

[0034] According to one embodiment of the present application, the step of performing lateral federated learning in cooperation with the client based on the first training set and obtaining a first splitting value corresponding to the training node comprises: generating a first feature subset available to the training node in the first training set and transmitting it to the client; receiving a feature value of each feature of the first feature subset transmitted from the client; determining each feature as a lateral splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset; and determining the first splitting value of the training node according to the lateral splitting value corresponding to each feature.

[0035] According to one embodiment of the present application, the step of determining each feature as a transverse splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset comprises: a step of determining a splitting threshold of any feature according to the feature value of any feature of the first feature subset; a step of obtaining a first data instance identifier set and a second data instance identifier set corresponding to the arbitrary feature according to the splitting threshold, wherein the first data instance identifier set includes a data instance identifier belonging to a first left subtree space and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space; and a step of determining the transverse splitting value corresponding to the arbitrary feature according to the first data instance identifier set and the second data instance identifier set.

[0036] According to one embodiment of the present application, the step of obtaining a first data instance identifier set and a second data instance identifier set corresponding to any feature according to the splitting threshold comprises: transmitting the splitting threshold to the client; receiving an initial data instance identifier set corresponding to the training node transmitted from the client, wherein the initial data instance identifier set is generated when the client splits the node for any feature according to the splitting threshold, and the initial data instance identifier set includes a data instance identifier belonging to the first left subtree space; and obtaining the first data instance identifier set and the second data instance identifier set based on the initial data instance identifier set and all data instance identifiers.

[0037] According to one embodiment of the present application, the step of performing longitudinal federated learning in cooperation with the client based on the second training set and obtaining a second split value corresponding to the training node comprises: a step of notifying the client to perform longitudinal federated learning based on the second training set; a step of receiving first gradient information of at least one third data instance identifier set of each feature transmitted from the client - wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, and the second left subtree space is a left subtree space formed by splitting one feature value among the features, and different feature values ​​correspond to different second left subtree spaces -; a step of determining a longitudinal split value of each feature according to the first gradient information of each feature and the total gradient information of the training node; and a step of determining the second split value of the training node according to the longitudinal split value corresponding to each feature.

[0038] According to one embodiment of the present application, the step of determining a longitudinal split value of each feature according to the first slope information of each feature and the total slope information of the training node comprises: for any feature, obtaining a second slope information corresponding to each first slope information according to the total slope information and each first slope information; for each first slope information, obtaining a candidate longitudinal split value of the arbitrary feature according to the first slope information and the second slope information corresponding to the first slope information; and selecting a maximum value among the candidate longitudinal split values ​​and setting it as the longitudinal split value of the arbitrary feature.

[0039] According to one embodiment of the present application, the verification set is mutually exclusive with the first training set and the second training set, respectively.

[0040] An embodiment of the first aspect of the present application provides a training method for a federated learning model, wherein the tendency of a server to automatically select a matching learning method by mixing a lateral splitting method and a longitudinal splitting method does not require attention to the data distribution method, and solves the technical problem that a conventional federated learning model training method cannot learn by fully utilizing all data present in the process and results in poor training effect due to insufficient data utilization, while simultaneously reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0041] To implement the above objective, an embodiment of the second aspect of the present application provides a method for training a federated learning model applied to a client, the method comprising: receiving a target splitting method transmitted when a server determines that a training node satisfies a preset splitting condition - said training node is a node of one of a plurality of boosting trees -; and performing node splitting on said training node based on said target splitting method.

[0042] According to one embodiment of the present application, prior to the step of performing node splitting for the training node based on the target splitting method, the method further comprises: a step of performing lateral federated learning based on a first training set and obtaining a first splitting value corresponding to the training node; a step of performing longitudinal federated learning based on a second training set and obtaining a second splitting value corresponding to the training node; and a step of transmitting the first splitting value and the second splitting value to the server.

[0043] According to one embodiment of the present application, the step of performing lateral federated learning based on the first training set and obtaining a first split value corresponding to the training node comprises: receiving a first feature subset available to the training node generated by the server in the first training set; transmitting a feature value of each feature of the first feature subset to the server; receiving a split threshold of each feature transmitted by the server; and obtaining an initial data instance identifier set corresponding to the training node based on the split threshold of each feature and transmitting the initial data instance identifier set to the server; wherein the initial data instance identifier set is used to instruct the server to generate a first data instance identifier set and a second data instance identifier set, and both the first data instance identifier set and the initial data instance identifier set include a data instance identifier belonging to a first left subtree space, and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space.

[0044] According to one embodiment of the present application, the step of obtaining an initial data instance identifier set corresponding to the training node based on the splitting threshold of each feature comprises, for any feature, comparing the splitting threshold of the arbitrary feature with the feature value of the arbitrary feature, obtaining a data instance identifier in which the feature value is smaller than the splitting threshold, and generating the initial data instance identifier set.

[0045] According to one embodiment of the present application, prior to the step of performing longitudinal federated learning based on a second training set and obtaining a second split value corresponding to the training node, the method comprises: receiving a gradient information request transmitted from the server; generating a second feature subset from the second training set according to the gradient information request; obtaining a first gradient information of at least one third data instance identifier set of each feature of the second feature subset, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting one feature value of the feature, and different feature values ​​correspond to different second left subtree spaces; and transmitting the first gradient information of the third data instance identifier set to the server.

[0046] According to one embodiment of the present application, the step of obtaining first slope information of at least one third data instance identifier set of each feature of the second feature subset comprises: for any feature, obtaining all feature values ​​of said arbitrary feature and bucketing said arbitrary feature based on said feature values; and obtaining first slope information of each bucketed third data instance identifier set of said arbitrary feature.

[0047] According to one embodiment of the present application, the step of performing node splitting for the training node based on the target splitting method comprises: receiving splitting information transmitted from the server - the splitting information includes the target splitting method, the target splitting feature selected as a feature splitting point, and the target splitting value -; and performing node splitting for the training node based on the splitting information.

[0048] According to one embodiment of the present application, after the step of performing node splitting for the training node, the method further includes the step of transmitting the split-generated left subtree space to the server.

[0049] According to one embodiment of the present application, if the training node is a leaf node, the method further comprises: receiving the weight of the leaf node transmitted from the server; determining the residual of each included data according to the weight of the leaf node; and using the residual as the residual input of the next boosting tree.

[0050] According to one embodiment of the present application, after the step of performing node splitting for the training node based on the target splitting method, the method further includes the step of receiving a validation set transmitted from the server and validating the target federated learning model based on the validation set.

[0051] According to one embodiment of the present application, the step of verifying the target model in cooperation with a verification client based on the verification set comprises: receiving a data instance identifier of one of the verification sets and splitting information of a verification node transmitted from the server - the verification node is a node of one of a plurality of boosting trees -; determining the node direction of the verification node according to the data instance identifier and the splitting information; and the node direction transmitted to the server causes the server to enter the next node according to the node direction and sets the next node as the updated verification node.

[0052] According to one embodiment of the present application, the step of determining the node direction of the verification node according to the data instance identifier and the splitting information comprises: the step of determining the feature value of each feature corresponding to the data instance identifier according to the data instance identifier; and the step of determining the node direction according to the splitting information and the feature value of each feature.

[0053] According to one embodiment of the present application, when all data instance identifiers of the verification set are verified, the method further comprises: receiving a model prediction value of a data instance represented by the data instance identifier transmitted from the server; obtaining a final verification result according to the model prediction value and comparing the verification result with a previous verification result to generate verification instruction information indicating whether to suspend or use the target federated learning model; and transmitting the verification instruction information to the server.

[0054] According to one embodiment of the present application, prior to the step of performing node splitting for the training node based on the target splitting method, the method further comprises: a step of performing lateral federated learning based on a first training set and obtaining a first splitting value corresponding to the training node; a step of performing longitudinal federated learning based on a second training set and obtaining a second splitting value corresponding to the training node; and a step of transmitting the first splitting value and the second splitting value to the server.

[0055] According to one embodiment of the present application, the step of performing lateral federated learning based on the first training set and obtaining a first split value corresponding to the training node comprises: receiving a first feature subset available to the training node generated by the server in the first training set; transmitting a feature value of each feature of the first feature subset to the server; receiving a split threshold of each feature transmitted by the server; obtaining an initial data instance identifier set corresponding to the training node based on the split threshold of each feature, and transmitting the initial data instance identifier set to the server, wherein the initial data instance identifier set is used to instruct the server to generate a first data instance identifier set and a second data instance identifier set, and both the first data instance identifier set and the initial data instance identifier set include a data instance identifier belonging to a first left subtree space, and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space.

[0056] According to one embodiment of the present application, the step of obtaining an initial data instance identifier set corresponding to the training node based on the splitting threshold of each feature comprises, for any feature, comparing the splitting threshold of the arbitrary feature with the feature value of the arbitrary feature, obtaining a data instance identifier in which the feature value is smaller than the splitting threshold, and generating the initial data instance identifier set.

[0057] According to one embodiment of the present application, prior to the step of performing longitudinal federated learning based on the second training set and obtaining a second split value corresponding to the training node, the method further comprises: receiving a gradient information request transmitted from the server; generating a second feature subset from the second training set according to the gradient information request; obtaining a first gradient information of at least one third data instance identifier set of each feature of the second feature subset, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting one of the feature values, and different feature values ​​correspond to different second left subtree spaces; and transmitting the first gradient information of the third data instance identifier set to the server.

[0058] According to one embodiment of the present application, the step of obtaining first gradient information of at least one third data instance identifier set of each feature of the second feature subset is,

[0059] For any feature, the method includes the step of obtaining all feature values ​​of the arbitrary feature and bucketing the arbitrary feature based on the feature values; and the step of obtaining first slope information of each bucketed set of third data instance identifiers of the arbitrary feature.

[0060] An embodiment of the second aspect of the present application provides a training method for a federated learning model, wherein a client receives a target splitting method transmitted when a training node satisfies a preset splitting condition from a server, and the training node is a node of one of a plurality of boosting trees, and performs node splitting on the training node based on the target splitting method, and by mixing a transverse splitting method and a longitudinal splitting method, the tendency to automatically select a matching learning method does not require interest in the data distribution method, and solves the technical problem that the training effect is poor due to insufficient data utilization and insufficient use of all data existing in the process of the training method of a conventional federated learning model, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0061] To implement the above objective, an embodiment of the third aspect of the present application provides a training device for a federated learning model applied to a server, the device comprising: an acquisition module that acquires a target splitting method corresponding to the training node when the training node satisfies a preset splitting condition, wherein the training node is a node of one of a plurality of boosting trees; a notification module that notifies a client to perform node splitting based on the target splitting method; a first training module that performs the next round of training by making the left subtree node created after the training node is split the training node again until the updated training node no longer satisfies the preset splitting condition; a second training module that performs the next round of training by making the other non-leaf node of the one boosting tree the training node again; and a generation module that stops training and creates a target federated learning model when the node data set of the plurality of boosting trees is all empty.

[0062] According to one embodiment of the present application, the acquisition module comprises: a first learning submodule that performs lateral federated learning in cooperation with the client based on a first training set to acquire a first splitting value corresponding to the training node; a second learning submodule that performs longitudinal federated learning in cooperation with the client based on a second training set to acquire a second splitting value corresponding to the training node; and a decision submodule that determines a target splitting method corresponding to the training node according to the first splitting value and the second splitting value.

[0063] According to one embodiment of the present application, the decision submodule comprises: a first decision unit that determines a relatively large value among the first splitting value and the second splitting value as a target splitting value corresponding to the training node; and a second decision unit that determines a splitting method corresponding to the training node according to the target splitting value.

[0064] According to one embodiment of the present application, the first learning submodule comprises: a generating unit that generates a first feature subset available to the training node in the first training set and transmits it to the client; a first receiving unit that receives a feature value of each feature of the first feature subset transmitted from the client; a third determining unit that determines each feature as a transverse splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset; and a fourth determining unit that determines the first splitting value of the training node according to the transverse splitting value corresponding to each feature.

[0065] According to one embodiment of the present application, the third decision unit comprises: a first decision sub-unit that determines a splitting threshold of any feature of the first feature sub-set according to a feature value of the any feature; a first acquisition sub-unit that acquires a first data instance identifier set and a second data instance identifier set corresponding to the any feature according to the splitting threshold, wherein the first data instance identifier set includes a data instance identifier belonging to a first left subtree space and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space; and a second decision sub-unit that determines a transverse splitting value corresponding to the any feature according to the first data instance identifier set and the second data instance identifier set.

[0066] According to one embodiment of the present application, the first acquisition sub-unit also transmits the splitting threshold to the client; receives an initial data instance identifier set corresponding to the training node transmitted from the client, wherein the initial data instance identifier set is generated when the client splits the node for any feature according to the splitting threshold, and the initial data instance identifier set includes data instance identifiers belonging to the first left subtree space; and is used to acquire the first data instance identifier set and the second data instance identifier set based on the initial data instance identifier set and all data instance identifiers.

[0067] According to one embodiment of the present application, the acquisition sub-unit is also used to compare each data instance identifier of the first data instance identifier set with the data instance identifier of the client and to acquire an anomalous data instance identifier; to perform preprocessing on the anomalous data instance identifier and to acquire the first data instance identifier set; and to acquire the second data instance identifier set based on all data instance identifiers and the first data instance identifier set.

[0068] According to one embodiment of the present application, the second learning submodule comprises: a notification unit that notifies the client to perform longitudinal federated learning based on the second training set; a receiving unit that receives first gradient information of at least one third data instance identifier set of each feature transmitted from the client, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting one of the feature values, and different feature values ​​correspond to different second left subtree spaces; a fifth decision unit that determines a longitudinal split value of each feature according to the first gradient information of each feature and the total gradient information of the training node; and a sixth decision unit that determines the second split value of the training node according to the longitudinal split value corresponding to each feature.

[0069] According to one embodiment of the present application, the fifth determination unit comprises: a second acquisition sub-unit that, for any feature, acquires second slope information corresponding to each first slope information according to the total slope information and each first slope information; a third acquisition sub-unit that, for each first slope information, acquires candidate longitudinal splitting values ​​of the arbitrary feature according to the first slope information and the second slope information corresponding to the first slope information; and a selection sub-unit that selects a maximum value among the candidate longitudinal splitting values ​​and sets it as the longitudinal splitting value of the arbitrary feature.

[0070] According to one embodiment of the present application, the first slope information includes a first slope sum of features corresponding to a data instance belonging to the second left subtree space and a second slope sum of features corresponding to a data instance belonging to the second left subtree space; and the second slope information includes a first slope sum of features corresponding to a data instance belonging to the second right subtree space and a second slope sum of features corresponding to a data instance belonging to the second right subtree space.

[0071] According to one embodiment of the present application, the device further includes: a determination module that determines the training node as a leaf node and obtains the weight of the leaf node when the training node does not satisfy a preset splitting condition; and a transmission module that transmits the weight of the leaf node to the client.

[0072] According to one embodiment of the present application, the decision module comprises: a first acquisition unit for acquiring a data instance belonging to the leaf node; and a second acquisition unit for acquiring first gradient information and second gradient information of the data instance belonging to the leaf node, and acquiring a weight of the leaf node according to the first gradient information and the second gradient information.

[0073] According to one embodiment of the present application, the decision submodule further includes a transmission unit that transmits splitting information to the client, and the splitting information includes the target splitting method, the target splitting feature selected as the feature splitting point, and the target splitting value.

[0074] According to one embodiment of the present application, the transmitting unit also transmits the splitting information to a client having a label; receives a left subtree space set transmitted from the client having a label; splits the second training set according to the left subtree space set; and is used to associate the training node with the identifier of the client having a label.

[0075] According to one embodiment of the present application, the acquisition module also receives a data instance identifier transmitted from the client; and, based on the data instance identifier, is used to determine a common data instance identifier between the clients, and the common data instance identifier is used to instruct the clients to determine the first training set and the second training set.

[0076] According to one embodiment of the present application, the notification module is also used to acquire the updated training node; the generation module is also used to determine that the updated training node satisfies a training stop condition, stop training, and generate a target federated learning model; the device further comprises a verification module that acquires a verification set and verifies the target federated learning model in cooperation with a verification client, wherein the verification client is one of the clients participating in the federated learning model training.

[0077] According to one embodiment of the present application, the verification module comprises: a first transmission submodule that transmits a data instance identifier of one of the verification sets and splitting information of a verification node to the verification client, wherein the verification node is a node of one of a plurality of boosting trees; a first reception submodule that receives a node direction corresponding to the verification node transmitted from the verification client, wherein the node direction is determined by the verification client according to the data instance identifier and splitting information; a node update submodule that enters the next node according to the node direction and sets the next node as the updated verification node; and a second transmission submodule that returns and transmits the data instance identifier and splitting information to the verification client until all data instance identifiers of the verification set are verified when the updated verification node satisfies the preset node splitting condition.

[0078] According to one embodiment of the present application, the verification module further includes an acquisition submodule that determines the updated verification node as a leaf node and obtains a model predicted value of a data instance represented by the data instance identifier when the updated verification node does not satisfy the preset node splitting condition.

[0079] According to one embodiment of the present application, the verification module further comprises: a third transmission sub-module that transmits a model prediction value of the data instance to the verification client when all data instance identifiers of the verification set are verified; a second receiving sub-module that receives verification instruction information transmitted from the verification client, wherein the verification instruction information is an instruction information obtained according to the model prediction value and indicates whether to suspend the model; and a decision sub-module that determines whether to suspend and use the target federated learning model according to the verification instruction information, and transmits the decision result to the client.

[0080] According to one embodiment of the present application, the acquisition module comprises: a first learning submodule that performs lateral federated learning in cooperation with the client based on a first training set to acquire a first splitting value corresponding to the training node; a second learning submodule that performs longitudinal federated learning in cooperation with the client based on a second training set to acquire a second splitting value corresponding to the training node; and a decision submodule that determines a target splitting method corresponding to the training node according to the first splitting value and the second splitting value.

[0081] According to one embodiment of the present application, the decision submodule comprises: a first decision unit that determines a relatively large value among the first splitting value and the second splitting value as a target splitting value corresponding to the training node; and a second decision unit that determines a splitting method corresponding to the training node according to the target splitting value.

[0082] According to one embodiment of the present application, the first learning submodule comprises: a transmitting unit that generates a first feature subset available to the training node in the first training set and transmits it to the client; a first receiving unit that receives a feature value of each feature of the first feature subset transmitted from the client; a third determining unit that determines each feature as a transverse splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset; and a fourth determining unit that determines the first splitting value of the training node according to the transverse splitting value corresponding to each feature.

[0083] According to one embodiment of the present application, the third decision unit comprises: a first decision sub-unit that determines a splitting threshold of any feature of the first feature sub-set according to a feature value of the any feature; a first acquisition sub-unit that acquires a first data instance identifier set and a second data instance identifier set corresponding to the any feature according to the splitting threshold, wherein the first data instance identifier set includes a data instance identifier belonging to a first left subtree space and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space; and a second decision sub-unit that determines a transverse splitting value corresponding to the any feature according to the first data instance identifier set and the second data instance identifier set.

[0084] According to one embodiment of the present application, the first acquisition sub-unit also transmits the splitting threshold to the client; receives an initial data instance identifier set corresponding to the training node transmitted from the client, wherein the initial data instance identifier set is generated when the client splits the node for any feature according to the splitting threshold, and the initial data instance identifier set includes a data instance identifier belonging to the first left subtree space; and is used to acquire the first data instance identifier set and the second data instance identifier set based on the initial data instance identifier set and all data instance identifiers.

[0085] According to one embodiment of the present application, the second learning submodule comprises: a notification unit that notifies the client to perform longitudinal federated learning based on the second training set; a second receiving unit that receives first gradient information of at least one third data instance identifier set of each feature transmitted from the client, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces; a fifth decision unit that determines a longitudinal split value of each feature according to the first gradient information of each feature and the total gradient information of the training node; and a sixth decision unit that determines the second split value of the training node according to the longitudinal split value corresponding to each feature.

[0086] According to one embodiment of the present application, the fifth determination unit comprises: a second acquisition sub-unit that, for any feature, acquires second slope information corresponding to each first slope information according to the total slope information and each first slope information; a third acquisition sub-unit that, for each first slope information, acquires candidate longitudinal splitting values ​​of the arbitrary feature according to the first slope information and the second slope information corresponding to the first slope information; and a selection sub-unit that selects a maximum value among the candidate longitudinal splitting values ​​and sets it as the longitudinal splitting value of the arbitrary feature.

[0087] According to one embodiment of the present application, the verification set is mutually exclusive with the first training set and the second training set, respectively.

[0088] An embodiment of the third aspect of the present application provides a training device for a federated learning model, and the tendency of the server to automatically select a matching learning method by mixing a transverse splitting method and a longitudinal splitting method does not require attention to the data distribution method, and solves the technical problem that a conventional federated learning model training method cannot learn by fully utilizing all the data present in the process and causes poor training effect due to insufficient data utilization, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0089] To implement the above objective, an embodiment of the fourth aspect of the present application provides a training device for a federated learning model applied to a client, the device comprising: a first receiving module that receives a target splitting method transmitted when a server determines that a training node satisfies a preset splitting condition - said training node is a node of one of a plurality of boosting trees -; and a splitting module that performs node splitting for said training node based on said target splitting method.

[0090] According to one embodiment of the present application, the splitting module comprises: a first learning submodule that performs lateral federated learning based on a first training set and obtains a first splitting value corresponding to the training node; a second learning submodule that performs longitudinal federated learning based on a second training set and obtains a second splitting value corresponding to the training node; and a transmission submodule that transmits the first splitting value and the second splitting value to the server.

[0091] According to one embodiment of the present application, the first learning submodule comprises: a first receiving unit that receives a first feature subset available to the training node generated by the server in the first training set; a first transmitting unit that transmits a feature value of each feature of the first feature subset to the server; a second receiving unit that receives a splitting threshold of each feature transmitted from the server; and a first acquiring unit that acquires an initial data instance identifier set corresponding to the training node based on the splitting threshold of each feature and transmits the initial data instance identifier set to the server; wherein the initial data instance identifier set is used to instruct the server to generate a first data instance identifier set and a second data instance identifier set, and both the first data instance identifier set and the initial data instance identifier set include a data instance identifier belonging to a first left subtree space, and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space.

[0092] According to one embodiment of the present application, the first acquisition unit is also used to compare, for any feature, the splitting threshold of the arbitrary feature with the feature value of the arbitrary feature, acquire a data instance identifier in which the feature value is smaller than the splitting threshold, and generate the first set of data instance identifiers.

[0093] According to one embodiment of the present application, the second learning submodule comprises: a third receiving unit that receives a gradient information request transmitted from the server; a generating unit that generates a second feature subset from a second training set in accordance with the gradient information request; a second acquiring unit that acquires first gradient information of at least one third data instance identifier set of each feature of the second feature subset, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces; and a second transmitting unit that transmits the first gradient information of the third data instance identifier set to the server.

[0094] According to one embodiment of the present application, the second acquisition unit comprises: a bucketing sub-unit that acquires all feature values ​​of any feature for any feature and buckets the any feature based on the feature values; and a first acquisition sub-unit that acquires first slope information of each bucketed set of third data instance identifiers of the any feature.

[0095] According to one embodiment of the present application, the splitting module comprises: a receiving sub-module that receives splitting information transmitted from the server, wherein the splitting information includes the target splitting method, the target splitting feature selected as a feature splitting point, and the target splitting value; and a splitting sub-module that performs node splitting for the training node based on the splitting information.

[0096] According to one embodiment of the present application, the splitting submodule is also used to transmit the split-generated left subtree space to the server.

[0097] According to one embodiment of the present application, when the training node is a leaf node, the invention further comprises: a second receiving module that receives the weight of the leaf node transmitted from the server; a decision module that determines the residual of each included data according to the weight of the leaf node; and an input module that uses the residual as the residual input of the next boosting tree.

[0098] According to one embodiment of the present application, the device further includes a verification module that receives a verification set transmitted from the server and verifies the target federated learning model based on the verification set.

[0099] According to one embodiment of the present application, the verification module comprises: a first receiving submodule receiving a data instance identifier of one of the verification sets transmitted from the server and splitting information of a verification node, wherein the verification node is a node of one of a plurality of boosting trees; a first determining submodule determining the node direction of the verification node according to the data instance identifier and the splitting information; and a first transmitting submodule wherein the node direction transmitted to the server causes the server to enter the next node according to the node direction and makes the next node the updated verification node.

[0100] According to one embodiment of the present application, the first decision submodule comprises: a first decision unit that determines a feature value of each feature corresponding to the data instance identifier according to the data instance identifier; and a second decision unit that determines the node direction according to the splitting information and the feature value of each feature.

[0101] According to one embodiment of the present application, the verification module further comprises: a second receiving sub-module that receives a model prediction value of a data instance represented by the data instance identifier transmitted from the server when all data instance identifiers of the verification set are verified; a generating sub-module that obtains a final verification result according to the model prediction value, compares the verification result with a previous verification result, and generates verification instruction information indicating whether to suspend or use the target federated learning model; and a second transmitting sub-module that transmits the verification instruction information to the server.

[0102] According to one embodiment of the present application, the splitting module comprises: a first learning submodule that performs lateral federated learning based on a first training set and obtains a first splitting value corresponding to the training node; a second learning submodule that performs longitudinal federated learning based on a second training set and obtains a second splitting value corresponding to the training node; and a third transmission submodule that transmits the first splitting value and the second splitting value to the server.

[0103] According to one embodiment of the present application, the first learning submodule comprises: a first receiving unit that receives a first feature subset available to the training node generated by the server in the first training set; a first transmitting unit that transmits a feature value of each feature of the first feature subset to the server; a second receiving unit that receives a splitting threshold of each feature transmitted from the server; and a first acquiring unit that acquires an initial data instance identifier set corresponding to the training node based on the splitting threshold of each feature and transmits the initial data instance identifier set to the server; wherein the initial data instance identifier set is used to instruct the server to generate a first data instance identifier set and a second data instance identifier set, and both the first data instance identifier set and the initial data instance identifier set include a data instance identifier belonging to a first left subtree space, and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space.

[0104] According to one embodiment of the present application, the first acquisition unit is also used to compare, for any feature, the splitting threshold of the arbitrary feature with the feature value of the arbitrary feature, acquire a data instance identifier in which the feature value is smaller than the splitting threshold, and generate the first set of data instance identifiers.

[0105] According to one embodiment of the present application, the second learning submodule comprises: a third receiving unit that receives a gradient information request transmitted from the server; a generating unit that generates a second feature subset from a second training set in accordance with the gradient information request; a second acquiring unit that acquires first gradient information of at least one third data instance identifier set of each feature of the second feature subset, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces; and a second transmitting unit that transmits the first gradient information of the third data instance identifier set to the server.

[0106] According to one embodiment of the present application, the second acquisition unit is,

[0107] A bucketing sub-unit that, for any feature, acquires all feature values ​​of the arbitrary feature and buckets the arbitrary feature based on the feature values;

[0108] It includes a second acquisition sub-unit for acquiring first slope information of each bucketed set of third data instance identifiers of the above arbitrary features.

[0109] An embodiment of the fourth aspect of the present application provides a training device for a federated learning model, wherein a client receives a target splitting method transmitted when a training node satisfies a preset splitting condition from a server, and the training node is a node of one of a plurality of boosting trees, and performs node splitting on the training node based on the target splitting method, and by mixing a transverse splitting method and a longitudinal splitting method, the tendency to automatically select a matching learning method does not require interest in the data distribution method, and solves the technical problem that causes poor training effects due to insufficient data utilization and inability to learn by fully utilizing all data existing in the process of the training method of a conventional federated learning model, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0110] To achieve the above objective, an embodiment of the fifth aspect of the present application provides an electronic device, said electronic device comprising a memory, a processor, and a computer program stored in said memory and executable on said processor, and when said processor executes said computer program, a training method of a federated learning model according to any one of the embodiments of the first aspect of the present application or the third aspect of the present application is implemented.

[0111] To achieve the above objective, an embodiment of the sixth aspect of the present application provides a computer-readable storage medium, and when the computer program said computer program is executed by a processor, a training method of a federated learning model according to either the embodiment of the first aspect of the present application or the embodiment of the third aspect of the present application is implemented. Brief explanation of the drawing

[0112] FIG. 1 is a schematic diagram of a federated learning application scene provided in one embodiment of the present application. FIG. 2 is a schematic flowchart of a training method for a federated learning model according to one embodiment of the present application. FIG. 3 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 4 is a schematic diagram of a data distribution according to one embodiment of the present application. FIG. 5 is a schematic diagram of a node splitting according to one embodiment of the present application. FIG. 6 is a schematic diagram of a data distribution according to one embodiment of the present application. FIG. 7 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 8 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 9 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 10 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 11 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 12 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 13 is a schematic diagram of bucketing according to a bucket mapping rule according to one embodiment of the present application. FIG. 14 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 15 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 16 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 17 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 18 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 19 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 20 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 21 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 22 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 23 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 24 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 25 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 26 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 27 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 28 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 29 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 30 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 31 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 32 is a schematic flowchart of a training method for a federated learning model according to another embodiment of the present application. FIG. 33 is a schematic diagram of the structure of a training device for a federated learning model according to one embodiment of the present application. FIG. 34 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 35 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 36 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 37 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 38 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 39 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 40 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 41 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 42 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 43 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 44 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 45 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 46 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 47 is a schematic diagram of the structure of a training device for a federated learning model according to another embodiment of the present application. FIG. 48 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specific details for implementing the invention

[0113] To better understand the technical means described above, exemplary embodiments of the present invention are described in more detail below with reference to the drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention is not limited to the described embodiments and can be implemented in various forms. Conversely, the provision of such embodiments is intended to facilitate a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0114] It should be understood that "and / or" mentioned in the embodiments of the present application describes the relationship between related objects and indicates that a three-way relationship may exist. For example, A and / or B may represent A existing alone, A and B existing simultaneously, or B existing alone, wherein A and B may be odd or plural. The character symbol " / " typically indicates the relationship between the preceding and succeeding related objects "or".

[0115] First, the partial vocabulary mentioned in the embodiments of the present application is introduced.

[0116] Homogeneous data: Data records held by different data providers possess identical characteristic attributes.

[0117] Ideological Data: Data records held by different data providers have all different characteristic attributes except for the fact that their data instance identifiers (IDs) are the same.

[0118] XGBoost: Abbreviated as XGB, it is a machine learning system scalable by a set of boosting trees.

[0119] Before introducing the technical means of the present application, the problems existing in the prior art and the technical conception process of the present application are first derived in combination with one specific application scene of the present application.

[0120] In actual applications, since it is difficult to completely guarantee all heterogeneous or all homogeneous data when performing federated learning together, partial homogeneous or heterogeneous data must be discarded when training federated learning with boosting trees, and lateral or longitudinal federated learning is used. However, because a relatively large amount of data is discarded, it leads to a decline in the performance of the model obtained based on federated learning training, and even when using lateral or longitudinal federated learning, the data labels must guarantee the existence of one side and must not exist randomly across multiple sides. However, since this is almost impossible in reality, current technology also limits the actual application of federated learning.

[0121] Regarding the above problem, the inventor discovered through research that a design for federated learning that combines horizontal federated learning and vertical federated learning can solve the problem of having to pay attention to the data distribution method in previous federated learning, and can also solve the problem of poor model effects obtained due to insufficient use of data.

[0122] Through the design of the means, when there is a relatively large amount of heterogeneous data, the means tends to use a longitudinal federated learning method (i.e., a longitudinal boosting tree) so that the model obtained by training has lossless features and can also use homogeneous data; when there is a relatively large amount of homogeneous data, the means tends to use a transverse federated learning method (i.e., a transverse boosting tree) and trains the model using heterogeneous data so that the model obtained by training has longitudinal lossless capabilities and improves the performance of the model.

[0123] For example, FIG. 1 is a schematic diagram of an application scene of a model training method for federated learning provided in the present application. As illustrated in FIG. 1, the application scene includes at least one client ( FIG. 1 represents three clients, each being client (111), client (112), and client (113)), a network (12), and a server (13). Here, each client and server (13) can communicate through the network (12).

[0124] It should be noted that FIG. 1 is merely a schematic diagram of an application scene provided in an embodiment of the present application, and the embodiment of the present application does not limit the devices included in FIG. 1, nor does it limit the positional relationship between the devices in FIG. 1. For example, in the application scene illustrated in FIG. 1, a data storage device may be further included, and said data storage device may be an external memory for the server (13) or an internal memory integrated into the server (13).

[0125] The present application provides a model training method, apparatus, and storage medium based on federated learning, and improves the performance of a model obtained by training through a design that combines lateral federated learning and longitudinal federated learning. The technical means of the present application are described in detail below through specific embodiments. It should be noted that some of the embodiments below may be combined with one another, and the same or similar concepts or processes are not described in partial embodiments.

[0126] The following describes a training method, apparatus, and electronic device for a federated learning model provided in an embodiment of the present application with reference to the drawings.

[0127] FIG. 2 is a schematic flowchart of a training method for a federated learning model according to one embodiment of the present application.

[0128] As illustrated in FIG. 2, a training method for a federated learning model provided in an embodiment of the present application is described, with the server as the execution body, and specifically includes steps S201 to S205.

[0129] In S201, if the training node satisfies a preset splitting condition, a target splitting method corresponding to the training node is obtained, where the training node is a node of one of the multiple boosting trees.

[0130] In an embodiment of the present application, it is explained that when a training node satisfies a preset splitting condition, the training node currently located must continue splitting, and in such a situation, a target splitting method corresponding to the training node can be obtained.

[0131] Here, the pre-set splitting condition can be configured according to the actual situation. For example, the pre-set splitting condition can be set as the material layer of the training node currently being processed failing to reach the maximum tree depth requirement and the loss function failing to satisfy the constraints.

[0132] Here, the target division method includes a transverse division method and a longitudinal division method.

[0133] Here, the Boosting Tree refers to an enhancement method that uses a counting model and a forward distribution algorithm, and uses a Decision Tree as a basis function.

[0134] In S202, the client is notified to perform node splitting based on the target splitting method.

[0135] In an embodiment of the present application, after obtaining a target splitting method corresponding to a training node, the server transmits the obtained target splitting method to a client and notifies the client to perform node splitting based on the target splitting method. Correspondingly, the client receives the target splitting method from the server and can perform node splitting for the training node based on the target splitting method.

[0136] In S203, until the updated training node no longer satisfies the preset splitting condition, the left subtree node generated after the training node is split is used again as the training node, and the next round of training is performed.

[0137] In an embodiment of the present application, the server uses the left subtree node generated after the training node is split as the training node again and performs the next round of training, determines whether the updated training node hierarchy satisfies a preset splitting condition, and determines that the updated training node must continue to split. That is, if the preset splitting condition is satisfied, the server continues to acquire a target splitting method corresponding to the updated training node, and informs the client to perform node splitting based on the target splitting method until the updated training node no longer satisfies the preset splitting condition. Here, the preset splitting condition may include a tree depth threshold, a quantity threshold of split samples, or an error threshold of a federated learning model.

[0138] In S204, the other non-leaf nodes of one boosting tree are used as training nodes again, and the next round of training is performed.

[0139] In an embodiment of the present application, the server may recall to another non-leaf node of the current boosting tree, set it back to the current training node, and perform the next round of training.

[0140] In S205, if the node datasets of multiple boosting trees are all empty, training is stopped and a target federated learning model is created.

[0141] In an embodiment of the present application, when the node data sets of a plurality of boosting trees are all empty, training can be stopped and a target federated learning model can be generated. Furthermore, until a preset number of training iterations is reached, the generated target federated learning model can be verified, information can be organized, and the model can be suspended.

[0142] Thus, in the training method of a federated learning model provided in this application, the tendency of the server to automatically select a matching learning method by mixing a lateral splitting method and a longitudinal splitting method does not require attention to the data distribution method and solves the technical problem of not being able to learn by fully utilizing all the data existing in the process of the training method of a conventional federated learning model and causing poor training effects due to insufficient data utilization, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0143] It needs to be explained that, in this application, when attempting to obtain a target splitting method corresponding to a training node, federated learning is performed in cooperation with a client to obtain the corresponding splitting value, and the target splitting method corresponding to the training node is determined according to the splitting value.

[0144] As an implementable method, as illustrated in FIG. 3, based on the above embodiment, the process of obtaining a target splitting method corresponding to a training node in the above step specifically includes steps S301 to S303.

[0145] In S301, based on the first training set, lateral federated learning is performed in cooperation with the client to obtain the first split value corresponding to the training node.

[0146] It needs to be explained that if a training node needs to continue splitting, the splitting method must be determined. That is, a decision must be made between transverse and longitudinal splitting. Here, most nodes have undergone a total of two candidate splits—one transverse and one longitudinal—and the splitting method with the relatively large splitting gain from the two candidate splits is selected and used as the final splitting method for that node.

[0147] When attempting a preliminary judgment, nodes meeting the following conditions undergo only vertical or horizontal splitting and are treated as the direct final splitting result.

[0148] (1) The deeper the tree, the smaller the total number of samples that need to be split at the node. The proportion of common samples in the samples of the node is determined, and if it is less than a preset value such as 10%, the minimum of the common samples is explained, the data distribution situation of the node is treated as a lateral distribution, and only lateral splitting is performed.

[0149] (2) If the proportion of joint samples in the node samples exceeds a preset value such as 98%, only longitudinal splitting can be performed.

[0150] It needs to be explained that the setting of the two prior judgment conditions above is intended to save training time, and the ratio of the two can be set in the training parameters.

[0151] The following explains the data mentioned in transverse, longitudinal, and two-type fission methods, respectively.

[0152] For example, as shown in Fig. 4, a total of 10,000 samples are trained, and when splitting reaches a single node, there are 100 samples. At this time, the data distribution is as follows. Of the 100 samples, 70 are shared samples from two platforms. Here, Platform A has 90 samples (including 20 proprietary samples and 70 shared samples), and Platform B has 80 samples (including 10 proprietary samples and 70 shared samples). Platform A has 18 features (including 10 proprietary features and 8 shared features), and Platform B has 18 features (including 10 proprietary features and 8 shared features).

[0153] For lateral splitting, all samples of the node in question participate in the splitting, and multiple candidate features and feature thresholds are set by the server.

[0154] For a common feature f and value v, platform A assigns 90 local samples to left and right subtrees, and platform B assigns 80 local samples to left and right subtrees, and each notifies the server of the sample classification status; the server calculates the splitting gain based on the left and right classification of the 100 samples and sets it as the gain of feature f.

[0155] For the reason for feature f and value v of platform A, platform A assigns 90 local samples to the left and right subtrees, platform B assigns 70 local samples to the left and right subtrees, and assigns all 10 samples with no feature f values ​​to the left or right subtree, classifies them into two types, and notifies the server of the sample classification status; the server calculates the splitting gain according to the two-type classification of the 100 samples and takes the relatively larger one as the gain of feature f.

[0156] The reasoning feature f and value v of platform B can be obtained in a manner similar to the above.

[0157] The server records the maximum gain value of all features as gain1.

[0158] For longitudinal splitting, only 70 samples from the nodes in question participate in the splitting. Similar to general longitudinal federated learning, the maximum feature splitting gain gain2 is calculated from the 70 samples.

[0159] Select the larger of the two gains, the maximum gain of the transverse split gain1 and the maximum gain of the longitudinal split gain2, and the node performs splitting using that splitting method. Then, proceed to the next node.

[0160] What needs to be explained is that whether a transverse splitting method is used or a longitudinal splitting method is chosen, the node sample set is split according to the following rules: the maximum gain is generated from platform A local feature f1; the maximum gain is generated from the common feature f2; and the maximum gain is generated from platform B local feature f3.

[0161] For example, as illustrated in FIG. 5(a), when a node splits according to f1, 90 samples of A are split to the left according to samples where the feature value is less than or equal to the threshold, and split to the right according to samples where the feature value is greater than the threshold; 10 samples of B have no feature value, and according to the method corresponding to maximum gain, if the missing value belongs to the right subtree of the samples, the 10 samples are split to the right, and if the missing value sample belongs to the left subtree, the 10 samples are split to the left.

[0162] As illustrated in Fig. 5(b), when a node splits according to f2, all samples have feature values, samples that are less than or equal to the threshold are split to the left, and samples that are greater than the threshold are split to the right.

[0163] As illustrated in FIG. 5(c), when a node splits according to f3, the specific process is similar to the process when a node splits according to f1, and is not described further here.

[0164] In an embodiment of the present application, the client may perform lateral federated learning based on a first training set to obtain a first split value corresponding to a training node and transmit it to a server. Correspondingly, the server may receive the first split value corresponding to a training node and obtain the first split value corresponding to a training node.

[0165] In S302, based on the second training set, longitudinal federated learning is performed in cooperation with the client to obtain the second split value corresponding to the training node.

[0166] In an embodiment of the present application, the client may perform lateral federated learning based on a second training set to obtain a second split value corresponding to a training node and transmit it to a server. Correspondingly, the server may receive the second split value corresponding to a training node and obtain the second split value corresponding to a training node.

[0167] In S303, the target splitting method corresponding to the training node is determined according to the first splitting value and the second splitting value.

[0168] It needs to be explained that traditional federated learning primarily includes lateral federated learning and longitudinal federated learning. Here, for lateral federated learning, the data used is multi-platform data with completely identical features. That is, it is lateral data, for example, the data (1.2)+(3)+(5.1) shown in FIG. 6. For longitudinal federated learning, the data used is multi-platform data with completely identical sample IDs (Identity Documents, identification numbers). That is, it is longitudinal data, for example, the data (2)+(3)+(4) shown in FIG. 6. Thus, in the prior art, only complete lateral or complete longitudinal data is modeled. That is, only partial data of FIG. 6 can be used.

[0169] In an embodiment of the present application, the first training set, i.e., the data participating in lateral federated learning, is all data samples from a plurality of clients. For example, it is the data (1)+(2)+(3)+(4)+(5) shown in FIG. 6; and the second training set, i.e., the data participating in longitudinal federated learning, is a common data sample from a plurality of clients, and all features of said sample participate in training. For example, it is the same as (2)+(3)+(4) shown in FIG. 6. Thus, the training method of the federated learning model provided in the present application can be applied to lateral and longitudinal cross data. That is, all data in FIG. 6 can be used.

[0170] Thus, in an embodiment of the present application, by performing lateral federated learning and longitudinal federated learning in cooperation with a client based on a first training set and a second training set, a first split value and a second split value corresponding to a training node are obtained.

[0171] What needs to be explained is that in the present application, when determining a target splitting method corresponding to a training node according to a first splitting value and a second splitting value, the target splitting value is determined by comparing the first splitting value and the second splitting value, and the corresponding target splitting method can be determined according to the target splitting value.

[0172] As an implementable method, as illustrated in FIG. 7, based on the above embodiment, the process of determining a target splitting method corresponding to a training node according to a first splitting value and a second splitting value in step S303 specifically includes steps S501 to S502.

[0173] In S501, the relatively large value between the first and second split values ​​is determined as the target split value corresponding to the training node.

[0174] In an embodiment of the present application, the server obtains a first split value and a second split value corresponding to a training node, compares the first split value and the second split value, and sets the relatively larger value among them as the target split value corresponding to the training node.

[0175] For example, if the obtained first split value is Gain1 and the second split value is Gain2, and Gain1 > Gain2, Gain1 can be set as the target split value corresponding to the training node.

[0176] In S502, the splitting method corresponding to the training node is determined based on the target splitting value.

[0177] In an embodiment of the present application, after setting a relatively large value among the first splitting value and the second splitting value as the target splitting value corresponding to the training node, the server can determine the splitting method corresponding to the training node according to the target splitting value.

[0178] Thus, in the training method of the federated learning model provided in the present application, by performing lateral federated learning and longitudinal federated learning in cooperation with the client, a first split value and a second split value are obtained, the value that is relatively large is set as the target split value corresponding to the training node, and the splitting method corresponding to the training node is determined according to the target split value, and the tendency to automatically select a matching learning method according to the target split value does not require interest in the data distribution method.

[0179] What needs to be explained is that in the present application, when a first split value corresponding to a training node is obtained by performing lateral federated learning in cooperation with a client based on a first training set, a lateral split value corresponding to each feature is obtained, and thus the first split value of the training node can be determined according to the lateral split value.

[0180] As an implementable method, as illustrated in FIG. 8, based on the embodiment, the process of obtaining a first split value corresponding to a training node by performing lateral federated learning in cooperation with a client based on a first training set in step S301 specifically includes steps S601 to S604.

[0181] In S601, a first feature subset available to the training node in the first training set is generated and sent to the client.

[0182] Optionally, the server may randomly generate a first feature subset available to the current training node from the first training set. For example, half of all features in the current first training set may be randomly generated to form a new feature set as the first feature subset, and the generated first feature subset may be transmitted to each client. Correspondingly, each client may receive the first feature subset and, based on the acquired first feature subset, it iterates to obtain the feature value of each feature in the set, and transmits it to the server based on the local data, that is, the feature value of the feature stored locally.

[0183] In S602, the feature value of each feature of the first feature subset transmitted from the client is received.

[0184] In an embodiment of the present application, a client may transmit a feature value of each feature of a first feature subset to a server. Correspondingly, the server may receive the feature value of each feature of the first feature subset transmitted from the client.

[0185] In S603, each feature is determined as a transverse splitting value corresponding to a splitting feature point, according to the feature value of each feature of the first feature subset.

[0186] As an implementable method, as illustrated in FIG. 9, based on the above embodiment, the process of determining each feature as a transverse division value corresponding to a division feature point according to the feature value of each feature of the first feature subset in step S603 specifically includes steps S701 to S703.

[0187] In S701, for any feature of the first feature subset, the splitting threshold of the arbitrary feature is determined according to the feature value of the arbitrary feature.

[0188] In an embodiment of the present application, the server may receive a feature value of each feature of a first feature subset transmitted from a client, and then generate a feature value list according to the feature value. Furthermore, for any feature of the first feature subset, one feature value may be randomly selected from the feature value labels and set as the global optimal splitting threshold of the current feature.

[0189] In S702, according to the splitting threshold, a first set of data instance identifiers and a second set of data instance identifiers corresponding to any feature are obtained, wherein the first set of data instance identifiers includes data instance identifiers belonging to a first left subtree space, and the second set of data instance identifiers includes data instance identifiers belonging to a first right subtree space.

[0190] As an implementable method, based on the embodiment as illustrated in FIG. 10, the process of obtaining a first data instance identifier set and a second data instance identifier set corresponding to any feature according to a splitting threshold in step S702 specifically includes steps S801 to S803.

[0191] In S801, the splitting threshold is transmitted to the client.

[0192] In an embodiment of the present application, after determining a splitting threshold for any feature, the splitting threshold may be broadcast to a client. Correspondingly, the client may receive the splitting threshold, determine a set of initial data instance identifiers corresponding to a training node based on the splitting threshold of each feature, and transmit the set of initial data instance identifiers to a server.

[0193] In S802, a set of initial data instance identifiers corresponding to the training node transmitted from the client is received, the set of initial data instance identifiers is generated when the client splits the node for any feature according to the splitting threshold, and the set of initial data instance identifiers includes data instance identifiers belonging to the first left subtree space.

[0194] In an embodiment of the present application, the server may receive an IL transmitted from the client, namely, an initial set of data instance identifiers IL including a data instance identifier belonging to a first left subtree space.

[0195] In S803, based on the initial data instance identifier set and all data instance identifiers, the first data instance identifier set and the second data instance identifier set are obtained.

[0196] As an implementable method, as illustrated in FIG. 11, based on the embodiment, the process of obtaining a first data instance identifier set and a second data instance identifier set based on an initial data instance identifier set in step S803 specifically includes steps S901 to S903.

[0197] In S901, each data instance identifier of the initial set of data instance identifiers is compared with the client's data instance identifier, and the strange data instance identifier is obtained.

[0198] Here, the strange data instance identifier may be a duplicate data instance identifier, a data instance identifier containing contradictions, etc.

[0199] In S902, preprocessing is performed on the strange data instance identifiers, and the first set of data instance identifiers is obtained.

[0200] In an embodiment of the present application, after receiving an IL, the server can determine the final IL by filtering out duplicate instance IDs from each set of ILs and processing ID information in which contradictions exist.

[0201] For example, if an instance ID is added to IL for client A, but an ID exists for client B and an instance is not added, it can be determined that the ID must exist in the corresponding IL.

[0202] In S903, based on all data instance identifiers and the first data instance identifier set, the second data instance identifier set is obtained.

[0203] In an embodiment of the present application, after obtaining a first data instance identifier set, if the first data instance identifier set IL is removed from all data instance identifiers, a second data instance identifier set IR can be obtained.

[0204] What needs to be explained is that in the present application, when longitudinal federated learning is performed in cooperation with the client based on a second training set and a second split value corresponding to the training node is obtained, the second split value of the training node can be determined according to the longitudinal split value corresponding to each feature.

[0205] As an implementable method, as illustrated in FIG. 12, based on the above embodiment, the process of obtaining a target splitting method corresponding to a training node in the above step specifically includes steps S1001 to S1004.

[0206] In S1001, the client is instructed to perform longitudinal federated learning based on the second training set.

[0207] In an embodiment of the present application, after the server notifies the client to perform longitudinal federated learning based on a second training set, the server sends a gradient information request to the client, G kv and H kv Information can be obtained. Correspondingly, the client obtains the unprocessed data of the current node according to that part data of the common ID, obtains a feature set randomly, performs a bucketing mapping for each sample according to each feature k of the set and each value v of all values ​​of that feature, and G of the left subtree space kv and H kv Calculate the first gradient information, and send it to the server after homomorphic encryption processing.

[0208] ;

[0209] ;

[0210] Here, x i,k is data instance x i Represents the value of the feature k.

[0211] For example, people with an initial value of 1-100 are mapped into three buckets: 20 years old or younger, 20-50 years old, and 50 years old or older. Samples from a single bucketing are either all assigned to the left or all assigned to the right. Theoretically, G and H transmitted to the server are cumulative sums. For example, in the above example, there must be three Gs transmitted, such as the sum of Gs for ages 1-20, the sum of Gs for ages 1-50, and the sum of Gs for ages 1-100 (corresponding to the G of the left subtree, respectively). Since homomorphic ciphertext operations are slow and the ciphertext becomes long, potentially increasing communication traffic, what is actually transmitted from the client is the G for each of the three bucketings, such as the sum of Gs for ages 1-20, the sum of Gs for ages 20-50, and the sum of Gs for ages 50 or older. The labeled platform receives the G of the three buckets, first decodes it into a text, and calculates the accumulated G for ages 1-20 / 1-50 / 1-100. The two formulas above represent the process and indicate the calculation of the sum of g for each bucket. s k,v is the current maximum value of the bucket (50 years old), and s k,v-1 Since is the maximum feature value (20 years old) of the previous bucket, x for ages 20-50 is filtered out. Through the feature bucketing method, the amount of computation can be reduced.

[0212] In S1002, first gradient information of at least one third data instance identifier set of each feature transmitted from the client is received, the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces.

[0213] In an embodiment of the present application, for any feature, the client obtains all feature values ​​of the arbitrary feature, buckets the arbitrary feature based on the feature values, and obtains first gradient information of each bucketed set of third data instance identifiers of the arbitrary feature. Correspondingly, the server may receive first gradient information of at least one set of third data instance identifiers among each feature transmitted from the client.

[0214] In S1003, the longitudinal split value of each feature is determined according to the first gradient information of each feature and the total gradient information of the training node.

[0215] In S1004, the second split value of the training node is determined according to the longitudinal split value corresponding to each feature.

[0216] The following explains slope information in an example format.

[0217] For example, as illustrated in FIG. 13, taking the feature number k of client A as an example, all samples of the current node can be sorted and arranged in order from smallest to largest based on the feature values ​​of feature k. Furthermore, according to bucket mapping rules, the samples can be divided sequentially into multiple data buckets (corresponding to multiple small feature thresholds to large feature thresholds). Furthermore, the total G of the first gradient g and the total H of the second gradient h of the v-th bucketing containing the samples can be calculated. That is, G corresponding to the v-th feature threshold v kv and H kv Calculate.

[0218] Here, G kv What is represented is the sum of the first gradients g of all samples in the v-th bucket after sorting, sorting node samples according to the value of the feature with number k, splitting them into multiple data buckets in order.

[0219] Here, H kv is the sum of the second slopes h of the corresponding samples.

[0220] It should be explained that the above bucket mapping rules are numerous, and the present application does not limit the specific method of the bucket mapping rules, but only ensures that samples with identical feature values ​​are guaranteed. For example, in FIG. 11, two samples with a value of 1 can be assigned to the same data bucket.

[0221] For example, samples of the same value can be grouped into one bucket. That is, if there are n samples and one feature has m types of values, they are divided into m buckets, and the threshold of the corresponding feature is the m types of values.

[0222] For example, the number of buckets can be limited. For instance, if there are many, it can be divided into m buckets; in that situation, if the value of feature k is less than m, it can be divided in the previous manner; if there are more than m, it can be divided into m buckets according to a method similar to equal division.

[0223] It needs to be explained that when attempting to determine the longitudinal split value of each feature, the maximum value among the candidate longitudinal split values ​​can be selected and used as the longitudinal split value of an arbitrary feature.

[0224] As an implementable method, as illustrated in FIG. 14, based on the above embodiment, the process of determining the longitudinal split value of each feature according to the first slope information of each feature and the total slope information of the training node in step S1003 specifically includes steps S1201 to S1203.

[0225] In S1201, for any feature, a second slope information corresponding to each first slope information is obtained according to the total slope information and each first slope information.

[0226] In S1202, for each first slope information, a candidate longitudinal split value of an arbitrary feature is obtained according to the first slope information and the second slope information corresponding to the first slope information.

[0227] In S1203, the maximum value among the candidate longitudinal splitting values ​​is selected and set as the longitudinal splitting value of an arbitrary feature.

[0228] Here, the first gradient information includes the first gradient sum of features corresponding to data instances belonging to the second left subtree space and the second gradient sum of features corresponding to data instances belonging to the second left subtree space; and the second gradient information includes the first gradient sum of features corresponding to data instances belonging to the second right subtree space and the second gradient sum of features corresponding to data instances belonging to the second right subtree space.

[0229] As an implementable method, in an embodiment of the present application, the server G to each client kv and H kv A request is made to acquire information. Correspondingly, the client acquires the unprocessed data of the current node according to that part of the common ID, acquires a random set of features, performs a bucketing mapping for each sample according to each feature k of the set and each value v of all values ​​of that feature, and G of the left subtree space kv and H kv It calculates and transmits to the server after homomorphic encryption. Additionally, the client calculates partial intermediate results of the loss function based on the shared data identifier and local data. For example, the first derivative g of the loss function i and second derivative h i Calculate and send to the server.

[0230] Furthermore, the server receives the G sent from the client. kv and H kv Decoder, the data corresponding to the current node's common ID, and all acquired g i and h i Accordingly, all g in the left subtree space of the current node i The sum of GL, the right subtree space, and all g iThe sum GR and the left subtree space, all h i The sum of HL, and all h of the right subtree space i You can calculate the sum of HR.

[0231] Taking XGB as an example, the target function is as shown in the formula below.

[0232]

[0233] We propose to represent the above formula similarly using a second-order Taylor expansion in XGB. The second-order form of the Taylor expansion is as follows.

[0234]

[0235] Here, g i and h i The calculation formula is as follows.

[0236]

[0237] g in the above Taylor expansion i is the first derivative, and h i It can be seen that is the second derivative. Here, GL and HL can be calculated using the following formulas, respectively.

[0238]

[0239]

[0240] Here, n represents the number of instances in the left subtree space. That is, in this situation, there are a total of n instances in the left subtree space.

[0241] Furthermore, the server calculates the optimal split point for each feature based on the above results, and can determine the global optimal split point (k, v, Gain) based on the split point information. Here, if several clients have the same feature, the server receives the G kv Randomly select G of the current feature kv Do it as, and H kvIt is processed in the same way.

[0242] Furthermore, the server may request IL information from the corresponding client based on (k, v, Gain). Correspondingly, the client receives split point information (k, v, Gain), searches to obtain the split point threshold value, and records the corresponding split point (k, value) information. The local data set splits the corresponding split point to obtain the IL set and transmits (record, IL, value) to the server. Here, record represents the index recorded on the client.

[0243] Furthermore, the server receives the (record, IL, value) information transmitted from the client, divides all instances of the common ID in the material node space, and associates the current node with the client through (client id, record). It records the information (client id, record_id, IL, feature_name, feature_value) and uses it as information for longitudinal splitting. That is, it uses it as the longitudinal splitting value of an arbitrary feature.

[0244] It needs to be explained that when node splitting selects features and values ​​corresponding to the optimal split point, samples of the current node can be assigned to left and right subtree nodes according to the value of the corresponding feature.

[0245] In S604, the first split value of the training node is determined according to the transverse split value corresponding to each feature.

[0246] It should be explained that, in the embodiments of the present application, a transverse division may be performed first, followed by a longitudinal division; and optionally, a longitudinal division may be performed first, followed by a transverse division.

[0247] Furthermore, since all data is used in the transverse splitting method and only data with the same ID is used in the longitudinal splitting method, it can be seen that in the transverse splitting method, more data is used and there is a greater probability of obtaining relatively good effects. In the longitudinal splitting method, the amount of data interaction between the client and the server is smaller and the speed is faster, so in the event of training interruption, transverse splitting can be performed first and then longitudinal splitting can be performed to obtain the deepest possible training results.

[0248] It is explained that in an embodiment of the present application, if a training node satisfies a preset splitting condition, the training node currently located must continue splitting, and in that situation, a target splitting method corresponding to the training node can be obtained. If a training node does not satisfy the preset splitting condition, it is explained that the training node currently located does not need to continue splitting, and in that situation, a leaf node can be determined and the weight of the leaf node can be transmitted to the client.

[0249] As an implementable method, based on the above embodiment as illustrated in FIG. 15, specifically includes steps S1301 to S1302.

[0250] In S1301, if the training node does not satisfy the preset splitting condition, the training node is determined as a leaf node, and the weight of the leaf node is obtained.

[0251] In an embodiment of the present application, if the training node does not satisfy a preset splitting condition, the server designates the material node as a leaf node, and the weight w of the said leaf node j Calculate , and the corresponding w j It stores the value of and can be used as the weight of the longitudinal leaf node.

[0252] Here, the weight W of the leaf node jis used to calculate the sample prediction score and can be calculated through the following formula.

[0253]

[0254] Here, G j g corresponding to all instances of node j i Represents the sum of, and H j h corresponding to all instances of node j i Represents the sum of.

[0255] In S1302, the weight of the leaf node is sent to the client.

[0256] In an embodiment of the present application, after the client obtains the weight of the leaf node, the client transmits the weight of the leaf node to the client and notifies each client that vertical leaf node splitting is no longer performed. That is, the node splitting operation is completed.

[0257] What needs to be explained is that, in an embodiment of the present application, splitting information may be transmitted to the client before attempting to notify the client to perform node splitting based on a target splitting method, wherein the splitting information includes a target splitting method, a target splitting feature selected as a feature splitting point, and a target splitting value.

[0258] As illustrated in FIG. 16, based on an embodiment, if the target splitting method is a longitudinal splitting method, steps S1401 to S1404 are specifically included prior to the step of notifying the client to perform node splitting based on the target splitting method.

[0259] In S1401, fragmentation information is transmitted to a client equipped with a label.

[0260] In an embodiment of the present application, a server may transmit splitting information to a client having a label. Correspondingly, the client having a label may receive the splitting information and, based on the splitting information, perform node splitting for a training node.

[0261] Regarding the longitudinal splitting method, optionally, the server notifies each client to perform a true node splitting operation based on whether the recorded longitudinal splitting information includes (client_id, record_id, IL, feature_name, feature_value). Here, the client corresponding to client_id knows all the information; that is, it knows (client_id, record_id, IL, feature_name, feature_value), while other clients only need to know the IL information. Furthermore, the server may designate the currently split left subtree node as the current processing node.

[0262] Accordingly, the client receives the IL or all information transmitted from the server. That is, it receives (client_id, record_id, IL, feature_name, feature_value) and performs a longitudinal node splitting operation; if the information (client_id, record_id, IL, feature_name, feature_value) exists, the client must record and store the relevant information when performing the split. Furthermore, after the split is completed, the client can set the split left subtree node as the current processing node.

[0263] For the lateral splitting method, optionally, the server can perform the splitting of the node in the lateral manner. That is, by performing a splitting operation on the current node based on the (k, value) information obtained through the lateral method, the server can obtain IL information and broadcast the IL to each client.

[0264] Correspondingly, the client receives (k, value) information from the server and can perform node splitting on the data with a shared ID. The splitting method is as follows: regarding the feature k of the shared ID data, if the value is smaller than value, the ID of the corresponding data is inserted into the IL set; otherwise, it is inserted into the IR set. If the feature k does not exist in the data, it is inserted into the right subtree space.

[0265] In S1402, a set of left subtree spaces transmitted from a client equipped with a label is received.

[0266] In an embodiment of the present application, a client with a label may perform node splitting on a training node and then transmit the split-generated left subtree space to a server. Correspondingly, the server may receive the set of left subtree spaces transmitted from the client with the label.

[0267] In S1403, splitting is performed on the second training set according to the left subtree space set.

[0268] In S1404, the identifier of a client equipped with a training node and a label is associated.

[0269] It needs to be explained that, in this application, the current training node can be initialized before responding to a preset splitting condition.

[0270] As an implementable method, based on the above embodiment as illustrated in FIG. 17, specifically includes steps S1501 to S1502.

[0271] In S1501, the data instance identifier transmitted from the client is received.

[0272] In an embodiment of the present application, a client may transmit a unique identifier ID for each of its data to a server. Correspondingly, the client may receive a unique identifier ID for each data, i.e., a data instance identifier.

[0273] In S1502, a joint data instance identifier between clients is determined based on the data instance identifier, and the joint data instance identifier is used to instruct the clients to determine the first training set and the second training set.

[0274] In an embodiment of the present application, the server may collect all instance IDs of each client, obtain a common ID between clients, and notify each client. Furthermore, the server may select one client to be a verification client, select a portion of data with labels from said client to be a verification data set, and since said portion of the data set does not exist in the data set of the common ID, correct the training data set list corresponding to said client and initialize the information of the verification data set. Then, the server may notify each client of the information of the verification ID list and the common ID list. Correspondingly, the client may receive the common ID list and the verification ID list (if any) transmitted from the server and initialize global local data information.

[0275] Furthermore, the server performs information initialization for each round of training for the current XGB forest list and training sequence, and performs information initialization for each tree for the current tree node and current XGB forest list, thereby notifying the client to perform information initialization for each round of training or initialization for each tree training.

[0276] Furthermore, after acquiring the target federated learning model, the generated target federated learning model can be verified.

[0277] Optionally, the server validates the target federated learning model in cooperation with a validation client based on a validation set, the validation client is one of the clients that participated in training the federated learning model, and the validation set mutually excludes the first training set and the second training set, respectively.

[0278] As a feasible implementation, the server may notify the client to perform a validation initialization operation. Correspondingly, the client performs the validation initialization.

[0279] Furthermore, the server may select one ID to start verification, initialize the XGB tree, and notify the client to start verification. Correspondingly, the client initializes the verification information.

[0280] Furthermore, the server can transmit split node information and a verified data ID to the verification client based on the current tree. Correspondingly, the client can obtain the corresponding data based on the data ID, determine whether to proceed to the left subtree or the right subtree based on the split node information transmitted from the server, and return the determination result to the server.

[0281] Furthermore, the server determines whether it has reached a leaf node after entering the next node based on the direction returned by the client; if it has not reached a leaf node, it selects one ID to re-verify, initializes the XGB tree, and notifies the client to re-verify. If it has reached a leaf node, it can record the weight of the leaf node, calculate the predicted value, and store it. If the currently predicted ID is not the last one among all predicted IDs, it selects one ID to re-verify, initializes the XGB tree, and notifies the client to re-verify; if the currently predicted ID is the last one among all predicted IDs, it can transmit all prediction results to the client. Correspondingly, the client receives all prediction results, calculates the final verification result, compares it with the previous verification result, determines whether to hold or use the current model, and notifies the server of the decision result.

[0282] Furthermore, based on the verification result returned by the client, the server determines whether to suspend or use the current model and broadcasts the decision result to all clients. Correspondingly, each client receives and processes the server's broadcast information.

[0283] Furthermore, the server determines whether the final prediction sequence has been reached; if it has not been reached, it performs information initialization for each round of training for the current XGB forest list heat exchange training sequence; if it has already reached the final prediction sequence, it terminates all training, processes information, and may suspend the model. Correspondingly, the client terminates all training, processes information, and suspends the model.

[0284] Thus, the training method of the federated learning model provided in this application combines a transverse splitting method and a longitudinal splitting method, and the tendency to automatically select a matching learning method does not require attention to the data distribution method. It solves the technical problem of the conventional federated learning model training method, which fails to learn by fully utilizing all data present in the process and results in poor training effects due to insufficient data utilization, while simultaneously reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0285] FIG. 18 is a schematic flowchart of a training method for a federated learning model according to one embodiment of the present application.

[0286] As illustrated in FIG. 18, a training method for a federated learning model provided in an embodiment of the present application is described, with the client as the performing body, and specifically includes steps S1601 to S1602.

[0287] In S1601, when the server determines that a training node satisfies a preset splitting condition, it receives a transmitted target splitting method, where the training node is a node of one of a plurality of boosting trees.

[0288] In an embodiment of the present application, it is explained that when a training node satisfies a preset splitting condition, the currently located training node must continue splitting. In such a situation, the server may obtain a target splitting method corresponding to the training node and notify the client to perform node splitting based on the target splitting method. Correspondingly, the client may receive the transmitted target splitting method when the server determines that the training node satisfies the preset splitting condition.

[0289] In S1602, node splitting is performed on the training nodes based on the target splitting method.

[0290] In an embodiment of the present application, the server may determine a target splitting method corresponding to a training node based on a first splitting value and a second splitting value. Correspondingly, the client may receive IL or (client_id, record_id, IL, feature_name, feature_value) information transmitted from the server and perform node splitting for the training node according to the target splitting method. Here, if the (client_id, record_id, IL, feature_name, feature_value) information exists, when performing node splitting for the training node, the client must record and store said information.

[0291] Furthermore, after the split is complete, the client can set the left subtree node after the split as the current processing node.

[0292] Thus, the training method of a federated learning model provided in this application, wherein a client receives a target splitting method transmitted when a training node satisfies a preset splitting condition from a server, and the training node is a node of one of a plurality of boosting trees, and performs node splitting on the training node based on the target splitting method, thereby mixing a horizontal splitting method and a vertical splitting method, and the tendency to automatically select a matching learning method, does not require interest in the data distribution method, and solves the technical problem that causes poor training effects due to insufficient data utilization and insufficient use of all data existing in the process of the training method of a conventional federated learning model, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0293] It needs to be explained that, in the present application, before performing node splitting on a training node based on a target splitting method, the client can perform federated learning in cooperation with the server and obtain the splitting value.

[0294] As an implementable method, based on the above embodiment as illustrated in FIG. 19, specifically includes steps S1701 to S1703.

[0295] In S1701, lateral federated learning is performed based on the first training set to obtain the first split value corresponding to the training node.

[0296] What needs to be explained is that in the present application, when attempting to obtain a first split value corresponding to a training node by performing lateral federated learning based on a first training set, an initial data instance identifier set corresponding to a training node is obtained, and the initial data instance identifier set can be transmitted to a server.

[0297] As an implementable method, based on the above embodiment as illustrated in FIG. 20, specifically includes steps S1801 to S1804.

[0298] In S1801, the server receives a first feature subset available to the training nodes generated from the first training set.

[0299] In an embodiment of the present application, the server may randomly generate a first feature subset available to the current training node from a first training set. For example, half of all features of the current first training set may be randomly generated, a new feature set may be formed to serve as the first feature subset, and the generated first feature subset may be transmitted to each client. Correspondingly, each client may receive the first feature subset.

[0300] In S1802, the feature value of each feature of the first feature subset is transmitted to the server.

[0301] In an embodiment of the present application, the client may, by iterating through the acquired first feature subset to obtain the feature value of each feature in the set, randomly select one of all values ​​of the feature according to the local data, i.e., the feature value of the feature stored locally, and transmit it to the server. Correspondingly, the server collects the feature value information transmitted from each client, forms a list of values, randomly selects one from the list to set as the global optimal splitting threshold of the current feature, and broadcasts the splitting threshold to each client.

[0302] In S1803, the splitting threshold of each feature transmitted from the server is received.

[0303] In an embodiment of the present application, the server may determine a splitting threshold for each feature based on the feature value of each feature of the received first feature subset and transmit it to the client. Correspondingly, the client may receive the splitting threshold for each feature transmitted from the server.

[0304] In S1804, based on the splitting threshold of each feature, an initial data instance identifier set corresponding to a training node is determined and the initial data instance identifier set is transmitted to a server; wherein the initial data instance identifier set is used to instruct the server to generate a first data instance identifier set and a second data instance identifier set, and both the first data instance identifier set and the initial data instance identifier set include data instance identifiers belonging to a first left subtree space, and the second data instance identifier set includes data instance identifiers belonging to a first right subtree space.

[0305] In an embodiment of the present application, when attempting to obtain an initial set of data instance identifiers corresponding to a training node based on the splitting threshold of each feature, for any feature, the splitting threshold of that feature is compared with the feature value of that feature, and data instance identifiers whose feature value is smaller than the splitting threshold are obtained, and an initial set of data instance identifiers is generated. Here, the splitting threshold can be set before starting training, depending on the actual situation.

[0306] As an implementable method, the client receives feature splitting threshold information, performs node splitting on the current feature, obtains IL, and notifies the server; if the corresponding feature does not exist on the client, it can return an empty IL.

[0307] Here, IL is the set of instance IDs in the left subtree space; the calculation method is as follows. Upon receiving the threshold value of feature k transmitted from the server, if the value of feature k corresponding to instance ID1 in the local data is smaller than the value, ID1 is added to the IL set. This can be expressed as a formula as follows.

[0308]

[0309] Here, ID k represents the value of instance ID feature A, and S IL represents set IL.

[0310] In S1702, longitudinal federated learning is performed based on the second training set to obtain the second split value corresponding to the training node.

[0311] What needs to be explained is that in the present application, when attempting to obtain a second split value corresponding to a training node by performing longitudinal federated learning based on a second training set, the first gradient information of at least one third data instance identifier set can be obtained and the first gradient information of the third data instance identifier set can be transmitted to a server.

[0312] As an implementable method, based on the above embodiment as illustrated in FIG. 21, specifically includes steps S1901 to S1904.

[0313] In S1901, a request for slope information transmitted from the server is received.

[0314] In an embodiment of the present application, the server transmits a tilt information request to the client, G kv and H kv Information can be obtained. Correspondingly, the client can receive a request for gradient information transmitted from the server.

[0315] In S1902, a second feature subset is generated from the second training set in response to a gradient information request.

[0316] In S1903, first gradient information of at least one third data instance identifier set of each feature of a second feature subset is obtained, the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces.

[0317] As an implementable method, based on the above embodiment as illustrated in FIG. 22, specifically includes steps S2001 to S2002.

[0318] In S2001, for any feature, all feature values ​​of the arbitrary feature are obtained, and bucketing is performed on the arbitrary feature based on the feature values.

[0319] In an embodiment of the present application, each sample bucket can be mapped according to each feature k of the set and each value v of all values ​​of the corresponding feature.

[0320] It should be explained that the above bucket mapping rules are numerous, and the present application does not limit the specific method of the bucket mapping rules, but only ensures that samples with identical feature values ​​are guaranteed. For example, in FIG. 11, two samples with a value of 1 can be assigned to the same data bucket.

[0321] For example, samples of the same value can be grouped into one bucket. That is, if there are n samples and one feature has m types of values, they are divided into m buckets, and the threshold of the corresponding feature is the m types of values.

[0322] For example, the number of buckets can be limited. For instance, if there are many, it can be divided into m buckets; in that situation, if the value of feature k is less than m, it can be divided in the previous manner; if there are more than m, it can be divided into m buckets according to a method similar to equal division.

[0323] In S2002, the first slope information of the third data instance identifier set of each bucketing of an arbitrary feature is obtained.

[0324] In an embodiment of the present application, a client obtains first gradient information of each bucketed third data instance identifier set of any feature. Correspondingly, a server may receive first gradient information of at least one third data instance identifier set of each feature transmitted from the client.

[0325] In S1904, the first slope information of the third data instance identifier set is transmitted to the server.

[0326] In an embodiment of the present application, the client obtains unprocessed data of the current node according to the part data of the common ID, obtains a feature set randomly, performs a bucketing mapping for each sample according to each feature k of the set and each value v of all values ​​of the corresponding feature, and G of the left subtree space kv and H kv Calculate and send to the server after homomorphic encryption processing.

[0327] In S1703, the first division value and the second division value are transmitted to the server.

[0328] In an embodiment of the present application, a client may perform lateral federated learning and longitudinal federated learning based on a first training set and a second training set to obtain a second split value corresponding to a training node, and then transmit the first split value and the second split value to a server. Correspondingly, the server may receive the first split value and the second split value.

[0329] It needs to be explained that, in this application, when attempting to perform node splitting on a training node based on a target splitting method, node splitting can be performed according to splitting information transmitted from the server.

[0330] As an implementable method, based on the above embodiment as illustrated in FIG. 23, specifically includes steps S2101 to S2102.

[0331] In S2101, fragmentation information transmitted from the server is received, wherein the fragmentation information includes a target fragmentation method, a target fragmentation feature selected as a feature fragmentation point, and a target fragmentation value.

[0332] In an embodiment of the present application, before attempting to notify the client to perform node splitting based on a target splitting method, the server may transmit splitting information to the client, wherein the splitting information includes a target splitting method, a target splitting feature selected as a feature splitting point, and a target splitting value. Correspondingly, the client may receive the splitting information transmitted from the server.

[0333] As an implementable method, the server can perform node splitting in a lateral manner. That is, by performing a splitting operation on the current node using the (k, value) information obtained according to the lateral method, IL information can be obtained and the IL can be broadcast to each client.

[0334] In S2102, node splitting is performed on the training nodes based on the splitting information.

[0335] As an implementable method, the client can perform node splitting on data with a shared ID by receiving (k, value) information from the server. The splitting method determines the value of feature k of the shared ID data: if the value is smaller than value, the ID of the corresponding data is inserted into the IL set; otherwise, it is inserted into the IR set. If feature k does not exist in the data, it is inserted into the right subtree space.

[0336] Furthermore, after performing node splitting on the training node, the client can send the split-generated left subtree space to the server. Correspondingly, the server can receive the split-generated left subtree space.

[0337] It is explained that in an embodiment of the present application, if the training node satisfies a preset splitting condition, the training node currently located must continue splitting; and if the training node does not satisfy the preset splitting condition, the training node currently located does not need to continue splitting, and in such a situation, the client uses the residual as the input for the next boosting tree and simultaneously performs node recall.

[0338] As an implementable method, based on the above embodiment as illustrated in FIG. 24, specifically includes steps S2201 to S2203.

[0339] In S2201, if the training node is a leaf node, the weight of the leaf node sent from the server is received.

[0340] In an embodiment of the present application, if the training node does not satisfy a preset splitting condition, the server designates the material node as a leaf node, and the weight w of the said leaf node j Calculate , and the corresponding w j It stores the value of and can be used as the weight of the longitudinal leaf node. Correspondingly, the client receives the weight w of the leaf node sent from the server. j Can receive.

[0341] In S2202, the residual of each data included is determined according to the weight of the leaf node.

[0342] In S2203, the residual is used as the input for the next boosting tree.

[0343] In an embodiment of the present application, the client [Ij(m), w jAccording to ], a new y'(t-1)(i) is calculated, recalled as a leaf node of the current tree, and can be set as the current node. Here, y'(t-1)(i) represents the Label residual corresponding to the i-th instance, t indicates that the current is the t-th tree, and t-1 indicates the previous tree.

[0344] Thus, the training method of a federated learning model provided in this application, wherein a client receives a target splitting method transmitted when a training node satisfies a preset splitting condition from a server, and the training node is a node of one of a plurality of boosting trees, and performs node splitting on the training node based on the target splitting method, thereby mixing a horizontal splitting method and a vertical splitting method, and the tendency to automatically select a matching learning method, does not require interest in the data distribution method, and solves the technical problem that causes poor training effects due to insufficient data utilization and insufficient use of all data existing in the process of the training method of a conventional federated learning model, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0345] It should be explained that, in the embodiments of the present application, the training process of the federated learning model mainly includes steps such as node splitting, model creation, and model verification. Below, the training method of the federated learning model provided in the embodiments of the present application is described, using the node splitting, model creation, and model verification steps as examples, where the server serves as the execution body and the verification client serves as the execution body, respectively, and the training of the federated learning model is performed.

[0346] With respect to the server being the execution body, as illustrated in FIG. 25, the training method of the federated learning model provided in the embodiment of the present application specifically includes steps S2301 to S2304.

[0347] In S2301, if the training node satisfies a preset splitting condition, a target splitting method corresponding to the training node is obtained; the training node is a node of one of the multiple boosting trees.

[0348] In S2302, the client is notified to perform node splitting based on the target splitting method, and the updated training nodes are obtained.

[0349] In S2303, it is determined that the updated training node satisfies the training stop condition, training is stopped, and the target federated learning model is created.

[0350] For clarification, the relevant details of steps S2301 to S2303 can be referenced in the above embodiments and are not described further here.

[0351] In S2304, a validation set is obtained, and the target federated learning model is validated in cooperation with a validation client, and the validation client is one of the clients participating in the training of the federated learning model.

[0352] Here, the validation set is typically a sub-sample of the training set. Optionally, a sample may be randomly sampled from the training set according to a preset ratio and used as the validation set. In an embodiment of the present application, the validation set includes a data instance identifier, and the validation set excludes the first training set and the second training set, respectively.

[0353] Thus, the training method for a federated learning model provided in this application, after the step of generating a federated learning model, has the server acquire a validation set to validate the target federated learning model in cooperation with a validation client, and when there is a relatively large number of homogeneous user data, combines training and validation, thereby reducing the validation loss of the federated learning model, improving the inference effect of the federated learning model, and further enhancing the validity and reliability of the federated learning model training process.

[0354] It should be explained that, in an embodiment of the present application, the updated training node includes a left subtree node generated after the training node split and other non-leaf nodes of one boosting tree. Here, the updated training satisfies the training stopping condition when the updated training node no longer satisfies the preset splitting condition; or when the updated training node is the last node of a plurality of boosting trees.

[0355] It should be explained that, in an embodiment of the present application, when attempting to verify a target federated learning model in cooperation with a verification client after obtaining a verification set, the data instance identifiers of the verification set can be verified one by one until all data instance identifiers of the verification set are verified.

[0356] As an implementable method, based on the above embodiment as illustrated in FIG. 26, the process of obtaining a verification set in step S2304 and verifying a target federated learning model in cooperation with a verification client specifically includes steps S2401 to S2405.

[0357] In S2401, the data instance identifier of one of the verification sets and the split information of the verification node are transmitted to the verification client, and the verification node is a node of one of the multiple boosting trees.

[0358] In an embodiment of the present application, the server may transmit any data instance identifier to the verification client and, at the same time, transmit splitting information of the verification node. Correspondingly, the verification client receives the data instance identifier and the splitting information of the verification node, obtains the corresponding data according to the data instance identifier, and determines the node direction corresponding to the verification node according to the splitting information. That is, it determines whether the node direction goes to the left subtree or the right subtree.

[0359] Here, the splitting information includes the features used for splitting and the splitting threshold.

[0360] In S2402, a node direction corresponding to the verification node transmitted from the verification client is received, wherein the node direction is determined by the verification client based on the data instance identifier and fragmentation information.

[0361] In an embodiment of the present application, after the step of determining a node direction corresponding to a verification node by a verification client, the node direction may be transmitted to a server. Correspondingly, the server may receive the node direction corresponding to a verification node transmitted from the verification client.

[0362] In S2403, enter the next node according to the node direction, and make the next node the updated verification node.

[0363] In an embodiment of the present application, the server may enter the next node according to the node direction returned by the verification client and make the next node the updated verification node. Furthermore, the server may determine whether the updated verification node satisfies a preset node splitting condition. If the updated verification node satisfies the preset node splitting condition, it explains that a leaf node has not been reached and performs step S2404; if the updated verification node does not satisfies the preset node splitting condition, it explains that a leaf node has already been reached and performs step S2405.

[0364] In S2404, if the updated verification node satisfies the preset node splitting condition, it returns and transmits the data instance identifiers and the splitting information to the verification client until all data instance identifiers of the verification set are verified.

[0365] In S2405, if the updated validation node does not satisfy the preset node splitting condition, the updated validation node is determined as a leaf node, and the model prediction value of the data instance represented by the data instance identifier is obtained.

[0366] In an embodiment of the present application, the server can determine the updated validation node as a leaf node, record the weight of the leaf node, and calculate and store the model prediction value of the data instance represented by the data instance identifier.

[0367] Here, the model prediction value of the data instance represented by the data instance identifier refers to the prediction value of each sample. When validating, if each sample goes to any leaf node in each tree, the Leaf Score of that leaf (leaf node score) is the score of the sample in that tree, and the sum of the scores of one sample in all trees is the prediction value.

[0368] Furthermore, after completing the above step S2404, it is possible to decide whether to withhold or use the target federated learning model.

[0369] As an implementable method, based on the above embodiment as illustrated in FIG. 27, steps S2404 are followed specifically by steps S2501 to S2503.

[0370] In S2501, if all data instance identifiers of the validation set are validated, the model prediction value of the data instance is sent to the validation client.

[0371] In an embodiment of the present application, when all data instance identifiers of the validation set are validated, that is, when the currently predicted data instance identifier is the last one among all predicted data instance identifiers, the model prediction value of the data instance can be transmitted to the validation client. Correspondingly, the validation client receives all prediction results and calculates the final validation result, compares it with the previous validation result to determine whether to hold or use the current target federated learning model, and generates validation instruction information according to the determination result.

[0372] To explain, when a client attempts to generate validation instruction information, it can obtain predicted values ​​by calculating all samples in the validation set. Since the validation client has its true label values, in this situation, the client calculates a related difference metric between the two based on the predicted values ​​and the label values. For example, it calculates metrics such as Accuracy (also called degree of accuracy) and RMSE (Root Mean Squared Error), and determines the model performance of the current epoch through these metrics.

[0373] Here, the current Epoch, also known as the current generation training, indicates that all training samples have undergone one forward and one backpropagation process in the neural network. In other words, one Epoch is a process of training all training samples once. Thus, if the obtained relevant difference metric is better than the metric from the previous Epoch, the currently obtained model can be retained; if the obtained relevant difference metric is inferior to the metric from the previous Epoch, the currently obtained model can be discarded.

[0374] In S2502, verification instruction information transmitted from a verification client is received, wherein the verification instruction information is obtained based on the model prediction value and is instruction information indicating whether to hold the model.

[0375] In an embodiment of the present application, a verification client may transmit verification instruction information to a server. Correspondingly, the server may receive the verification instruction information transmitted from the verification client.

[0376] In S2503, based on the verification instruction information, it determines whether to hold or use the target federated learning model, and transmits the decision result to the client.

[0377] In an embodiment of the present application, the server determines whether to withhold or use the target federated learning model based on verification instruction information, and transmits the decision result to all clients.

[0378] With respect to the verification client being the performing body, the training method of the federated learning model provided in the embodiment of the present application, as illustrated in FIG. 28, specifically includes steps S2601 to S2603.

[0379] In S2601, when the server determines that a training node satisfies a preset splitting condition, it receives a transmitted target splitting method, and the training node is a node of one of the multiple boosting trees.

[0380] In S2602, node splitting is performed on the training nodes based on the target splitting method.

[0381] For clarification, the relevant details of steps S2601 to S2602 can be referenced in the above embodiments and are not described further here.

[0382] In S2603, the validation set transmitted from the server is received, and the target federated learning model is validated based on the validation set.

[0383] In an embodiment of the present application, the server may acquire a validation set and transmit it to a validation client. Correspondingly, the validation client may receive the validation set transmitted from the server and, based on the validation set, validate the target federated learning model.

[0384] Here, the validation set transmitted from the server is received, and the target federated learning model is validated based on the validation set.

[0385] Thus, the training method for a federated learning model provided in this application performs node splitting on a training node based on a target splitting method, and then the verification client receives a verification set transmitted from the server, thereby verifying the target federated learning model based on the verification set; and when there is a relatively large number of homogeneous user data, by combining training and verification, the verification loss of the federated learning model is reduced, the inference effect of the federated learning model is improved, and the validity and reliability of the federated learning model training process are further enhanced.

[0386] It should be explained that, in an embodiment of the present application, when a verification client attempts to verify a target federated learning model based on a verification set, it may verify the data instance identifiers of the verification set one by one until all data instance identifiers of the verification set are verified.

[0387] As an implementable method, as illustrated in FIG. 29, the process of verifying a target federated learning model based on a verification set in step S2603, based on the above embodiment, specifically includes steps S2701 to S2703.

[0388] In S2701, the data instance identifier of one of the validation sets transmitted from the server and the splitting information of the validation node are received, where the validation node is a node of one of the multiple boosting trees.

[0389] In an embodiment of the present application, the server may transmit any data instance identifier to a verification client and, at the same time, transmit splitting information of a verification node. Correspondingly, the verification client may receive said data instance identifier and splitting information of a verification node.

[0390] Here, the splitting information includes the features used for splitting and the splitting threshold.

[0391] In S2702, the node direction of the verification node is determined based on the data instance identifier and splitting information.

[0392] In an embodiment of the present application, the verification client obtains the corresponding data according to the data instance identifier and can determine the node direction corresponding to the verification node according to the splitting information. That is, it determines whether the node direction goes to the left subtree or the right subtree.

[0393] As an implementable method, as illustrated in FIG. 30, the process of determining the node direction of a verification node according to the data instance identifier and splitting information in step S2702, specifically steps S2801 to S2802, is included.

[0394] In S2801, the feature value of each feature corresponding to the data instance identifier is determined according to the data instance identifier.

[0395] For clarification, the relevant details of step S2801 can be referenced in the above embodiment and are not described further here.

[0396] In S2802, the node direction is determined based on the splitting information and the feature value of each feature.

[0397] In an embodiment of the present application, the verification client determines the features used for splitting based on splitting information, and determines the node direction based on the feature value and splitting threshold of said features.

[0398] In S2703, the node direction is transmitted to the server, causing the server to enter the next node according to the node direction, and the next node is set as the updated verification node.

[0399] In an embodiment of the present application, a verification client may determine a node direction corresponding to a verification node and then transmit the node direction to a server. Correspondingly, the server may receive the node direction corresponding to a verification node transmitted from the verification client.

[0400] Furthermore, after completing the above step S2703, it is possible to decide whether to withhold or use the target federated learning model.

[0401] As an implementable method, based on the above embodiment as illustrated in FIG. 31, steps S2703 are followed specifically by steps S2901 to S2903.

[0402] In S2901, if all data instance identifiers of the validation set are validated, the model prediction value of the data instance represented by the data instance identifier transmitted from the server is received.

[0403] In an embodiment of the present application, when all data instance identifiers of the validation set are validated, that is, when the currently predicted data instance identifier is the last one among all predicted data instance identifiers, the model prediction value of the data instance can be transmitted to the validation client. Correspondingly, the validation client receives all prediction results and calculates the final validation result, compares it with the previous validation result to determine whether to hold or use the current target federated learning model, and generates validation instruction information according to the determination result.

[0404] It should be explained that when a validation client attempts to generate validation instruction information, it can obtain predicted values ​​by calculating all samples in the validation set. Since the validation client has its true label values, in this situation, the client calculates a related difference indicator between the two based on the predicted values ​​and the label values. For example, it calculates indicators such as Accuracy (also called degree of accuracy) and RMSE (Root Mean Squared Error), and determines the model performance of the current epoch through these indicators.

[0405] Here, the current Epoch, also known as the current generation training, indicates that all training samples have undergone one forward and one backpropagation process in the neural network. In other words, one Epoch is a process of training all training samples once. Thus, if the obtained relevant difference metric is better than the metric from the previous Epoch, the currently obtained model can be retained; if the obtained relevant difference metric is inferior to the metric from the previous Epoch, the currently obtained model can be discarded.

[0406] In S2902, a final validation result is obtained based on the model prediction value, and validation instruction information is generated by comparing the validation result with the previous validation result to indicate whether to hold or use the target federated learning model.

[0407] In an embodiment of the present application, a verification client may transmit verification instruction information to a server. Correspondingly, the server may receive the verification instruction information transmitted from the verification client.

[0408] In S2903, verification instruction information is sent to the server.

[0409] In an embodiment of the present application, the server determines whether to suspend or use the target federated learning model based on verification instruction information transmitted from a verification client, and can transmit the decision result to all clients.

[0410] FIG. 32 is a schematic flowchart of a training method for a federated learning model according to one embodiment of the present application.

[0411] As illustrated in FIG. 32, the training method of a federated learning model provided in an embodiment of the present application, exemplified by all processes for training a federated learning model for a server and a client (including a verification client), specifically includes steps S3001 to S3006.

[0412] In S3001, the server and client each perform initialization processing.

[0413] Optionally, the client sends a data identifier to the server. Specifically, the client sends to the server a data identifier for each piece of data that distinguishes each of its data. Correspondingly, the server receives the data identifiers sent from each client.

[0414] Furthermore, the server determines a shared data identifier between clients based on the received data identifier. Here, the shared data identifier is the same data identifier provided on different clients, determined by the server based on the data identifier reported by each client.

[0415] Furthermore, the server sends a shared data identifier to the client.

[0416] Furthermore, the client obtains the derivative of the loss formula based on the shared data identifier and local data, and proceeds with homomorphic encryption processing. Specifically, the client calculates some intermediate results of the loss function based on the shared data identifier and local data. For example, the first derivative g of the loss function i and second derivative h i Calculate . Here, g i and h i The calculation formula is as follows.

[0417]

[0418] Here, y i is the predicted result of sample i; the meaning of each symbol may refer to the related technology.

[0419] The formula for calculating the loss function is as follows.

[0420]

[0421] We propose expressing the above formula similarly in XGB using a second-order Taylor expansion. The second-order form of the Taylor expansion is as follows.

[0422]

[0423] Furthermore, the client sends the encrypted derivative to the server. Correspondingly, the server receives the encrypted derivative sent from the client.

[0424] Furthermore, the server performs decoder processing on the received derivative after encryption and proceeds to calculate the average value for the decoded derivative to obtain the average value. For the same shared data identifier, the derivatives corresponding to each client are accumulated to calculate the average value. For example, as described above, the calculation method involves the first derivative g corresponding to each shared data identifier. i and second derivative h i The average value is calculated by accumulating. Specifically, it includes the following formula.

[0425]

[0426]

[0427] Here, n represents the number of shared data IDs, and g i (j) is the first derivative g of data j i Represents, and h i (j) is the second derivative h of data j i It represents.

[0428] Furthermore, the server sends the average value to the client. For example, the server sends the average value to the client in the form of a list. In one implementation, the first derivative g i and second derivative h i can coexist in the same list; in other implementations, the first derivative g i and second derivative h i They each exist in different lists. For example, the first derivative g i exists in list A, and the second derivative h i It exists in list B. Correspondingly, the client receives the average value sent from the server.

[0429] Furthermore, the client updates the average value stored locally.

[0430] In S3002, the server and client start lateral XGB processing.

[0431] Optionally, the server determines whether the current material tree node should continue splitting. For example, the server determines whether to continue splitting the tree node based on whether the material hierarchy of the current material tree node has reached the maximum tree depth; if the tree node does not need to continue splitting, the server designates the material node as a leaf node and the weight w of the said leaf node j Calculate , and the corresponding w j It stores the value of and uses it as the weight of the leaf node of the horizontal XGB; if the tree node needs to continue splitting, the server generates a set of features available to the current node from all sets of features and sends the set of features to each client.

[0432] Furthermore, the server randomly generates a set of features available to the current node from all sets of features and sends the corresponding set of features to each client.

[0433] Furthermore, the client iterates through each feature of the feature set according to the acquired feature set, randomly selects one of all values ​​of the corresponding feature based on local data, and sends it to the server.

[0434] Furthermore, the server receives feature value information transmitted from each client, constructs a list of values, randomly selects one from the list of values ​​to set as the global optimal splitting threshold of the current feature, and broadcasts the said global optimal splitting threshold to each client.

[0435] Furthermore, the client performs node splitting on the current feature based on the received feature splitting threshold information, obtains the IL, and notifies the server; if the corresponding feature does not exist in the client, it returns an empty IL.

[0436] Here, IL is the set of instance IDs in the left subtree space, and the calculation method is as follows. The client receives the global optimal splitting threshold value of feature k transmitted from the server, and if the value of feature k corresponding to instance ID1 in the local data is smaller than the global optimal splitting threshold value, ID1 is added to the IL set. The formula is as follows.

[0437]

[0438] Here, represents the value of instance ID feature k, and S IL represents the set IL.

[0439] Furthermore, regarding the current feature, the server receives the IL transmitted from each client, filters out duplicate instance IDs in each IL, and processes contradictory ID information. For some clients, the instance ID can be added to the IL; however, if an ID exists for some clients but is not added, it is determined that the ID must exist in the IL, and the final IL and IR are determined. If the current feature does not exist in the data of some clients, the instance ID of the data of those clients is input into the IR, GL, GR, HL, and HR are calculated, and the split value Gain of the current feature is obtained.

[0440]

[0441] Here, GL is all first derivatives g in the left subtree space i It is the sum of; and GR is all first derivatives g in the right subtree space i It is the sum of; and HL is all second derivatives h in the left subtree space i It is the sum of; HR is all second derivatives h in the right subtree space i It is the sum of. The calculation method is as follows.

[0442]

[0443]

[0444] Here, n1 represents the number of instances in the left subtree space, and n2 represents the number of instances in the right subtree space.

[0445] The server, in cooperation with the client, iterates through each feature of a randomly selected feature set and, based on the splitting value calculated for each feature as a splitting node, sets the feature with the maximum splitting value as the optimal effect feature of the current node. Additionally, the server also knows threshold information corresponding to the splitting nodes and sets information such as the splitting threshold, splitting value gain, and selected feature as the optimal splitting information for the current node in a lateral XGB.

[0446] Furthermore, the server performs processing operations after the lateral node splitting, and the server notifies each client that it will no longer perform the lateral node splitting operation. In other words, the lateral node splitting operation has been completed.

[0447] Furthermore, the server designates the material node as a leaf node, and the weight w of the said leaf node j Calculate , and the corresponding w j It stores the value of and uses it as the weight of the leaf node of the horizontal XGB.

[0448] Here, , G m g corresponding to all instances of node m i Represents the sum of, and H m h corresponding to all instances of node m i Represents the sum of.

[0449] Furthermore, the server informs each client that it will no longer perform the lateral node splitting operation. In other words, it completes the lateral node splitting operation.

[0450] Furthermore, the client completes the processing after the lateral node split.

[0451] In S3003, the server and client start longitudinal XGB processing.

[0452] Optionally, the server informs the client to process the longitudinal XGB.

[0453] Furthermore, the server G to each client kv and H kv Request to obtain information.

[0454] Furthermore, the client obtains the unprocessed data of the current node according to that part of the common ID, obtains a random feature set, performs bucketing mapping for each sample according to each feature k of the set and all values ​​v of that feature, and G of the left subtree space kvand H kv Calculate , perform homomorphic encryption, and send to the server. Optionally, sort the values ​​of feature k of the dataset, perform a bucket insertion operation, and {s k,1 , s k,2 , s k,3 , 쪋, s k,v-1 It can be divided by}; G kv and H kv The calculation formula is as follows.

[0455]

[0456]

[0457] Here, x i,k is data x i Represents the value of the feature k.

[0458] Furthermore, the server [[G sent from each client kv ]] and [[H kv Decoder ]], and that part data of the current node's common data identifier and all previously obtained g i and h i Accordingly, the G and H of the current node can be calculated. Optionally, if the partial clients are equipped with the same features, the server receives the G kv Randomly select one of the current features' G kv Do it as, and H kv It is also processed in this way. Furthermore, G, H, and G kv , H kv Based on this, the optimal split point of each feature can be calculated, and based on the split point information, the global optimal split point (k, v, Gain) can be calculated.

[0459] In an embodiment of the present application, the received information is compared with a preset threshold, and if the Gain is less than or equal to the threshold, step S3027 is performed without performing longitudinal node splitting; if the Gain is greater than the threshold, step S3023 can be performed.

[0460] Furthermore, the server requests IL from the corresponding client based on (k, v, Gain).

[0461] Furthermore, Client C receives split point information (k, v, Gain), searches to obtain the split point threshold value, and records the information for the corresponding split point (k, value). The local data set is split according to the corresponding split point to obtain the IL, and (record, IL, value) is transmitted to the server. Here, record represents the index recorded on the client, and the IL calculation method is as described above.

[0462] Furthermore, the server receives information (record, IL, value) transmitted from client C, divides instances of all common IDs in the material node space, and associates the current node with client C through (client id, record). In an embodiment of the present application, the server may record (client id, record_id, IL, feature_name, feature_value) to use as information for longitudinal splitting and perform step S3027.

[0463] Furthermore, the server designates the material node as a leaf node, and the weight w of the said leaf node j Calculate , and the corresponding w j It stores and uses it as the weight of the longitudinal leaf node.

[0464] Furthermore, the server notifies each client not to perform vertical leaf node splitting any further; or, completes the node splitting operation.

[0465] Furthermore, each client proceeds with processing after the longitudinal node split.

[0466] In S3004, the server and client start mixed processing of horizontal XGB and vertical XGB.

[0467] Optionally, the server determines whether the current node is split. In an embodiment of the present application, if the current node does not need to be split, the server's location node is designated as a leaf node, and the weight w of the said leaf node j Calculate and information [Ij(m), w j ] is sent to all clients; if the current node needs to be split, the server obtains Gain by lateral XGB and obtains Gain by vertical XGB, and determines the target Gain based on the Gain, thereby determining the node splitting method and performing node splitting.

[0468] Furthermore, the server obtains a Gain through lateral XGB and obtains a Gain through vertical XGB, and obtains a target Gain according to the Gain, thereby determining the node splitting method and performing node splitting. In an embodiment of the present application, if the method is lateral XGB, the server performs the node splitting in the lateral manner. That is, according to the (k, value) information obtained by the lateral method, it performs a splitting operation for the current node, obtains IL information, and broadcasts the IL to each client; if the method is vertical XGB, the client notifies each client to perform a true node splitting operation according to the (client_id, record_id, IL, feature_name, feature_value) of the vertical splitting recorded by the server.

[0469] Furthermore, the client notifies each client to perform a true node splitting operation based on the (client_id, record_id, IL, feature_name, feature_value) of the longitudinal split recorded by the server. Here, the client corresponding to client_id must know all the information of (client_id, record_id, IL, feature_name, feature_value), while other clients only need to know the IL information.

[0470] The server uses the currently split left subtree node as the current processing node.

[0471] Furthermore, the client receives IL or (client_id, record_id, IL, feature_name, feature_value) transmitted from the server and performs a longitudinal node splitting operation; if the information (client_id, record_id, IL, feature_name, feature_value) exists, the client must record and store the said information when splitting. In an embodiment of the present application, after splitting is completed, the split left subtree node is set as the current processing node.

[0472] Furthermore, the server returns to step S3002 and continues subsequent processing, and the client returns to step S3002 and waits for a server message. It must be explained that since only the splitting of the current node has been completed at this point, the splitting of the left and right subtree nodes of the next hierarchy must also be performed. Therefore, the process returns to step S3002 and performs the node splitting of the next node.

[0473] Furthermore, the server performs node splitting in a lateral manner. That is, based on the (k, value) information obtained by the above lateral method, it performs a splitting operation on the current node, obtains IL information, and broadcasts the IL to each client. Here, IL can be represented by the following formula.

[0474]

[0475] Here, represents the value of instance ID feature k, and represents the set IL.

[0476] Furthermore, the client can receive the IL broadcast from the server, combine the data of the local non-cooperative ID according to the IL, determine the IL and IR of the current node, and perform a node splitting operation. It should be explained that the determination of the IL and IR is based on the ID of the local non-cooperative ID data, and if it is not in the set of ILs transmitted by the server, it is explained that it is in the set of IRs.

[0477] Furthermore, if the server performs node splitting in a lateral manner, it is performed according to the selected feature k and threshold value, and broadcasts (k, value) to the client.

[0478] Furthermore, each client receives the (k, value) information from the server and performs node splitting on the data with the shared ID, where the splitting method is feature k of the shared ID data. Here, if the value is smaller than value, the ID of the corresponding data must be inserted into the IL set. Otherwise, it is inserted into the IR set. If the data does not exist in feature k, the right subtree space is inserted.

[0479] Furthermore, the server returns to step S3002 to continue performing the splitting operation of the next node, and the client returns to step S3002 to wait for a server message.

[0480] Furthermore, the server designates the material node as a leaf node, and the weight w of the said leaf node j Calculate and information [Ij(m), w j Sends ] to all clients. Here, Ij(m) is the set of instance IDs of the current node space, and w j is the weight of the current node.

[0481] Furthermore, the client [Ij(m), w jBased on ], calculate a new y'(t-1)(i), traverse to other non-leaf nodes of the current tree, and set them as the current nodes. Here, y'(t-1)(i) represents the Label residual corresponding to the i-th instance, t indicates that the current is the t-th tree, and t-1 indicates the previous tree.

[0482] Furthermore, the server traverses to the leaf nodes of the current tree and sets them as the current nodes.

[0483] Furthermore, if the current node after the traversal exists and is not empty, return to step S3002 and proceed with the next processing, and the client returns to step S3002 and waits for a server message.

[0484] Furthermore, if the current node is empty, the model validation operation is performed.

[0485] In S3005, the server and the validation client perform validation of the target federated learning model.

[0486] Optionally, the server notifies the validation client to perform a validation initialization action.

[0487] Furthermore, the verification client performs verification initialization.

[0488] Furthermore, the server selects one ID and begins verification. It initializes the XGB tree and notifies the verification client to start verification.

[0489] Furthermore, the verification client initializes the verification information.

[0490] Furthermore, the server sends the split node information and the verified data ID to the verification client according to the current XGB tree.

[0491] Furthermore, the verification client retrieves the corresponding data based on the data ID, determines whether to proceed to the left subtree or the right subtree based on the split node information transmitted from the server, and returns it to the server.

[0492] Furthermore, the server enters the next node according to the direction returned by the validation client. It then determines whether it has reached a leaf node; if it has, the server records the weight of the leaf node, calculates and stores the predicted value; if it has not reached, the server selects one ID and starts validation. It initializes the XGB tree and notifies the validation client to start validation.

[0493] Furthermore, the server records the weights of the leaf nodes, calculates the predicted values, and stores them. If the currently predicted ID is the last of all predicted IDs, the server sends all prediction results to the client; otherwise, the server selects one ID and starts verification. It initializes the XGB tree and notifies the verification client to start verification.

[0494] Furthermore, the server sends all prediction results to the client.

[0495] Furthermore, the validation client receives all prediction results, calculates the final validation result, compares it with the previous validation result, determines whether to hold or use the current model, and notifies the server of the decision result.

[0496] Furthermore, the server determines whether to suspend or use the current model based on the verification results returned by the verification client, and this is done for all clients.

[0497] Furthermore, each client receives and processes the server's broadcast information.

[0498] Furthermore, the server determines whether the final predicted sequence number has been reached, and if it has, performs step S3006.

[0499] In S3006, the server and client each terminate training and put the target federated learning model on hold.

[0500] Optionally, the server terminates all training, processes information, and suspends the model.

[0501] Furthermore, the client terminates all training, processes the information, and suspends the model.

[0502] As described above, in the training method of a federated learning model according to the embodiment of the present application, the tendency to automatically select a matching learning method by mixing a transverse splitting method and a longitudinal splitting method does not require attention to the data distribution method and solves the technical problem of not being able to learn by fully utilizing all data existing in the process of the training method of a conventional federated learning model and causing poor training effects due to insufficient data utilization, while simultaneously reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0503] Based on the same application concept, an embodiment of the present application further provides an apparatus corresponding to a training method of a federated learning model.

[0504] FIG. 33 is a schematic diagram of the structure of a training device for a federated learning model provided in an embodiment of the present application.

[0505] As illustrated in FIG. 33, the training device (1000) of the federated learning model is applied to a server and includes an acquisition module (110), a notification module (120), a first training module (130), a second training module (140), and a generation module (150).

[0506] Here, the acquisition module (110) is used to acquire a target splitting method corresponding to the training node when the training node satisfies a preset splitting condition; the training node is a node of one of a plurality of boosting trees.

[0507] The notification module (120) is used to notify the client to perform node splitting based on the target splitting method.

[0508] The first training module (130) is used to perform the next round of training by making the left subtree node created after the training node is split the training node again, until the updated training node no longer satisfies the preset splitting condition.

[0509] The second training module (140) is used to make other non-leaf nodes of the one boosting tree into training nodes again and to perform the next round of training.

[0510] The generation module (150) is used to stop training and generate a target federated learning model when the node data sets of the plurality of boosting trees are all empty.

[0511] According to one embodiment of the present application, as shown in FIG. 34, the acquisition module (110) of FIG. 33 is,

[0512] A first learning submodule (111) that performs lateral federated learning in cooperation with the client based on a first training set to obtain a first split value corresponding to the training node;

[0513] A second learning submodule (112) that performs longitudinal federated learning in cooperation with the client based on the second training set to obtain a second split value corresponding to the training node; and

[0514] It includes a decision submodule (113) that determines a target splitting method corresponding to the training node according to the first splitting value and the second splitting value.

[0515] According to one embodiment of the present application, as illustrated in FIG. 35, the decision submodule (113) of FIG. 34 is,

[0516] A first decision unit (1131) that determines a relatively large value among the first split value and the second split value as the target split value corresponding to the training node;

[0517] It includes a second decision unit (1132) that determines a splitting method corresponding to the training node according to the target splitting value.

[0518] According to one embodiment of the present application, as illustrated in FIG. 36, the first learning submodule (111) of FIG. 33 is,

[0519] A generating unit (1111) that generates a first feature subset available to the training node in the first training set and transmits it to the client;

[0520] A first receiving unit (1112) that receives the feature value of each feature of the first feature subset transmitted from the client;

[0521] A third determination unit (1113) that determines each feature as a transverse division value corresponding to a division feature point according to the feature value of each feature of the first feature subset; and

[0522] It includes a fourth determination unit (1114) that determines the first split value of the training node according to the transverse split value corresponding to each of the above features.

[0523] According to one embodiment of the present application, as illustrated in FIG. 37, the third determination unit (1113) of FIG. 36 is,

[0524] A first determination sub-unit (11131) that determines a splitting threshold of any feature of the first feature subset according to the feature value of the arbitrary feature;

[0525] A first acquisition sub-unit (11132) that acquires a first data instance identifier set and a second data instance identifier set corresponding to the arbitrary feature according to the above splitting threshold - the first data instance identifier set includes a data instance identifier belonging to a first left subtree space, and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space - ; and

[0526] It includes a second determination sub-unit (11133) that determines the transverse division value corresponding to the arbitrary feature according to the first data instance identifier set and the second data instance identifier set.

[0527] According to one embodiment of the present application, a first acquisition sub-unit (11132) also transmits the splitting threshold to the client; receives an initial data instance identifier set corresponding to the training node transmitted from the client, wherein the initial data instance identifier set is generated when the client splits the node for any feature according to the splitting threshold, and the initial data instance identifier set includes data instance identifiers belonging to the first left subtree space; and is used to acquire the first data instance identifier set and the second data instance identifier set based on the initial data instance identifier set and all data instance identifiers.

[0528] According to one embodiment of the present application, the first acquisition sub-unit (11132) is also used to compare each data instance identifier of the first data instance identifier set with the data instance identifier of the client and to acquire an anomalous data instance identifier; to perform preprocessing on the anomalous data instance identifier and to acquire the first data instance identifier set; and to acquire the second data instance identifier set based on all data instance identifiers and the first data instance identifier set.

[0529] According to one embodiment of the present application, as illustrated in FIG. 38, the second learning submodule (112) of FIG. 34 is,

[0530] A notification unit (1121) that notifies the client to perform longitudinal federated learning based on the second training set;

[0531] A second receiving unit (1122) receiving first gradient information of at least one third data instance identifier set of each feature transmitted from the above client - the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces - ;

[0532] A fifth determination unit (1123) that determines the longitudinal splitting value of each feature according to the first gradient information of each feature and the total gradient information of the training node; and

[0533] It includes a sixth determination unit (1124) that determines the second split value of the training node according to the longitudinal split value corresponding to each feature.

[0534] According to one embodiment of the present application, as illustrated in FIG. 39, the fifth determination unit (1123) of FIG. 38 is,

[0535] For any feature, a second acquisition sub-unit (11231) that acquires second slope information corresponding to each first slope information according to the total slope information and each first slope information;

[0536] For each first slope information, a third acquisition sub-unit (11232) that acquires a candidate longitudinal split value of the arbitrary feature according to the first slope information and the second slope information corresponding to the first slope information; and

[0537] It includes a selection sub-unit (11233) that selects the maximum value among the above candidate longitudinal splitting values ​​and sets it as the longitudinal splitting value of the above arbitrary feature.

[0538] According to one embodiment of the present application, the first slope information includes a first slope sum of features corresponding to a data instance belonging to the second left subtree space and a second slope sum of features corresponding to a data instance belonging to the second left subtree space; and the second slope information includes a first slope sum of features corresponding to a data instance belonging to the second right subtree space and a second slope sum of features corresponding to a data instance belonging to the second right subtree space.

[0539] According to one embodiment of the present application, as illustrated in FIG. 40, the combined model training device (1000) is,

[0540] A decision module (160) that determines the training node as a leaf node and obtains the weight of the leaf node when the training node does not satisfy a preset splitting condition;

[0541] It further includes a transmission module (170) that transmits the weight of the leaf node to the client.

[0542] According to one embodiment of the present application, as illustrated in FIG. 41, the determination module (160) of FIG. 40 is,

[0543] A first acquisition submodule (161) for acquiring a data instance belonging to the leaf node;

[0544] It includes a second acquisition submodule (162) that acquires first gradient information and second gradient information of a data instance belonging to the leaf node, and acquires the weight of the leaf node according to the first gradient information and the second gradient information.

[0545] According to one embodiment of the present application, as illustrated in FIG. 35, the decision submodule (113) of FIG. 34 is,

[0546] The apparatus further includes a transmitting unit (1133) that transmits splitting information to the above client, wherein the splitting information includes the target splitting method, the target splitting feature selected as the feature splitting point, and the target splitting value.

[0547] According to one embodiment of the present application, a transmitting unit (1133) also transmits the splitting information to a client having a label; receives a left subtree space set transmitted from the client having a label; splits the second training set according to the left subtree space set; and is used to associate the training node with the identifier of the client having a label.

[0548] According to one embodiment of the present application, the acquisition module (110) also receives a data instance identifier transmitted from the client; and, based on the data instance identifier, determines a common data instance identifier between the clients, and the common data instance identifier is used to instruct the clients to determine the first training set and the second training set.

[0549] Thus, the tendency of the training device and server of the federated learning model provided in this application to mix the transverse splitting method and the longitudinal splitting method and automatically select the matching learning method solves the technical problem that the training effect is poor due to insufficient data utilization and the inability to learn by fully utilizing all data existing in the process of the training method of the conventional federated learning model, without needing to pay attention to the data distribution method, while simultaneously reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0550] Based on the same application concept, an embodiment of the present application provides an apparatus corresponding to a model training method of another federated learning model.

[0551] FIG. 42 is a schematic diagram of the structure of a training device for a federated learning model provided in an embodiment of the present application. As shown in FIG. 42, a model training device (2000) for a federated learning model applied to a client includes a first receiving module (210) and a splitting module (220).

[0552] Here, the first receiving module (210) is used to receive the target splitting method transmitted when the server determines that the training node satisfies the preset splitting condition, and the training node is a node of one of the boosting trees among a plurality of boosting trees.

[0553] The splitting module (220) is used to perform node splitting on the training node based on the target splitting method.

[0554] According to one embodiment of the present application, as shown in FIG. 43, the splitting module (220) of FIG. 42 is,

[0555] A first learning submodule (221) that performs lateral federated learning based on a first training set and obtains a first split value corresponding to the training node;

[0556] A second learning submodule (222) that performs longitudinal federated learning based on a second training set and obtains a second split value corresponding to the training node; and

[0557] It includes a transmission submodule (223) that transmits the first division value and the second division value to the server.

[0558] According to one embodiment of the present application, as illustrated in FIG. 44, the first learning submodule (221) of FIG. 43 is,

[0559] A first receiving unit (2211) that receives a first feature subset available to the training node generated by the server in the first training set;

[0560] A first transmitting unit (2212) that transmits the feature values ​​of each feature of the first feature subset to the above server;

[0561] A second receiving unit (2213) for receiving the splitting threshold of each feature transmitted from the server; and

[0562] It includes a first acquisition unit (2214) that acquires a set of initial data instance identifiers corresponding to the training node based on the splitting threshold of each of the above features, and transmits the set of initial data instance identifiers to the server.

[0563] The first set of data instance identifiers is used to instruct the server to generate a first set of data instance identifiers and a second set of data instance identifiers, wherein both the first set of data instance identifiers and the first set of data instance identifiers include data instance identifiers belonging to a first left subtree space, and the second set of data instance identifiers includes data instance identifiers belonging to a first right subtree space.

[0564] According to one embodiment of the present application, the first acquisition unit (2214) is also used to compare, for any feature, the splitting threshold of the arbitrary feature with the feature value of the arbitrary feature, acquire a data instance identifier in which the feature value is smaller than the splitting threshold, and generate the first set of data instance identifiers.

[0565] According to one embodiment of the present application, as illustrated in FIG. 45, the second learning submodule (222) of FIG. 43 is,

[0566] A third receiving unit (2221) that receives a slope information request transmitted from the above server;

[0567] A generating unit (2222) that generates a second feature subset from a second training set in accordance with the above gradient information request;

[0568] A second acquisition unit (2223) for acquiring first gradient information of at least one third data instance identifier set of each feature of the second feature subset above - the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting a feature value of one of the features, and different feature values ​​correspond to different second left subtree spaces - ; and

[0569] It includes a second transmission unit (2224) that transmits the first slope information of the third data instance identifier set to the server.

[0570] According to one embodiment of the present application, as shown in FIG. 46, the second acquisition unit (2223) of FIG. 45 is,

[0571] A bucketing sub-unit (22231) that, for any feature, obtains all feature values ​​of the arbitrary feature and buckets the arbitrary feature based on the feature values;

[0572] It includes a first acquisition sub-unit (22232) for acquiring first slope information of each bucketed set of third data instance identifiers of the above arbitrary features.

[0573] According to one embodiment of the present application, as shown in FIG. 43, the splitting module (220) of FIG. 42 is,

[0574] A receiving submodule (224) that receives splitting information transmitted from the above server - the splitting information includes the target splitting method, the target splitting feature selected as the feature splitting point, and the target splitting value - ;

[0575] It further includes a splitting submodule (225) that performs node splitting for the training node based on the splitting information above.

[0576] According to one embodiment of the present application, the splitting submodule (225) is also used to transmit the split-generated left subtree space to the server.

[0577] According to one embodiment of the present application, as illustrated in FIG. 47, the combined model training device (1000) is,

[0578] If the above training node is a leaf node, a second receiving module (230) receives the weight of the leaf node transmitted from the server;

[0579] A decision module (240) that determines the residual of each included data according to the weight of the leaf node above; and

[0580] It further includes an input module (250) that uses the above residual as the residual input for the next boosting tree.

[0581] Thus, in the training device of the federated learning model provided in the present application, the client receives a target splitting method transmitted when the server determines that the training node satisfies a preset splitting condition, the training node is a node of one of a plurality of boosting trees, and the tendency to perform node splitting on the training node based on the target splitting method, thereby mixing the transverse splitting method and the longitudinal splitting method and automatically selecting a matching learning method, solves the technical problem that the training method of the conventional federated learning model does not need to be concerned with the data distribution method and fails to learn by sufficiently utilizing all the data present in the process of the training method of the federated learning model, and causes poor training effects due to insufficient data utilization, while reducing the loss of the federated learning model and improving the performance of the federated learning model.

[0582] Based on the same application concept, embodiments of the present application further provide an electronic device.

[0583] FIG. 48 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in FIG. 48, the electronic device (3000) includes a memory (310), a processor (320), and a computer program stored in the memory (310) and executable on the processor (320), and when the processor executes the computer program, a training method of the federated learning model is implemented.

[0584] It should be understood by those skilled in the art that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may use the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Additionally, the present application may use the form of a computer program product executed on a computer-usable storage medium (including, but not limited to, magnetic disk storage devices, CD-ROMs, optical storage devices, etc.) containing program code usable by one or more computers.

[0585] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and combinations of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. By providing said computer program instructions to a processor of a general-purpose computer, a dedicated computer, an embedded processor, or other programmable data processing device to create a machine, the instructions executed by the processor of the computer or other programmable data processing device are made to create a device of a specified function in one or more processes of the flowchart and / or one or more blocks of the block diagram.

[0586] Since the computer program instruction may also be stored in computer-readable memory that guides a computer or other programmable data processing device to operate in a specific manner, the instruction stored in said computer-readable memory creates a product including said instruction device, and said instruction device implements a specified function in one or more processes of a flowchart and / or one or more blocks of a block diagram.

[0587] Since the computer program instruction can also be loaded into a computer or other programmable data processing device, a series of operation steps are performed on the computer or other programmable device to generate processing implemented by the computer, thereby providing a step of implementing a specified function in one or more processes of a flowchart and / or one or more blocks of a block diagram in the instruction performed by the computer or other programmable device.

[0588] It should be noted that, in a claim, a reference signal located between parentheses should not be assumed to be a limitation to the claim. The word "includes" does not exclude the existence of components or steps not listed in the claim. The word "one" or "one" preceding a component does not exclude the existence of multiple such components. The present application may be implemented by hardware comprising several different components and a suitably programmed computer. In a claim listing several units of a device, several of said devices may be specifically implemented through the same hardware. The use of words first, second, third, etc., does not indicate any order. Such words may be interpreted as names.

[0589] Although preferred embodiments of this application have been described, once a person skilled in the art becomes aware of the basic creative concept, other changes and modifications to said embodiments may be made. Accordingly, the appended claims are intended to describe the preferred embodiments and all changes and modifications included within the scope of this application.

[0590] Those skilled in the art may make variations and modifications to this application within the spirit and scope of the invention. Thus, if such amendments and modifications to this application fall within the claims and the same technical scope as this application, this application is intended to include such variations and modifications.

Claims

Claim 1 A training method for a federated learning model applied to a server, comprising: a step of obtaining a target splitting method corresponding to a training node when the training node satisfies a preset splitting condition; the training node being a node of one of a plurality of boosting trees; a step of notifying a client to perform node splitting based on the target splitting method; a step of performing the next round of training by making the left subtree node created after the training node split the training node again the training node until the updated training node no longer satisfies the preset splitting condition; a step of performing the next round of training by making the other non-leaf node of the one boosting tree the training node again the training node; and a step of stopping training and creating a target federated learning model when the node data set of the plurality of boosting trees is all empty. Claim 2 A training method for a federated learning model according to claim 1, wherein the step of obtaining a target splitting method corresponding to the training node comprises: a step of obtaining a first splitting value corresponding to the training node by performing lateral federated learning in cooperation with the client based on a first training set; a step of obtaining a second splitting value corresponding to the training node by performing longitudinal federated learning in cooperation with the client based on a second training set; and a step of determining a target splitting method corresponding to the training node according to the first splitting value and the second splitting value. Claim 3 A training method for a federated learning model according to claim 2, wherein the step of determining a target splitting method corresponding to the training node according to the first splitting value and the second splitting value comprises: determining a relatively large value among the first splitting value and the second splitting value as the target splitting value corresponding to the training node; and determining a splitting method corresponding to the training node according to the target splitting value. Claim 4 A training method for a federated learning model according to claim 2, wherein the step of performing lateral federated learning in cooperation with the client based on the first training set to obtain a first splitting value corresponding to the training node comprises: generating a first feature subset available to the training node in the first training set and transmitting it to the client; receiving a feature value of each feature of the first feature subset transmitted from the client; determining each feature as a lateral splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset; and determining the first splitting value of the training node according to the lateral splitting value corresponding to each feature. Claim 5 In claim 4, the step of determining each feature as a transverse splitting value corresponding to a splitting feature point according to the feature value of each feature of the first feature subset comprises: a step of determining a splitting threshold for any feature of the first feature subset according to the feature value of the arbitrary feature; a step of obtaining a first data instance identifier set and a second data instance identifier set corresponding to the arbitrary feature according to the splitting threshold, wherein the first data instance identifier set includes a data instance identifier belonging to a first left subtree space and the second data instance identifier set includes a data instance identifier belonging to a first right subtree space; and a step of determining the transverse splitting value corresponding to the arbitrary feature according to the first data instance identifier set and the second data instance identifier set; characterized by comprising: a training method for a federated learning model. Claim 6 A training method for a federated learning model according to claim 5, wherein the step of obtaining a first data instance identifier set and a second data instance identifier set corresponding to the arbitrary feature according to the splitting threshold comprises: transmitting the splitting threshold to the client; receiving an initial data instance identifier set corresponding to the training node transmitted from the client, wherein the initial data instance identifier set is generated when the client splits the node for the arbitrary feature according to the splitting threshold, and the initial data instance identifier set includes a data instance identifier belonging to the first left subtree space; and obtaining the first data instance identifier set and the second data instance identifier set based on the initial data instance identifier set and all data instance identifiers. Claim 7 A method for training a federated learning model according to claim 2, wherein the step of performing longitudinal federated learning in cooperation with the client based on the second training set and obtaining a second split value corresponding to the training node comprises: a step of notifying the client to perform longitudinal federated learning based on the second training set; a step of receiving first gradient information of at least one third data instance identifier set of each feature transmitted from the client, wherein the third data instance identifier set includes a data instance identifier belonging to a second left subtree space, the second left subtree space is a left subtree space formed by splitting one feature value among the features, and different feature values ​​correspond to different second left subtree spaces; a step of determining a longitudinal split value of each feature according to the first gradient information of each feature and the total gradient information of the training node; and a step of determining the second split value of the training node according to the longitudinal split value corresponding to each feature. Claim 8 A training method for a federated learning model according to claim 7, wherein the step of determining a longitudinal split value of each feature according to the first gradient information of each feature and the total gradient information of the training node comprises: for any feature, obtaining a second gradient information corresponding to each first gradient information according to the total gradient information and each first gradient information; for each first gradient information, obtaining a candidate longitudinal split value of the arbitrary feature according to the first gradient information and the second gradient information corresponding to the first gradient information; and selecting a maximum value among the candidate longitudinal split values ​​and setting it as the longitudinal split value of the arbitrary feature. Claim 9 A training method for a federated learning model according to claim 1, further comprising: a step of determining the training node as a leaf node and obtaining the weight of the leaf node when the training node does not satisfy a preset splitting condition; and a step of transmitting the weight of the leaf node to the client. Claim 10 A training method for a federated learning model according to claim 9, wherein the step of obtaining the weight of the leaf node comprises: a step of obtaining a data instance belonging to the leaf node; a step of obtaining first gradient information and second gradient information of the data instance belonging to the leaf node, and a step of obtaining the weight of the leaf node according to the first gradient information and the second gradient information. Claim 11 A training method for a federated learning model according to claim 3, further comprising the step of transmitting splitting information to the client prior to the step of notifying the client to perform node splitting based on the target splitting method, wherein the splitting information includes the target splitting method, the target splitting feature selected as a feature splitting point, and the target splitting value. Claim 12 A training method for a federated learning model according to claim 11, wherein, prior to the step of notifying the client to perform node splitting based on the target splitting method when the target splitting method is a longitudinal splitting method, the method further comprises: the step of transmitting splitting information to a client having a label; the step of receiving a left subtree space set transmitted from the client having the label; the step of splitting the second training set according to the left subtree space set; and the step of associating the training node with the identifier of the client having the label. Claim 13 A method for training a federated learning model according to claim 1, further comprising: a step of obtaining the updated training node after the step of notifying the client to perform node splitting based on the target splitting method; a step of determining that the updated training node satisfies a training stop condition, stopping the training, and generating a target federated learning model; and a step of obtaining a validation set and validating the target federated learning model in cooperation with a validation client - the validation client is one of the clients participating in the federated learning model training - Claim 14 A training method for a federated learning model applied to a client, comprising: receiving a target splitting method transmitted when a server determines that a training node satisfies a preset splitting condition - said training node is a node of one of a plurality of boosting trees -; and performing node splitting on said training node based on said target splitting method; characterized by comprising. Claim 15 A training method for a federated learning model according to claim 14, further comprising the step of, after the step of performing node splitting for the training node based on the target splitting method, receiving a validation set transmitted from the server and validating the target federated learning model based on the validation set. Claim 16 A training method for a federated learning model according to claim 15, wherein the step of verifying the target federated learning model in cooperation with a verification client based on the verification set comprises: receiving a data instance identifier of one of the verification sets and splitting information of a verification node transmitted from the server, wherein the verification node is a node of one of a plurality of boosting trees; determining the node direction of the verification node according to the data instance identifier and the splitting information; and the node direction transmitted to the server causes the server to enter the next node according to the node direction, and sets the next node as the updated verification node. Claim 17 A training method for a federated learning model according to claim 16, wherein the step of determining the node direction of the verification node according to the data instance identifier and the splitting information comprises: the step of determining the feature value of each feature corresponding to the data instance identifier according to the data instance identifier; and the step of determining the node direction according to the splitting information and the feature value of each feature. Claim 18 A training method for a federated learning model according to claim 16, further comprising: receiving a model prediction value of a data instance represented by the data instance identifier transmitted from the server when all data instance identifiers of the verification set are verified; obtaining a final verification result according to the model prediction value and comparing the verification result with a previous verification result to generate verification instruction information indicating whether to suspend or use the target federated learning model; and transmitting the verification instruction information to the server. Claim 19 An electronic device comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein when the processor executes the computer program, a training method of a federated learning model according to any one of claims 1 to 13 or claims 14 to 18 is implemented. Claim 20 A computer-readable storage medium in which a computer program is stored, characterized in that when the computer program is executed by a processor, a training method of any one of claims 1 to 13 or claims 14 to 18 is implemented. Claim 21 delete Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 delete Claim 26 delete Claim 27 delete Claim 28 delete Claim 29 delete Claim 30 delete Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete Claim 35 delete Claim 36 delete Claim 37 delete Claim 38 delete Claim 39 delete Claim 40 delete Claim 41 delete Claim 42 delete Claim 43 delete Claim 44 delete Claim 45 delete Claim 46 delete Claim 47 delete Claim 48 delete Claim 49 delete Claim 50 delete Claim 51 delete Claim 52 delete Claim 53 delete Claim 54 delete Claim 55 delete Claim 56 delete Claim 57 delete Claim 58 delete Claim 59 delete Claim 60 delete Claim 61 delete Claim 62 delete Claim 63 delete Claim 64 delete Claim 65 delete Claim 66 delete Claim 67 delete Claim 68 delete Claim 69 delete Claim 70 delete Claim 71 delete Claim 72 delete Claim 73 delete Claim 74 delete Claim 75 delete Claim 76 delete Claim 77 delete Claim 78 delete Claim 79 delete Claim 80 delete Claim 81 delete Claim 82 delete Claim 83 delete Claim 84 delete Claim 85 delete Claim 86 delete Claim 87 delete Claim 88 delete Claim 89 delete Claim 90 delete Claim 91 delete Claim 92 delete

Images