Method, device and system for secure vertical federated learning

Abstract

A machine learning model is learned using secure longitudinal federated learning in the following manner: a network machine learning model receives a set of private machine learning model outputs from a plurality of private machine learning models. The set of private machine learning model outputs is based on data owned exclusively by each of the plurality of private machine learning models. The set of private machine learning model outputs is aligned based on sample IDs of the data. The network machine learning model provides a prediction, the prediction being the output of the network model based on the set of private machine learning model outputs. The network model transmits the prediction to one of the plurality of private machine learning models, the one of the plurality of private machine learning models including a label. The network model receives a loss based on the label and the prediction from the one of the plurality of private machine learning models, calculates a gradient based on the loss, and updates parameters of the network model based on the loss.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 897,808, filed on September 9, 2019, entitled “METHOD AND SYSTEM FOR SECURE VERTICAL FEDERATED LEARNING,” and U.S. Patent Application No. 17 / 014,204, filed on September 8, 2020, entitled “METHOD AND SYSTEM FOR SECURE VERTICALFEDERATED LEARNING,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to the field of machine learning, and in particular to a method and system for performing secure longitudinal federated learning to train a machine learning model using a supervised learning algorithm and a training dataset, wherein the training dataset has samples distributed among multiple different computing systems. Background Art

[0004] A machine learning model is a computerized representation of a real-world process. To train a machine learning model, data (called training data) is collected and a machine learning algorithm is executed to learn the parameters of the machine learning model (such as weights and biases). For many real-world applications, the amount of available data is limited or the data quality is low, making training machine learning models difficult.

[0005] Consolidating data from multiple data owners (i.e., data owners) is one way to obtain sufficient data (i.e., training data) to train machine learning models. However, due to increasing concerns and restrictions on data sharing and privacy, such as Europe's General Data Protection Regulation (GDPR) and China's Cybersecurity Law, transferring, merging, and integrating data from different data owners can be difficult (or even impossible). Many data owners are companies; however, these companies consider their data to be a non-shareable corporate asset. Companies are often reluctant to share their data with competitors or third parties due to concerns that the data could be copied or used to benefit competitors.

[0006] As a result, large amounts of dispersed, siloed data across different data owners create difficulties. Restrictions on data sharing and privacy make it difficult to transfer, merge, or fuse data samples acquired from different data owners at a common site. One such difficulty involves generating a dataset of training data (called a training dataset) large enough to effectively train the associated machine learning model.

[0007] Therefore, there is a need to legally and effectively use data obtained from dispersed and isolated data owners to train machine learning models while protecting the privacy, confidentiality and security of the data.

[0008] This background information is provided to reveal information believed by the applicant to be potentially relevant to the present invention. It is not necessarily intended, nor should it be construed, that any of the foregoing information constitutes prior art against the present invention. Summary of the Invention

[0009] Embodiments of the present invention provide methods, computing devices, and systems for secure vertical federated learning (SVFL) and subsequent operation of SVFL systems. Security is provided by allowing data owners to securely maintain the confidentiality of their data. Vertical federated learning is a type of machine learning that involves training a machine learning model using a supervised learning algorithm and data samples from multiple data owners. Different data owners may possess the same or overlapping sets of data samples, but record different feature sets for at least one data sample within these sets. The various learning operations described below according to embodiments of the present invention can be performed iteratively and in an appropriate order by centralized and private machine learning models or corresponding computing devices. Learning operations are performed by computing systems and (e.g., cloud-based) servers belonging to different data owners. Each data owner may be associated with one or more virtual or physical machines that execute the learning algorithm to learn model parameters during model training. The servers may also include one or more such machines. The present invention also provides for subsequent operation of the trained (via machine learning) computing systems and servers. These operations are also performed in a vertically federated manner that maintains the privacy of data and model parameters.

[0010] Embodiments of the present invention provide a method for secure longitudinal federated learning of a machine learning model, which can be a central machine learning model or a private machine learning model. The method includes: a server running a central machine learning model receives a set of outputs. The set of outputs is received from each of a plurality of computing systems, each computing system running a different corresponding one of a plurality of private machine learning models. Different computing systems in the plurality of computing systems transmit different corresponding outputs from the set of outputs generated based on different corresponding portions of an identified data sample. The set of outputs is aligned based on a sample ID that identifies the data sample. The method also includes: the server providing a prediction calculated by the central machine learning model based on the set of outputs. The method also includes: receiving a loss at the server, the loss calculated based on a comparison of the prediction with a label corresponding to the data sample. The method also includes: initiating an update of current model parameters of the machine learning model based on the loss. This may include: the server updating the central machine learning model or transmitting a message to another computing device, the message indicating the loss or the gradient of the loss, and the message causing the update to occur at the other computing device.

[0011] In some embodiments, the method includes: the server calculating the gradient of the loss with respect to model parameters of the central machine learning model. Then, directly or indirectly updating the current model parameters of the machine learning model being trained based on the gradient of the loss. Updating based indirectly on the gradient may include calculating further gradients using the chain rule of derivatives and performing the update based on the further gradients.

[0012] In some embodiments, providing the prediction includes transmitting the prediction to a designated computing system (e.g., corresponding to a task owner) among the plurality of computing systems. Specifically, the designated computing system stores the label. The loss is calculated by the designated computing system, and receiving the loss includes receiving the loss from the designated computing system.

[0013] In some embodiments, providing the prediction includes providing the prediction to a training function of the server. The training function stores the label, and receiving the loss includes receiving the loss from the training function. In such embodiments, the method further includes, prior to receiving the prediction, the server receiving the label from a designated one of the computing systems and storing the label in a memory accessible to the server; and calculating the loss using the training function.

[0014] In some embodiments, the machine learning model being trained is the central machine learning model, and the method further includes: the server updating the current model parameters of the central machine learning model based on the loss.

[0015] In other embodiments, the machine learning model being trained is one of the multiple private machine learning models, and initiating an update of the current model parameters of the machine learning model being trained includes: the server transmitting an indication of the loss or an indication of the gradient of the loss to a computing system running one of the multiple private machine learning models.

[0016] In some embodiments, the gradient of the loss is a gradient with respect to model parameters of the central machine learning model. In such embodiments, updating the current model parameters of the one of the private machine learning models is directly based on the gradient of the loss with respect to the current model parameters, the gradient of the loss with respect to the current model parameters being calculated from the gradient with respect to the model parameters of the central machine learning model using a calculation implementing the chain rule of derivatives.

[0017] In various embodiments, the method further comprises: repeating the training method until a convergence condition is reached.

[0018] An embodiment of the present invention provides a server comprising: a computer processor operably coupled to a memory; and a communication interface. The server runs a central machine learning model. The server is configured to receive a set of outputs of a private machine learning model from each of a plurality of computing systems (each computing system running a different corresponding one of a plurality of private machine learning models), wherein different computing systems among the plurality of computing systems transmit different corresponding outputs of the set of outputs generated based on different corresponding parts of an identified data sample, and the set of outputs is aligned based on a sample ID identifying the data sample. The server is also configured to provide a prediction calculated by the central machine learning model based on the set of outputs. The server is also configured to receive a loss calculated based on a comparison of the prediction with a label corresponding to the data sample. The server is also configured to initiate an update of the current model parameters of the machine learning model being trained based on the loss.

[0019] According to various embodiments, the server may be configured to operate according to various embodiments of the above-described method.

[0020] Embodiments of the present invention provide a method for training a machine learning model. The method includes: a first computing system running a designated private machine learning model (e.g., of a task owner) transmits to a server running a central machine learning model: a sample ID; and an output of the designated private machine learning model. The output is generated based on a portion of a data sample identified by the sample ID. Each of a plurality of computing systems (including the first computing system) runs a different corresponding one of a plurality of private machine learning models (including the designated private machine learning model). Each of the plurality of computing systems transmits a different corresponding one of a set of private machine learning model outputs (including the output). Each of the set of outputs is generated based on a different corresponding portion of the data sample, and the set of outputs is aligned based on the sample ID. The method also includes: the computing system receiving, from the server, a prediction calculated by the central machine learning model based on the combined set of outputs. The method also includes: the computing system providing a loss to the server. The loss is calculated based on a comparison of the prediction with a label corresponding to the data sample. The label may be stored in a memory privately accessible to the first computing system and inaccessible to the server or other computing systems in the plurality of computing systems. Subsequently, an update of current model parameters of the trained machine learning model is initiated based on the loss.

[0021] According to various embodiments of the above method, the machine learning model being trained is the central machine learning model or one of the multiple private machine learning models.

[0022] According to various embodiments of the above-mentioned method, the updating of the current model parameters of the machine learning model under training is performed based on the calculated gradient of the loss relative to the model parameters of the machine learning model under training.

[0023] According to various embodiments of the above method, each of the multiple computing systems includes a corresponding private data storage space, and each corresponding private data storage space: is accessible by a corresponding one of the multiple computing systems and is not accessible by the server and other computing systems in the multiple computing systems; and stores a corresponding one of the different corresponding parts of the data sample.

[0024] Embodiments of the present invention provide a combination of methods, including methods performed by a server running a central machine learning model and one or more computing systems running a private machine learning model. Embodiments of the present invention provide an apparatus such as a computing system or server, or a system of such apparatuses, for operating according to one or more of the above methods. Each private machine learning model is run using a corresponding (private) computing system. The central machine learning model is run using a corresponding (e.g., cloud-based) server. Embodiments of the present invention also provide a computer program product comprising a (e.g., non-transitory) computer-readable medium having statements and instructions recorded thereon. When the statements and instructions are executed by one or more computers, the corresponding computer or computers implement one or more of the above methods.

[0025] The potential technical effects of embodiments of the present invention are as follows. Machine learning can be performed based on data from multiple entities (data owners) without requiring the data owners to share their data in encrypted or unencrypted form. Secure, accurate, and efficient general-purpose machine learning can potentially be supported. Data owners do not need to send their private data or their machine learning model parameters. While machine learning model outputs can be shared, this is not considered to constitute sharing of private data. No approximation is required, no encryption is required, and the method is compatible with a variety of machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Further features and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 An example embodiment of the present invention provides vertical partitioned data of two data owners that require vertical federated learning;

[0028] Figure 2 A Secure Vertical Federated Learning (SVFL) system for training a machine learning model for a specific task using a supervised learning algorithm is shown in one embodiment of the present invention.

[0029] Figure 3A An embodiment of the present invention provides Figure 2 The method performed by the SVFL system to train the machine learning model for the specific task;

[0030] Figure 3B Some embodiments of the present invention provide Figure 3A Further details of the method shown;

[0031] Figure 4An embodiment of the present invention provides a trained SVFL system that can be used to generate predictions for new data;

[0032] Figure 5A An embodiment of the present invention provides Figure 4 The method shown is executed by the trained SVFL system to generate predictions;

[0033] Figure 5B Some embodiments of the present invention provide Figure 5A Further details of the method shown;

[0034] Figure 6 An alternative embodiment of the present invention provides a secure vertical federated learning (SVFL) system for training a machine learning model for a specific task using a supervised learning algorithm.

[0035] Figure 7 Another alternative embodiment of the present invention provides a secure vertical federated learning (SVFL) system for training a machine learning model for a specific task using a supervised learning algorithm.

[0036] Figure 8A An embodiment of the present invention provides Figure 7 The method performed by the SVFL system to train the machine learning model for the specific task;

[0037] Figure 8B Some embodiments of the present invention provide Figure 8A Further details of the method shown;

[0038] Figure 9 The figure shows the relationship between the data owner model and the cloud model provided by one embodiment of the present invention;

[0039] Figure 10 A computing device that can be used to implement the SVFL and methods described herein is shown.

[0040] It should be noted that throughout the drawings, like features are identified by like reference numerals. DETAILED DESCRIPTION

[0041] Embodiments of the present invention provide a Secure Vertical Federated Learning (SVFL) method and system for securely training private machine learning models for supervised machine learning tasks. Training is performed using private data owned and stored by multiple data owners. The data owners run a computing system that interacts with a central machine learning model, which runs on a central server. Training also uses the central machine learning model for the supervised machine learning task without infringing on the data privacy and model privacy of these data owners. Model privacy refers to the privacy of the structure and parameters used by private data owners to represent data for use. The use of data may include, for example, using the data to generate a private machine learning model, using the model to generate inferences or predictions based on observations (i.e., new data samples) input into the private machine learning model, or a combination thereof.

[0042] As used herein, the term "data sample" refers to stored data that relates to a specific entity, such as a person, object, or other identifiable entity that has observable characteristics that can be recorded as data. Each data sample can be identified using a corresponding sample ID. A data sample set refers to a plurality of different data samples. A data sample may have a plurality of data features (or simply "features") that describe different attributes of the corresponding entity. For example, different data features may be associated with different financial, social, or behavioral data of a person. If a data sample is represented as a vector, the data features may be represented as components of the vector. In various embodiments, for training purposes, sampling from a data sample set is not required. Instead, it can be assumed that the data sample set is sampled from the underlying distribution of all data.

[0043] Typically, according to an embodiment of the present invention, a central (shared) machine learning model on a central server is first trained by interacting with the computing system of the data owner. After the central machine learning model has been sufficiently trained (e.g., to convergence), for example, the private machine learning models running on the computing system of each data owner are trained in turn. The process of training the central machine learning model first and then the private machine learning models can then be iteratively repeated until sufficient training (e.g., convergence) is achieved. For example, in an outer loop, the central machine learning model is trained, and each private machine learning model is trained in turn, and the process is repeated until convergence is achieved. There may be one or more inner loops. In each inner loop, either the central machine learning model or the private machine learning model is trained.

[0044] In various embodiments, the interactions that occur during the training of the central machine learning model include: providing the output from the private machine learning model to the central machine learning model; providing the current prediction of the central machine learning model to the task owner's computing system based on the output; and providing the loss (feedback indicating the prediction performance) from the task owner's computing system back to the central machine learning model. The task owner can provide the loss based on its saved labels. The task owner is also referred to as the designated owner in this article, and the task owner's computing system (e.g., the computing system running the task owner's private machine learning model) is also referred to as the designated computing system.

[0045] It should be understood that here and elsewhere, running machine learning models and communicating between different machine learning models involves running a computing system that is private to the machine learning model (e.g., a task owner's computing system or a data owner's computing system), running a central server, or a combination thereof.

[0046] In various embodiments, the interactions that occur during the training of a private machine learning model may include: providing output from the private machine learning model to the central machine learning model; providing a current prediction from the central machine learning model based on the output to the task owner's computing system; providing a loss from the task owner's computing system back to the central machine learning model; providing the gradient of the loss to the trained private machine learning model; and providing an updated output from the trained private machine learning model to the central machine learning model. The updated output is generated after adjusting model parameters of the trained private machine learning model.

[0047] After training, the machine learning model can be used to make predictions, for example. The computing system of the task owner (e.g., the private data owner) sends the output of its private machine learning model to a central server (i.e., the central machine learning model). The output is based on its corresponding feature portion of the sample that needs to be predicted. In addition, the sample ID is also provided. Next, a collaboration is performed, in which each computing system of each data owner provides its machine learning model output for its own portion of the same sample. (Recall that the computing systems of different data owners have different features for the same sample). The output of the machine learning model is provided to the central machine learning model. The central machine learning model then generates a prediction, which is the output of the central machine learning model generated based on the output of the private machine learning model. The prediction of the central machine learning model is provided to the computing system of the task owner, and the computing system of the task owner makes a final prediction based on the prediction.

[0048] In some embodiments, each private machine learning model can be considered a feature extractor. The central machine learning model then concatenates the features (e.g., feature maps) provided by these private machine learning models and feeds the concatenated feature maps into its model. The term "feature" in this paragraph may be distinguished from the term used elsewhere in this document.

[0049] The various terms used in this document should be interpreted generally in the context of machine learning. For example, machine learning models, parameters, predictions, losses, loss functions, labels, samples, features, etc. can all be interpreted as machine learning terms that are readily understood by those skilled in the art, for example, see Christopher M. Bishop's Pattern Recognition and Machine Learning (Springer, 2006).

[0050] For example, a label can refer to the value of some prediction made by a machine learning model. In some cases, some data may be labeled a priori, such as in supervised machine learning. In this case, this might be the case with training data. The predictions output by the machine learning model can be compared to these labels, and the error between the two can be quantified as a loss fed back to the machine learning model. The label can be interpreted as the true prediction result for a data sample. Labels can be stored locally, in the cloud, or both.

[0051] According to various embodiments, each data owner does not have to share their data with other parties (including other data owners). Similarly, a data owner does not have to share the parameters (i.e., weights and biases) of their machine learning model with other parties. Approximations do not have to be used in calculations. Since proprietary data is not shared, encryption is not necessarily required, and communication can use unencrypted or plain text data.

[0052] Some embodiments focus on the general scenario of vertically partitioned data, where different data owners privately own and store different feature sets for the same data sample set. An illustrative example is smart retail, where a retailer (e.g., ) can be obtained by using a bank (e.g. TD ) and financial data from social networking companies (e.g. ) significantly improves the performance of personalized product recommendations. Similarly, the bank may be able to significantly improve the accuracy of credit score assessments by using purchase data from the retailer and social relationship data from the social networking company. The social networking company may be able to more accurately push content of interest to users by analyzing financial data from the bank and purchase data from the retailer. Therefore, data sharing between data owners can potentially improve the performance of each data owner's system. However, data owners (such as the retailer, the bank, and the social networking company) are reluctant to disclose their private data or private machine learning models.

[0053] A vertical federated learning system can be viewed as a set of N data owners {F1, F2, ..., F N}, any data owner may want to train a machine learning model by integrating his own data with data from other data owners. N} indicates that D i Indicates the data owner F i If all data were shared freely among N data owners, they would be able to jointly train a machine learning model M SUM That is, M SUM Will use D1∪, D2∪…∪D N In contrast, the present invention provides a method for vertical federated learning, in which the computing systems of data owners collaborate to train the machine learning model M. FED , so that during training, any data owner F i Data D i Will not be exposed to other data owners F k , where k≠i. In addition, according to various embodiments of the present invention, M FED The accuracy (expressed as V FED ) can be close enough to the hypothetical fully collaborative machine learning model (i.e., the model learned when the models are jointly trained using all the data that is freely shared) M SUM The accuracy of V SUM More formally, let δ be a non-negative real number. When the machine learning model (“federated learning model”) M learned using federated learning FED With the hypothetical fully collaborative machine learning model M SUM The accuracy loss between is expressed as:

[0054] |V FED –V SUM |<δ.

[0055] It can then be said that the federated learning model has a δ accuracy loss relative to the fully collaborative model, where the fully collaborative model is used as a baseline.

[0056] Figure 1 The vertical partitioning of data for vertical federated learning provided by one embodiment of the present invention is shown. Figure 1 It shows multiple data owners {F1, F2, ..., F N} has data 100{D1,D2,…,D N}. In this case, for illustrative purposes, the number of data owners is N=2. Each dataset can be viewed as having multiple data samples 102, as shown vertically. Each data sample can have multiple data features 104. For example, each data sample can correspond to a person. An online retailer will have data related to personalized product recommendations for its users, and different data entries can be represented as data features of different people (data samples). Similarly, a bank will have financial data of its users, such as credit scores. A social media company will have social relationship data of its users. In the case where the same person (data sample) has data (features) accessible to multiple or all data owners, different data owners will have different data features for the same data sample.

[0057] like Figure 1 As shown, data owner 1 owns data 106 corresponding to its multiple users (people), and data owner 2 owns data 108 corresponding to its multiple users. The vertical span of box 110 represents the people (data samples) for whom both data owner 1 and data owner 2 have data. However, data owner 1 has different (possibly overlapping) data features of these common people compared to data owner 2. Box 110 represents the data features of these common people that both data owners have in common. When box 110 is roughly as shown in the figure, the data is vertically partitioned and vertical federated learning as described in this article can be used. According to the vertical partitioning, different datasets {D1, D2, ..., D N} contains different feature subsets of the same data sample set. {D1,D2,…,D N The feature subsets contained in} may or may not overlap with each other.

[0058] In order to identify and align data samples that may have data from multiple different data owners, a unique sample ID or key can be used to identify each data sample. For a person, the sample ID can be, but is not limited to, name, date of birth, age, address, government identification number, or a combination of these data, which allows data for a person to be combined from multiple data sources.

[0059] Figure 2FIG2 shows an SVFL system 200 for training a machine learning model for a specific task T provided by one embodiment of the present invention. During the training phase, the SVFL system 200 learns a private machine learning model for task T and further learns a central machine learning model for task T. The SVFL system 200 includes a computing system 116 of a task owner and computing systems (118-1, ..., 118-k) of multiple data owners, which interact with a central machine learning model 204. The central machine learning model 204 can be located in a central server 205 running in the cloud (e.g., a data center) or other locations. There are any number k of data owners, Figure 2 Only the computing systems of three data owners 1, 2 and k are shown and labeled. Figure 1 Combined, computing system 116 may be considered to correspond to the data owner (and task owner) that owns data 106, and computing system 118 may be considered to correspond to the data owner that owns data 108.) Although the task owner is designated as data owner j=1, it should be understood that this is an arbitrary index for purposes of illustration only.

[0060] The term "cloud" as used herein refers to computing resources (i.e., data storage and computing power) that are available on demand and not located at any data owner. The "cloud" can be a public cloud computing platform such as Amazon AWS, Microsoft Azure, a private cloud, or a hybrid cloud. The computing resources of the "cloud" can include physical servers, service networks, distributed servers, virtual servers, or other forms of network servers and service technologies known in the art. These servers can be located in a data center or instantiated using components located in a data center. These servers include computing resources for processing and storing software (i.e., the software of the central machine learning model) and data (e.g., parameters of the central machine learning model, outputs of the private machine learning model, losses of the private machine learning model, predictions generated by the central machine learning model, and network infrastructure for communicating with data owners).

[0061] The task owner corresponding to computing system 116 is the data owner who privately owns the private machine learning model for a specific task T. The task owner privately owns and stores (on computing system 116) the labels 208 associated with task T and privately defines a loss function for its private machine learning model. The task owner privately owns and stores (on computing system 116) the private machine learning model associated with task T and its dataset, which is used to train its private machine learning model for task T during the training phase. Before training the machine learning model, the task owner initializes the private machine learning model 212 by setting each parameter of its private machine learning model to an initial value. In various embodiments, any data owner can become a task owner by initiating a task using the SVFL system 200. Initiating a task may include initiating the process of launching the SVFL system 200 to train the machine learning model for the specific task T, including each private machine learning model and the central model for the specific task T. For example, initiating a task may include performing the necessary operations to start the process of training an image classifier. Initiating a task may include defining a task and transmitting at least a message to the central machine learning model, which triggers the machine learning operations corresponding to the task. Defining the tasks may include the initialization operations described herein and the actions of the SVFL system 200. The data owner's computing system may initiate any number of tasks, thereby initiating the training of corresponding machine learning models for any number of tasks. However, in various embodiments, each task is required to have only one task owner at a time.

[0062] As will be readily understood by those skilled in the art, a loss function is a function (equivalent to a cost function, error function, or objective function) that receives the output of a machine learning model (e.g., a prediction) and calculates the loss (the output of the loss function calculated based on the received output). In supervised learning, each training data includes an input (e.g., a training data sample) and an output (a label, which can be a true output or an expected output). The loss reflects the difference between the output of the machine learning (e.g., a prediction) and the label. The loss is used for backpropagation to update the model parameters of the machine learning model. Various loss functions can be used, such as loss functions that determine mean squared error, hinge loss, cross entropy loss, and the like. The appropriate loss function for training a machine learning model depends on the specific task for which the machine learning model is being trained.

[0063] For data owners 2 to k, each of these k-1 data owners stores their own private data set (224-1, ..., 224-k) on their corresponding computing systems (118-1, ..., 118-k). Each private data set owned and stored by the k-1 data owners is vertically partitioned (e.g., as described above in conjunction with Figure 1Generally described). The data owner's computing system can train private machine learning models for any number of tasks as needed, but it may be required that a task has only one task owner at a time. Each k-1 data owner privately owns and stores a private machine learning model (222-1, ..., 222-k), which is trained using private data sampled from the data owner's private dataset (224-1, ..., 224-k) during the training phase. All data owners interact with the central machine learning model 204 running on the central server 205 in the SVFL system 200 using their private data and private machine learning models. For example, the interaction itself can be performed by exchanging machine learning outputs, losses, and loss gradients. This is done to achieve the overall machine learning model for task T without violating the data privacy and model privacy of any data owner.

[0064] The central machine learning model 204 (i.e., model 0) for task T is located in the central server 205 and is trained during the training phase. The purpose of performing the training is that, after training, the central machine learning model can be used to make predictions during the subsequent inference phase. The trained central machine learning model can be stored in the cloud, for example, in the central server 205. For security and privacy reasons, private datasets are not stored in the cloud or anywhere other than the private storage resources of their respective data owners. The cloud includes the necessary computing resources (i.e., virtual machines and storage space) for storing the central machine learning model 204 for task T and performing training of the central machine learning model 204 for task T.

[0065] In various embodiments, there is no direct connection between the computing systems of different data owners. Instead, interactions occur only in pairs between the central machine learning model (i.e., the server running the central machine learning model) and the computing systems of the corresponding data owners. The interaction between the central machine learning model 204 and the computing system 116 of the task owner may include or consist of the following steps:

[0066] The computing system 116 sends the output 216 of its private machine learning model 212 (Model 1) for task T to the central machine learning model 204 (Model 0) for task T. The output 216 may be the prediction made by the private machine learning model.

[0067] The central machine learning model 204 (Model 0) of task T sends the gradient 218 of the loss relative to the output 216 of the task owner’s private machine learning model 212 (e.g., Model 1) to the computing system 116.

[0068] The central machine learning model 204 for task T sends the prediction 209 it generates for task T to the computing system 116 .

[0069] The computing system 116 sends the loss value 210 determined for its private machine learning model 212 (e.g., Model 1) to the central machine learning model 204 for task T.

[0070] The interaction between the central server 205 hosting the central machine learning model 204 of task T and the computing systems (118-1, ..., 118-k) of data owners who are not task owners is illustrated with reference to the computing system 118-1 of data owner 2 and includes or consists of the following steps:

[0071] Computing system 118-1 sends output 226-1 of its private machine learning model 222-1 (e.g., Model 2) for task T to central machine learning model 204 (e.g., Model 0) for task T. Output 216-1 may be the prediction made by the private machine learning model.

[0072] The central machine learning model 204 (e.g., Model 0) of Task T sends the gradient 228 of the loss relative to the output 226-1 of the private machine learning model 222-1 (e.g., Model 2) of Data Owner 2 to the computing system 118-1.

[0073] Specific symbols may be used to explain the processes used by embodiments of the SVFL system 200. Some of these symbols are as follows:

[0074] ·X j (where j∈{1,…,k}) represents the set of training data for data owner j's computing system (referred to as a training dataset). A training dataset may include some or all of the data collected and stored in storage by data owner j's computing system, including some or all of the features of the data samples. Training data refers to data specifically used to train a machine learning model. As mentioned above, a feature can refer to a portion of a data sample. In the case of vertically partitioned data, different data owners' computing systems may store different features for the same data sample.

[0075] If the full data training of the SVFL method is adopted, then X j Represents the set of all training data stored by the computing system of data owner j. Full data training refers to the case where all available data is used to train the machine learning model.

[0076] If a random training method is adopted during the training of the SVFL system 200, then X jIt can represent the set of randomly sampled training data samples owned and stored by the computing system of data owner j. Random training refers to the case where a machine learning model is trained using a random subset of all available data.

[0077] (X1, ..., X k All data samples in the dataset are aligned using unique sample IDs. Sample IDs are assumed to be consistent across the data owner's computing systems. That is, if two computing systems have two corresponding data entries for the same sample (e.g., a person), both data entries are indexed using the same sample ID.

[0078] ·θ j (where j∈{1,…,k}) represents the set of parameters of the private machine learning model (e.g., 212, 222-1,…, 222-k) j of the computing system of data owner j. The set of parameters of each private machine learning model j includes internal variables that can be learned during the training phase of the private machine learning model j.

[0079] ·g j (X j │θ j )(where j∈{1,…,k}) represents the private machine learning model of the computing system of data owner j (e.g., 212, 222-1,…, 222-k).

[0080] o j =g j (X j │θ j )(where j∈{1,…,k}) represents the value relative to X j The outputs of the private machine learning model j (e.g., 216, 226-1, ..., 226-k). The machine learning model outputs can be configured according to the task, but are generally related to the desired predictions or information generated by the model based on the inputs provided therein. Examples of model outputs can be: classification outputs indicating which category the input belongs to, and regression outputs indicating the predicted outcome variable based on one or more inputs. Output o j can be considered as input X j Abstract features of , as the output of a private machine learning model.

[0081] θ0 represents the set of parameters of the central machine learning model 204 (eg, Model 0). The set of parameters of the central machine learning model 204 is learned during the training phase of the SVFL method of the present invention.

[0082] ·g0(o1,…,o k │θ0) represents the central machine learning model 204 (Model 0) stored in the cloud.

[0083] o0=g(o1,…,o k │θ0) represents the output 209 of the central machine learning model 204 (Model 0). Here, o0 also represents the prediction generated by the central machine learning model 204 for task T.

[0084] Figure 3A and Figure 3B An embodiment is shown provided by Figure 2 The method 300 performed by the SVFL system 200 shown to train private machine learning models (212, 222-1, ..., 222-k) and the central machine learning model 204. In step 302, the SVFL system 200 is initialized. For example, anonymous sample IDs can be generated by the computing system of each data owner j in a consistent manner among data owners. A hashing method can be used to generate a unique hash key for each sample ID to be used as the anonymous sample ID. If two hash keys are the same, the sample IDs they correspond to should be the same. In this way, the computing system of each data owner can calculate the hash key privately by itself and only send the hash key to the cloud. The central machine learning model 204 (e.g., model 0) collects the anonymous sample IDs from all data owners j and aligns (X1, ..., X2). k ) in the data sample.

[0085] In various embodiments, the sample ID is provided via request / response messaging. As previously described, each sample has a unique sample ID. For example, a sample ID can be generated using a hashing technique. For example, a person's service card ID number can be that person's unique ID. In some embodiments, a sample ID can be anonymized using a hashing technique. For example, an anonymous service card ID can be used as the sample ID.

[0086] Next, the computing system 116 of data owner 1 becomes the task owner by initiating task T. The computing system 116 of data owner 1 initializes (e.g., randomly) the parameters θ1 of its private machine learning model 212 (e.g., model 1) and sends a task initiation request to the central machine learning model 204 (e.g., model 0). Based on the task initiation request received from the task owner, the central server 205 initializes (e.g., randomly) the parameter set θ0 of the central machine learning model 204 (e.g., model 0) and broadcasts the task initiation request to the computing systems (e.g., 118-1, ..., 118-k) of data owners 2 to k. Based on the task initiation request received from the central machine learning model 204 (e.g., model 0), each computing system (118-1, ..., 118-k) of each data owner 2 to k initializes (e.g., randomly) the parameters of its private machine learning models 222-1 (e.g., model 2), ..., 222-k (e.g., model k). The central server 205 may be provided by a cloud computing platform that runs computer devices (e.g., computing resources) and may be co-located or integrated with the computer devices (e.g., computing resources) that run the central machine learning model.

[0087] For example, a task initiation request can be made by sending a TCP / IP connection request from the task owner to the central server 205, and then sending a predefined message from the task owner to the central server 205 requesting identification as the task owner. Simultaneously, the task owner can also send a set of anonymous sample IDs to the server, so that the server can further use this set of anonymous sample IDs to coordinate with other data owners. The task initiation request broadcast by the server to the computing systems of other data owners is similar, but in this case, it is the server that sends the TCP / IP connection request to the computing systems of the other data owners. The server will also send the set of anonymous sample IDs received from the task owner to the computing systems of the other data owners. In this way, the computing systems of the data owners will understand which set of data is being requested for the task.

[0088] In step 304, combined Figure 3B , trains a central machine learning model 204 (e.g., model 0). Each computing system (116, 118-1, ..., 118-k) of data owners 1 to k retrieves (332) a batch of data samples from a storage space accessible only to that computing system, and provides the retrieved batch of data samples as input to their own private machine learning models (e.g., 212, 222-1, ..., 222-k). Each corresponding private machine learning model (e.g., 212, 222-1, ..., 222-k) computes (334) outputs (e.g., outputs o1, ..., o2) for such input data samples. kThese outputs (o1, ..., o k ) is sent (336) to the central machine learning model 204 (Model 0) and used as input to the central machine learning model. Then, based on the outputs (o1, ..., o2) received from the computing systems (116, 118-1, ..., 118-k) of the data owners 1 to k, k ), the central machine learning model 204 (Model 0) calculates (338) its prediction o0=g0(o1,…,o k │θ0), and sends (340) the prediction o0 to the computing system 116 of the task owner (i.e., data owner 1). The prediction o0 is the prediction of the task owner given the current model parameters θ0 and in response to the received input (i.e., the output o1, ..., o2 of the private machine learning model). k ) is the output of the central machine learning model 204.

[0089] It should be noted that the central machine learning model 204 is the one that receives the outputs of other machine learning models (o1, ..., o k ) as its input. This is different from receiving data samples (X1, ..., X k ) as input. That is, the central machine learning model is trained based on the output of the private machine learning model, which maintains the data privacy of the data owner because the data samples are not shared with other data owners. In addition, the private machine learning model is trained in coordination with the (at least partially) trained central machine learning model. The interactive iteration of training operations between the machine learning models further facilitates training.

[0090] Based on the prediction o0 received from the central machine learning model 204 (Model 0), the task owner's computing system 116 calculates the loss associated with its task T using the labels associated with the retrieved data samples and sends the loss back to the central server 205 running the central machine learning model 204 (e.g., Model 0). Based on the loss calculated by the computing system 116, the central server 205 calculates the gradient of the loss with respect to its current parameters, expressed as Also referred to as the "gradient" of the central machine learning model 204 (e.g., Model 0). The central server 205 then uses the gradient to update the parameters θ0 of the central machine learning model 204 (e.g., Model 0). Step 304 is iterated (e.g., using different samples) until sufficient convergence of the central machine learning model 204 (e.g., Model 0) is achieved.

[0091] Updating the parameters θ0 of the central machine learning model 204 can be performed using various methods, such as backpropagation, which adjusts the parameters θ0 of the central machine learning model based on supervised learning feedback, as will be readily understood by those skilled in the art. For example, the weights of the neural network nodes can be adjusted based on the feedback so that the model generates predictions that improve the results, as measured by the feedback (loss).

[0092] Full convergence of the central machine learning model 204 may be considered achieved under a variety of conditions. In some embodiments, full convergence may occur when the number of iterations exceeds a maximum number of iterations specified by the user or the SVFL system 200. In some embodiments, full convergence may occur when the change in loss between the last iteration and the current iteration is less than a specified threshold. In some embodiments, full convergence may occur when the gradient Sufficient convergence can occur when the 2-norm (Euclidean norm) of is less than a specified threshold. Various other convergence criteria can be used to indicate that machine learning has approached a steady state or sufficiently optimal conditions.

[0093] After the central machine learning model is sufficiently converged, the method 300 proceeds to steps 306 to 312, which include training each of the private machine learning models (212, 222-1, ..., 222-k) running on each of the computing systems of the private data owners 1 to k (e.g., computing systems 116, 118-1, ..., 118-k). First, the computing systems (116, 118-1, ..., 118-k) of the private data owners 1 to k retrieve (e.g., a batch of) data samples from the storage space and input the data samples into their corresponding private machine learning models. Then, the private machine learning models (212, 222-1, ..., 222-k) are respectively used to calculate outputs (o1, ..., o2) based on the retrieved data samples. k Then, these outputs (o1, ..., o k ) are sent to the central server 205 where they are used as input to the central machine learning model. Based on the outputs (o1, ..., o2) received from the computing systems (116, 118, 202) of the private data owners 1 to k k ), the central server 205 stores the output as a set S = {o1, ..., o k Based on the set S, the central machine learning model 204 (Model 0) is used to calculate the prediction o0.

[0094] The prediction o0 provided by the central machine learning model is then sent (by the central server 205) to the task owner's computing system 116. Based on the prediction o0 received from the central server 205 running the central machine learning model 204 (e.g., Model 0), the task owner's computing system 116 uses its labels 208 to calculate the loss associated with its task T. The task owner's computing system 116 then sends the loss back to the central server 205. Based on the loss received from the task owner's computing system 116, the central server 205 implementing (e.g., running) the central machine learning model 204 (e.g., Model 0) calculates the gradient And the central server 205 calculates the gradient Sent to the computing system (118-1, ..., 118-k) of the private data owner j. Based on the gradient received from the central server 205 For example, computing system 118-j of private data owner j (where j is an integer from 1 to k-1) updates its private machine learning model parameters θ by performing backpropagation j Then, the computing system 118-j of the private data owner j computes the new output o j ', and the new output o j 'Sent to the central server 205. Based on the new output o received from the computing system 118-j of the private data owner j j ', the central machine learning model 204 (e.g., Model 0) converts o j ∈S is replaced by o j 'To update the set S. Parameter θ j The update of corresponds to the training of the private machine learning model 222-j of the private data owner j in the current iteration.

[0095] In some embodiments, the chain rule of derivatives can be used to calculate the gradient of the loss for some i≠j: Calculate the loss gradient of a certain j (Note that here either i or j may be equal to zero.) This may involve computationally applying the formula exactly or approximately For example, as described above, based on the loss received from the task owner, the central machine learning model can calculate the gradient This calculation may involve calculating Then apply the chain rule formula to calculate

[0096] After the private machine learning model 222-j of the private data owner j is trained, the private machine learning model 222-j+1 of the next private data owner (j+1)_ will be trained. The above process can be repeated until the private machine learning models k of the task owner and all k-1 private data owners are trained to convergence 314. The convergence of the private machine learning models (212, 222-1, ..., 222-k) can be considered to be achieved under various conditions. When the number of iterations of the learning process exceeds the maximum number of iterations specified by the user or the SVFL system, it can be considered that convergence has occurred. When the change in loss between the last iteration and the current iteration is less than a specified threshold, it can be considered that convergence has occurred. When for all j∈{0, ..., k}, the gradient When the 2-norms of are all less than the specified threshold, it can be considered that convergence occurs.

[0097] like Figure 3A As shown, the training sequence is as follows. After training (304) the central machine learning model to convergence, each private machine learning model is trained in turn by initializing (306) j to 1 and training (308) the private machine learning model g of data owner j. j , increase (310) j; if j is still not greater than k (the number of data owners), as shown in comparison 312, then train the next private machine learning model g for data owner j j . This process is performed until all k private machine learning models are trained. Then, the convergence condition is checked (314). If the convergence condition is met, the training ends, otherwise the process is repeated starting from step 304. The training of each private machine learning model in step 308 can include performing a single training iteration, as described above.

[0098] although Figure 3A The description above specifies a particular sequence of training operations and convergence checks, but it will be appreciated that the training iterations and convergence checks may vary in a variety of ways. For example, each instance of step 304 may involve a single iteration (or a finite number of iterations) of training the central machine learning model, without necessarily requiring a convergence check as part of step 304. For another example, convergence checks may be performed intermittently or at any point in the training cycle. For another example, private machine models may be trained in any order, rather than training the data owner's private machine learning models in a consistent order. For another example, in a given iteration of training a private machine learning model, any subset of the private machine learning models may be trained without necessarily training other private machine learning models in that iteration.

[0099] Figure 4An SVFL system 400 provided by one embodiment of the present invention is shown, operating in a training (e.g., prediction) mode. For example, the SVFL system 400 can be an instance of the SVFL system 200, such as after the system is trained according to the method 300. The SVFL system 400 includes: a task owner's computing system 116, which includes the task owner's trained private machine learning model 212-1 (i.e., the private machine learning model trained during the execution of the method 300); k-1 of the k data owners' computing systems (118-1, ..., 118-k), each computing system (118-1, ..., 118-k) including a trained private machine learning model 222-1, ..., 222-k (i.e., the private machine learning model trained during the execution of the method 300); and a trained central machine learning model 204 (i.e., the central machine learning model trained during the execution of the method 300).

[0100] The task owner's computing system 116 is the computing system that initializes and privately owns the data owner of task T. In the SVFL system 400, the task owner initiates a prediction request for a new data sample and outputs a prediction generated by its private machine learning model 212 based on the new data sample. The task owner owns and stores its private machine learning model 212 and dataset 214.

[0101] Each of the computing systems (118-1, ..., 118-k) of the other k-1 data owners stores a private data set (224-1, ..., 224-k). The private data sets (224-1, ..., 224-k) owned and stored by the k-1 data owners are vertically partitioned (see Figure 1 Each data owner privately owns and stores its trained private machine learning model (222-1, ..., 222-k). Each private machine learning model has full access to the private data of its data owner. All data owners use their own private data and their own trained private machine learning models to collaborate with the trained central machine learning model to achieve predictions for task T.

[0102] The trained central machine learning model 204 generates the final prediction for the task T. For example, the central machine learning model 204 can be stored and run in the central server 205. Unlike the computing systems (116, 118-1, ..., 118-k) of the task owner and the k-1 data owners, the central server 205 does not necessarily store any private datasets of the data owners. In fact, to ensure privacy and security, it is generally considered best not to store private datasets in the central server 205 or in association with the central machine learning model 204.

[0103] In various embodiments, there is no direct connection or interaction between the computing systems of different data owners. Necessary interactions occur only between the computing systems of private data owners (including the computing system 116 of the task owner) and the central server 205. The interaction between the central server 205 and the computing system 116 of the task owner includes: the task owner initiates a prediction request for a sample by sending the corresponding sample ID to the central server; the task owner sends its output regarding the sample to the central server. In response, the central server sends the final prediction of the central machine learning model for task T to the computing system 116 of the task owner.

[0104] The interaction between the central server and the computing systems (118-1, ..., 118-k) of k-1 data owners who are not task owners includes: the central server 205 sends the sample ID to each of the computing systems (118-1, ..., 118-k); each of the computing systems (118, 202) sends its output regarding the sample ID to the central server.

[0105] The symbols used in the trained SVFL system 400 are similar to the symbols used in the untrained SVFL system 200. However, for the sake of clarity, these symbols are defined as follows.

[0106] ·x i,j represents the data sample x stored by the computing system of data owner j i The eigenvector of .

[0107] i∈{1,…,n} represents the data sample x i The unique sample ID of the

[0108] n represents the number of data samples.

[0109] j∈{1,…,k} represents the index of the jth data owner and its computing system.

[0110] k represents the number of data owners and corresponding computing systems.

[0111] ·θ j represents the set of learned model parameters of the trained private machine learning model of data owner j.

[0112] ·g j (x i,j │θ j ) represents the trained private machine learning model of data owner j.

[0113] o j =g j(x i,j │θ j ) represents the trained private machine learning model g j Relative to x i,j Output.

[0114] θ0 represents the set of learned model parameters of the trained central machine learning model 204.

[0115] ·g0(o1,…,o k │θ0) represents the trained central machine learning model 204.

[0116] o0=g(o1,…,o k │θ0) represents the trained central machine learning model g0 204 relative to (o1, ..., o k Here, o0 also represents the prediction generated by the trained central machine learning model 204 of the SVFL system 400.

[0117] Using this notation, the prediction method 500 employed by the SVFL system 400 will now be described.

[0118] Figure 5A Shown by Figure 4 The embodiment of the present invention performs the following steps of the prediction method. The method includes the following steps: Step 502: Initialize the prediction task. As part of the initialization, the task owner's computing system 116 performs the prediction on the data sample x. i The prediction method is initiated. The trained private machine learning model g1 of the task owner (for simplicity, assume that the task owner j=1) receives the data sample x i , and output its response to the data sample x i The prediction o1=g1(x i,1 The task owner's computing system 116 sends the prediction o1 and the sample ID i to the central server 205 .

[0119] Step 504: The embodiment of this step is Figure 5B As shown in more detail in FIG, the collaboration involving computing systems (118-1, ..., 118-k) of other data owners is involved. Based on the sample ID i from the computing system 116 of the task owner, the central server 205 transmits (522) (e.g., broadcasts) the sample ID i to the computing systems (118-1, ..., 118-k) of data owners 2 to k (i.e., data owners other than the task owner). The computing system 118-j of each data owner j (where j∈{2, ..., k}) computes (524) its output o for its trained private machine learning model. j =g j (xi,j │θ j ), and output it j Send (526) to the central server 205. Based on the outputs (o1, ..., o2) of the computing systems (116, 118-1, ..., 118-k) of all data owners 1 to k k ), the central server uses the trained central machine learning model g0 to calculate the prediction o0=g0(o1,…,o k |θ0). The central server sends the calculated prediction o0 to the task owner 116. Based on the prediction o0 received from the central server, the task owner outputs the prediction received from the cloud as the data sample x of task T. i The final prediction is 506.

[0120] Figure 6 FIG2 shows an SLVF system for training a machine learning model for a specific task using a supervised learning algorithm provided by another alternative embodiment of the present invention. Figure 2 Similarly, the illustrated SLVF system 600 is shown during a training phase where both the private machine learning model and the concentrator learning model are trained. Figure 2 In the embodiment of , the task owner 116 owns and stores the tags associated with the task T, and the central machine learning model 204 communicates with the task owner's computing system 116 to calculate the loss. Figure 6 In the illustrated embodiment of the SVFL system 600, the task owner's computing system 116 shares certain information with the central server 205 via communication 604. This information may include labels 602 and the definition of a loss function associated with the task T. Upon receiving the information 604, the central server 205 may calculate the loss independently of the task owner's computing system 116. The above embodiment reduces the need for the central server to communicate with the task owner's computing system 116. Before performing a learning operation, the information 604 may be transmitted to the central server 205, for example, using one or more secure messages. Because the task owner's computing system 116 shares the labels and loss function definition with the central server 205, the central server 205 may calculate the loss independently. This reduces the communication overhead between the central server 205 and the task owner's computing system 116.

[0121] The central server 205 may include a training function operatively coupled to the central machine learning model 204. The training function may be provided using a computer processor operatively coupled to a memory and executing appropriate stored computer program instructions. The training function is configured to receive predictions from the central machine learning model 204, compare the predictions to associated labels, determine a loss associated with the predictions, and provide the loss to the central machine learning model 204 as training feedback.

[0122] exist Figure 6 In an embodiment of the invention, the SVFL system 600 stores the labels 602 and the definition of the loss function (both of which are associated with the learning task T) in a local location of the central machine learning model 204. This location can be in the memory of the central server or (equivalently) in the cloud. The central server or another associated computer instantiated in the cloud can use this stored information to directly calculate the loss associated with the task T. Figure 2 In the example, the prediction 209 is transmitted from the central machine learning model 204 to the computing system 116 of the task owner (e.g., data owner 1), and the computing system 116 of the task owner transmits the loss 210 back to the central machine learning model 204. Figure 2 On the contrary, Figure 6 In the example, the prediction 606 can be transmitted from the central machine learning model 204 to a local or integrated computing component (e.g., of the central server 205), which determines the loss based on the locally stored labels 602 and transmits the loss 608 back to the central machine learning model 204 without further involvement of the computing system 116 of the task owner (e.g., data owner 1). Accordingly, the current training iteration (possibly over multiple iterations) of the central machine learning model 204 can be performed with reduced further communication with the data owner (note that each iteration may also require outputs o1 to o2). k , described below). Since the labels and loss function definitions are stored locally, training can be performed with reduced communication overhead.

[0123] Figure 6 The embodiments of the SVFL system 600 allow for an alternative secure longitudinal federated learning approach to train private machine learning models and central machine learning models. The training approach is similar to the one in the example above except for the following differences: Figure 3A The described method proceeds similarly. First, during initialization 302, the task owner's computing system 116 additionally sends (604) the labels 208 and the loss function associated with the task T to the central server 205. Second, after the central machine learning model 204 generates its predictions o0=g0(o1, ..., o kAfter the task owner's computing system 116 receives the prediction 606 from the central machine learning model 204, the central server 205 or other associated computing device or cloud computing platform stores the labels 602. The central server 205 (or other computing device or cloud computing platform) calculates the loss based on the labels and provides the loss back to the central machine learning model 204. This can occur during the training of the central machine learning model 204 in step 304 or during the training of one or more private machine learning models (212, 222-1, ..., 222-k) in step 308.

[0124] Figure 7 Another embodiment of an SVFL system is shown that trains a private machine learning model along with a central machine learning model for a supervised machine learning task T during pre-training and training phases. Figure 7 In the illustrated embodiment, the private machine learning models (212, 222-1, ..., 222-k) of the data owners of the SVFL system 700 are pre-trained. Each data owner's computing system (116, 118-1, ..., 118-k) has its corresponding pre-trained machine learning model (712, 724-1, ..., 724-k) associated with its private machine learning model. The labels 208 stored in the computing system 116 of data owner 1 (i.e., task owner) are used to provide loss feedback shared with the pre-trained machine learning models (712, 724-1, ..., 724-k) of the data owners in the SVFL system 700. Sharing losses with computing systems of other data owners is not necessarily expected to constitute an instance of sharing private information between data owners. This is because the losses can be provided as raw data without context or meaningful information. Details of the pre-training operations described in accordance with various embodiments of the present invention are provided below in conjunction with Figure 8.

[0125] Figure 8A and Figure 8B Shown including pre-training Figure 7 The method 800 of the SVFL system 700 is shown. Figure 3A The training method 300 shown adds a pre-training step 802, and the step 802 is applied to the computing systems (116, 118-1, ..., 118-k) of some or all data owners. Figure 8B As shown), the computing system of each data owner j initializes 812 the parameter set ω of its corresponding pre-trained machine learning model j The pre-trained machine learning model of data owner j is denoted as f j (o j |ω j), and with its private machine learning model g j Here, o j =g j (X j |θ j ) is a private machine learning model g j The output of . Based on a set of (e.g. randomly sampled) training data samples X j , the computing system storing the private machine learning model j calculates (814) the output o of its private machine learning model for each sampled data sample j , and the output of its private machine learning model o j Sent to its associated pre-trained machine learning model f j Based on private machine learning models g j Output o j , pre-trained machine learning model f j Calculate (816) its predicted p j =f j (o j |ω j ), and predict it p j Send (818) to the task owner. Here, if the computing system of data owner j is the computing system 116 of the task owner, then there is no need to send its prediction p to the outside. j Instead, the prediction is kept internally and based on the p received from the computing system 118-j of the data owner j. j The task owner calculates the loss using its labels and loss function and sends the calculated loss back to the pre-trained machine learning model 724-j of the data owner j. Here, if the computing system of the data owner j is the computing system 116 of the task owner, the loss can be calculated directly without transmitting the prediction p to the outside. j and said losses.

[0126] Based on the loss received from the task owner, a pre-trained machine learning model f j (i.e. 724-j) calculate (820) gradient and Then, pre-train the machine learning model f j (ie 724-j) the gradient Send to private machine learning model g j Based on the pre-trained machine learning model f j Gradient Private machine learning models j Calculate (822) gradient Applying gradients to pre-trained machine learning models To update its parameter set ωj . Basically at the same time, private machine learning models g j Applying a gradient To update its parameter set θ j . Iterate the above steps until the pre-trained machine learning model f j (i.e. 724-j) to achieve convergence. After convergence, the pre-trained model f can be deleted j (ie, 724 - j ). The method may be repeated for each data owner in the system 700 . Figure 8A The rest of the steps are as follows: Figure 3A Training and pre-training can be thought of as processes run by a computer and can involve forward propagation and backpropagation.

[0127] By using pre-training, the model parameters θ1, θ2, ... θ can be obtained before further training as described in other parts of this article. k This improves convergence characteristics such as convergence speed and solution quality. Without pretraining, model parameters are initialized by randomly or arbitrarily assigning values to each parameter. This results in arbitrary and potentially highly inaccurate initial conditions. With pretraining, the initial values of the model parameters are not random but rather obtained by training the model using a limited training dataset. This often results in improved initial conditions.

[0128] Figure 9 The relationship between the private machine learning model and the central machine learning model 204 provided by one embodiment of the present invention is shown. The private machine learning model g of task T of data owner j is shown. j (X j │θ j )(where j∈{1,…,k}). X j (where j∈{1,…,k}) 902 is the set of training data samples for the private machine model. j (where j∈{1,…,k}) 904 is the set of parameters of the private machine learning model j of data owner j. Each training data sample 902 is input into its corresponding model 212 to produce an output o j =g j (X j │θ j )(where j∈{1,…,k})906, the output is the private machine learning model j relative to the training data sample X jThe central machine learning model 204 receives the outputs of the private machine learning models of all data owners (where j∈{1,…,k}) and initializes its parameters θ0 908. The parameters θ0 are the set of parameters of the central machine learning model g0. The output o0 of the central machine learning model g0 = g(o1,…,o k │θ0)910 is the central machine learning model g0 relative to (o1, ..., o k The gradient 912 with respect to the output of each data owner's computing system is sent from the central machine learning model g0 to the data owner's private machine learning model g j .

[0129] An embodiment of the present invention provides a method for training a machine learning model to perform a task. The method is performed by a central machine learning model. The method includes: the central machine learning model receives multiple sets of outputs. Each set of outputs is received from each of multiple corresponding private machine learning models. The set of outputs received from each private machine learning model is generated based on data exclusively owned by the owner corresponding to the specific private machine learning model. The multiple sets of outputs are aligned based on the sample IDs of the data. The method includes: the central machine learning model generates a prediction. The prediction is the output of the central machine learning model and is generated based on the received multiple sets of outputs. The method includes: the central machine learning model transmits the prediction to one of the private machine learning models. The multiple recipient private machine learning models (and typically each private machine learning model) have associated labels. The method includes: the central machine learning model receives a loss determined based on the label and the prediction from the recipient private machine learning model. The method includes: the central machine learning model calculates a gradient based on the loss; and updates the parameters of the central machine learning model based on the loss.

[0130] In a further embodiment, the central machine learning model stores the labels locally.

[0131] According to another aspect, an embodiment includes a method for training a machine learning model. The method is performed by a private machine learning model in collaboration with a central machine learning model. The method includes: the private machine learning model transmitting an output of the private machine learning model to a central machine learning model. The output is generated based on data exclusively owned by an owner corresponding to the private machine learning model. The output of the private machine learning model also includes a sample ID. The method includes: the private machine learning model receiving a prediction calculated by the central machine learning model from the central machine learning model. The prediction is an output of the central machine learning model generated based on the private machine learning model output. The prediction is aligned with a set of other private machine learning model outputs based on the provided sample ID and the sample IDs of the other private machine learning model outputs. Similarly, the other private machine learning model outputs are generated based on data exclusively owned by the other private machine learning models. The method includes: the private machine learning model calculating a loss based on the label and the prediction. The method includes: the private machine learning model transmitting the loss to the central machine learning model. The method includes: the private machine learning model receiving, from the central machine learning model, a gradient of the output of the private machine learning model based on the loss. The method includes: the private machine learning model calculating the gradient of the loss with respect to parameters of the private machine learning model by using the chain rule of derivatives, or performing a comparable calculation. The method includes: the private machine learning model updating the parameters of the private machine learning model based on the loss.

[0132] In a further embodiment, the private machine learning model includes local data storage space for data exclusively owned by the private machine learning model.

[0133] In a further embodiment, each of the other private machine learning models includes local data storage space for data exclusively owned by each of the other owners corresponding to the other private machine learning models.

[0134] According to another aspect, an embodiment includes a method for generating predictions using a central machine learning model. The method includes: the central machine learning model receives a set of private machine learning model outputs and a sample ID from a private machine learning model. The set of private machine learning model outputs is based on data exclusively owned by the owner of the private machine learning model. The method includes: the central machine learning model transmits the sample ID to multiple other private machine learning models. The method includes: the central machine learning model receives multiple other private machine learning model outputs generated by the other private machine learning models based on the sample ID from the other private machine learning models. The method includes: the central machine learning model calculates a prediction. The prediction is output by the central machine learning model based on the set of private machine learning model outputs and the set of other private machine learning model outputs. The method includes: the central machine learning model transmits the prediction to the private machine learning model.

[0135]

[0014] An embodiment according to another aspect includes a method for generating predictions using a private machine learning model. The method includes the private machine learning model computing a set of private machine learning model outputs associated with a sample ID. The method includes the private machine learning model transmitting the set of private machine learning model outputs and the sample ID to a central machine learning model. The method includes the private machine learning model receiving a prediction from the central machine learning model. The prediction is computed based on the set of private machine learning model outputs and a plurality of other private machine learning model outputs, the plurality of other private machine learning model outputs being generated by a corresponding plurality of other private machine learning models based on the sample ID. The method includes the private machine learning model computing a private (final) prediction based on the prediction.

[0136] An embodiment of the present invention provides a method for training a central machine learning model, the method comprising: a server running the central machine learning model receives a set of outputs of the private machine learning model from each of a plurality of computing systems (each computing system running a different corresponding one of a plurality of private machine learning models), the set of outputs of the private machine learning model being generated based on a portion of an identified data sample, wherein different computing systems among the plurality of computing systems transmit different corresponding outputs generated based on different corresponding portions of the data sample, and the set of outputs is aligned based on a sample ID of the data sample; the central machine learning model provides a prediction, wherein the prediction is generated by the central machine learning model based on the set of outputs received from each of the plurality of private machine learning models; the central machine learning model receives a loss, the loss being calculated based on a comparison of the prediction with a label corresponding to the data sample, wherein the central machine learning model updates parameters of the central machine learning model based on the loss.

[0137] According to some embodiments, the method further comprises: the central machine learning model calculating the gradient of the loss relative to the parameters of the central machine learning model, and then updating the parameters of the central machine learning model based on the gradient. According to some embodiments, providing the prediction comprises: transmitting the prediction to a designated computing system among the plurality of computing systems, the designated computing system storing the label, and the loss being calculated by the designated computing system; and receiving the loss comprises: receiving the loss from the designated computing system.

[0138] According to some embodiments, the method further comprises: providing the prediction comprises providing the prediction to a training function of the server, the training function storing the label, and the loss calculated by the training function; and receiving the loss comprises receiving the loss from the training function. The method further comprises: prior to receiving the prediction, the server receiving the label from a designated one of the computing systems and storing the label in a memory accessible to the server.

[0139] An embodiment of the present invention provides a method for training a private machine learning model, the method comprising: a server running a partially or fully trained central machine learning model receives from multiple computing systems running corresponding multiple private machine learning models (including the private machine learning model being trained): a sample ID identifying a data sample; the outputs of the multiple private machine learning models, wherein different private machine learning models among the multiple private machine learning models transmit different corresponding outputs among the outputs generated based on different corresponding parts of the data sample; the central machine learning model provides a prediction, the prediction is calculated by the central machine learning model based on the outputs of the multiple private machine learning models; the loss is received at the server, the loss is calculated based on a comparison of the prediction with a label corresponding to the data sample; the server transmits a gradient of the loss relative to the current parameters of the central machine learning model to one of the multiple computing systems running the private machine learning model being trained, the gradient being determined by the server based on the loss, wherein the current model parameters of the private machine learning model being trained are updated based on the gradient.

[0140] In some embodiments of the above method, providing the prediction includes: transmitting the prediction to a specified computing system among the multiple computing systems, the specified computing system storing the label, and the loss being calculated by the specified computing system; receiving the loss includes: receiving the loss from the specified computing system.

[0141] In other embodiments of the above method, providing the prediction includes: providing the prediction to a training function of the server, the training function saving the label, and the loss calculated by the training function; receiving the loss includes: receiving the loss from the training function; the method also includes: before receiving the prediction, the server receiving the label from a specified computing system in the computing system; storing the label in a memory accessible to the server.

[0142] An embodiment of the present invention provides a method for training a central machine learning model, the method comprising: a computing system running a private machine learning model transmits to a server running the central machine learning model: one or more sample IDs identifying one or more data samples; one or more outputs of the private machine learning model, the outputs being generated based on a portion of the identified one or more data samples accessible to the private machine learning model, wherein different computing systems among multiple computing systems running corresponding multiple private machine learning models (including the private machine learning model) transmit different corresponding outputs generated based on different corresponding portions of the one or more data samples to the server, the different corresponding outputs including the one or more outputs of the private machine learning model; receiving a prediction from the central machine learning model, the prediction being calculated based on the combined different corresponding outputs, the different corresponding outputs being aligned based on the one or more sample IDs; providing a loss to the server, the loss being calculated based on a comparison of the prediction with labels corresponding to the one or more data samples, wherein the current model parameters of the central machine learning model are updated based on the loss.

[0143] According to some embodiments of the above method, updating the current model parameters based on the loss includes: the server determining the gradient of the loss relative to the current parameters of the central machine learning model; and the server updating the current model parameters of the central machine learning model based on the gradient of the loss.

[0144] According to some embodiments of the above method, the computing system that runs the private machine learning model includes a private data storage space, which stores a portion of the one or more data samples used to generate the one or more outputs of the private machine learning model.

[0145] According to some embodiments of the above method, each of the multiple computing systems running the corresponding multiple private machine learning models includes a corresponding private data storage space, and each private data storage space stores a corresponding portion of the different corresponding portions of the one or more data samples.

[0146] According to some embodiments, the method further comprises: initiating a task and training the central machine learning model for the task. According to some embodiments, the method further comprises: repeating the training method until a convergence condition is reached.

[0147] According to some embodiments of the above method, the prediction is performed by the computing system running the private machine learning model, the providing of the loss includes transmitting the loss calculated by the computing system running the private machine learning model to the server, and the label is privately accessible by the computing system running the private machine learning model.

[0148] According to some embodiments of the above method, the receiving the prediction and the providing the loss are performed by the server, and the method further includes: before the receiving the prediction, providing the label to the server from the computing system running the private machine learning model, and storing the label in a memory accessible to the server.

[0149] An embodiment of the present invention provides a method for training a private machine learning model, the method comprising: a computing system running the private machine learning model transmits to a server running a partially or fully trained central machine learning model: one or more sample IDs identifying one or more data samples; one or more outputs of the private machine learning model, the outputs being generated based on a portion of the identified one or more data samples accessible to the private machine learning model, wherein different computing systems among multiple computing systems running corresponding multiple private machine learning models (including the private machine learning model) transmit different corresponding outputs generated based on different corresponding portions of the one or more data samples to the server, the different corresponding outputs including the one or more outputs of the private machine learning model; receiving a prediction from the central machine learning model, the prediction being calculated based on the combined different corresponding outputs, the different corresponding outputs being aligned based on the one or more sample IDs; providing a loss to the server, the loss being calculated based on a comparison of the prediction with labels corresponding to the one or more data samples; the private machine learning model receiving from the central machine learning model a gradient of the loss relative to the current parameters of the central machine learning model, the gradient being determined by the central machine learning model based on the loss, wherein the current model parameters of the private machine learning model are updated based on the gradient.

[0150] According to some embodiments, the above method also includes: calculating a revised output based on the updated parameters, and transmitting the revised output to the central machine learning model, wherein the central machine learning model uses the revised output in subsequent training operations.

[0151] According to some embodiments, the above method also includes: the private machine learning model calculates the gradient of the loss relative to the parameters of the private machine learning model by using the chain rule of derivatives; and updates the one or more parameters of the private machine learning model based on the gradient of the loss relative to the parameters of the private machine learning model.

[0152] According to some embodiments, the above method further comprises: repeating the training method until a convergence condition is reached.

[0153] According to some embodiments of the above method, receiving the prediction is performed by the computing system running the private machine learning model, providing the loss includes transmitting the loss calculated by the computing system running the private machine learning model to the server, and the label is privately accessible by the computing system running the private machine learning model.

[0154] According to some embodiments of the above method, the receiving the prediction and the providing the loss are performed by the server, and the method further includes: before the receiving the prediction, providing the label to the server from the computing system running the private machine learning model, and storing the label in a memory accessible to the server.

[0155] According to some embodiments of the above method, receiving the prediction is performed by one of the multiple computing systems running one of the multiple private machine learning models (other than the private machine learning model), providing the loss includes transmitting the loss calculated by the one of the multiple computing systems to the server, and the label is privately accessible by the one of the multiple computing systems.

[0156] An embodiment of the present invention provides a method for training a central machine learning model, the method comprising: transmitting a sample ID and a set of outputs of the private machine learning model from each of a plurality of computing systems (each computing system running a different corresponding one of a plurality of private machine learning models) to a server running the central machine learning model, the set of outputs of the private machine learning model being generated based on a portion of an identified data sample corresponding to the sample ID, wherein different computing systems among the plurality of computing systems transmit different corresponding outputs generated based on different corresponding portions of the data sample, and the set of outputs being aligned based on the sample ID; the server receiving the set of outputs and the sample ID; the central machine learning model providing a prediction, wherein the prediction is generated by the central machine learning model based on the set of outputs received from each of the plurality of private machine learning models; the central machine learning model receiving a loss, the loss being calculated based on a comparison of the prediction with a label corresponding to the data sample, wherein the central machine learning model updates parameters of the central machine learning model based on the loss.

[0157] According to some embodiments of the above method, providing the prediction includes: transmitting the prediction to a designated computing system among the multiple computing systems, and the designated computing system saves the label; receiving the loss includes: receiving the loss from the designated computing system; the method also includes: the designated computing system calculates the loss.

[0158] According to some embodiments of the above method, providing the prediction includes: providing the prediction to a training function of the server, and the training function saves the label; receiving the loss includes: receiving the loss from the training function; the method also includes: before receiving the prediction, the server receives the label from a specified computing system in the computing system; storing the label in a memory accessible to the server; and calculating the loss through the training function.

[0159] An embodiment of the present invention provides a method for training a private machine learning model, the method comprising: transmitting from each of a plurality of computing systems running corresponding multiple private machine learning models (including the private machine learning model being trained) to a server running a partially or fully trained central machine learning model: a sample ID identifying a data sample; the outputs of the multiple private machine learning models, wherein different private machine learning models among the multiple private machine learning models transmit different corresponding outputs among the outputs generated based on different corresponding parts of the data sample; the server receiving the set of outputs and the sample ID; the central machine learning model providing a prediction, the prediction being calculated by the central machine learning model based on the outputs of the multiple private machine learning models; receiving a loss at the server, the loss being calculated based on a comparison of the prediction with a label corresponding to the data sample; the server transmitting a gradient of the loss relative to a current parameter of the central machine learning model to one of the multiple computing systems running the private machine learning model being trained, the gradient being determined by the server based on the loss; the one of the multiple computing systems running the private machine learning model being trained updating the current model parameters of the private machine learning model being trained based on the gradient.

[0160] According to some embodiments of the above method, providing the prediction includes: transmitting the prediction to a designated computing system among the multiple computing systems, and the designated computing system saves the label; receiving the loss includes: receiving the loss from the designated computing system; the method also includes: the designated computing system calculates the loss.

[0161] According to some embodiments of the above method, providing the prediction includes: providing the prediction to a training function of the server, the training function saves the label, and the loss is calculated by the training function; receiving the loss includes: receiving the loss from the training function; the method also includes: before receiving the prediction, the server receives the label from a specified computing system in the computing system; storing the label in a memory accessible to the server.

[0162] An embodiment of the present invention provides a method for generating predictions using a trained private machine learning model, the method comprising: a computing system running the private machine learning model transmitting to a server running a trained central machine learning model: a sample ID of an identified data sample; a private machine learning model output generated by the private machine learning model based on a portion of the identified data sample, wherein different computing systems among multiple computing systems running corresponding multiple trained private machine learning models (including the trained private machine learning model) transmit different corresponding outputs generated based on different corresponding portions of the identified data sample to the central machine learning model, the different corresponding outputs including the private machine learning model output; the computing system running the private machine learning model receives a prediction from the central machine learning model, the prediction being calculated based on the combined different corresponding outputs, wherein a final prediction is generated by the computing system running the private machine learning model based on the prediction.

[0163] An embodiment of the present invention provides a computing system, which includes: a computer processor operably coupled to a memory; a communication interface; the computing system runs a private machine learning model and is used to: transmit to a server running a trained central machine learning model: a sample ID of an identified data sample; a private machine learning model output generated by the private machine learning model based on a portion of the identified data sample, wherein different computing systems among a plurality of computing systems running corresponding multiple trained private machine learning models (including the trained private machine learning model) transmit different corresponding outputs generated based on different corresponding portions of the identified data sample to the central machine learning model, the different corresponding outputs including the private machine learning model output; receive a prediction from the central machine learning model, the prediction being calculated based on the combined different corresponding outputs, wherein a final prediction is generated based on the prediction by the computing system running the private machine learning model.

[0164] According to some embodiments, the above method includes outputting (or the above computing system is also used to output) the prediction from the central machine learning model as the final prediction.

[0165] According to some embodiments of the above method or computing system, before transmitting the sample ID and the private machine learning model output, the private machine learning model, the multiple other private machine learning models and the central machine learning model are collaboratively trained using vertically partitioned training data, wherein the vertically partitioned training data includes other data samples, each data sample includes multiple features, and different subsets of the multiple features can be accessed by different private machine learning models among the multiple private machine learning models.

[0166] According to some embodiments of the above method or computing system, the sample ID and the private machine learning model output exclude the portion of the data sample and further exclude the training parameters of the private machine learning model.

[0167] According to some embodiments of the above method or computing system, the private machine learning model output is generated in response to the portion of the identified data sample based on the input-output relationship established into the private machine learning model through pre-training.

[0168] The machine learning models described herein can be executed on many computing systems, whether physical or virtual. The computing systems can be centralized or distributed, and each computing system (116, 118, 202) of the data owner is used to store the model, data, and labels. Figure 10 is a block diagram of a computing device that can be used to implement embodiments of the methods disclosed herein. In some embodiments, computing functions can be performed on multiple virtual and / or physical computing devices located in multiple geographic locations. Various physical or virtual computing resources can be used to implement embodiments of the present invention. Therefore, embodiments of the present invention provide a computing device or system of computing devices that can be used to perform the computing and communication operations described elsewhere herein, for example, with respect to the various methods specified above. Each data owner can have a plurality of virtual and / or physical computing devices located in multiple geographic locations. Figure 10 The central machine learning model may also use one or more computing devices such as Figure 10 It can be implemented by one or more computing devices described in .

[0169] Specific computing devices may use all of the components shown or only a subset of the components, and the level of integration may vary from device to device. In addition, computing devices may contain multiple instances of components, such as multiple processing units, processors, memories, network interfaces, etc. Computing device 1000 typically includes a processor 1010, a bus and memory 1020, and may optionally include a mass storage device 1030, a video adapter 1040, and an I / O interface 1060 (each shown in dashed lines to indicate that they are optional). The computing system may also include one or more network interfaces 1050 for connecting the computing system to a communication network 1055.

[0170] The processor 1010 may include any type of electronic data processor and may include one or more cores or processing elements. The processor 1010 may be a central processing unit (CPU), a tensor processor unit (TPU), a neural processing unit (NPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA). The memory may include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In an embodiment, the memory may include a ROM used at power-on and a DRAM for storing programs and data used when executing a program. The bus may be one or more of several bus architectures of any type including a memory bus or a memory controller, a peripheral bus, or a video bus.

[0171] The mass storage device can include any type of non-transient storage device for storing data, programs, and other information and making the data, programs, and other information accessible via the bus. For example, the mass storage device can include one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, or an optical disk drive.

[0172] The video adapter 1040 and the I / O interface 1060 provide optional interfaces for coupling external input and output devices to the processing unit. Examples of input and output devices include a display 1070 coupled to the video adapter 1040 and an I / O device 1080, such as a touch screen, coupled to the I / O interface 1060. Other devices can be coupled to the processing unit and additional or fewer interfaces can be used. For example, a serial interface such as a universal serial bus (USB) (not shown) can be used to provide an interface for external devices. Alternatively, the computing system can rely on a network interface to connect to available mass storage devices, video adapters, and I / O interfaces in the network.

[0173] In addition, additional components may be included, such as one or more of a graphics processor, a tensor processor, an application-specific integrated circuit, a field-programmable gate array, a neuromorphic processor, or other electronic or photonic processing components, which may be used in conjunction with or in place of the processor 1010 to perform processing operations. The processing operations may include machine learning operations, other operations that support the machine learning operations, or a combination thereof.

[0174] Unless defined otherwise, all technical and / or scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art.

[0175] Through the description of the above embodiments, the present invention can be implemented only by using hardware, or by using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile or non-transient storage medium, which can be a read-only optical disc (compact disk read-only memory, CD-ROM), a USB flash drive, or a removable hard disk. The software product includes many instructions that enable a computer device (a personal computer, a server, or a network device) to execute the method provided by the embodiment of the present invention. For example, such execution can correspond to a simulation of the logical operations described herein. The software product may additionally or optionally include a plurality of instructions that enable a computer device to execute operations for configuring or programming a digital logic device according to an embodiment of the present invention.

[0176] Although the present invention has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations of the present invention may be made without departing from the present invention. Therefore, the specification and drawings are to be regarded only as illustrative of the present invention as defined by the appended claims, and are intended to include any and all modifications, variations, combinations or equivalents within the scope of the present invention.

Claims

1. A method for secure longitudinal federated learning to train a machine learning model, characterized in that The method comprises: a server running a central machine learning model receiving a set of outputs of the private machine learning model from each of a plurality of computing systems, wherein each computing system runs a different respective one of the plurality of private machine learning models, different computing systems of the plurality of computing systems transmitting different respective ones of the set of outputs generated based on different respective portions of the identified data sample, the set of outputs aligned based on a sample ID identifying the data sample; The server provides a prediction computed by the central machine learning model based on the set of outputs; receiving, at the server, a loss calculated based on a comparison of the prediction to a label corresponding to the data sample; Initiate an update of current model parameters of the private machine learning model based on the loss.

2. The method according to claim 1, characterized in that Also includes: The server calculates the gradient of the loss relative to the model parameters of the central machine learning model, wherein the updating of the current model parameters of the machine learning model being trained is directly or indirectly based on the gradient of the loss.

3. The method according to claim 1 or 2, characterized in that Providing the prediction includes: transmitting the prediction to a designated computing system among the plurality of computing systems, the designated computing system storing the tag, and the loss being calculated by the designated computing system; receiving the loss includes: receiving the loss from the designated computing system.

4. The method according to claim 1, wherein Providing the prediction includes: providing the prediction to a training function of the server, the training function storing the label; receiving the loss includes: receiving the loss from the training function; the method also includes: before receiving the prediction, the server receiving the label from a specified computing system in the computing system and storing the label in a memory accessible to the server; using the training function to calculate the loss.

5. The method according to claim 1, wherein The machine learning model being trained is the central machine learning model, and the method further includes: the server updating the current model parameters of the central machine learning model based on the loss.

6. The method according to claim 1, characterized in that The machine learning model being trained is one of the multiple private machine learning models, wherein initiating an update of the current model parameters of the machine learning model being trained includes: the server transmitting an indication of the loss or an indication of the gradient of the loss to a computing system running one of the multiple private machine learning models.

7. The method according to claim 6, characterized in that The gradient of the loss is the gradient relative to the model parameters of the central machine learning model, and the updating of the current model parameters of the one of the private machine learning models is directly based on the gradient of the loss relative to the current model parameters, and the gradient of the loss relative to the current model parameters is derived from the gradient of the model parameters relative to the central machine learning model using a calculation that implements the chain rule of derivatives.

8. The method according to claim 1, characterized in that Also includes: The method is repeated until convergence conditions are reached.

9. A server, characterized in that: include: a processor operatively coupled to the memory; Communication interface; The server runs a central machine learning model and is used to: receiving a set of outputs of a private machine learning model from each of a plurality of computing systems, wherein each computing system executes a different respective one of the plurality of private machine learning models, different computing systems in the plurality of computing systems transmitting different respective ones of the set of outputs generated based on different respective portions of the identified data sample, the set of outputs aligned based on a sample ID identifying the data sample; providing a prediction computed by the central machine learning model based on the set of outputs; receiving a loss calculated based on a comparison of the prediction to a label corresponding to the data sample; Initiate an update of current model parameters of the private machine learning model based on the loss.

10. The server according to claim 9, wherein: Also used to: calculate the gradient of the loss relative to the model parameters of the central machine learning model, wherein the updating of the current model parameters of the machine learning model being trained is directly or indirectly based on the gradient of the loss.

11. The server according to claim 9 or 10, characterized in that: Providing the prediction includes: transmitting the prediction to a designated computing system among the plurality of computing systems, the designated computing system storing the label, and the loss being calculated by the designated computing system; receiving the loss includes: receiving the loss from the designated computing system.

12. The server according to claim 9, wherein: Providing the prediction includes: providing the prediction to a training function of the server, the training function saving the label; receiving the loss includes: receiving the loss from the training function; the server is also used to: before receiving the prediction, receive the label from a specified computing system in the computing system; store the label in a memory accessible to the server; and calculate the loss using the training function.

13. The server according to claim 9, wherein: The machine learning model that is being trained is the central machine learning model, and the server is further used to: update the current model parameters of the central machine learning model based on the loss.

14. The server according to claim 9, wherein: The machine learning model being trained is one of the multiple private machine learning models, wherein initiating an update of the current model parameters of the machine learning model being trained includes: the server transmitting an indication of the loss or an indication of the gradient of the loss to a computing system running one of the multiple private machine learning models.

15. The server according to claim 14, wherein: The gradient of the loss is the gradient relative to the model parameters of the central machine learning model, and the updating of the current model parameters of the one of the private machine learning models is directly based on the gradient of the loss relative to the current model parameters, and the gradient of the loss relative to the current model parameters is derived from the gradient of the model parameters relative to the central machine learning model using a calculation that implements the chain rule of derivatives.

16. The server according to claim 9, wherein: Also used for: Repeating a training method until convergence conditions are reached.

17. A method for secure longitudinal federated learning to train a specified private machine learning model for a specific task, characterized in that The method comprises: The first computing system running the designated private machine learning model transmits to a server running a central machine learning model: a sample ID; an output of the designated private machine learning model, the output being generated based on a portion of the data sample identified by the sample ID; wherein each of a plurality of computing systems runs a different corresponding one of a plurality of private machine learning models, the plurality of computing systems including the first computing system, the plurality of private machine learning models including the designated private machine learning model; each of the plurality of computing systems transmits a different corresponding one of a set of private machine learning model outputs, the set of private machine learning model outputs including the output; each of a set of outputs is generated based on a different corresponding portion of the data sample, the set of outputs being aligned based on the sample ID; The first computing system receives, from the server, a prediction computed by the central machine learning model based on the combined set of outputs; the first computing system providing a loss to the server, the loss calculated based on a comparison of the prediction and a label corresponding to the data sample, the label stored in a memory privately accessible to the first computing system and inaccessible to the server and other computing systems of the plurality of computing systems; The first computing system updates current model parameters of the specified private machine learning model based on the loss.

18. The method according to claim 17, characterized in that The updating of the current model parameters is performed based on the calculated gradients of the loss with respect to the model parameters of the machine learning model being trained.

19. The method according to claim 17 or 18, characterized in that Each of the multiple computing systems includes a corresponding private data storage space, each corresponding private data storage space: accessible by a corresponding one of the multiple computing systems and inaccessible to the server and other computing systems in the multiple computing systems; and storing a corresponding one of the different corresponding parts of the data sample.

20. A computer-readable medium, characterized in that The computer-readable medium comprises instructions which, when executed by a processor of a computing system, cause the computing system to perform the method according to any one of claims 17 to 19.

21. A computing system, characterized in that include: a processor operatively coupled to the memory; The communication interface; the computing system runs a specified private machine learning model and is used to: Transmitting to a server running a central machine learning model: a sample ID; an output of the designated private machine learning model, the output being generated based on a portion of the data sample identified by the sample ID; wherein each of a plurality of computing systems runs a different corresponding one of a plurality of private machine learning models; the plurality of computing systems includes a first computing system, the plurality of private machine learning models including the designated private machine learning model; each of the plurality of computing systems transmits a different corresponding one of a set of private machine learning model outputs, the set of private machine learning model outputs including the output; each of a set of outputs is generated based on a different corresponding portion of the data sample, the set of outputs being aligned based on the sample ID; receiving, from the server, a prediction computed by the central machine learning model based on the combined set of outputs; providing a loss to the server, the loss calculated based on a comparison of the prediction and a label corresponding to the data sample, the label stored in a memory privately accessible to the computing system and inaccessible to the server and other computing systems of the plurality of computing systems; Subsequently, an update of current model parameters of the specified private machine learning model is initiated based on the loss.

22. The computing system according to claim 21, wherein: The machine learning model being trained is the central machine learning model or one of the multiple private machine learning models.

23. The computing system according to claim 21 or 22, characterized in that The updating of the current model parameters of the machine learning model under training is performed based on the calculated gradient of the loss relative to the model parameters of the machine learning model under training.

24. The computing system of claim 21, wherein: Each of the multiple computing systems includes a corresponding private data storage space, each corresponding private data storage space: accessible by a corresponding one of the multiple computing systems and inaccessible to the server and other computing systems in the multiple computing systems; and storing a corresponding one of the different corresponding parts of the data sample.

25. A method for secure longitudinal federated learning to train a central machine learning model, characterized in that The method comprises: receiving, at a server running the central machine learning model, a sample ID and a set of outputs of a private machine learning model from each of a plurality of computing systems, the set of outputs of the private machine learning model being generated based on a portion of the identified data sample corresponding to the sample ID, wherein each computing system runs a different respective one of the plurality of private machine learning models, different ones of the plurality of computing systems transmitting different respective outputs generated based on different respective portions of the data sample, the set of outputs being aligned based on the sample ID; The server receives the set of outputs and the sample ID; the central machine learning model providing a prediction, wherein the prediction is generated by the central machine learning model based on the set of outputs received from each of the plurality of private machine learning models; The central machine learning model receives a loss calculated based on a comparison of the prediction to a label corresponding to the data sample; Wherein, the central machine learning model updates the parameters of the central machine learning model based on the loss.

26. The method according to claim 25, characterized in that Providing the prediction includes: transmitting the prediction to a designated computing system among the multiple computing systems, and the designated computing system saves the label; receiving the loss includes: receiving the loss from the designated computing system; the method also includes: the designated computing system calculates the loss.

27. The method according to claim 25 or 26, characterized in that Providing the prediction includes: providing the prediction to a training function of the server, the training function saving the label; receiving the loss includes: receiving the loss from the training function; the method also includes: before receiving the prediction, the server receiving the label from a specified computing system in the computing system; storing the label in a memory accessible to the server; and calculating the loss through the training function.

28. A server, characterized in that: include: processor; A memory storing instructions, which, when executed by the processor, causes the server to perform the method according to any one of claims 25 to 27.

29. A method for secure longitudinal federated learning to train private machine learning models, characterized in that The method comprises: transmitting, from each of a plurality of computing systems running a corresponding plurality of private machine learning models to a server running a partially or fully trained central machine learning model: a sample ID identifying a data sample; outputs of the plurality of private machine learning models, wherein the plurality of private machine learning models includes the trained private machine learning model, different ones of the plurality of private machine learning models transmitting different ones of the outputs generated based on different ones of the data samples; The server receives a set of outputs and the sample ID; The central machine learning model provides a prediction, the prediction calculated by the central machine learning model based on the outputs of the plurality of private machine learning models; receiving, at the server, a loss calculated based on a comparison of the prediction to a label corresponding to the data sample; The server transmits, to one of the plurality of computing systems running the private machine learning model being trained, a gradient of the loss with respect to current parameters of the central machine learning model, the gradient determined by the server based on the loss; The one of the plurality of computing systems running the private machine learning model under training updates current model parameters of the private machine learning model under training based on the gradient.

30. The method according to claim 29, wherein Providing the prediction includes: transmitting the prediction to a designated computing system among the multiple computing systems, and the designated computing system saves the label; receiving the loss includes: receiving the loss from the designated computing system; the method also includes: the designated computing system calculates the loss.

31. The method according to claim 29 or 30, characterized in that Providing the prediction includes: providing the prediction to a training function of the server, the training function saving the label, and the loss calculated by the training function; receiving the loss includes: receiving the loss from the training function; the method also includes: before receiving the prediction, the server receiving the label from a specified computing system in the computing system; storing the label in a memory accessible to the server.

32. A computing system, characterized in that include: a processor operatively coupled to the memory; Communication interface; the computing system runs a private machine learning model and is used to perform the method according to any one of claims 29 to 31.

33. A computer-readable medium, characterized in that The computer-readable medium comprises instructions which, when executed by a processor of a computing system, cause the computing system to perform the method according to any one of claims 29 to 31 .

34. A method for generating predictions using a trained private machine learning model, characterized in that The method comprises: The computing system running the private machine learning model transmits to the server running the trained central machine learning model: the sample ID of the identified data sample; the private machine learning model output generated by the private machine learning model based on a portion of the identified data sample, wherein different computing systems among the multiple computing systems running the corresponding multiple trained private machine learning models transmit different corresponding outputs generated based on different corresponding portions of the identified data sample to the central machine learning model, and the different corresponding outputs include the private machine learning model output; the multiple trained private machine learning models include the trained private machine learning model; the computing system running the private machine learning model receives a prediction from the central machine learning model, and the prediction is calculated based on the combined different corresponding outputs, wherein the computing system running the private machine learning model generates a final prediction based on the prediction.

35. The method according to claim 34, wherein Also includes: The computing system running the private machine learning model outputs the prediction from the central machine learning model as the final prediction.

36. The method according to claim 34 or 35, characterized in that Prior to transmitting the sample ID and the private machine learning model output, the private machine learning model, multiple other private machine learning models, and the central machine learning model are collaboratively trained using vertically partitioned training data, wherein the vertically partitioned training data includes other data samples, each data sample includes multiple features, wherein different subsets of the multiple features can be accessed by different private machine learning models among the multiple private machine learning models.

37. The method according to claim 34, wherein The sample ID and the private machine learning model output exclude the portion of the data sample and further exclude training parameters of the private machine learning model.

38. The method according to claim 34, wherein The private machine learning model output is generated in response to the portion of the identified data sample based on an input-output relationship built into the private machine learning model through pre-training.

39. A computing system, characterized in that include: a processor operatively coupled to the memory; Communication interface; the computing system runs a private machine learning model and is used to perform the method according to any one of claims 34 to 38.

40. A computer-readable medium, characterized in that The computer-readable medium comprises instructions which, when executed by a processor of a computing system, cause the computing system to perform the method according to any one of claims 34 to 38.

Images