Systems and methods for cross-site learning

Abstract

Systems and methods for cross-site learning, data samples are transmitted from a central server to at least one local server device. The central server receives a set of predictions based on the transmitted set of data samples from the at least one local server device. The central server trains a central model based on the received set of predictions. The central model or a portion of the central model corresponding to the task of interest can then be sent to at least one local server device. Neither local data from the local site nor the trained model from the local site is transmitted to the central server. This ensures protection and security of the data at the local site.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to machine learning and, more particularly, to ensuring data privacy in federated learning models. Background Art

[0002] In machine learning models, data can be hosted locally across multiple sites. Using this local data, models can be trained locally to build a central model that leverages the benefits of local models. However, data privacy and security concerns prohibit local data from being moved out of the local site.

[0003] Federated learning systems have been shown to have high inherent user privacy protection qualities. The reason for this is mainly due to the fact that user data is kept on the user's device or device. The part of federated learning that is most susceptible to user privacy degradation is the access to model updates that move between the user's device and the backend server.

[0004] Although federated learning has made progress, these technologies do not guarantee that the privacy of local data will be protected. There is a problem of obtaining local data or trained learning models from local sites. This poses a potential data privacy issue.

[0005] It would therefore be desirable to be able to provide a system that addresses at least some of the above-mentioned issues. Summary of the invention

[0006] Aspects of the disclosed embodiments relate to learning a central model from a large number of locally trained models at multiple local sites. This object is solved by the subject matter of the independent claims. Further advantageous modifications can be found in the dependent claims.

[0007] According to a first aspect, the above and further objects and advantages are obtained by an apparatus. In one embodiment, the apparatus includes a processor. The processor is configured to: transmit a set of data samples to at least one local server device; receive a set of predictions from the local server device, the set of predictions being based on the transmitted set of data samples; and train a central model based on the received set of predictions. In this way, the central model is learned from a large number of locally trained models at multiple local sites without obtaining any local data or trained machine learning models from the local sites.

[0008] In a possible implementation of the device, the processor is further configured to transmit at least a portion of the central model to at least one local server device. The integrated data set of the central model can be very large. Sending only a portion of the large integrated data set related to a specific local node or server can be effective and efficient in terms of time and cost.

[0009] In a possible implementation form of the device, the processor is further configured to: determine at least one task of interest at at least one local server device; identify a portion of the central model corresponding to the at least one task of interest; and transmit the identified portion of the central model corresponding to the at least one task of interest to the at least one local server device. By identifying a specific task of interest at a local node, only the data of the central model relevant to the task needs to be sent to the corresponding local node, rather than sending the entirety of the potentially large integrated data set. This provides efficiencies in terms of time and cost.

[0010] In a possible implementation of the device, the set of data samples is publicly available data samples. Using publicly available data samples enables training to be performed at a local site without infringing data privacy.

[0011] In a possible implementation form of the device, the set of data samples is specific to the task of interest at at least one local server device.Using publicly available data samples enables training to be performed at the local site without violating data privacy.

[0012] In a possible implementation form of the device, the device includes a training / inference server.

[0013] In a possible implementation form of the device, the processor is configured to transmit data samples corresponding to the task of interest at at least one local server device.

[0014] In a possible implementation form of the device, the processor is configured to form an integrated data set corresponding to the task of interest from the received set of predictions.

[0015] In a possible implementation form of the device, at least one local server device trains a local model using local data.

[0016] In a possible implementation form of the device, the processor is configured to train the central model.

[0017] In a possible implementation form of the device, the processor is configured to initiate the knowledge distillation process using at least one local server device.

[0018] In a possible implementation form of the device, during the distillation process, the processor is configured to send data to at least one local server device.

[0019] In a possible implementation form of the device, the processor is configured to pre-download data to at least one local server device.

[0020] In a possible implementation form of the device, at least one local server device uses such data to infer its latest local model and obtains the inferred result.

[0021] In a possible implementation form of the device, the processor is configured to receive a result of the inference.

[0022] In a possible implementation form of the device, the processor is configured to aggregate the inferred results.

[0023] In a possible implementation form of the device, the aggregated results are used to train a central model.

[0024] In a possible implementation form of the device, the processor comprises a central server device in communication with the central model.

[0025] According to a second aspect, the above and further objects and advantages are obtained by a method. In one embodiment, the method comprises: transmitting a set of data samples from a central server to at least one local server device; receiving a set of predictions at the central server from at least one local server, the set of predictions being based on the transmitted set of data samples; and training a central model at the central server based on the received set of predictions.

[0026] In a possible implementation form of the method, the method further comprises: transmitting at least a portion of the central model from the central server to at least one local server device. The integrated data set of the central model can be very large. Sending only a portion of the large integrated data set relevant to a specific local node or server can be effective and efficient in terms of time and cost.

[0027] In a possible implementation form of the method, the method further comprises: determining at least one task of interest at at least one local server device; identifying a portion of the central model corresponding to the at least one task of interest; and transmitting the identified portion of the central model corresponding to the at least one task of interest to the at least one local server device. By identifying a specific task of interest at a local node, only the data of the central model relevant to the task needs to be sent to the corresponding local node, rather than sending the entirety of the potentially large integrated data set. This provides efficiencies in terms of time and cost.

[0028] In a possible implementation form of the method, the set of data samples comprises publicly available data samples.

[0029] In a possible implementation form of the method, the set of data samples is specific to a task of interest at at least one local server device.

[0030] In a possible implementation form of the method, the central server includes a training / inference server.

[0031] In a possible implementation form of the method, the transmitted set of data samples corresponds to a task of interest at at least one local server device.

[0032] In a possible implementation form of the method, the method further comprises: forming an integrated data set corresponding to the task of interest from the received set of predictions.

[0033] In a possible implementation form of the method, at least one local server device trains a local model using local data.

[0034] In a possible implementation form of the method, the central server trains the central model.

[0035] In a possible implementation form of the method, the central server may initiate the knowledge distillation process using at least one local server device.

[0036] In a possible embodiment of the method, during the distillation process, the central server sends data to at least one local server device.

[0037] In a possible implementation form of the method, the data may be pre-downloaded to at least one local server device.

[0038] In a possible implementation form of the method, at least one local server device uses such data to infer its latest local model and obtains the inferred result.

[0039] In a possible implementation form of the method, a central server receives the inferred result.

[0040] In a possible implementation form of the method, the central server is configured to aggregate the inferred results.

[0041] In a possible implementation of the method, the aggregated results are used to train a central model.

[0042] According to a third aspect, the above and further objects and advantages are obtained by a computer program product. In one embodiment, the computer program product comprises a non-transitory computer readable medium having program instructions stored thereon, which, when executed by a processor, causes the processor to perform a method according to any possible implementation form described herein.

[0043] These and other aspects, implementations, and advantages of the exemplary embodiments will become apparent from the embodiments described herein considered in conjunction with the accompanying drawings. However, it should be understood that the description and drawings are designed for illustrative purposes only and not as a definition of the limitations of the disclosed invention, which should be referred to the appended claims. Additional aspects and advantages of the invention will be set forth in the following description and will be apparent in part from the description or may be understood by practice of the invention. Moreover, aspects and advantages of the invention may be realized and obtained by the means and combinations particularly pointed out in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In the following detailed portion of the present disclosure, the present invention will be described in more detail with reference to example embodiments shown in the accompanying drawings, in which:

[0045] Figure 1 A schematic block diagram of an exemplary device incorporating aspects of the disclosed embodiments is illustrated.

[0046] Figure 2 An exemplary privacy-preserving cross-hospital consortium learning setting incorporating aspects of the disclosed embodiments is illustrated.

[0047] Figure 3 Exemplary methods incorporating aspects of the disclosed embodiments are illustrated. DETAILED DESCRIPTION

[0048] See also Figure 1 , illustrates a schematic block diagram of an exemplary device 100 incorporating aspects of the disclosed embodiments. Aspects of the disclosed embodiments are configured to learn a central model from a large number of locally trained models at multiple local sites. Data privacy is explicitly protected by not using local data or moving trained machine learning models from local sites.

[0049] like Figure 1 By way of example, in one embodiment, the device 100 includes: at least one processor 102: and a corresponding memory 104, which is used to store one or more programs, which can be executed by the processor 102 to perform the methods described herein. The device 100 also includes at least one input device 106 for receiving input data and at least one output device 108 for outputting data. In alternative embodiments, the device 100 may include any other device or component suitable for a machine learning environment. In one embodiment, the device 100 includes a training / inference server.

[0050] As will be generally described herein, in one embodiment, processor 102 is configured to transmit a set of data samples to at least one local server device, e.g. Figure 2 The processor 102 is configured to receive a set of predictions from the local server 204, the set of predictions being based on the transmitted set of data samples. The processor 102 is then configured to train a central model based on the received set of predictions and transmit the central model to at least one local server device 204. The set of data samples in this example is a publicly available set of data samples. In this way, local data or trained machine learning models are not taken or removed from a local site such as the local server device 204, and the privacy of the data at the local site is maintained.

[0051] Examples of processor 102 include, but are not limited to, microprocessors, microcontrollers, complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, or any other type of processing circuit. Optionally, processor 102 may be one or more separate processors, processing devices, and various elements associated with the processing devices that may be shared by other processing devices. In addition, one or more separate processors, processing devices, and elements are arranged in various architectures to respond to and process instructions of drive device 100.

[0052] In one embodiment, one or more of the input module 106 and / or the output module 108 may include one or more communication networks or modules to enable communication and transfer of information to and from the device 100. The communication network may be a wired or wireless communication network. Examples of communication networks may include, but are not limited to, wireless fidelity (Wi-Fi) networks, local area networks (LANs), wireless personal area networks (WPANs), wireless local area networks (WLANs), wireless wide area networks (WWANs), cloud networks, long term evolution (LTE) networks, plain old telephone service (POTS), metropolitan area networks (MANs), and / or the Internet. Exemplary communication protocols may include wired and wireless communication protocols, such as, but not limited to, transmission control protocol and Internet protocol (TCP / IP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), file transfer protocol (FTP), ZigBee, EDGE, infrared (IR), IEEE 802.11, 802.16, long term evolution (LTE), light fidelity (Li-Fi), and / or other cellular communication protocols or Bluetooth (BT) communication protocols, including variants thereof.

[0053] The memory 104 may generally include suitable logic, circuitry, interfaces and / or code that may be configured to store instructions executable by the processor 102. Exemplary implementations of the memory 104 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), a hard disk drive (HDD), flash memory, and / or a secure digital (SD) card. Computer-readable storage media for providing non-transitory memory may include, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing.

[0054] Figure 2 An exemplary system 200 incorporating aspects of the disclosed embodiments is illustrated. In this example, the system 200 generally includes a distributed framework for training machine learning models. Figure 2As shown, the system 200 includes at least a central server 202 and a plurality of local nodes 204a-204n. Although only three local nodes 204a-204n are shown in this example, aspects of the disclosed embodiments are not limited thereto. In alternative embodiments, the system 200 may include any suitable or desired number of local nodes other than three. It should be understood that the system 200 forms or is part of a machine learning network.

[0055] The local nodes or sites 204a-204n will be configured to host their own data and training / inference services locally. The local nodes 204a-204n are also configured to communicate with the central server 202, which is also referred to as a centrally located training / inference server.

[0056] At each local site 204a-204n, the local data will be used to train the corresponding local model 206a-206n using the local training server. In one embodiment, the corresponding local model 206a-206n can then be deployed for internal artificial intelligence (AI) services.

[0057] like Figure 2 As illustrated, each local site 204a-204n is configured to be able to receive messages from and transmit messages to the central server 202. However, in accordance with aspects of the disclosed embodiments, local data at the local sites 204a-204n will not be transmitted to the central server 202.

[0058] For purposes of the description herein, the system 200 will generally be referred to as a cross-hospital federated learning setting. In this example, the local sites or nodes 204a-204n may generally be referred to as "hospital sites" or "hospital servers." Although aspects of the disclosed embodiments will be generally described with respect to a health care or hospital environment, aspects of the disclosed embodiments are not limited thereto. In alternative embodiments, aspects of the disclosed embodiments may be applied to or implemented in any machine learning system in which data privacy is a concern.

[0059] Figure 2 The illustrated central server or node 202 is configured to coordinate the training of a central model, generally shown as 210. The central model 210 and local models 206a-206n may include any suitable machine learning models, such as a neural network architecture.

[0060] Aspects of the disclosed embodiments involve two main processes. The first is to train local models 206a-206n on corresponding local data. Then, the knowledge from the training of local models 206a-206n is distilled into the central model 210 trained at the central node 202.

[0061] For example, see also Figure 3 In one embodiment, at any point during the training of the central model, the central node 202 may initiate 302 a knowledge distillation process with one or more of the local nodes 204a-204n. In one embodiment, the knowledge distillation process initiated by the central node 202 may occur at the end of various epochs in training a local model, such as one or more of the local models 206a-206n. In an alternative embodiment, the knowledge distillation process initiated by the central node 202 may occur at the end of the training of one or more local models 206a-206n.

[0062] During the distillation process, the central server 202 sends 304 data to one or more of the local nodes 204a-204n. In one embodiment, the data can be pre-downloaded to one or more local nodes 204a-204n. The data sent during the distillation process includes public data, rather than private data. Typically, there are repositories of data sets for different tasks available online. These repositories are configured to index available public data sets with their links. These resources are used to download data to a central server such as the central server 202.

[0063] Each of the one or more local nodes 204a-204n can be configured to use this data to infer its latest local model and obtain 306 the inferred results, such as activations, predictions, features, or any kind of description. The inferred results will be sent 308 back to the central node 202.

[0064] The central node 202 is configured to aggregate 310 the inferred results from one or more local nodes 204a-204n. Using the inferred aggregated results, the central model 210 is trained 312, updated and / or fine-tuned using certain knowledge / explanation distillation objective functions. The privacy of the local data of the local nodes 204a-204n is protected throughout the training process because no local data, partial or complete local models are transmitted to the central node 202.

[0065] A multifaceted approach is used to train the central model 210. At the central node 202, a large data set is prepared and collected. The large data set will be used to train the central model 210.

[0066] The large data set used to train the central model 210 will not have any data from any local node 204a-204n. Instead, it is assumed that the specific task of interest at each local node 204a-204n is known. Given this knowledge, data is collected from different sources related to the specific task of interest. The data samples present in the data set can be annotated manually or automatically. Manual annotation refers to one or more individuals annotating the data. For example, an individual can annotate the location of a lung nodule in the downloaded data. Automatic annotation refers to the use of an automated process, such as using an existing algorithm to generate annotations.

[0067] This process is repeated for all interesting tasks at all local nodes 204a-204n. The result can be a large integrated data set, also referred to as a central large-scale data set. The large data set will generally correspond to the interesting tasks at all local nodes 204a-204n. Aspects of the disclosed embodiments are configured to enable a local node among the local nodes 204a-204n to receive a data set corresponding to a task of interest at the local node, or alternatively, to receive the entire large integrated data set when the task of interest is unknown.

[0068] There may be different tasks of interest at each node 204a-204n. The central data set may correspond to one or more tasks of interest at the local nodes 204a-204n. If the local tasks are known, the corresponding local node in the local nodes 204a-204n will receive the data corresponding to the tasks of interest. If the local tasks are unknown, the local node will receive the entire data set.

[0069] In one embodiment, see also the above Figure 3 In step 304, the large integrated data set may be preloaded onto each local node 204a-204n. During the knowledge distillation process initiated by the central node 202, the central node 202 only sends a process initiation signal or a set of data indexes of the large integrated data set, such as a subset, to the local nodes 204a-204n.

[0070] In one embodiment, during the knowledge distillation process initiated by the central node 202, the data index of the integrated data set sent by the central node 202 may be determined in a guided manner. The data index for the next distillation process may be selected based on the results of the current distillation process. For example, only data with unsatisfactory consistency from all local nodes 204a-204n may be selected for the next round of distillation.

[0071] Given this central data set, predictions are now obtained on each sample from each local node learning model. This involves the central node 202 sending each data sample to each individual local inference server in the corresponding local node 204a-204n and retrieving the prediction information. In this process, the local nodes 204a-204n do not transmit any local proprietary data outside the corresponding local network. Instead, the central node 202 sends data 203a-203n from the pre-collected integrated data set and retrieves the inference results 205a-205n only from the trained local model 206a. In this way, the privacy requirements of the local nodes 204a-204n are met.

[0072] The inference results retrieved from the inference server of the corresponding local node 204a-204n may include, but are not limited to, a prediction result from a machine learning model, a feature map activation from a specific intermediate layer of the machine learning model, or feature map activations from multiple intermediate layers of the machine learning model. For example, the prediction result may be whether the image is a cat or a dog.

[0073] In one embodiment, given all prediction information retrieved from all local nodes 204a-204 about all data samples in the integrated data set, the method may include: training a central machine learning model in a student and integrated teacher manner. In this example, there are multiple teacher models from all local nodes 204a-204n such as a hospital site.

[0074] In one embodiment, a specific training goal of the central model 210 is to ensure that the predictions of the central model 210 are as close as possible to the ensemble predictions of all teacher models from all local nodes 204a-204n.

[0075] In one embodiment, once the training process of the central model 210 has converged, the central model 210 is now ready to be deployed. At this stage, the central node 202 transmits to each of the local nodes 204a-204n that the new machine learning model is ready to be deployed. The transmission may also include the corresponding accuracy statistics of the model, such as the new performance on the existing test data set. The central node 202 waits for a signal from the manager of each local node 204a-204n.

[0076] When the corresponding site manager transmits the need to update the corresponding local model to the central server 202, the central server 202 transmits the new centrally trained model 210 to the corresponding local nodes 204a-204n in a predefined form via the communication channel. Then, the process of deploying the updated machine learning model is repeated for all local sites that request to update their local machine learning models.

[0077] Aspects of the disclosed embodiments are configured to repeat the process of training the central model and updating the local models when necessary. Specifically, if any individual local node or even multiple local nodes decide to deploy a new locally trained model in its inference server, aspects of the disclosed embodiments can check for this occurrence. In one embodiment, the central server 202 can be configured to periodically communicate with each local node 204a-204n to request any changes to the corresponding local models 206a-206n. If there are changes, the local node 204a-204n or the corresponding local site manager can respond. The process of training a new central model will be triggered, which repeats all of the above steps.

[0078] Aspects of the disclosed embodiments relate to learning a central model from a large number of locally trained models at multiple local sites. Data privacy is protected by not fetching or transmitting any local data or trained machine learning models from the local sites. Student-teacher integrated knowledge techniques are used to learn a privacy-preserving central machine learning model from multiple locally trained machine learning models.

[0079] Thus, although the basic novel features of the present invention as applied to the exemplary embodiments of the present invention have been shown, described and pointed out, it should be understood that those skilled in the art may make various omissions, substitutions and changes in the form and details of the exemplified devices and methods and their operation without departing from the spirit and scope of the presently disclosed invention. Further, it is expressly contemplated that all combinations of those elements that perform substantially the same functions in substantially the same manner to achieve the same results are within the scope of the present invention. Moreover, it should be recognized that the structures and / or elements shown and / or described in conjunction with any disclosed form or embodiment of the present invention may be incorporated into any other disclosed or described or suggested form or embodiment as a general matter of design choice. Therefore, the present invention is intended to be limited only by the scope indicated by the scope of the appended claims.

Claims

1. A device for cross-site learning, comprising: A processor configured to: transmitting a set of public data samples to at least one local server device; The local server device includes a local machine learning model trained by a private data sample; the sample is an image; receiving, from the at least one local server device, a set of predictions generated by the local machine learning model based on the transmitted set of public data samples; the predictions comprising annotation information for the image; and A central machine learning model is trained based on the received set of predictions; the private data samples used to train at least one local server device are not transmitted to the central machine learning model.

2. The device according to claim 1, wherein: The processor is further configured to transmit at least a portion of the central model to the at least one local server device.

3. The device according to claim 2, wherein: The processor is also configured to: determine at least one task of interest at the at least one local server device; identify a portion of the central machine learning model corresponding to the at least one task of interest; and transmit the identified portion of the central machine learning model corresponding to the at least one task of interest to the at least one local server device.

4. The device according to claim 1, wherein: The set of data samples is specific to a task of interest at the at least one local server device.

5. The device according to claim 1, wherein: The apparatus comprises a training or derivation server.

6. The device according to claim 1, wherein: The transmitted set of data samples corresponds to a task of interest at the at least one local server device.

7. The device according to claim 1, wherein: The processor is further configured to form an integrated data set corresponding to at least one task of interest from the received set of predictions.

8. A method for cross-site learning, comprising: transmitting a set of public data samples from the central server to at least one local server device; The local server device includes a local machine learning model trained by a private data sample; the sample is an image; receiving, in the central server, from the at least one local server device, a set of predictions generated by the local machine learning model based on the transmitted set of public data samples; the predictions comprising annotation information for the image; as well as A central machine model is trained in the central server based on the received set of predictions; the private data samples used to train at least one local server device are not transmitted to the central machine learning model.

9. A computer program product for cross-site learning, comprising a non-transitory computer-readable medium having program instructions stored thereon, which when executed by a processor cause the processor to perform the method according to claim 8.

Images