Medical information processing device, medical information processing system, and medical information processing method

Abstract

To improve accuracy of a model.SOLUTION: A medical information processing apparatus comprises processing circuitry. The processing circuitry is configured to: obtain first trained parameters for a model; obtain second trained parameters for the model; determine a first evaluation value by inputting data from a first data cohort into a model having the first trained parameters; and determine a second evaluation value by inputting data from the first data cohort into a model having the second trained parameters.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The embodiments described herein generally relate to a medical image processing device, a medical image processing system, and a medical image processing method for training a model at multiple facilities. [Background technology]

[0002] In the field of artificial intelligence (AI), it is known to train an AI model (e.g., a neural network) using data from one or more facilities. For example, an AI model can be trained on medical imaging data collected at and stored by multiple hospitals or other medical facilities.

[0003] Training an AI model on data from one or more facilities can be performed using what we will refer to hereafter as “traditional federated learning algorithms.”

[0004] In traditional distributed learning algorithms, an AI model (e.g., a neural network) learns on one or more cohorts simultaneously without transferring data between such cohorts. The collection of facilities providing data is called a "federation."

[0005] Copies of the AI ​​model are sent to each facility in the federation from a central location server, which can be any server that is not participating in the learning. The central location server acts as a "central connection point."

[0006] A copy of the model sent to each site is trained on the data cohort at that site over a period of time, after which each site returns the trained model parameters to the central location server.

[0007] The central location server aggregates the trained model parameters from multiple facilities to form a new model, and the training cycle then repeats with the model being pushed out to each facility.

[0008] 1 is a schematic diagram of a number of facilities 10A to 10F each maintaining a cohort of individual data 12A to 12F. A central location server 14 communicates with each of such facilities 10A to 10F.

[0009] In some of the following descriptions, actions (e.g., training a model) are referred to as being performed by a facility. References to actions as being performed by a facility may be considered to relate to actions performed by one or more devices having access to data maintained by such facility. For example, the one or more devices may be computers located at such facility or computers having access to data maintained by such facility.

[0010] References to data maintained by a facility may refer to data stored at or on behalf of such facility. Data maintained by a facility may be data the use of which is restricted to such facility and / or data to which such facility controls access.

[0011] FIG. 2 is a schematic diagram of a conventional distributed learning algorithm.

[0012] The central portion of Figure 2 illustrates a central server 14, a facility 10, and a cohort 12 of data maintained by the facility 10. Arrows 16, 18 represent a cycle of training an AI model using the cohort 12 of data. Arrow 16 represents sending an AI model from the central server 14 to the facility 10. The AI ​​model is trained at the facility 10. Arrow 18 represents returning the trained model parameters to the central server 14.

[0013] The left side of Fig. 2 represents the process performed at the central server 14. Although only one facility 10 is illustrated in Fig. 2, in practice, the AI ​​model is trained at each of multiple facilities 10 to obtain individual trained model parameters 20 from each facility 10. Fig. 2 shows three sets of trained model parameters 20A, 20B, 20C. Such trained model parameters 20A, 20B, 20C are sent to the central server 14.

[0014] The central server 14 obtains a weighted average 22 of the trained model parameters 20A, 20B, 20C. In traditional distributed training algorithms, the trained model parameters 20A, 20B, 20C are weighted by data cohort size: trained model parameters obtained by training on small data cohorts are given a smaller weight in the weighted average 22 than trained model parameters obtained by training on larger data cohorts.

[0015] A weighted average 22 of the trained model parameters from one learning cycle is used to generate a new model that can then be sent to the facilities 10 for further learning. Typically, multiple cycles of model training are performed. In each cycle, the model is sent to each of the facilities 10 for training, and each facility 10 returns a separate set of trained model parameters 20.

[0016] The right side of Figure 2 represents the process carried out at each facility 10. The facilities 10 receive models 30 from the central server 14. Such models 30 received from the central server 14 can be trained and / or applied to patient data.

[0017] 2 shows a training process in which a model 30 is trained on data that is local to the facility 10. The training process outputs a set of feedback parameters 20, which may also be described as "trained parameters" or "model parameters." Such feedback parameters 20 are provided to the central server 14 and combined with parameters from other facilities 10.

[0018] 2 shows a process 32 in which the model 30 is used to evaluate local data and obtain a diagnosis. For example, a model 30 that has been trained using data from multiple facilities 10 can then be used at one facility 10 to analyze patient data.

[0019] A traditional distributed learning algorithm can be expressed in pseudocode as follows:

number

[0020] b g is the bias of the global model. w g are the global model weights. Together, the biases and weights constitute the set of model parameters. Similarly, b iis the bias of the model at facility i, and w i is the model weight at facility i.

[0021] This pseudocode highlights one means of training (SGD). Other distributed training methods can use different model training processes.

[0022] Distributed learning can be important in tasks where large amounts of data cannot be easily transferred to a single location. This can include the medical field, where datasets are typically highly confidential. Due to legal, political, technical, and / or economic reasons, it can be difficult to move a dataset outside the hospital or facility where it was generated.

[0023] Distributed training attempts to train a model without any data transfer. Traditional distributed training has been shown to be successful in training models.

[0024] Traditional distributed learning operates under the assumption that all datasets in a federation are sampled from the same distribution, which is rarely the case: systematic differences between centers often exist, for example due to differences in scanner models, protocols, or judgments.

[0025] Facilities with data cohorts that differ significantly from the majority of data cohorts at other facilities in the association are referred to as "outlying facilities" or "outliers."

[0026] Out-of-range facilities can cause two problems in traditional distributed learning. First, models may perform worse at out-of-range facilities because model parameters learned from other facilities are not all useful to the out-of-range facilities. Second, during the combining phase, parameters returned from out-of-range facilities may contaminate the new model with parameters learned at such out-of-range facilities. Parameters learned at out-of-range facilities may be of limited usefulness to other facilities. Including parameters learned at out-of-range facilities may slow down learning across the entire federation. [Prior art documents] [Patent documents]

[0027] [Patent Document 1] JP 2010-020445 A Summary of the Invention [Problem to be solved by the invention]

[0028] One of the problems that the embodiments disclosed in this specification and in the drawings attempt to solve is improving the accuracy of the model. [Means for solving the problem]

[0029] A medical information processing device according to an embodiment includes a processing circuit configured to: acquire first learned parameters for a model, the first learned parameters being generated by training the model using data from a first data cohort; acquire second learned parameters for the model, the second learned parameters being generated by training the model using data from a second, different data cohort; input the data from the first data cohort into a model having the first learned parameters to determine a first evaluation value; and input the data from the first data cohort into a model having the second learned parameters to determine a second evaluation value. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a schematic diagram showing multiple facilities, each connected to a central server. [Diagram 2] FIG. 1 is a schematic diagram showing model training by conventional distributed training. [Diagram 3] FIG. 1 is a schematic diagram of a system according to an embodiment. [Figure 4] FIG. 1 is a schematic diagram of an apparatus according to an embodiment. [Diagram 5] 1 is a flowchart illustrating an overview of a learning method according to an embodiment. [Figure 6] FIG. 1 is a schematic diagram of a three-fold cross-validation process. [Figure 7] 1 is a flowchart illustrating an overview of a method according to an embodiment. [Figure 8A] Here is a normal MNIST digit. [Figure 8B] Shows brightness-reversed MNIST digits. [Figure 9A] FIG. 13 is a plot of results from a first experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 9B] FIG. 13 is a plot of results from a first experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 9C] FIG. 13 is a plot of results from a first experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 10A] FIG. 13 is a plot of effects obtained in a first experiment using a method according to an embodiment. [Figure 10B] FIG. 13 is a plot of effects obtained in a first experiment using a method according to an embodiment. [Figure 10C] FIG. 13 is a plot of effects obtained in a first experiment using a method according to an embodiment. [Figure 11] 4 shows an example of a Gaussian noise sample. [Figure 12] FIG. 13 is a plot of effects obtained in a second experiment using a method according to an embodiment. [Figure 13A] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13B] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13C] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13D] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13E] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13F] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13G] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13H] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 13I] 13 shows plots of results from a second experiment in which models were trained on MNIST data using a traditional distributed training method and using a method according to an embodiment. [Figure 14] FIG. 1 is a plot in which associations of facilities are plotted in similarity space. [Figure 15] FIG. 13 is a plot showing a federation in which one of the facilities is not connected to the other facilities. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] FIG. 3 is a schematic diagram of a system including a plurality of facilities 40A, 40B, 40C, 40D. Each facility 40A, 40B, 40C, 40D has a respective data cohort 42A, 42B, 42C, 42D. The devices at each of the facilities 40A, 40B, 40C, 40D are connected to each other via a communication network 44. Such a communication network may include any suitable type or types of connections, such as, for example, any suitable wired or wireless connections. For example, such a communication network may form part of the Internet. The system shown in FIG. 3 is an example of a medical information processing system. The devices at each of the facilities 40A, 40B, 40C, 40D are examples of learning devices. The communication network 44 connects the plurality of learning devices.

[0032] FIG. 3 shows an example of four facilities. In other embodiments, any number of facilities can be connected. Such facilities can form a federation to perform the distributed learning method according to the embodiment. In this embodiment, such facilities are hospitals. In other embodiments, at least some of the facilities may be, for example, companies or universities.

[0033] In further embodiments, one or more of the facilities 40A, 40B, 40C, 40D can be replaced by any entity configured to store or use data cohorts. Entities include, for example, parts of facilities, hospitals, universities, corporations, etc. For example, in some situations, each entity may comprise a separate department, system, or device within a single facility. Such entities may be such that the data cohorts maintained by each entity are separate from the data cohorts maintained by other entities.

[0034] An apparatus 50 according to an embodiment is illustrated diagrammatically in FIG. 4. In this embodiment, a separate apparatus 50 is used by each of the facilities 40A, 40B, 40C, 40D. The apparatus 50 is configured to train a model (e.g., a neural network) and to combine trained model parameters obtained from training on the apparatus 50 itself and from training on other apparatuses 50 in other facilities. In this embodiment, the apparatus is also configured to apply the trained model, for example to aid in diagnosis. In other embodiments, different apparatuses can be used to apply the trained model. In further embodiments, any apparatus or combination of apparatuses can be used in each of the facilities.

[0035] The device 50 comprises a computing device 52, in this case a personal computer (PC) or a workstation. The device 50 is connected to at least one scanner 54 via a data store 60. The device 50 is also connected to one or more display screens 56 and one or more input devices 58, such as a computer keyboard, mouse, or trackball. The device 50 is an example of a medical information processing device.

[0036] At least one scanner 54 may comprise any scanner configured to perform medical imaging. The or each scanner 54 is configured to generate medical image data, which may comprise two-dimensional, three-dimensional, or four-dimensional data in any imaging modality. For example, the scanner 54 may comprise a magnetic resonance (MR or MRI) scanner, a CT (computed tomography) scanner, a cone-beam CT scanner, an X-ray scanner, an ultrasound scanner, a PET (positron emission tomography) scanner, or a SPECT (single photon emission computed tomography) scanner.

[0037] Image data sets acquired by the at least one scanner 54 are stored in a data store 60 and then provided to the computing device 52 .

[0038] In this embodiment, data store 60 stores a cohort of training data comprising a plurality of training image datasets, each of which has an associated set of ground truth data. For example, such ground truth data may comprise classification and / or segmentation data. Such ground truth data may have been obtained by manual annotation of the training image datasets or by any suitable automated or semi-automated method.

[0039] In alternative embodiments, the training image data sets and / or other image data sets are sourced from one or more further data stores (not shown), which may form part of a Picture Archiving and Communication System (PACS). The data store 60 or further data stores may comprise any suitable form of memory storage.

[0040] Typically, access to the training image datasets and other image datasets stored in the data store 60 is restricted to a single facility. A device 50 only has access to data from its own facility. For example, a device 50 that has access to data from facility 40A does not have access to data from facilities 40B, 40C, 40D. Further data access restrictions may be implemented. For example, access to the training image datasets and / or other image datasets at a given facility may be restricted to a single device 50 or a group of devices 50 that may be located within such facility. Access may be restricted to specific users. In some circumstances, a device 50 or a group of devices may not have access to other data (e.g., medical records) associated with the training image datasets and / or other image datasets.

[0041] That is, a first data cohort is held by an entity that is local to the medical device, or the first data cohort is comprised of data collected at an entity that is local to the medical device, and a second data cohort is held by a further entity that is remote from the medical device, or the second data cohort is comprised of data collected at a further entity that is remote from the medical device, and data from the second data cohort is not permitted to be provided to the entity that is local to the medical device, or in other words, the medical device does not have the capability to receive data from the second data cohort.

[0042] For example, if the medical information processing device is a device 50 that has access to data from facility 40A, a first data cohort may consist of data collected at a single device 50 or group of devices 50, and a second data cohort may consist of data collected at facilities 40B, 40C, and 40D. Data from the second data cohort may not be permitted to be provided to the single device 50 or group of devices 50, and the device 50 may not be capable of receiving data from the second data cohort.

[0043] The computing device 52 includes a processing unit 62 for processing data, including image data. Such a processing unit may include a central processing unit (CPU) and a graphical processing unit (GPU). In other embodiments, the processing unit may include a CPU without a GPU.

[0044] Processing device 62 includes training circuitry 64 configured to train a model on data from a data cohort of training image datasets stored in data store 60, similarity circuitry 66 configured to determine a measure of similarity between sites and / or data cohorts, aggregation circuitry 68 configured to combine model parameter values ​​to obtain an updated model, and application circuitry 69 configured to apply the trained model to new data. Circuits 64, 66, 68, 69 are examples of processing circuitry.

[0045] In this embodiment, the circuits 64, 66, 68, 69 are each implemented in a CPU and / or GPU by a computer program having computer readable instructions executable to perform the method of the embodiment. In other embodiments, the various circuits may be implemented as one or more ASICs (application specific integrated circuits) or FPGAs (field programmable gate arrays).

[0046] Computing device 52 also includes a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0047] In the embodiment of FIG. 4, data store 60 stores a data cohort comprising medical imaging data acquired from scanner 54. In other embodiments, the data in the data cohort can comprise any type of imaging data, which may not be medical imaging data. In further embodiments, any type of data can be stored by data store 60. Such data may not be imaging data. Such data may not be medical data. Such data may have any suitable number of dimensionalities.

[0048] Returning to Figure 3, each facility 40A, 40B, 40C, 40D has one or more individual devices 50 as described above with reference to Figure 4. Each of the devices has access to an individual data cohort at that individual facility. The data cohort for a facility may comprise data collected by scanners installed at that facility. The data cohort for a facility may comprise data related to patients treated by that facility.

[0049] Typically, devices at a facility only have access to data cohorts related to that facility. Devices do not have access to data cohorts held at other facilities. For example, data transfer between facilities may be restricted for data protection and medical privacy reasons. In some cases, data cohorts may be very large. For example, medical imaging scans may generate very large data files. Thus, it may be difficult or impossible to share or transfer data cohorts between facilities. Devices at a facility may not have the capability to receive data cohorts from another facility.

[0050] In this embodiment, each device is located at its individual facility, in other embodiments, the devices may be located away from their individual facilities but have access to the data cohorts associated with those facilities.

[0051] The devices 50 in the facilities 40A, 40B, 40C, 40D are configured to communicate with each other via the communications network 44. In other embodiments, any suitable number or type of communications network can be used to connect the devices in the facilities 40A, 40B, 40C, 40D. Unlike traditional distributed learning, a central server is not used to communicate with all the devices 50. Instead, the devices 50 can communicate with each other directly over the communications network 44.

[0052] The system of Figure 4 is configured to perform a method that may be referred to as "soft distributive training." Soft distributive training is described below with reference to Figures 5 through 15.

[0053] 5 is a flow chart illustrating an overview of a learning method according to an embodiment, which is performed by devices 50 in a plurality of facilities 40A, 40B, 40C, 40D, which are connected in a network 44 as described above with reference to FIG.

[0054] For example, assume we have multiple data cohorts 42A, 42B, 42C, 42D, as illustrated in FIG. 3. We want to compare the similarities of these cohorts 42A, 42B, 42C, 42D to each other without transferring data. The similarities then provide the influence each facility receives from each other facility during model aggregation. Methods for calculating and aggregating such influences are described below.

[0055] Similarity is calculated by training models at each facility 40A, 40B, 40C, 40D and testing these models at all facilities 40A, 40B, 40C, 40D (including the facilities at which the models were trained). Performance scores are obtained by performing the models above random chance accuracy. After such performance scores are calculated, the scores are normalized for each facility to obtain an impact value. Because facilities change over time, such impact values ​​can be recalculated at intervals to ensure that the values ​​are up to date.

[0056] Prior to the start of the process represented by the flowchart of FIG. 5, a model is provided to each facility 40A, 40B, 40C, 40D. Such provided model can be referred to as a "global model." The same model is provided to each facility 40A, 40B, 40C, 40D. Such model can be stored or generated at a first one of such facilities (e.g., 40A) and provided by the first one of such facilities 40A to the other facilities 40B, 40C, 40D. In some embodiments, the model is generated at an external facility that is not part of the federation. The model can be pre-trained at the external facility on a data cohort that is similar to the data cohorts of at least some of the facilities 40A, 40B, 40C, 40D in the federation.

[0057] The model may comprise any suitable deep learning or artificial intelligence model, such as, for example, a neural network, such as a convolutional neural network.

[0058] A model is described by a set of model parameters. In this embodiment, the set of model parameters comprises a set of biases and a set of weights. In the following description, references to a model may be interchangeable with references to a set of model parameters. For example, references to a model that is transferred between facilities may be considered as references to a set of model parameters that are transferred between facilities.

[0059] The model may be intended to be trained to perform any suitable task or tasks, such as, for example, one or more tasks to be used in clinical practice. For example, the model may be intended to perform any suitable classification or regression. In the context of medical imaging, the model may be configured to perform, for example, classification and / or segmentation, etc. The model may be configured to obtain location and / or bounding boxes. The model may be used, for example, in Computer Aided Detection (CADe) systems, Computer Aided Diagnosis (CADx) systems, Clinical Decision Support Systems (CDSS), etc. to assist in diagnosis in medical institutions.

[0060] At stage 70 of FIG. 5, a model is trained at each of the facilities 40A, 40B, 40C, 40D.

[0061] Consider the case of facility 40A. A learning device 50A at facility 40A receives data from a data cohort 42A maintained by facility 40A. Such data may comprise a subset of the data maintained in data cohort 42A. In this embodiment, data cohort 42A comprises scanner data and ground truth data (e.g., ground truth classification and / or ground truth segmentation).

[0062] The learning circuit 64A of the learning device 50A provides data to the model and performs a model learning process for the purpose of learning such a model to generate a desired output. Any suitable model learning process may be used. For example, stochastic gradient descent may be used. In other embodiments, Adadelta, Adam, or Nadam learning methods may be used.

[0063] The output of the model training process is a set of learned parameters 20A for the model.

[0064] At stage 71, the learning device 50A stores a copy of the learned parameters 20A obtained by training the model on data received from its own facility's data cohort. The learning device 50A also sends copies of the learned parameters 20A obtained by training the model on its own facility's data cohort to devices at each of the other facilities 40B, 40C, and 40D. For example, if the medical information processing device is a device 50 that has access to data from the facility 40A, the data cohort 42A is an example of a first data cohort, and the learned parameters 20A are an example of a first trained parameter. That is, the learning device 50A obtains a first trained parameter for the model, and the first trained parameter is generated by training the model using data from the first data cohort.

[0065] Corresponding training processes for the models are performed by devices 50B, 50C, 50D at each of the other facilities 40B, 40C, 40D, each training process resulting in a separate set of trained model parameters 20B, 20C, 20D.

[0066] The learning device 50A in the first facility 40A receives copies of the learned parameters 20B, 20C, 20D obtained by learning the model at that facility from the devices in each of the other facilities 40B, 40C, 40D. Similarly, the learning devices 50B, 50C, 50D in the other facilities also receive the learned parameters previously obtained at each of the other facilities.

[0067] Training the model on the different data cohorts 42A, 42B, 42C, and 42D is expected to result in different trained parameters 20A, 20B, 20C, and 20D being obtained. The different data cohorts may have different numbers of data samples, such as data from different numbers of scans. The data held in the different data cohorts may differ, for example, in terms of the scanner type and / or scanner settings and / or scanner protocols used. For example, if the medical information processing device is a device 50 having access to data from a facility 40A, each of the different data cohorts 42A, 42B, 42C, and 42D is an example of a second different data cohort. Also, each of the trained parameters 20B, 20C, and 20D is an example of a second trained parameter. That is, the training device 50A obtains second trained parameters for the model, and the second trained parameters are generated by training the model using data from the second different data cohort.

[0068] Two facilities can be considered similar if a model trained at one of them produces good results at the other facility.

[0069] At stage 72, the similarity circuit 66A performs a similarity determination procedure to obtain a similarity between the model trained at the first facility 40A and the models trained at each of the facilities 40A, 40B, 40C, 40D, which includes pairing the first facility with itself.

[0070] Devices 50B, 50C, 50D in the other facilities perform a similar similarity determination procedure to obtain similarities between all pairs of facilities, including pairings with themselves.

[0071] The case of a comparison between a first facility and a second, different facility can be considered the normal case. Consider a similarity circuit 66A of a learning device 50A at a first facility 40A. The similarity circuit 66A receives trained parameters 20B from a second facility 40B. The trained parameters 20B received from the second facility 40B were previously obtained by training a model using data from a data cohort 42B held by the second facility 40B.

[0072] The similarity circuit 66A generates a model using the learned parameters 20B from the second facility 40B. The similarity circuit 66A tests the model using data from a data cohort 42A maintained by the first facility 40A. Such data used for testing may be a subset of the data cohort 42A.

[0073] The similarity circuit 66A determines a performance score, which represents a degree of accuracy in the execution of the model. Such a degree of accuracy is determined by comparing the output of the model to the ground truth. For example, the percentage of correct predictions by the model for a given task may be evaluated against the ground truth.

[0074] The performance score is an independent measure of accuracy. The performance score is an independent measure of how accurate a model is.

[0075] Similarly, similarity circuitry 66B of device 50B at site 40B tests model parameters 20A from the model trained on data cohort 42A at site 40A.

[0076] In this embodiment, training and testing are performed using three-fold cross-validation. In other embodiments, any number of folds can be used. The results of such folds are averaged to give a final performance score.

[0077] The training circuitry 64A at the first facility trains on all but one split of its data cohort 42A. The similarity circuitry 66B at the second facility receives the trained parameters 20A from the first facility and tests such parameters on one of the splits of its data cohort 42B. This is repeated until all splits of the data cohort 42B have been used for testing.

[0078] The second facility's training circuitry 64B trains on all but one partition of its data cohort 42B. The first facility's similarity circuitry 66A receives the trained parameters 20B from the second facility and tests such parameters on one of the partitions of its data cohort 42A. This is repeated until all partitions of the data cohort 42A have been used for testing.

[0079] For each fold, a performance score is given depending on its accuracy in testing, which is then averaged across all three folds to get a final performance score.

[0080] Moving to the case where a model trained at one facility is being compared to itself, for example, the similarity procedure is applied to a model trained by device 50A at facility 40A using test data from cohort 40B at facility 40A. The case of comparing facility 40 to itself can be considered a special case.

[0081] The training data cohort 42A in the facility 40A is divided into several partitions. The number of such partitions is at least 2. In this embodiment, the training data cohort is divided into three partitions.

[0082] The training circuit 64A and similarity circuit 66A train and test on the data 42A using n-fold (three-fold in this embodiment) cross-validation. The training circuit 64A trains all but one fold. The similarity circuit 66A receives the trained parameters 20A from the training circuit 64A and tests on the excluded fold. This is repeated until all folds have been used for testing.

[0083] For each fold, a performance score is obtained depending on its accuracy in the test, which is then averaged across all folds to give a final performance score.

[0084] 6 is a schematic diagram of an example of training and testing at sites 40A and 40B using three-fold cross-validation. Similar training and testing can be performed with each pair of sites.

[0085] Facilities 40A and 40B are represented by triangles. In the example shown, facility 40A has a data cohort 42A with 900 training cases. Facility 40B has a data cohort 42B with 30 training cases. Each arrow represents one training / evaluation run for soft distributed training.

[0086] A three-fold evaluation divides each data cohort into three equal subdivisions (more generally: an n-fold would be divided into n subdivisions). The model is then trained on two (n-1) of these subdivisions and tested on one (the one not used for training, in the case of same-site evaluation). Each subdivision uses two different training and evaluation subdivisions.

[0087] The three arrows 80 represent three cases of training the model at a first site 40A and testing at a second site 40B. In each case, the model is trained on two of the splits of data cohort 42A, so the model is trained on 600 samples. The model is tested on one split of data cohort 42B, so the model is tested on 10 evaluation samples.

[0088] Three further arrows 82 represent three cases of training the model at the second site 40B and testing at the first site 40A. In each case, the model is trained on two of the splits of data cohort 42B, so the model is trained on 20 samples. The model is tested on one split of data cohort 42A, so the model is tested on 300 evaluation samples.

[0089] The three arrows 84 represent three cases of training the model at a first site 40A and testing at the same site 40A. In each case, the model is trained on two of the splits of the data cohort 42A, so the model is trained on 600 samples. The model is tested on one split of the data cohort 42A, so the model is tested on 300 evaluation samples.

[0090] The three arrows 86 represent three cases of training the model at a second site 40B and testing at the same site 40B. In each case, the model is trained on two of the splits of the data cohort 42B, so the model is trained on 20 samples. The model is tested on one split of the data cohort 42B, so the model is tested on 10 evaluation samples.

[0091] The process as shown in Figure 6 is repeated for each pair of facilities.

[0092] At stage 73, each and every one of the facilities 40A, 40B, 40C, 40D determines a set of performance scores, where each performance score represents the degree of similarity between that facility and each other of the facilities in the network (including itself).

[0093] Once all the performance scores have been calculated, one can think of there being an m by m matrix of performance scores, where m is the number of centers, since every pair of centers has a score. Note that in general, this matrix will not be symmetric. The performance scores of a model trained at a first center and applied to a data cohort at a second center may differ from the performance of a model trained at the first center and applied to a data cohort at the first center.

[0094] Note that in practice no facility may maintain the entire matrix of performance scores: each facility may maintain only the performance scores that are relevant to that facility.

[0095] That is, the similarity circuit of the learning device at each facility inputs data from a first data cohort into a model having first learned parameters to determine a first evaluation value, and inputs data from the first data cohort into a model having second learned parameters to determine a second evaluation value. For example, if the medical information processing device is device 50 having access to data from facility 40A, the first evaluation value is the similarity (performance score) between facility 40A and facility 40A itself, and the second evaluation value is the similarity (performance score) between facility 40A and the other facilities (40B, 40C, 40D).

[0096] At stage 74, the performance scores are used to generate a set of impact values, which may also be referred to as "ratings values."

[0097] Consider one device 50A in one facility 40A. The similarity circuit 66A stores individual performance scores for each of the facilities 40A, 40B, 40C, 40D. The similarity circuit 66A converts each performance score into an accuracy above random chance. The performance scores are an independent measure of performance. Accuracy above random chance is a relative measure. It is relative to the performance that would be achieved by random chance. Random chance may also be called random chance accuracy.

[0098] To convert a performance score into accuracy above random chance, the similarity circuit first determines the random chance for the task the model is configured to perform. For example, consider a classification task. The random chance is the highest percentage of a particular class in the dataset. In a binary classification task where there are two classes that appear equally frequently, the random chance would be 0.5. On the other hand, if there is one class that is 90% of all classes, the random chance would be 0.9.

[0099] If the performance score is below random chance, the accuracy of being above random chance as determined is 0.

[0100] If performance exceeds random chance, accuracy above random chance is determined as follows. Accuracy above random chance = (PR) / (1-R) where P is the performance score and R is the random chance.

[0101] Similarity circuit 66A then normalizes all of the retained performance scores so that each accuracy above random chance is a value between 0 and 1. Such normalization is such that for each site, the accuracy above random chance values ​​for models tested at that site sum to 1.

[0102] The normalized measure of accuracy above random chance is called the "influence value" or "rating value."

[0103] The influence value can be thought of as a measure of how useful a model trained at one facility is to another facility (or to itself). Models that are considered useful have a high influence value. Models that are considered not useful have a low influence value.

[0104] That is, the similarity circuit determines a degree of similarity between the first data cohort and the second data cohort based on the determined first evaluation value and second evaluation value. The first evaluation value represents the performance of a model having first learned parameters when applied to data from the first data cohort. The second evaluation value represents the performance of a model having second learned parameters when applied to data from the first data cohort. The performance represented by the evaluation value comprises an accuracy of the model compared to random chance. Accuracy above random chance is an example of a degree of similarity between the first data cohort and the second data cohort and an accuracy of the model compared to random chance.

[0105] It is again emphasized that different facilities will have different sets of performance scores that will be normalized to produce different impact values. For example, a given set of trained model parameters may be more useful for a first facility than for a second facility if the first facility is more similar to the facility where the model parameters are trained. In some situations, a facility may find that a model trained at another facility is more useful than a model trained locally at that facility. For example, the other facility may train a model on a larger dataset.

[0106] Turning again to FIG. 3, FIG. 3 shows a set of arrows 46. Each of the arrows represents an individual influence value, with thicker arrows representing higher influence values. From FIG. 3, it can be seen that each of the facilities has an influence value for itself. It can also be seen that the influence value from a first facility to a second facility may be different from the influence value in the opposite direction from the second facility to the first facility. For example, arrow 46E from the first facility 40A to the second facility 40B is thinner than arrow 46F from the second facility 40B to the first facility 40A. The influence value for the model trained on 40B when applied to 40A is higher than the influence value for the model trained on 40A when applied to 40B.

[0107] At stage 75, the aggregation circuitry 68 at each facility runs a distributed learning algorithm using the influence values ​​obtained at stage 108. Such a distributed learning algorithm is run separately at each facility.

[0108] The distributed learning algorithm includes an aggregation step 76, in which the aggregation circuit 68 aggregates the model parameters obtained from each of the facilities. The aggregation comprises weighting the model parameters from each of the facilities according to their influence values. That is, the aggregation circuit 68 obtains an updated model by combining the first learned parameter with the second learned parameter, the combination of the first learned parameter with the second learned parameter being weighted according to the first evaluation value and the second evaluation value.

[0109] In the embodiment of Figure 5, there is no explicit weighting of model parameters by the number of training cases each model has previously trained on. There are cases where models trained on more training cases may have more useful parameters and therefore a higher influence value. However, in some situations, models trained on more training cases may not have a high influence value, for example when the training facility uses a different scanner than the facility where the aggregation is being performed.

[0110] By aggregating using influence values, it can be ensured that new models formed at each facility are specialized to work at that facility, and such specialization may lead to higher performance.

[0111] The output of Figure 5 is an individual updated model for each facility. At each facility, an updated model is formed by aggregating the model parameters from all of the facilities according to their influence values.

[0112] FIG. 5 shows only one learning cycle. In practice, multiple learning cycles can be performed. For example, in an embodiment, each facility sends its updated model to all of the facilities for learning. Each of the facilities returns further model parameters previously obtained by training the updated model on data held at such facility. Each facility tests the received model parameters on its own data, as described above with reference to stage 72, to obtain updated influence values. The updated influence values ​​are then aggregated. Such a cycle can be repeated, for example, multiple times until the model converges. For example, if the medical information processing device is device 50 having access to data from facility 40A, device 50A is configured to iteratively repeat the learning cycle. The learning cycle comprises sending the updated model to at least one device different from device 50A, obtaining further learned parameters by further training the model, testing the further learned parameters and learned parameters obtained from devices different from device 50A, and obtaining a further updated model by combining the further learned parameters.

[0113] FIG. 7 is a further flow chart illustrating the embodiment of FIG.

[0114] At stage 90, the learning circuit 64 of the device 50 receives the model. At stage 92, the learning circuit 64 trains the model on data from a data cohort held local to the device 50 to obtain a set of model parameters. At stage 94, the similarity circuit 66 receives the model parameters trained on the device 50 and also receives sets of model parameters trained for devices at other facilities. At stage 96, the similarity circuit 66 tests all of the model parameters on data from the local data cohort. At stage 98, the similarity circuit 66 processes the results of the tests to obtain an influence value for all of the facilities. At stage 100, the aggregation circuit 68 aggregates the model parameters using weights that depend on the influence values. At stage 102, the aggregation circuit 68 outputs an updated model.

[0115] At stage 104, the training circuitry 64 determines if training is complete. For example, the training circuitry 64 may determine a measure of convergence. If training is not complete, the flow chart returns to stage 90. The training circuitry 64 receives an updated model for training.

[0116] When training is complete, training circuitry 64 outputs the trained model at stage 106. At stage 108, application circuitry 69 applies the trained model to new data, which may be data that is not part of the training data on which the model was previously trained. Such new data may comprise any suitable imaging data set or sets.

[0117] When applied to a new dataset, the trained model performs the task it learned to perform, such as classification and / or segmentation and / or location. A clinician can use the results of the trained model to obtain information about the imaging dataset. Such a clinician can use the results of the trained model to assist in diagnosis.

[0118] It may be expected that some aspects of the facility may change over time. It may therefore be beneficial to re-run the model training process of Figures 5 and 7 at intervals to update the model parameters and / or weights. At stage 110 of Figure 7, the training circuit 64 begins the model training process again. Several training cycles may be performed, for example, according to stages 90 to 104 as described above. A new trained model can be generated by the model training process of stage 110.

[0119] The impact values ​​from a given facility may change over time, for example due to new scanners, so model training using a soft distributed training method may be performed at regular intervals, for example at six month or yearly intervals, to recalculate the impact values.

[0120] In some embodiments, model learning is performed each time a facility leaves or joins a federation (group, entity) or when the facility undergoes a major change (e.g., a new scanner).

[0121] By using the soft distributed learning model described above with reference to Figures 5 and 7, the trained model obtained at each facility may be more appropriate for that facility than a model trained centrally using traditional distributed learning. The resulting trained model may have better performance in the learning task. The trained model can be tailored to an individual facility. More weight can be given to models trained on other similar facilities than models trained on dissimilar facilities.

[0122] Soft distributional learning methods do not assume that all data comes from the same distribution. Soft distributional learning methods take into account differences between different data cohorts. Outliers may have a reduced effect on the distributional learning process.

[0123] Generally, medical data is too sensitive to be easily transferred to a central location. Distributed learning algorithms allow data to remain at a facility. Data cohorts from different facilities typically have systematic differences that are not taken into account in traditional distributed learning. Because medical data from different hospitals typically may have systematic differences, traditional distributed learning may not provide good performance when such differences are significant.

[0124] Soft distributed learning may be considered to modify traditional distributed learning algorithms and correct for differences between centers.

[0125] The soft distributed learning algorithm is deployed across multiple facilities (e.g., hospitals) and used to compare data cohorts between facilities. Such comparison results in an influence value between facilities. A traditional distributed learning algorithm may be considered to be modified according to an influence measure. Instead of a facility receiving equal weighting from all facilities, the weighting is now adjusted by the influence value. The soft distributed learning method can eliminate the need for a central server during use. The soft distributed learning method can protect against compromised or malfunctioning facilities. The soft distributed learning method can improve performance at outlier facilities.

[0126] The soft distributed training method of Figures 5 and 7 differs from traditional distributed training algorithms in that all models are sent to all facilities instead of to a central server. In the methods of Figures 5 and 7, no central server is used.

[0127] The advantages of removing the central server may be related to control and / or privacy. Facilities may no longer depend on a third party (the central server) functioning (and functioning correctly) for distributed learning to occur. Facilities can communicate directly with each other without going through a central server. Since no third parties receive the models, such model data remains entirely within a group of facilities, e.g., a hospital federation. The absence of a third party receiving the models may result in fewer privacy concerns.

[0128] Model aggregation is done locally at the facility, which is an environment where end users have more control compared to central servers in different unknown locations.

[0129] This can come at a cost in computational complexity: since every model is sent to every other facility during soft distributed training, it takes O(m 2 ) algorithm, where m is the number of facilities. However, this computational cost is now shared among m facilities. Moreover, if the influence between two facilities is zero, there is no need to transfer such a model between facilities (as it will be weighted by zero). This is comparable to the computational cost of O(m) for traditional distributed learning.

[0130] Calculating the impact of every pair of facilities requires training the model in O(m) time (divided among m facilities) and O(m 2 ) testing costs (again divided among m facilities) can be costly. Traditional distributed training does not feature this step.

[0131] 5 and 7, a prototype has been created to test the proposed method described above. The prototype has been applied to illustrative test data from the Modified National Institute of Standards and Technology (MNIST) dataset, which comprises images of handwritten digits. The MNIST dataset is often used in machine learning as a training and testing dataset.

[0132] In experiments conducted, results show that soft distributed learning outperforms traditional distributed learning, and such experiments and their results are described below with reference to Figures 8A through 13.

[0133] The first experiment concerns the difference in cohort size and domain. In this first experiment, data from the MNIST dataset is used to simulate data cohorts at three sites. The first site (site 1) has a data cohort of 30 data samples. The second site (site 2) has a data cohort of 3000 training samples. The third site (site 3) has a data cohort of 3000 training samples. To make the third site an outlier, the training samples of the third site are inverted in intensity. FIG. 8A shows an example of a normal MNIST image. FIG. 8B shows an intensity-inverted version of the MNIST image of FIG. 8A.

[0134] The data cohorts in the first experiment are constructed such that the smaller site (site 1) has less influence on the larger site (site 2) than vice versa. The data cohorts are constructed such that site 3 is an outlier.

[0135] A simulated training process is performed that includes using simulated data cohorts at three sites to train and test models using soft distributed training as described above with reference to FIG. 5. In each of a number of training cycles, models are trained by each of the sites. Each site then tests all of the trained models and determines an individual impact value for each of the trained models.

[0136] 9A, 9B, and 9C are plots showing accuracy (as a percentage) on validation data versus number of learning cycles. The sites are labeled Site 1, Site 2, and Site 3.

[0137] Figure 9A relates to a first facility, facility 1. Line 111 shows the performance of traditional distributed learning.

[0138] Line 112 shows the performance of the global pooling method, where a single model is trained on all of the data. The data at all sites is pooled in one place, and then the model is trained on this pooled data. Line 112 is the same in Figures 9B and 9C because the global pooling evaluation is a single evaluation, rather than one evaluation for each site.

[0139] Line 114 shows local accuracy, which is the performance of models trained within site 1 without model aggregation or sharing. Line 113 shows ensemble accuracy, which is the average performance of the set of models, where the set includes locally trained models for all three sites.

[0140] Line 115 shows the performance of soft distributive learning as described above with reference to Figures 5 and 7. The shaded areas around the line represent error bars.

[0141] In Fig. 9A, the local accuracy is relatively low due to the small amount of training data at such sites. The two distributed methods (soft distributed training 115 and traditional distributed training 111) perform well, showing their strengths in small sites. Soft distributed training performs better than traditional distributed training because it recognizes site 3 as an outlier.

[0142] 9B relates to a second facility, facility 2. Line 116 shows the performance of traditional distributive learning. Line 112 shows the performance of global pooling. Line 118 shows the ensemble accuracy. Line 119 shows the local accuracy for facility 2. Line 120 shows the performance of soft distributive learning.

[0143] 9C is for a third facility. Line 121 shows the performance of traditional distributive learning. Line 112 shows the performance of global pooling. Line 123 shows the ensemble accuracy. Line 124 shows the local accuracy for facility 3. Line 125 shows the performance of soft distributive learning.

[0144] 10A, 10B, and 10C show the amount of influence each site received from the other sites in the first experiment.

[0145] Figure 10A shows the influence values ​​calculated by site 1, which is a site with a training sample of 30. Figure 10A shows that site 1 receives a medium amount of influence from itself (bar 130), a large amount of influence from site 2 (bar 132), and a small amount of influence from site 3 (bar 134).

[0146] It is predicted that site 1 will be highly influenced by site 2, which is a site with 3000 data samples. A model trained on 3000 data samples is predicted to be more useful when the samples are generally similar compared to a model trained on 30 data samples.

[0147] It is also predicted that site 1 will be less influenced by site 3. Site 3 is an outlier because its data samples are intensity inverted.

[0148] Moving to Figure 10B, which shows the amount of influence experienced by site 2. Site 2 is influenced by a relatively small amount from site 1 (bar 136), which may be expected due to the relatively small cohort size at site 1. Site 2 is influenced by a large amount from itself (bar 138). Site 2 is not influenced at all by site 3, which is an outlier.

[0149] Turning to Figure 10C, which depicts the impact of site 3, it can be seen that site 3 is only impacted by itself (bar 140). It is expected that site 3, having a large cohort of brightness-inverted samples, may find its own trained model more useful than those of sites 1 and 2, which train their models on non-inverted samples.

[0150] It can be concluded that in experiment 1, the soft coalition learned that site 3 is an outlier and therefore has minimal influence over sites 1 and 2. This offers an improvement over traditional distributed training. Traditional distributed training typically uses cohort size to weight combinations of model parameters. This weighting does not take into account differences in the type of data in the cohorts (here, intensity-reversed and non-inverted). Traditional distributed training can be adversely affected by outlier sites.

[0151] In the first experiment, site 2 is less influenced by site 1 than site 1 is by site 2. This is due to the size difference between the sites, ensuring that the relative sizes are captured during influencing. This is the strength of the soft distributed learning in the first experiment.

[0152] In experiment 1, the model trained on site 1 is slightly overfitted on its own data since it does not have many training examples. This leads to poor performance when tested on site 2. On the other hand, the model for site 2 has a lot of training data and therefore performs well on site 1.

[0153] The results in Figures 9A to 9C show that soft distributed learning narrowly outperforms traditional distributed learning at site 1. Performance is similar at sites 2 and 3.

[0154] A second experiment was performed. The second experiment can be described as a noise test. In the second experiment, nine facilities were simulated. The first facility had a cohort of 30 MNIST training samples, each with an image of a handwritten digit. The data cohort provided to the other eight simulated facilities each had 30 individual examples. The 30 examples had no signal, only Gaussian noise. That is, each of the 30 examples did not include an image of a digit, but instead included only noise. Figure 11 shows an example of a Gaussian noise sample. In such a Gaussian noise sample, no digits are present.

[0155] The goal of the second experiment was to determine whether soft distributed learning could recognize when a facility had useless data and ignore facilities with useless data.

[0156] The training procedure was carried out for nine simulated facilities according to the method described above with reference to Figures 5 and 7. Figure 12 is a plot of the influence values ​​for the first facility, which was the facility with the cohort of MNIST samples. Facility 1 has an influence value of 1 (bar 150) for its own trained model and no influence value for any of the models trained at the other facilities. The influence values ​​for the other facilities were found to be random and are not shown here.

[0157] Figure 13 includes nine plots (Figures 13A-13I), each plot showing the accuracy (as a percentage) of the validation data versus the number of learning cycles for a separate one of the nine simulated sites of the second experiment.

[0158] Plot 160 (FIG. 13A) is for the first facility, which is the facility with the MNIST data cohort. Plots 170 (FIG. 13B), 172 (FIG. 13C), 174 (FIG. 13D), 176 (FIG. 13E), 178 (FIG. 13F), 180 (FIG. 13G), 182 (FIG. 13H), 184 (FIG. 13I) each are for a separate one of the other facilities, which can be described as "noisy facilities." It can be seen that all the noisy facilities have low results. This is expected since there is no signal in those data samples.

[0159] In plot 160 (FIG. 13A), line 161 represents the performance of traditional distributive learning. Line 162 represents the performance of global pooling. Line 163 represents the ensemble accuracy. Line 164 represents the local accuracy. Line 165 represents the performance of soft distributive learning.

[0160] At site 1 (plot 160, FIG. 13A), line 165 representing soft distributed learning is much higher than line 161 representing traditional distributed learning, proving that the use of influence leads to improved performance. In a second experiment, shown in FIGS. 13A-13I, soft distributed learning performs much better than traditional distributed learning due to the recognition of other sites as outliers. This highlights the strength of soft distributed learning over traditional distributed learning.

[0161] Line 164 is the performance of a model trained only on local data, which is slightly better than using a soft-distributed approach. In the first experiment, it was found that the model trained only on local data performed much worse than the soft-distributed approach.

[0162] In various further embodiments, models can be trained at different facilities to obtain different sets of model parameters. The model parameters can be used to determine the degree of similarity between the facilities and / or their data cohorts. For example, influence values ​​can be determined as described above. A high influence value may indicate a high degree of similarity. A lower influence value may indicate a lower degree of similarity. In some circumstances, the similarity values ​​can be determined without subsequently using the similarity values ​​to weight the set of trained models.

[0163] In many cases, influence values ​​are not symmetric. For example, the influence value for a model parameter trained at site A and tested at site B may not be the same as the influence value for a model parameter trained at site B and tested at site A. In some embodiments, the influence values ​​for pairs of sites can be combined, e.g., averaged, to obtain a measure of similarity between the sites.

[0164] In some circumstances, it may be expected that a pair of facilities may be similar. For example, such facilities may have similar scanners and similar patient populations. If a measure of similarity between facilities is found to be low, such low similarity may indicate problems with one or both of the facilities, such as, for example, problems with the scanners, and / or data processing methods, and / or model learning methods.

[0165] In other circumstances, pairs of facilities are expected to differ, for example, they may use different scanners or different scanning methods, and other differences in the data cohorts may exist.

[0166] The similarity between facilities can be plotted in a similarity space. The matrix of influence values ​​between all facilities can be projected into a two-dimensional similarity space through dimension reduction techniques. Figure 14 shows multiple facilities 190 as points in a similarity space. A similarity space is a space where the distance between the points represents the similarity between the facilities. The similarity space is all relative and has any scale. The relative distance between the facilities indicates the dissimilarity between them.

[0167] Facilities that are closer to each other in similarity space are more similar. Facilities that are farther from each other in similarity space are more dissimilar. A facility that is not close to any of the other facilities in similarity space may be an outlier.

[0168] In Figure 14, a link is drawn between a pair of facilities if the influence value between the two facilities is non-zero. The link can indicate that the two facilities have data cohorts that are similar enough to receive models from each other.

[0169] A representation of the facilities in similarity space (eg, the plot of FIG. 14) can be displayed to the user in some circumstances.

[0170] The group of facilities used to train a common model can be called a "federation."

[0171] FIG. 14 also depicts the disconnection of facility 190Y from the Union.

[0172] In some circumstances, one or more facilities 190 may be removed from the federation. To remove a facility, the facility's point in the similarity space is removed and all connections to that facility are removed. Models within the removed facility can continue to learn locally, without being influenced or affected by other facilities in the federation.

[0173] This may result in some facilities being completely disconnected from the federation. In the example of Figure 14, if the lowest-ranked facility 190X is removed, facility 190Y will be disconnected and removed from the federation. Such facility 190Y does not have a non-zero influence value with any facility 190 other than 190X.

[0174] FIG. 15 represents a new facility being added to an existing federation. The location of the new facility in the similarity space is found by determining the influence value with all other facilities. The new facility is linked in the similarity space to facilities with which it has a non-zero influence value. If the new facility is connected to at least one other facility, it becomes part of the federation. If the new facility is not connected to any of the other facilities, it is rejected. In FIG. 15, facility 190Z is not connected to any of the other facilities and is therefore rejected.

[0175] When a new facility is added to the federation, it receives the model using the normal method when the current distributed learning cycle is completed, and then participates in the subsequent distributed learning cycle.

[0176] In some situations, there may be facilities that are believed to behave as rogue facilities. A rogue facility may be one whose data or models do not look like they should. This may be due to errors in the software or hardware used, incompatibility issues, or simply human misuse of the software (either accidental or intentional), leading to invalid ground truth or assessments.

[0177] A dishonest institution in traditional distributed learning can be a serious risk. Such an institution can provide wrong parameters that ruin what the federation is trying to learn. This has a particular problem when the institution is a large institution, such as a large hospital that generates a lot of data. In traditional distributed learning, a large institution with a lot of data is weighted highly during the combination of models.

[0178] Under soft distributive learning, dishonest establishments are quickly identified as outliers in the similarity space. They will have low influence values. They can be disassociated from the federation.

[0179] Certain embodiments provide a method for training a model, comprising receiving a first training model generated based on a first training dataset collected from a first cohort and a second training model generated based on a second training dataset collected from a second cohort, and calculating a first evaluation value and a second evaluation value by inputting the datasets collected from the first cohort into the first training model and the second training model.

[0180] The first learning model and the second learning model can be synthesized based on the first evaluation value and the second evaluation value.

[0181] Weights of the first and second learning models can be calculated based on the first and second evaluation values, and the combined procedure can be performed based on the weights.

[0182] Certain embodiments provide a system comprising a plurality of facilities, each equipped for machine learning, each having access to a cohort of data, connected in a communications network, the facilities communicating updates to their learned parameters, the updates being dependent on the training cohort at each facility.

[0183] Parameter updates may be a function of influence between cohorts. Influence may be a function of the performance of a model trained at one center and tested at another center or on itself. There may be no need for a central pooling server. New centers can be added or removed at any time.

[0184] Although particular circuits have been described herein, in alternative embodiments, the functionality of one or more of these circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. A reference to a single circuit encompasses multiple components that provide the functionality of that circuit, regardless of whether such components are separate from one another, and a reference to multiple circuits encompasses single components that provide the functionality of those circuits.

[0185] Although certain specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the present invention. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Moreover, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the present invention. The appended claims and their equivalents are intended to cover such forms or modifications as fall within the scope of the present invention. [Explanation of symbols]

[0186] 10, 10A…Facilities 12, 12A…Cohort 14…Central server 20…Feedback parameters 20A, 20B, 20C…Model parameters 22…Average 30…Model 32…Process 40A, 40B, 40C, 40D…Facilities 42A, 42B, 42C, 42D… Data cohort 44…Communications network 50...device 52...Calculating device 54…Scanner 56...Display screen 58…Input device 60…Data store 62... Processing equipment 64…Learning circuit 66…Similarity circuit 68…Aggregation circuit 69…Applicable circuit

Claims

1. obtaining first learned parameters for a model, the first learned parameters being generated by training the model using data from a first data cohort; obtaining second learned parameters for the model, the second learned parameters being generated by training the model using data from a second, different data cohort; determining a first evaluation value by inputting the data from the first data cohort into a model having the first trained parameters; determining a second evaluation value by inputting the data from the first data cohort into a model having the second trained parameters; obtaining an updated model by combining the first learned parameters and the second learned parameters; A medical information processing device comprising a processing circuit configured to perform the above.

2. The medical image processing apparatus according to claim 1 , wherein the combination of the first learned parameter and the second learned parameter is weighted according to the first evaluation value and the second evaluation value.

3. 2. The medical information processing device of claim 1, wherein the processing circuitry is further configured to determine a degree of similarity between the first data cohort and the second data cohort based on the determined first evaluation value and the determined second evaluation value.

4. 2. The medical information processing device of claim 1, wherein the first evaluation value represents a performance of the model having the first learned parameters when applied to the data from the first data cohort, and the second evaluation value represents a performance of the model having the second learned parameters when applied to the data from the first data cohort.

5. The medical image processing apparatus of claim 4 , wherein the performance represented by each evaluation value comprises an accuracy of the model compared to random chance.

6. The medical information processing device of claim 1 , wherein the first data cohort is maintained by an entity local to the medical information processing device, and / or the first data cohort is comprised of data collected at an entity local to the medical information processing device.

7. The medical information processing device of claim 6, wherein the second data cohort is held by a further entity remote from the medical information processing device and / or the second data cohort is comprised of data collected at a further entity remote from the medical information processing device.

8. The medical information processing device of claim 6 , wherein the data from the second data cohort is not permitted to be provided to the entity local to the medical information processing device.

9. The medical information processing device of claim 6 , wherein the medical information processing device is incapable of receiving data from the second data cohort.

10. The medical image processing device is configured to repeatedly perform a learning cycle; The medical image processing device of claim 2, wherein the learning cycle comprises: transmitting the updated model to at least one further medical image processing device; training the updated model to obtain further learned parameters; testing the further learned parameters with further learned parameters from the at least one further medical image processing device; and obtaining a further updated model by combining the further learned parameters.

11. a training circuit for obtaining first learned parameters for a model, the first learned parameters being generated by training the model using data from a first data cohort; a training circuit for obtaining second learned parameters for the model, the second learned parameters being generated by training the model using data from a second, different data cohort; a similarity circuit determining a first evaluation value by inputting the data from the first data cohort into a model having the first learned parameters; a similarity circuit determining a second evaluation value by inputting the data from the first data cohort into a model having the second learned parameters; an aggregating circuit combining the first learned parameters and the second learned parameters to obtain an updated model; A medical information processing method comprising:

12. a plurality of learning devices, each associated with a distinct entity and having access to a cohort of distinct data held by the entity; a communication network connecting the plurality of learning devices; A system comprising: The learning device for each entity is training a model on data from the cohort of data for the entity, thereby obtaining a set of trained parameters; receiving further trained parameters from other learning devices, the further trained parameters having been obtained by training the model on data from a cohort of the individual data held by each other entity; inputting the data from the cohort of data into a model having the set of trained parameters to determine an individual evaluation value for each set of trained parameters; obtaining an updated model by combining the set of trained parameters with the further trained parameters; A medical information processing system comprising a processing circuit configured to perform the steps of:

13. the processing circuitry of the learning device in each of the entities is further configured to obtain an individual updated model for that entity by combining the sets of learned parameters; The medical information processing system of claim 12 , wherein the combinations of the learned parameters are weighted according to the evaluation values ​​obtained for such entities.

14. 1. A method for training a model, comprising: training a model with a separate device at each of a plurality of entities, the training being performed on a separate cohort of data held at the entity to obtain a separate set of trained parameters from each of the plurality of entities; transmitting, by a respective medical image processing device at each entity, the set of learned parameters for the entity to each other of the plurality of entities; determining, by an individual medical information processing device at each entity, an individual evaluation value for each set of trained parameters by inputting data from the data cohort at that entity into a model having the set of trained parameters; obtaining an individual updated model by combining the trained parameters by an individual medical image processing device in each of the entities; A medical information processing method comprising:

15. The medical information processing method described in Claim 14, wherein the combinations of the learned parameters in each entity are weighted according to the evaluation values ​​obtained in the relevant entity.

Images