Training a model to perform tasks on medical data

Abstract

According to one aspect, a method is provided for training a model to perform a task on medical data using a distributed machine learning process, where a global model is updated based on training performed on local copies of the model at a plurality of clinical sites. The method includes: a) transmitting 302 information to a plurality of clinical sites that enables each of the plurality of clinical sites to create a local copy of the model and train the local copy of the model on the training data of the clinical site. The method then includes b) receiving 304 from each of the plurality of clinical sites: i) local updates to parameters of the model obtained by training the local copy of the model on the training data of the respective clinical site, and ii) metadata related to the quality of the training performed at the respective clinical site; and c) updating 306 parameters of the global model based on the received local updates to the parameters and the received metadata.

Description

Technical Field

[0001] Embodiments of the present invention relate to training a model using a distributed machine learning process.

Background Art

[0002] Learning from large amounts of patient data can significantly increase the ability to generate and test hypotheses related to healthcare. Prediction models are used to capture and use the knowledge contained in large amounts of patient data. The model can be trained using a machine learning process on large amounts of data from previously treated patients. A model trained in this way has the potential to be used, among other things, for making predictions in many medical fields, such as image segmentation and diagnosis. Such models can be used to better personalize healthcare.

Summary of the Invention

Problems to be Solved by the Invention

[0003] One of the major hurdles in enabling personalized medicine by using models trained using machine learning processes is obtaining enough patient data to train the models. Data from a single hospital is unlikely to be sufficient to develop a model that can be used for various types of patients (e.g., potentially spread across the globe). However, acquiring data from different hospitals and patient groups is time-consuming, increasing the time from model planning to deployment. In the deep learning domain, model performance improves as the number of training data samples increases. Therefore, to ensure the best possible model to assist physicians, model performance can be actively improved with more data. However, combining data from multiple clinical sites (e.g., hospitals, physician surgeries, etc.) can be difficult due to ethical, legal, political, and administrative barriers associated with data sharing. One way to mitigate such problems is to train models using distributed machine learning processes, such as the federated learning process described, for example, in the 2019 paper "Towards Federated Learning at Scale: System Design" by Bonawitz et al. Distributed learning allows models to be trained using data from different clinical sites, without the data ever leaving the same location. [Means for solving the problem]

[0004] The distributed machine learning process described above can be used to train a model (also known as a “machine learning model”) on training data located at various sites without the need to move the training data from each site. Those skilled in the art are familiar with distributed learning and distributed learning processes, such as federated machine learning, which are succinctly illustrated in Figure 1, which shows a central server 102 communicating with multiple clinical sites 104-112. The central server coordinates the training of the model using the distributed learning process with the training data located at each clinical site 104-112. The central server maintains a “global” or central copy of the model and can send information about the global model to each clinical site, such as parameters that allow local copies of the model to be created (114). Each clinical site can then create a local copy of the model and train that local copy on the training data at its respective clinical site. Each clinical site 104-112 can then send updates to one or more parameters of the model to the central server (116). The central server combines the updates from each clinical site, for example, through averaging, to update the global model. This allows a global model in a central server 102 to be trained, updated, and improved based on training data from multiple clinical sites 104-112, without the need for such training data to leave each clinical site. An object of the embodiments of the present specification is to improve such a process for training a model to perform a task on medical data using a distributed machine learning process.

[0005] Accordingly, according to the first aspect, there is a method for training a model to perform a task on medical data using a distributed machine learning process, wherein the global model is updated based on training performed on local copies of the model at multiple clinical sites. The method comprises the steps of: a) sending information to multiple clinical sites that enables each of the multiple clinical sites to create a local copy of the model and train the local copy of the model on the training data of the clinical site; b) receiving from each of the multiple clinical sites i) local updates of parameters in the model obtained by training the local copy of the model on the training data of the clinical site, and ii) metadata relating to the quality of training performed at the clinical site; and c) updating the parameters in the global model based on the received local updates of parameters and the received metadata.

[0006] Therefore, when combining local updates to generate updates for the global model, metadata related to the quality of training performed at each clinical site can be used. In this way, different local updates can be given different significance (e.g., through the use of weighting) depending on the quality of training performed at each clinical site. This can lead to improved training, an improved model, and therefore improved clinical outcomes in the clinical process using the model. Since the model is trained on data from different sites, there may be irregularities in the data, which can lead to bias and model drift. By considering appropriate metadata while merging weights, model drift can be avoided, which leads to a better quality model.

[0007] According to a second aspect, there is a method at a clinical site for training a model to perform a task on medical data using a distributed machine learning process, wherein a global model on a central server is updated based on training performed on a local copy of the model at the clinical site. The method comprises the steps of: receiving information from a central server that enables the creation of a local copy of the model at the clinical site and the training of the local copy on training data; training the local copy of the model according to the information; and transmitting to the central server i) an update of the model based on training the local copy of the model on the training data at the clinical site, and ii) metadata relating to the quality of the training performed at the clinical site.

[0008] According to the third aspect, there is the use of a model trained according to the first or second aspect to perform a task on medical data.

[0009] According to a fourth aspect, there is an apparatus for training a model to perform tasks on medical data using a distributed machine learning process, wherein a global model is updated based on training performed at multiple clinical sites. The apparatus comprises a memory having instruction data representing a set of instructions, and a processor configured to communicate with the memory and execute the set of instructions. When the set of instructions is executed by the processor, the processor is caused to perform the following steps: a) send information to the multiple clinical sites that enables each of the multiple clinical sites to create a local copy of the model and train the local copy of the model on the training data of the clinical sites; b) receive from each of the multiple clinical sites i) local updates of parameters in the model obtained by training the local copy of the model on the training data of the clinical sites, and ii) metadata relating to the quality of training performed at each clinical site; and c) update the parameters in the global model based on the received local updates of parameters and the received metadata.

[0010] According to the fifth aspect, there is a computer program product including a computer-readable medium, wherein the computer-readable medium has computer-readable code embodied therein, and the computer-readable code is configured to cause a computer or processor to execute the methods of the first and second aspects when an appropriate computer or processor executes an instruction.

[0011] These and other embodiments will be evident from the embodiments described below and will be explained with reference thereto.

[0012] Here, exemplary embodiments will be described with reference to the following drawings, merely as examples. [Brief explanation of the drawing]

[0013] [Figure 1] A diagram illustrating the distributed learning process for training a model. [Figure 2] Figures illustrating apparatus according to several embodiments of the present specification. [Figure 3] A diagram illustrating a method according to several embodiments of the present specification. [Figure 4] A figure illustrating a method for determining model drift according to several embodiments of the present specification. [Figure 5] Figures illustrating apparatus according to several embodiments of the present specification. [Figure 6] A diagram illustrating a method according to several embodiments of the present specification. [Figure 7] A diagram illustrating a system according to several embodiments of the present specification. [Figure 8] A figure showing liver image segmentation by a model trained according to embodiments of the present specification. [Modes for carrying out the invention]

[0014] As described above, embodiments of this specification aim to improve methods for training clinical models to perform tasks on medical data using distributed machine learning processes.

[0015] Referring to Figure 2, in some embodiments, and according to some embodiments of the present specification, there is a device 200 used to train a model that performs a task on medical data using a distributed machine learning process. Generally, the device may form part of a computer device or system, such as a laptop, desktop computer, or other computing device. In some embodiments, the device 200 may form part of a distributed computing device or cloud.

[0016] The device comprises a memory 204 having instruction data representing a set of instructions, and a processor 202 (e.g., processing circuit or logic) configured to communicate with the memory and execute the set of instructions. Generally, when the set of instructions is executed by the processor, the processor can be made to execute any of the embodiments of Method 300, as described below.

[0017] Embodiments of the device 200 may be used to train a model that performs tasks on medical data using a distributed machine learning process, in which case the global model is updated based on training performed on local copies of the model at multiple clinical sites. More specifically, the set of instructions, when executed by the processor 202, causes the processor to perform the following steps: a) send information to multiple clinical sites so that each of the multiple clinical sites can create a local copy of the model at its individual clinical site and train each local copy of the model on training data; b) receive from each of the multiple clinical sites: i) local updates to parameters in the model obtained by training the local copy of the model on training data at each clinical site, and ii) metadata related to the quality of training performed at each clinical site; and c) update parameters in the global model based on the received local updates to the parameters and the received metadata.

[0018] The processor 202 may have one or more processors, processing units, multicore processors, or modules configured or programmed to control the apparatus 200 in the manner described in this specification. In certain implementations, the processor 202 may have multiple software and / or hardware modules, each configured to perform or for performing individual or multiple steps of the method described in this specification. The processor 202 may have one or more processors, processing units, multicore processors, and / or modules, each configured or programmed to control the apparatus 200 in the manner described in this specification. In some implementations, for example, the processor 202 may have multiple (e.g., interoperable) processors, processing units, multicore processors, and / or modules configured for distributed processing. It will be understood by those skilled in the art that such processors, processing units, multicore processors, and / or modules may each be located in different locations and may each perform different steps and / or different parts of a single step of the method described in this specification.

[0019] Memory 204 is configured to store program code that can be executed by processor 202 to execute the methods described in this specification. Alternatively, or in addition, one or more memories 204 may be external to device 200 (e.g., may be separate from or remote from the device). For example, one or more memories 204 may be part of another device. Memory 204 can also be used to store a global model, received local updates, received metadata, and / or any other information or data received, calculated, or determined by processor 202 of device 200 or from any interface, memory, or device external to device 200. Processor 202 can be configured to control memory 204 to store a global model, received local updates, received metadata, and / or any other information or data described in this specification.

[0020] In some embodiments, memory 204 can have multiple sub - memories, and each sub - memory can be capable of storing a part of the instruction data. For example, at least one sub - memory can store instruction data representing at least one instruction of a set of instructions, and at least one other sub - memory can store instruction data representing at least one other instruction of the set of instructions.

[0021] FIG. 2 shows only the components necessary to explain this aspect of the present disclosure, and it should be understood that in an actual implementation, apparatus 200 can have additional components with respect to the illustrated components. For example, apparatus 200 can further have a display. The display can include, for example, a computer screen and / or a screen on a mobile phone or tablet. The apparatus can further have a user input device such as a keyboard, a mouse, or other input devices that enable a user to interact with the apparatus, for example, to provide initial input parameters used in the methods described herein. Apparatus 200 can have a battery or other power source for powering apparatus 200, or means for connecting apparatus 200 to a main power source.

[0022] Referring to FIG. 3, there is a computer-implemented method 300 used to train a model that performs a task (e.g., processes medical data) on medical data using a distributed machine learning process, where a global model is updated based on training performed on local copies of the model at a plurality of clinical sites.

[0023] Embodiments of method 300 can be performed, for example, by an apparatus such as apparatus 200 described above.

[0024] In short, step a) Method 300 includes step 302 transmitting information to multiple clinical sites that enables each of the multiple clinical sites to create a local copy of the model at its respective clinical site and train each local copy of the model on training data. Step b) Method 300 includes step (304) receiving from each of the multiple clinical sites i) local updates to parameters in the model obtained by training the local copy of the model on training data at each clinical site, and ii) metadata relating to the quality of training performed at each clinical site. Step c) Method includes step 306 updating parameters in the global model based on the received local updates to the parameters and the received metadata.

[0025] As mentioned above, since the model is trained on data from different sites, there may be irregularities in the data between sites, which can lead to bias and model drift, resulting in differences in the limits of decision used to perform the task (e.g., classification / segmentation) across different epochs of training. Generally, bias describes how well the model fits the training set. A model with high bias does not fit the dataset closely, while a model with low bias fits the dataset very closely. Bias arises from models that are overly simplistic and fail to capture trends present in the dataset. Model drift can be broadly categorized into two types. The first type is called "conceptual drift." Conceptual drift means that the statistical properties of the target variable that the model is trying to predict change unexpectedly over time. This causes problems because the accuracy of the prediction decreases over time. "Data drift": If the underlying variables are changing, the model is doomed to fail. This occurs when the statistical properties of the predictors change.

[0026] By merging weights while considering appropriate metadata, model drift can be avoided, resulting in a higher-quality model. Therefore, metadata can be used in relation to the quality of training performed at each site when combining individual local updates to generate updates for the global model. In this way, each different local update can be given different significance (e.g., through the use of weighting) depending on the quality of training performed at each clinical site.

[0027] More specifically, the model can include any type of model that can be trained using a machine learning process. Examples of models include, but are not limited to, neural networks, F-Nets, U-Nets and convolutional neural networks, random forest models and support vector machine (SVM) models.

[0028] Those skilled in the art are familiar with machine learning and machine learning models, but simply put, machine learning can be used to find a predictive function for a given dataset, which is typically a mapping between a given input and output. The predictive function (or mapping function) is generated during a training phase, which involves providing the model with exemplary inputs and ground truth (e.g., correct) outputs. The testing phase involves predicting the output for a given input. Applications of machine learning include, for example, curve fitting, facial recognition, and spam filtering.

[0029] In some embodiments of this specification, the model includes a neural network model, such as a deep neural network model. Those skilled in the art will be familiar with neural networks, but simply put, a neural network is a type of machine learning model that can be trained to predict a desired output for given input data. A neural network is trained by providing training data having exemplary input data and a desired corresponding "correct" or ground truth result. A neural network has multiple layers of neurons, each neuron representing a mathematical operation applied to the input data. The output of each layer in the neural network is fed to the next layer to produce an output. For each training data, the weights associated with the neurons are adjusted (e.g., using processes such as backpropagation and / or gradient descent) until optimal weights are found that produce predictions of the training example that reflect the corresponding ground truth.

[0030] As described above, the methods and systems described in this specification relate to training a model such as any of the models described above using a distributed learning process. The distributed learning process is described above with reference to Figure 1, and its details are understood to apply to embodiments of apparatus 200 and method 300. Examples of distributed learning processes include, but are not limited to, federated learning and distributed data parallel processing methods.

[0031] In some embodiments, the device 200 may have a server that coordinates training performed by servers at multiple clinical sites, in other words, a “central server”. Here, method 300 can be executed or initiated by a user, a company, or any other designer or orchestrator of the training process, for example, using the device 200. Using terminology generally related to distributed learning plans, the central server (e.g., device 200) may have a “master” of the scheme, and the multiple clinical sites may have “workers” or nodes.

[0032] A central server (e.g., device 200) can store and / or maintain (e.g., update) the global model. The global model (or global copy of the model) has a master copy or a central copy of the model. As described in more detail below, the results of training performed at each of the multiple clinical sites (e.g., local updates) are sent to the central server and incorporated into the global model. Thus, the global model represents the current "combined" results of all training performed at the multiple clinical sites.

[0033] In this context, a clinical site may include a hospital, surgery center, clinic, and / or data center, or other computing site suitable for storing medical data arising from such a clinical site.

[0034] As described above, this model is for performing tasks on medical data. In this context, medical data can include, but is not limited to, any type of data that can be used, generated, and / or acquired in a medical environment, including, for example, clinical diagnostic data such as a patient's vital signs, or physiological parameters, medical images, medical files (e.g., patient records), and / or outputs from medical devices (e.g., operational or diagnostic data from medical devices).

[0035] The model can accept one or more of the above-described types of medical data as input and perform tasks on the medical data. These tasks may include, for example, classification or segmentation tasks. For example, the model may predict the classification of medical data and / or provide output classifications. In embodiments of this specification, the model may, for example, output a patient diagnosis based on input medical data. In embodiments where the medical data includes medical images, the model may, for example, output segmentation of the medical images, the location of features of interest within the medical images, or a diagnosis based on the medical images. However, those skilled in the art will understand that these are merely examples and that the model may accept different types of medical data as input and provide different types of output (e.g., perform different tasks) in the above examples.

[0036] Returning to Method 300, as described above, Method 300 includes step 302 of sending information to each of the clinical sites that enables each of the clinical sites to create a local copy of the model and train the local copy of the model on training data.

[0037] For example, the information may include model information indicating the type of model and / or the values ​​of parameters within the model. For example, in embodiments where the model includes a neural network, the information may include, but is not limited to, several layers within the neural network model, the input and output channels of the model, and the values ​​of weights and biases within the neural network model. In general, the information transmitted in step a) is sufficient to allow each of multiple clinical sites to create a local copy of the model.

[0038] The information may further include instructions on how each clinical site should train the model. For example, the information could indicate, for instance, the number of training epochs to be performed, the amount of training data to be used to train the model, and the type of data to be used to train the model.

[0039] In step b), method 300 has step 304 to receive from each of a plurality of clinical sites: i) local updates to parameters in the model obtained by training a local copy of the model on training data at each individual clinical site, and ii) metadata related to the quality of training performed at each individual clinical site.

[0040] Local updates to parameters within a model may include the results of training a local copy of the model on training data at individual clinical sites. For example, changes in the model parameters resulting from training. In embodiments where the model includes a neural network, the parameters may include weights or biases in the neural network, or changes to be applied to the weights or biases in the neural network. Thus, in some embodiments, step b) includes receiving updated values ​​wi (or changes in value Δwi) of one or more weights or biases in the neural network model.

[0041] The metadata relates to the quality of training performed at each clinical site. In some embodiments, the metadata provides metrics for the performance of individual local copies of the trained model. For example, such metrics are indicators of the accuracy of the local model at individual clinical sites.

[0042] In some embodiments, metadata provides performance metrics for each local copy of the trained model on one or more subsets of training data that share common characteristics expected to affect the model's error. These characteristics may be expected to affect the model's error by, for example, making it easier (or more difficult) for the model to perform tasks on medical data (e.g., classify / segment). For instance, metadata could include performance metrics for individual local models when classifying medical data with different quality levels or levels of completeness (e.g., complete images compared to partial images).

[0043] In another embodiment, the metadata may include medical statistics that could affect training errors. In other words, the metadata may have static data related to the characteristics of the training data at each medical site that could affect the accuracy of individual local models. For example, the number of high-quality training data samples may be compared to the number of low-quality training data samples.

[0044] In some embodiments, metadata provides an indicator of the quality of training data at each clinical site. For example, metadata can provide an indicator of the distribution of training data at clinical sites across different output classifications of the model. In this sense, output classifications can include labels or categories output by the model. For example, metadata can describe whether the training data is evenly distributed across different output classifications, or whether the training data is skewed towards a particular classification (e.g., using more training data associated with some labels compared to others).

[0045] For example, consider a classification problem with five classes (or labels). Each clinical site has a different proportion of data for each class, and the trainable data changes as distributed learning is performed. The returned metadata may include the number of samples per class present at each node during weight updates. This can provide an indicator of how balanced (e.g., across different classes) the training data used to train each local model is. Local updates resulting from a more balanced training dataset can be given greater weights than local updates resulting from a less balanced training dataset.

[0046] In step c) of method 300, the method includes step 306, which updates the parameter in the global model based on the received local update and the received metadata for the parameter.

[0047] Generally, metadata is used to perform parameter merging on a central server. Therefore, the merged parameters may include a function of parameters received from clinical sites and their corresponding metadata. In other words, in some embodiments: Merged parameters = function (metadata, parameters received from clinical site) Mathematically, the function can be expressed as follows: Consider n clinical sites N1, N2, N3, ... with parameters W1, W2, W3, ... Each clinical site has quality measures α1, α2, α3, ... where the α value varies between 0 and 1 and is calculated from metadata sent from the clinical sites to the central server. Thus, the merged parameters can be calculated as follows: Merged parameters = (α1*W1+α2*W2+α3*W3+...) / (α1+α2+α3+...)

[0048] In other words, in some embodiments, the step of combining local updates with parameters to determine updates to the global model includes determining parameters for the global model according to the following: Global parameter = (α1*W1 + α2*W2 + α3*W3 + ... + αN*WN) / (α1 + α2 + α3 + ... αN); Here, WN includes local updates to the model parameters, such as those determined by the nth clinical site, and αN includes real numbers in the range 0 ≤ αN ≤ 1. Furthermore, αN is determined from metadata associated with the updates to the model parameters determined by the nth site. To avoid misunderstanding, other parameters may be used when calculating the α value. For example, in the case of batch training, one approach to calculating αi is: αi = the number of relevant samples in the i-th node / global batch size Here, the available samples at the nth node can be obtained from the scan's metadata information (slice thickness, resolution, etc.). In some embodiments, step c) may include combining local updates to parameters to determine an update to the global model by weighting each local update according to the individual metadata, such that local updates associated with metadata indicating high-quality training results have a higher weight (e.g., a higher value of α as described above) compared to updates associated with metadata indicating low-quality training results. For example, generally, local updates associated with a more accurate local model can be given a higher weight compared to local updates associated with a less accurate local model.

[0049] In one embodiment, the medical data includes computed tomography (CT) scan data. In such an embodiment, the metadata can provide an indicator of the performance of individual local copies of the model when classifying CT images of different radiation doses. For example, the metadata can provide an indicator of the model's performance when classifying high-dose and / or low-dose CT scans (or classifying high-dose CT scans relative to low-dose CT scans). In such an example, the model can be expected to classify high-dose CT images more accurately than low-dose CT images. In this embodiment, in step c) of method 300, such metadata can be used to prioritize updates received from a first clinical site having a local model that performs better against high-dose CT scans, even if the model at the first clinical site performs relatively poorly against low-dose CT scans, compared to updates received from a second clinical site having a local model that performs worse against high-dose CT scans.

[0050] In another embodiment, the metadata may describe the number of training data samples that are low-dose or high-dose for contrast enhancement. As described above, if the model makes a mistake with respect to low-dose CT images, this error is given less weight compared to a model that makes a mistake with respect to high-dose CT images (it is expected that the algorithm will perform very well with respect to high-dose CT images, and a small number of mistakes with respect to low-dose CT images will be acceptable).

[0051] In another example, metadata may have an indicator of the model's performance when classifying training data at different levels of completeness. For example, in an embodiment where a model is trained to perform segmentation of anatomical features in medical imaging data, the metadata may include an indicator of the model's performance when segmenting complete and / or partial images of anatomical features. In this embodiment, in step c) of method 300, such metadata may be used to prioritize updates received from a first clinical site that performs well in segmenting complete images of anatomical features over updates received from a second clinical site that has a local model that performs poorly in segmenting complete images of anatomical features, even if the first model performs relatively poorly in segmenting partial images of anatomical features.

[0052] In an embodiment where the medical data includes CT scan data and the model is for liver segmentation within the CT scan data, the metadata may include, for example, the following information: 1. Errors in low-dose CT and high-dose CT 2. Errors based on segmentation regions

[0053] When observing a CT volume, not the entire liver is visible on every slice of the CT volume. If the model makes a mistake when the liver is only partially visible, this error should be acceptable and smaller than the error that occurs with respect to image slices when the entire liver is visible. That is, during distributed learning of liver segmentation, if one node makes a mistake when the liver is only partially visible, and a second node makes mistakes between the scenarios where the liver is only partially visible and where it is fully visible, the algorithm should give higher weight to updates coming from the former.

[0054] Therefore, as described above, the global model can be updated using metadata that provides deeper insights into the quality of local updates determined by multiple clinical sites in a distributed learning scheme. As described above, when training data originates from different clinical sites, there is a possibility of bias and / or model drift affecting the model. The method described herein demonstrates a means of mitigating this model bias and data heterogeneity.

[0055] Referring here to other embodiments, in some embodiments of this specification, Method 300 can be further improved by detecting, based on an analysis of the visualization output, whether the global model is drifting during the training process. For example, if the region of interest activated / considered by the model when determining classification or labeling continues to change, the relevant drift can be identified. The variation values ​​can be calculated based on benchmark training data, which is supplied through the global model at each different point in time during the training process (e.g., supplied at time t0, with the change in variation obtained at t1). The variation calculation is obtained with respect to coordinate values ​​or with respect to the area below the bounding box of the region of interest activated / considered by the model.

[0056] For example, model drift can be determined as follows: Model drift: |(coordinates at t0)-(coordinates at t1)|>dynamic threshold

[0057] In other words, prior to steps a), b), and c) (for example at time t0), step 300 may have the step of determining a first region of the test image to be used by the global model to perform a task on the test image. The method may then further have the step of determining a second region of the test image to be used by the updated global model to perform a task on the test image, and comparing the first region of the test image with the second region of the test image to determine a measure of model drift, following steps a), b), and c).

[0058] The comparison step may include, for example, comparing coordinates related to the first and second regions (e.g., the center or edge of the region or bounding box), or comparing regions within the first and second regions and determining whether the region has changed by, for example, a statistically significant amount, or whether the change is greater than a threshold amount.

[0059] In this context, dynamic thresholds can be determined based on the current content and different models, as well as the type of the model. Therefore, this is not static for all application / model types.

[0060] This is shown in Figure 4, which shows an image of liver 402 containing lesion 404. The model is used to classify (e.g., locate) the lesions. The model is trained according to method 300 described above. Prior to steps a), b), and c), at time t0, the model classifies the lesions based on the region in image 406a. At time t1, it classifies the same lesions based on the region in image 406b. Differences in the location and size of regions 406a and 406b may indicate that the model has drifted. Thus, model drift can be determined by comparing and monitoring changes in regions between different training epochs / updates.

[0061] In other embodiments, steps a), b), and c) may be repeated, for example, to provide a series of training epochs. For example, steps a), b), and c) may be repeated periodically or whenever new training data becomes available at a clinical site.

[0062] Turning to other embodiments, in some embodiments, the method can be enhanced through the use of active learning. Those skilled in the art are familiar with active learning, which, simply put, trains the model on training data that has been pre-classified or misclassified with a low probability of accuracy. Thus, it effectively concentrates training on areas of weakness in the model.

[0063] In some embodiments, the method may therefore include repeating steps a), b), and c) for each subset of training data at each clinical site classified by a model having a certainty lower than a threshold certainty level. For example, certainty can be measured using the certainty level output by the model. In other embodiments, the certainty that the model classified the data can be calculated using a measure of entropy. A measure of entropy can reflect the amount of information in a dataset. Thus, the higher the entropy, the greater the amount of information in the dataset. For example, if a dataset has high entropy, the dataset has diversity in its content.

[0064] In general, an ambiguity zone can be defined, which contains training data where classification is uncertain. Training data within such an ambiguity zone can be used in subsequent epochs of model training. It should be noted that the ambiguity zone can be dynamic and may change between epochs as the (global) model improves.

[0065] Furthermore, in these embodiments, if optimized (e.g., active) distributed learning is performed and the model is trained on misclassified training samples each time, the metadata and quality measure (α) may change from training epoch to training, as described above.

[0066] In this way, the most important training data is considered in each training epoch, adding more value to the model. Thus, "optimized" distributed learning can be performed, considering only misclassified examples to update weights in subsequent epochs. The proposed concept captures dataset variability while ensuring the data does not leave the hospital premises. Furthermore, the new concept is designed to ensure that models that perform well with less data are trained.

[0067] Generally, one problem with distributed learning processes is that as the model's predictions improve, parameter updates become smaller (weight updates are negated by applying a simple or weighted average). This can sometimes prevent distributed learning models from performing similarly to centralized models. The use of active learning, as described above, helps overcome this problem by considering only misclassified images without proper segmentation or classification for retraining (a characteristic of active learning). This has several advantages: training time becomes faster each time the training data decreases; the loss function can be more concentrated and the gradients used to update the model can be better, since the algorithm trains only on misclassified data.

[0068] Generally, it is unrealistic to assume that data from all clinical sites are very similar. Therefore, considering the quality of misclassified data (based on active learning principles) as metadata and using this information while merging weights helps build a better global model.

[0069] The idea of ​​combining distributed learning and active learning follows the principle of "thinking globally" and "acting locally." A global model can be trained using training data from different hospitals located globally through distributed learning, and the model's performance at individual nodes is improved through active learning. Distributed learning captures the variability of data across (potentially) globally located human populations, while active learning improves performance at local nodes using less data.

[0070] Now, shifting to the perspective of a clinical site where local training is performed as described above, Figure 5 shows a device 500 used at a clinical site to train a model that performs tasks on medical data using a distributed machine learning process, according to some embodiments of the present specification. Generally, the device can form part of a computer device or system, such as a laptop, desktop computer, or other computing device. In some embodiments, the device 500 can form part of a distributed computing configuration or cloud.

[0071] The device includes a memory 504 having instruction data representing a set of instructions, and a processor 502 (e.g., processing circuit or logic) configured to communicate with the memory and execute the set of instructions. Generally, when the set of instructions is executed by the processor, it can cause the processor to execute any of the embodiments of Method 600, as described below.

[0072] Embodiments of the device 500 can be used at clinical sites to train a model that performs tasks on medical data using a distributed machine learning process, where a global model on a central server is updated based on training performed on local copies of the model at the clinical sites. More specifically, the set of instructions, when executed by a processor, causes the processor to perform the steps of: receiving information from a central server that enables local copies of the model to be created and trained on training data at the clinical sites; training the local copies of the model according to the information; and sending to the central server i) an update to the model based on the training of the local copies of the model on training data at the clinical sites, and ii) metadata relating to the quality of training performed at each clinical site.

[0073] The processor 502 may have one or more processors, processing units, multicore processors, or modules configured or programmed to control the apparatus 500 in the manner described in this specification. In certain implementations, the processor 502 may have multiple software and / or hardware modules, each configured or performing individual or multiple steps of the method described in this specification. The processor 502 may have one or more processors, processing units, multicore processors, and / or modules configured or programmed to control the apparatus 500 in the manner described in this specification. In some implementations, for example, the processor 502 may have multiple (e.g., interoperable) processors, processing units, multicore processors, and / or modules configured for distributed processing. It will be understood by those skilled in the art that such processors, processing units, multicore processors, and / or modules may be located in different positions and may perform different steps and / or different parts of a single step of the method described in this specification.

[0074] Memory 504 is configured to store program code that can be executed by processor 502 to perform the method described herein. Alternatively or additionally, one or more memories 504 may be outside of apparatus 500 (i.e., separate from or away from apparatus). For example, one or more memories 504 may be part of another apparatus. Memory 504 may be used to store a local copy of the model, training data, training output, and / or any other information or data received, computed, or determined by processor 502 of apparatus 500 or from any interface, memory, or device outside of apparatus 500. Processor 502 may be configured to control memory 504 to store a local copy of the model, training data, training output, and / or any other information or data generated by or used in conjunction with method 600 as described below.

[0075] In some embodiments, the memory 504 may have multiple submemories, each capable of storing a portion of the instruction data. For example, at least one submemory may store instruction data representing at least one instruction of the instruction set, and at least one other submemory may store instruction data representing at least one other instruction of the instruction set.

[0076] Figure 5 shows only the components necessary to illustrate this aspect of the present invention, and it should be understood that in actual implementations, the device 500 may have additional components beyond those shown. For example, the device 500 may further have a display. The display may include, for example, a computer screen and / or a screen on a mobile phone or tablet. The device may further have user inputs, such as a keyboard, mouse, or other input device, which allows a user to interact with the device to provide initial input parameters used in the method described herein. The device 500 may have a battery or other power source to supply power to the device 500, or means for connecting the device 500 to a mains power source.

[0077] Referring to Figure 6, there is a computer implementation 600 used in training a model to perform a task on medical data using a distributed machine learning process, wherein a global model on a central server is updated based on training performed on local copies of the model at clinical sites. Embodiments of method 600 can be performed by devices such as the apparatus 500 described above.

[0078] In short, in the first step 602, Method 600 includes receiving information from a central server that enables the creation of a local copy of the model at a clinical site and its training on training data. In the second step 604, Method includes training the local copy of the model according to the information. In the third step 606, Method includes sending to the central server i) an update to the model based on the training of the local copy of the model on training data at the clinical site, and ii) metadata relating to the quality of the training performed at each clinical site.

[0079] The methods and apparatus for the central server are described above with respect to Figures 2 and 3, and it is understood that the details thereof are equally applicable to the methods used at clinical sites.

[0080] In this context, the clinical site 500 may have servers (e.g., “clinical servers”) or data centers associated with a hospital, surgery center, clinic, or any other medical facility. The clinical site may be a data center such as a hospital data center (HDC), or any other compute data center suitable for storing medical data.

[0081] The information received in step 602 is described above with respect to Figures 2 and 3, and it will be understood that the details apply equally to apparatus 500 and method 600. Using the information, the clinical site creates a local copy of the model and trains the local copy of the model using training data at the clinical site (for example, according to the information received from the central server).

[0082] Those skilled in the art are familiar with methods for training machine learning models, including, but not limited to, gradient descent and backpropagation.

[0083] The clinical site obtains metadata related to the quality of training performed at each clinical site for the local model, and in step 606, sends i) updates to the model (e.g., training results) based on training of the local copy of the model on the training data at the clinical site, and ii) the metadata to the central server. The metadata is described in detail above with respect to apparatus 200 and method 300, and it will be understood that the details therein apply equally to apparatus 500 and method 600.

[0084] Referring here to another embodiment, Figure 7 illustrates a method for training a model using a distributed learning process according to several embodiments of the present specification. In this embodiment, researchers or other users are located on a computer or server 700, a central server 702, and multiple clinical sites (or nodes) 704. For clarity, only one clinical site 704 is shown in Figure 7. In this embodiment, the model includes a neural network. The method is as follows:

[0085] The researchers develop the model and deploy it on the server with 708 pre-initialized weights. The following processes are performed:

[0086] 710. The researcher sends the model and initialized weights to server 702. This starts the server. The server waits for node 704 to connect.

[0087] 712. When the server connects the deep learning model to the node, the connection between the server and the node is encrypted. The model is received by node 704 (714).

[0088] 716. The node creates a local copy of the model and performs training on the local copy of the model. Training proceeds using an active learning method, and the initialized model is used in the node to make predictions (or classifications) on the training data. If a prediction has a confidence level lower than a certain threshold confidence level (which is assigned by the researcher and is, for example, a dice score less than 0.95), it is used for further training of the model. The model is trained for several epochs, as described in the model file. Generally, training involves receiving weight values ​​(718), fitting them for different epochs (720), and obtaining the final weights and metadata (722).

[0089] 724. The weights are returned to the central server 702 along with metadata related to the quality of the training performed.

[0090] 726. The returned weights are merged with the help of metadata returned from the node, using averaging, weighted averaging, or other statistical methods, which are used by researchers to fit and update the global model (e.g., the version of the model stored in the central server 602). The information describing the updated global model is then sent back to node 704 for training again with the new merged weights (726).

[0091] This process is performed iteratively until the model converges. Transmission between the central server 704 and node 706 can log into a database (which can also be recorded on a blockchain so that the records cannot be deleted). This step can be used to maintain privacy. Once the model converges, the final weights can be sent to the researchers (730). It should be noted that if the central server has local training data stored on it, the central server can also perform training 732 on its own local copy of the model (for example, in some embodiments, the central server may have servers at clinical sites that train its own local copy of the model while coordinating a distributed training process across multiple clinical sites).

[0092] Referring here to another embodiment, in some embodiments, a model trained according to any of the methods or apparatus described herein (e.g., Method 300, 600, or 700, or Apparatus 200, or 500) is used to perform a task on medical data. Use may be performed in addition to or separately from the methods described herein. Examples of use include, but are not limited to, segmenting images (such as CT scans of the liver) using a model trained according to any of the methods described herein; or classifying medical records (e.g., making a diagnosis or any other classification) using a model trained according to any of the methods described herein.

[0093] Referring now to Figure 8, we see the segmentation of liver 802 output produced by a model trained using a conventional distributed learning process, compared with the segmentation of liver 804 output by a model trained using methods 300 and 600 described above.

[0094] In another embodiment, a computer program product is provided which includes a computer-readable medium having computer-readable code embodied therein, the computer-readable code being configured to instruct the computer or processor to execute one or more of the methods described herein when executed by a suitable computer or processor.

[0095] Accordingly, it will be understood that this disclosure also applies to computer programs adapted to implement embodiments, in particular computer programs on or within a carrier. The programs may be in the form of source code, object code, code intermediate source, and, for example, partially compiled object code, or any other form suitable for use in implementing the methods by which the embodiments described herein are constructed.

[0096] Furthermore, it will be understood that such programs can have many different architectural designs. For example, program code that implements the functionality of a method or system can be subdivided into one or more subroutines. Many different ways of distributing functionality among these subroutines will be apparent to those skilled in the art. Subroutines can be stored together in an executable file to form a self-contained program. Such an executable file may contain computer executable instructions, e.g., processor instructions and / or interpreter instructions (e.g., Java® interpreter instructions). Alternatively, one, one, or all of the subroutines can be stored in at least one external library file and linked to the main program statically or dynamically, for example, at runtime. The main program contains at least one call to at least one of the subroutines. Subroutines can also have function calls to each other.

[0097] The carrier of a computer program can be any entity or device capable of carrying the program. For example, the carrier may include a ROM, such as a CD-ROM or semiconductor ROM, or a data storage device such as a magnetic recording medium, such as a hard disk drive. Furthermore, the carrier may be a transmittable carrier, such as an electrical or optical signal, which can be carried via an electrical or optical cable, or by wireless or other means. When the program is embodied in such a signal, the carrier may consist of such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, which is adapted to perform the relevant method or is used to perform the relevant method.

[0098] Variations of the disclosed embodiments can be understood and implemented by those skilled in the art in carrying out the principles and techniques described herein, based on a review of the drawings, disclosures, and appended claims. In the claims, the word “comprising” does not preclude other components or steps, and the indefinite article “a” or “an” does not preclude plurality. A single processor or other unit can perform the functions of several items enumerated in the claims. The mere fact that certain means are described in mutually different dependent claims does not imply that combinations of these means cannot be used advantageously. Computer programs can be stored or distributed on suitable media such as optical storage media or solid-state media supplied together with or as part of other hardware, but they can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. No reference numerals in the claims should be construed as limiting in scope.

[0099] [Appendix 1] Experimental data Experiment 1: Sample size: Number of samples at each node (class imbalance) Model Type: A neural network trained on the MINST (Modified National Institute of Standards and Technology) database. MINST contains handwritten images of digits from 0 to 9. The model is trained to classify each image based on its numerical content. In the experiment, medical data was available at two nodes. The training dataset had 10 different classes. The first node had 9 classes with similar prevalences and a small number of samples from class 10. In the second node, the data for the first 9 classes was very sparse, while the data for the 10th class had a high prevalence. In the example, the models were merged using two approaches. 1) Simple average merge: The models were merged using metadata information. The merged models give a 20% accuracy. 2) Weighted average merge: Models are merged using metadata information (class prevalence). This model achieved accuracy of 90% and 88.9% on the train and test data, respectively. The results are summarized below. Node 1 Node 2 Merge Merge (Simple average) (Weighted average) Training accuracy 0.8971 0.09915 0.19358333 0.90055 Test accuracy: 0.886500 0.100900 0.200300 0.889700

[0100] Experiment 2: Acquisition / Image Scanner Setup Model: Neural Network Assumption: Data is available at two nodes. The datasets at nodes 1 and 2 are acquired using different CT machines at two different locations. In the example above, data is obtained from two different locations. The first dataset has an average HU intensity of 60 HUs, and the second dataset has an average HU intensity of 100. To perform federated learning, preprocessing techniques based on metadata must be used. For example, even if the average intensity of one site is slightly different from the expected value, the data can still be used, but the weighting of that site will be lower (e.g., scaled based on degree or difference from the expected value). In other words, updating the weight from this site indicates that it should be given lower priority because the distribution is not as expected. If the data is on completely different scales, the weights can be set to zero without harming the model at all. In general, if the data from two sites is different or has different properties, the model is more likely to fail. Therefore, the weights assigned to sites can be modified based on statistical heterogeneity to find the best model from a given site. This improves the accuracy of the resulting global model.

[0101] Experiment 3: Image Quality (CT Scan) Model: Neural Network Assumption: Data is available at two nodes. The datasets at nodes 1 and 2 are acquired using different CT machines at two different locations. The quality of the data acquired from the two centers is completely different. A weighted merge can be performed based on intensity histogram compression. Here, higher weights are assigned based on similarity to the overall sample dataset. This allows for a penalty to the model if errors occur on lower-quality data by assigning lower weights. This improves the accuracy of the resulting global model. The various forms of the present invention are described below. (Note 1) A computer implementation for training a model to perform a task on medical data using a distributed machine learning process, The global model is updated based on training performed on local copies of the model at multiple clinical sites. The model is used to predict the classification of the medical data, or, if the medical data includes medical images, the model is used to segment the medical images. The aforementioned computer implementation method a) A step of transmitting to the plurality of clinical sites information that enables each of the plurality of clinical sites to create a local copy of the model and to train the local copy of the model on the training data of the clinical site, b) A step of receiving from each of the plurality of clinical sites: i) local updates of parameters in the model obtained by training the local copy of the model on the training data at the clinical site, and ii) metadata related to the quality of the training performed at the clinical site. c) A step of updating the parameters in the global model based on the received local updates of the parameters and the received metadata, wherein the update to the global model is determined by combining the local updates of the parameters, weighting each local update according to the respective metadata such that local updates related to metadata indicating high-quality training results have a higher weight than local updates related to metadata indicating low-quality training results. A method of having. Note (Note 2) The step of determining the update to the global model by combining the local updates of the parameters is, Global parameter = (α1*W1 + α2*W2 + α3*W3 + ... + αN*WN) / (α1 + α2 + α3 + ... + αN) The method according to Appendix 1, comprising determining the parameters of the global model, wherein WN includes local updates to the parameters in the model determined by the nth clinical site, and αN includes a real number in the range 0 ≤ αN ≤ 1, and the value of αN is determined from metadata related to the updates to the parameters in the model determined by the nth clinical site. (Note 3) The method according to Appendix 1 or 2, wherein the metadata provides an indicator of the performance of individual local copies of the trained model for one or more subsets of training data at individual clinical sites having common characteristics that are expected to affect model error. (Note 4) The method according to Appendix 3, wherein the medical data includes computed tomography (CT) scans, and the metadata includes an indicator of the performance of the local copy of the model when classifying CT scans with different radiation doses. (Note 5) The method according to Appendix 3, wherein the medical data includes medical images, the model is used to segment the medical images in order to obtain segmentation of anatomical features within the medical image data, and the metadata includes an indicator of the model's performance in segmenting the whole image and / or partial images of the anatomical features. (Note 6) The method according to any one of Appendix 1 to 5, wherein the metadata provides an indicator of the quality of the training data at each of the clinical sites. (Note 7) The method according to Appendix 6, wherein the metadata provides an indicator of the distribution of the training data at the clinical site between different output classifications of the model. (Note 8) The medical data includes a medical image, and the method is Prior to steps a), b), and c), the step of determining a first region of the test image to be used by the global model to perform the task on the test medical image, Following steps a), b) and c), the step of determining a second region of the test medical image to be used by the updated global model to perform the task on the test medical image, The steps include: determining the scale of model drift by comparing the first region of the test image with the second region of the test image; The method described in any one of the appendices 1 to 7, having the following characteristics. (Note 9) The method according to any one of Appendix 1 to 8, further comprising the step of repeating steps a), b), and c) for a subset of training data for each clinical site classified by a model having a certainty below a threshold certainty level. (Note 10) A computer implementation at a clinical site for training a model to perform a task on medical data using a distributed machine learning process, The global model on the central server is updated based on training performed on local copies of the model at the clinical sites. The model is used to predict the classification of the medical data, or, if the medical data includes medical images, the model is used to segment the medical images. The aforementioned computer implementation method The steps include receiving information from the central server that enables the creation of a local copy of the model and its training on training data at the clinical site, A step of training the local copy of the model according to the information, i) an update of the model based on training the local copy of the model on the training data of the clinical site, and ii) metadata relating to the quality of the training performed at the clinical site, to be transmitted to the central server. A method of having. (Note 11) The method according to any one of Appendix 1 to 10, wherein the model includes a neural network model and the parameters include weights or biases in the neural network model. (Note 12) A device for training models to perform tasks on medical data using a distributed machine learning process, The global model is updated based on training performed on local copies of the model at multiple clinical sites. The model is used to predict the classification of the medical data, or, if the medical data includes medical images, the model is used to segment the medical images. The aforementioned device A memory containing instruction data representing a set of instructions, A processor configured to communicate with the memory and execute the set of instructions, It has, When the aforementioned set of instructions is executed by the processor, the processor will be instructed to: The steps include transmitting to the plurality of clinical sites information that enables each of the plurality of clinical sites to create a local copy of the model and to train the local copy of the model on the training data of the clinical site, The steps include receiving from each of the plurality of clinical sites: i) local updates of parameters in the model obtained by training the local copy of the model on the training data of the clinical site, and ii) metadata related to the quality of training performed at the clinical site. A step of updating parameters in the global model based on the received local updates of parameters and the received metadata, wherein the update to the global model is determined by combining the local updates of the parameters, weighting each local update according to the respective metadata such that local updates related to metadata indicating high-quality training results have a higher weight than updates related to metadata indicating low-quality training results. A device that performs an action. (Note 13) A computer-readable medium having a computer-readable code, wherein the computer-readable code, when executed by a suitable computer or processor, causes the computer or processor to perform the method described in any one of appendices 1 to 11.

Claims

1. A computer implementation for training a model to perform a task on medical data using a distributed machine learning process, The global model is updated based on training performed on local copies of the model at multiple clinical sites. The aforementioned model is used to predict the classification of the medical data, wherein the medical data includes medical images. The aforementioned computer implementation method a) A step of transmitting to the plurality of clinical sites information that enables each of the plurality of clinical sites to create a local copy of the model and to train the local copy of the model on the training data of the clinical site, b) Steps of receiving from each of the plurality of clinical sites: i) local updates of parameters in the model obtained by training the local copy of the model on the training data at the clinical site; and ii) metadata that provides an indicator of the quality of the training data at the clinical site by providing an indicator of how well-balanced the training data at the clinical site is between different output classifications of the model. c) A step of updating the parameters in the global model based on the received local updates of the parameters and the received metadata, wherein the update to the global model is determined by combining the local updates of the parameters, weighting each local update according to its respective metadata such that local updates from a more balanced training dataset are given greater weight than local updates from a less balanced training dataset. A method of having.

2. The step of determining the update to the global model by combining the local updates of the parameters is, Global parameter = (α1*W1 + α2*W2 + α3*W3 + ... + αN*WN) / (α1 + α2 + α3 + ... + αN) The method according to claim 1, comprising determining the parameters of the global model in accordance with, where WN includes local updates to the parameters in the model determined by the nth clinical site, and αN includes a real number in the range 0 ≤ αN ≤ 1, the value of αN is determined from metadata related to the updates to the parameters in the model determined by the nth clinical site.

3. The method according to claim 1 or 2, wherein the metadata has the number of samples per class present at the clinical site during the weight update.

4. The method according to any one of claims 1 to 3, further comprising the step of repeating steps a), b), and c) for a subset of training data of each clinical site classified by a model having a certainty lower than a threshold certainty level.

5. A computer implementation method at a clinical site for training a model to perform a task on medical data using a distributed machine learning process, The global model on the central server is updated based on training performed on local copies and metadata of the model at the clinical sites. The aforementioned model is used to predict the classification of the medical data, wherein the medical data includes medical images. The aforementioned computer implementation method The steps include receiving information from the central server that enables the creation of a local copy of the model and its training on training data at the clinical site, A step of training the local copy of the model according to the information, i) updating the model based on training the local copy of the model on the training data of the clinical site, and ii) providing metadata that provides an indicator of the quality of the training data at the clinical site by providing an indicator of how well-balanced the training data at the clinical site is between the different output classifications of the model, and transmitting these to the central server; A method of having.

6. The method according to claim 5, wherein the metadata has the number of samples per class present at the clinical site during the weight update.

7. The method according to any one of claims 1 to 4, wherein the model includes a neural network model, and the parameters include weights or biases in the neural network model.

8. A device for training models to perform tasks on medical data using a distributed machine learning process, The global model is updated based on training performed on local copies of the model at multiple clinical sites. The aforementioned model is used to predict the classification of the medical data, wherein the medical data includes medical images. The aforementioned device A memory containing instruction data representing a set of instructions, A processor configured to communicate with the memory and execute the set of instructions, It has, When the aforementioned set of instructions is executed by the processor, the processor will be instructed to: a) A step of transmitting to the plurality of clinical sites information that enables each of the plurality of clinical sites to create a local copy of the model and to train the local copy of the model on the training data of the clinical site, b) Receiving from each of the plurality of clinical sites: i) local updates of parameters in the model obtained by training the local copy of the model on the training data of the clinical site, and ii) metadata that provides an indicator of the quality of the training data at the clinical site by providing an indicator of how well-balanced the training data at the clinical site is between the different output classifications of the model; c) A step of updating the parameters in the global model based on the received local updates of the parameters and the received metadata, wherein the update to the global model is determined by combining the local updates of the parameters, weighting each local update according to its respective metadata such that local updates from a more balanced training dataset are given greater weight than local updates from a less balanced training dataset. A device that performs an action.

9. A computer program having computer-readable code, wherein the computer-readable code, when executed by a suitable computer or processor, causes the computer or processor to perform the method described in any one of claims 1 to 7.

Images