Federated learning system for training machine learning algorithm and maintaining patient privacy

Abstract

To provide a method for using a federated learning classifier in digital pathology.SOLUTION: A method in a federated learning digital pathology system 100 includes: distributing, by a centralized server 110, a global model 112, 114 to a plurality of client devices 120-140. The client devices further train the global model using a plurality images of a specimen and corresponding annotations to generate at least one further trained model. The client devices provide further trained models to the centralized server. The centralized server aggregates the further trained models with the global model to generate an updated global model. The updated global model is then distributed to the plurality of client devices.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to digital pathology, and in particular to machine learning techniques for federated learning. [Background technology]

[0002] Digital pathology involves scanning pathology slides (e.g., histopathology or cytopathology glass slides) bearing tissues and / or cells into digital images for use in evaluation. The tissues and / or cells in the digital images can then be examined and / or interpreted by a pathologist using digital pathology image analysis for a variety of reasons, including diagnosing disease, assessing response to treatment, and developing drugs to combat disease. To examine the tissues and / or cells in the digital images (which are substantially transparent), pathology slides can be prepared using color stains (e.g., immunostains) that selectively bind to tissue and / or cellular components. Immunohistochemistry (IHC) is a common application of immunostaining and involves the process of selectively identifying antigens (proteins) in cells of tissue sections by utilizing the principle of antibodies and other compounds (or substances) that specifically bind to antigens in biological tissues. In some assays, the target antigen for the stain in the specimen is sometimes called a biomarker. Digital pathology image analysis can then be performed on the digital images of the stained tissues and / or cells to identify and quantify staining for antigens in biological tissues (e.g., biomarkers indicative of tumor cells).

[0003] Machine learning techniques have shown great promise in digital pathology image analysis, such as tumor region identification, metastasis detection, and patient prognosis. Many computing systems equipped with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis, such as tumor region and metastasis detection. For example, a CNN can have a series of convolutional layers as hidden layers, and this network structure enables the extraction of representational features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques have also been implemented for image segmentation. Image segmentation is the process of dividing a digital image into multiple segments (sets of pixels, also known as image objects). The typical purpose of segmentation is to simplify and / or change the representation of an image to one that is more meaningful and easier to analyze. For example, image segmentation is often used to find objects such as tumors (or other tissue types) and boundaries (lines, curves, etc.) within an image. To perform image segmentation for large data sets (e.g., whole-slide pathology images), the image is first divided into many small patches. A computing system equipped with machine learning techniques is trained to classify these patches, and all patches within the same class are combined into one segmented region. Machine learning techniques can then be further implemented to predict or classify the segmented region (e.g., negative tumor cells or tumor cells with no staining expression) based on the phenotypic features associated with the segmented region.

[0004] Various machine learning techniques require training data to establish ground truth for performing classification. In the medical field, patient data is often difficult to obtain due to privacy concerns and legal requirements. Therefore, properly training a classifier can be a challenge. Federated learning is a decentralized machine learning technique that involves providing a base classifier to one or more client devices. Each device can then operate using the base classifier. Once the classifier is utilized on each device, a user provides input related to the output provided by the classifier. The user can provide input to each classifier based on the output, and each of the respective classifiers can be updated according to the user input. The updated classifier can then be provided to update the base classifier. The updated classifier can then be distributed to the client devices. Thus, a federated learning system can be updated without the need to pass data between entities. Summary of the Invention

[0005] In various embodiments, a computer-implemented method is provided.

[0006] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0007] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0008] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0009] Some embodiments of the present disclosure include a computer-implemented method for using a federated learning classifier, including: distributing, by a centralized server, a global model configured to classify pathology images to a plurality of client devices; receiving, by the centralized server, an updated model from at least one of the plurality of client devices, where the updated model has been further trained on at least one of the plurality of client devices using a plurality of slide images and a plurality of corresponding annotations; aggregating, by the centralized server, the updated model with the global model to generate an updated global model; and distributing the updated global model to at least one of the plurality of client devices.

[0010] Some embodiments of the present disclosure include a computer-implemented method in which aggregating the updated model with the global model to generate the updated global model includes performing averaging of at least one weight of the global model with at least one weight of the updated model.

[0011] Some embodiments of the present disclosure include a computer-implemented method, wherein performing the averaging includes performing a weighted average of at least one weight of the updated model and at least one weight of the global model according to the number of slide images used to further train the updated model and the total number of images used to train the global model.

[0012] Some embodiments of the present disclosure include computer-implemented methods in which annotations are provided by a user observing the output of a global model on a slide image, and the annotations include modifications to the output generated by the global model.

[0013] Some embodiments of the present disclosure include a computer-implemented method further including receiving, by a centralized server, metadata associated with the plurality of slide images, and wherein aggregating further includes normalizing the trained model according to the metadata.

[0014] Some embodiments of the present disclosure include a computer-implemented method that further includes validating, by the centralized server, the performance improvement of the updated global model against the global model using the validation dataset.

[0015] Some embodiments of the present disclosure include a computer-implemented method for using a federated learning classifier by a client device, the method including receiving a global model configured to classify pathology images from a centralized server, receiving stained tissue images, where the stained tissue images are divided into image patches, performing image analysis on the image patches using the global model, training the global model using the image patches and at least one corresponding user annotation to generate an updated model, where the at least one corresponding user annotation includes a correction to the classification generated by the global model, transmitting the updated model to the centralized server, receiving the updated global model, and validating performance improvements of the updated global model using a client-specific validation dataset.

[0016] Some embodiments of the present disclosure include computer-implemented methods in which the correction of the classification produced by the global model is a reclassification of at least one of cell type, tissue type, or tissue boundary.

[0017] Some embodiments of the present disclosure include computer-implemented methods in which the updated model does not include individual patient information.

[0018] Some embodiments of the present disclosure include a computer-implemented method further including generating metadata associated with a plurality of images and providing the metadata to a centralized server.

[0019] Some embodiments of the present disclosure include computer-implemented methods in which the metadata includes at least one of the area of ​​the slide or tissue to which the image corresponds, the type of staining performed, the concentration of the staining, and the equipment used for staining or scanning.

[0020] Some embodiments of the present disclosure include computer-implemented methods in which sending the updated model is performed after a threshold number of iterations, a length of time, or after the model has changed beyond a threshold amount.

[0021] Some embodiments of the present disclosure include a computer-implemented method for using a federated learning classifier in digital pathology, the method including: distributing, by a centralized server, a global model to a plurality of client devices; training, by a client device from the plurality of client devices, the global model using a plurality of images of a specimen to generate at least one further trained model, where one or more of the plurality of images include at least one annotation; providing, by the client devices, the further trained model to the centralized server; aggregating, by the centralized server, the further trained model with the global model to generate an updated global model; and distributing the updated global model to the plurality of client devices.

[0022] Some embodiments of the present disclosure include a computer-implemented method further performing: generating, by a client device, metadata associated with a plurality of images; and providing, by the client device, the metadata to a centralized server, wherein aggregating, by the centralized server, further training models with the global model to generate an updated global model further includes normalizing the further training models according to the metadata.

[0023] Some embodiments of the present disclosure include computer-implemented methods in which the metadata includes at least one of the area of ​​the slide or tissue to which the image corresponds, the type of staining performed, the concentration of the staining, and the equipment used for staining or scanning.

[0024] Some embodiments of the present disclosure include a computer-implemented method further configured to validate, by the centralized server, the performance of the updated global model against the global model using the validation dataset.

[0025] Some embodiments of the present disclosure include a computer-implemented method further configured to roll back the updates to the global model if the performance of the updated global model is lower than the global model.

[0026] Some embodiments of the present disclosure include a computer-implemented method in which aggregating the updated model with the global model to generate the updated global model includes performing averaging of at least one weight of the global model with at least one weight of the updated model.

[0027] Some embodiments of the present disclosure include a computer-implemented method, wherein performing the averaging includes performing a weighted average of at least one weight of the updated model and at least one weight of the global model according to the number of slide images used to further train the updated model and the total number of images used to train the global model.

[0028] Some embodiments of the present disclosure include computer-implemented methods in which sending the updated model is performed after a threshold number of iterations, a length of time, or after the model has changed beyond a threshold amount.

[0029] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of any features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0030] The present disclosure is described in conjunction with the accompanying drawings, in which:

[0031] [Figure 1] 1 illustrates an example of a federated learning digital pathology system according to various embodiments.

[0032] [Figure 2] FIG. 1 shows a block diagram illustrating a computing environment for non-tumor segmentation and image analysis using deep convolutional neural networks according to various embodiments.

[0033] [Figure 3] 3 shows an exemplary diagram 300 illustrating a model architecture for non-target region segmentation according to various embodiments.

[0034] [Figure 4] 1 depicts examples of staining variation across different H&E slide images according to various embodiments.

[0035] [Figure 5] 1 illustrates a process for training a predictive model according to various embodiments.

[0036] [Figure 6] 1 illustrates a process for a round of FL training of a predictive model according to various embodiments.

[0037] [Figure 7] 1 illustrates results generated after multiple rounds of FL training of a predictive model according to various embodiments.

[0038] [Figure 8] 1 illustrates a process for a round of FL training of a predictive model according to various embodiments.

[0039] [Figure 9] 1 illustrates a process for receiving an updated model from a client according to various embodiments.

[0040] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used in this specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0041] This disclosure describes techniques for a digital pathology (DP) federated learning (FL) system. FL is a distributed machine learning approach in which multiple client devices collaboratively train a deep learning model (global model) for image analysis without sharing training data. A server is configured to distribute the global model to one or more clients. The server is configured to maintain, update, and redistribute the global model as part of an iterative process. In each iteration (or round), each client can receive the global model and perform DP image analysis on local data (e.g., patient data including pathology slides). The clients can further train the global model using locally available data (e.g., patient data and user input). Updated models can be periodically sent from one or more clients to the server. The updated models may be incorporated into the global model to generate an updated global model. The updated global model can then be distributed to the clients. The iterations continue indefinitely, or until, for example, training converges. In some examples, the received updated models may not be incorporated into the global model.

[0042] Immunohistochemical (IHC) slide staining is widely used to study different types of cells, such as cancerous cells and immune cells, within biological tissues because it can be used to identify proteins within cells in tissue sections. It is possible to evaluate IHC-stained cells in tissue sections under a microscope at high magnification and / or automatically analyze digital images of biological specimens using digital pathology algorithms. Often, in whole-slide analysis, evaluation of the stained biological specimen requires segmentation of the stained biological specimen, including identification of target regions (e.g., positive and negative tumor cells) and exclusion of non-target regions (e.g., normal tissue or blank slide regions). In some instances, the excluded non-target regions contain biological material or structures that may be very difficult to distinguish from other biological material or structures in the target region and are therefore excluded from the evaluation of the biological specimen. As a result, in such cases, pathologists typically provide manual tumor annotation while excluding non-target regions. However, manual tumor annotation is prone to error, pathologist bias, and laborious due to the large size of whole-slide images at high magnification and the large amount of data to be processed.

[0043] Automated segmentation and classification of tumors and tumor cells can be challenging for a variety of reasons. For example, tumors and tumor cells can vary greatly across patients with respect to size, shape, and localization. This prohibits the use of strong prior information about shape and localization that is commonly used for robust image analysis in many other applications, such as face recognition or navigation. As a result, traditional image analysis algorithms typically provide undesirable detection results (e.g., overdetection or misclassification) of these challenging regions.

[0044] To address these limitations and issues, a wide variety of large amounts of training data are required. Given privacy concerns regarding medical data, obtaining large amounts of training data has proven difficult. The technology of the FL DP system of this embodiment involves the use of a machine learning architecture that allows data to be used at the client location for training without the need to transmit the data to a centralized location. Thus, patient information does not leave its original location, mitigating privacy concerns. One exemplary embodiment of the present disclosure relates to a computer-implemented method for automatically performing image analysis on pathology slides, including performing pre-processing, image analysis, and post-processing. For example, the FL DP system may include one or more deep learning architectures that utilize FL to improve performance while not transferring the underlying training data between entities. For example, the FL DP system may include a deep learning pre-processing system (e.g., for performing image segmentation to remove or mask specific regions), a deep learning system for image processing (e.g., for identifying regions of the image having desired features), and / or a deep learning system for performing post-processing (e.g., utilizing the identified regions of the image to perform further analysis). Thus, an FL DP system can include multiple models of each client device, each model capable of utilizing FL.

[0045] In some embodiments, the computer-implemented method can include the use of one or more models. The model can have, for example, a convolutional neural network (CNN) architecture or model that utilizes a two-dimensional segmentation model (e.g., a modified U-Net or other suitable architecture) to automatically detect and exclude biological structures or non-tumor cells before running standard image analysis algorithms to learn and recognize target regions. Post-analysis can then be performed to provide or assist in a diagnosis or further course of action. The convolutional neural network architecture or model can be trained using pre-labeled images. As a result, the model (e.g., a trained convolutional neural network architecture or model) can be used to segment non-target regions, which can then be masked from full-slide analysis before, during, or after inputting the image into the image analysis algorithm. The image analysis model (e.g., CNN) performs the classification task and outputs a tumor readout of the target region. A post-processing model performs further classification based on the tumor readout. Advantageously, the proposed architecture and technique can improve the accuracy of tumor cell classification by improving the models used at all stages of image analysis.

[0046] As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of the something.

[0047] As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily entirely, specified (and including) what is specified entirely, as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" can be replaced with "within [percentage]" of what is specified, where percentage includes 0.1, 1, 5, and 10%.

[0048] As used herein, the terms "sample," "biological sample," or "tissue sample" refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans, veterinary animals such as cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies of tissue), cell samples (such as cytological smears, such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biomolecules derived from an initial biological sample. In certain embodiments, the term "biological sample," as used herein, refers to a sample prepared from a tumor or portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0049] As used herein, the term "biological material or structure" refers to a naturally occurring material or structure that contains all or part of a biological structure (e.g., a cell nucleus, cell membrane, cytoplasm, chromosome, DNA, cell, cell mass, etc.).

[0050] As used herein, the term "non-target region" refers to a region of an image having image data that is not intended to be evaluated in an image analysis process. A non-target region can include a non-tissue region of an image that corresponds to a substrate, such as glass, without a sample, for example, when only white light from the imaging source is present. A non-target region may additionally or alternatively include a tissue region of an image that corresponds to one or more biological materials or structures that are not intended to be analyzed in the image analysis process or that are difficult to distinguish from biological materials or structures in the target region (e.g., lymphoid aggregates).

[0051] As used herein, the term "target region" refers to a region of an image containing image data intended to be evaluated in an image analysis process. Target regions include any region, such as a tissue region, of an image intended to be analyzed in an image analysis process.

[0052] As used herein, the term "tile" or "tile image" refers to a single image corresponding to a portion of an entire image or an entire slide. In some embodiments, a "tile" or "tile image" refers to a region of an entire slide scan or a region of interest having (x,y) pixel dimensions (e.g., 1000 pixels by 1000 pixels). For example, consider an entire image divided into M columns of tiles and N rows of tiles. Each tile in the MxN mosaic contains a portion of the entire image; i.e., tiles at locations M1, N1 contain a first portion of the image, tiles at locations M3, N4 contain a second portion of the image, and the first and second portions are different. In some embodiments, the tiles can each have the same dimensions (pixel size by pixel size).

[0053] As used herein, the term "patch" or "image patch" refers to a container of pixels corresponding to a portion of a tile image, an entire image, or an entire slide. In some embodiments, a "patch" or "image patch" refers to a region or region of interest of a tile image having (x,y) pixel dimensions (e.g., 256 pixels by 256 pixels). For example, a 1000 pixel by 1000 pixel tile image divided into 100 pixel by 100 pixel patches will contain 100 patches (each patch containing 1000 pixels). In other examples, the patches may overlap.

[0054] In some embodiments, a federated learning (FL) system for digital pathology (DP) can be utilized to generate and distribute global models (e.g., aggregated global models) without exchanging sensitive or identifying data (e.g., patient data) between clients and / or a centralized system (e.g., a server). The server is configured to maintain and distribute the global models in an iterative process as updated models are received from clients. FIG. 1 depicts an exemplary FL DP system 100 including one or more servers 110 configured to maintain and distribute one or more global models 112, 114. The server 110 communicates with one or more client systems 120, 130, 140, each of which may include various DP equipment such as workstations 122, 132, 142, microscopes 124, 134, 144, digital slide scanners 126, 136, 146, and any other necessary equipment as would be understood by one of ordinary skill in the art. Each of the client systems can utilize one or more local models 128, 138, 148, 150 based on the global models 112, 114. The client systems 120, 130, 140 can be utilized to further train the local models 128, 138, 148, 150. For example, the client systems 120, 130, 140 can receive patient data, classify the patient data using the local models 128, 138, 148, 150, receive user input regarding the classified patient data (e.g., from a pathologist or other medical professional utilizing a graphical user interface that displays the classified data), and update the local models 128, 138, 148, 150 based on the user input (e.g., each client retrains the global model using a local training dataset). In various embodiments, the client devices are configured to periodically provide their local models 128, 138, 148, 150 to the centralized server 110.The centralized server 110 can then use the local models 128, 138, 148, 150 to update the global models 112, 114 (e.g., by updating weights in the global models) and distribute the updated global models 112, 114 to the client systems 120, 130, 140.

[0055] In some embodiments, after each iteration, the performance of each of the updated local models 128, 138, 148, 150 can be confirmed using a validation data set. If the local models 128, 138, 148, 150 are determined to provide improved performance on the validation data set, the local models can be incorporated into the global models 112, 114. The performance of the updated global models 112, 114 may also be verified using the validation data set. If the global models 112, 114 are improved, the updated global models 112, 114 may be distributed to all or some of the client devices 120, 130, 140. In some embodiments, clients can choose not to share their updated local models 128, 138, 148, 150 but still receive the updated global models 112, 114. In other embodiments, clients can choose to share their local models 128, 138, 148, 150 but not receive the updated global models 112, 114. In other embodiments, clients can choose not to share their updated local models 128, 138, 148, 150 and not receive the updated global models 112, 114. As such, models generated at client sites are not controlled by the centralized server 110 and are shared with the centralized server 110 at the discretion of the client. Each client can have an independent validation data set that can be used to test the performance of the model based on their quality criteria. Based on this validation, the client can decide whether to deploy the global models 112, 114.

[0056] In some embodiments, after each iteration, the performance of each of the updated local models 128, 138, 148, 150 can be confirmed using a validation data set. If the local models 128, 138, 148, 150 are determined to provide improved performance on the validation data set, the local models can be incorporated into the global models 112, 114. The performance of the updated global models 112, 114 may also be verified using the validation data set. If the global models 112, 114 are improved, the updated global models 112, 114 may be distributed to all or some of the client devices 120, 130, 140. In some embodiments, clients can choose not to share their updated local models 128, 138, 148, 150 but still receive the updated global models 112, 114. In other embodiments, clients can choose to share their local models 128, 138, 148, 150 but not receive the updated global models 112, 114. In other embodiments, clients can choose not to share their updated local models 128, 138, 148, 150 and not receive the updated global models 112, 114. As such, models generated at client sites are not controlled by the centralized server 110 and are shared with the centralized server 110 at the discretion of the client. Each client can have an independent validation data set that can be used to test the performance of the model based on their quality criteria. Based on this validation, the client can decide whether to deploy the updated global models 112, 114.

[0057] FIG. 2 illustrates a block diagram of a computing environment 200 for non-tumor segmentation and image analysis using deep convolutional neural networks according to various embodiments. The computing environment 200 may include an analysis system 205 for training and executing predictive models, e.g., two-dimensional CNN models. More specifically, the analysis system 205 may include training subsystems 210a-n (where "a" and "n" represent any natural numbers) that build and train respective predictive models 215a-n (which may be referred to herein individually as predictive models 215 or collectively as predictive models 215) for use by other components of the computing environment 200. The predictive models 215 may be machine learning ("ML") or deep learning ("DL") models, e.g., deep convolutional neural networks (CNNs) such as U-Net neural networks, primordial neural networks, residual neural networks ("Resnet"), or recurrent neural networks, e.g., long short-term memory ("LSTM") models or gated recurrent unit ("GRU") models. The predictive model 215 may also be any other suitable ML model trained to segment non-target regions (e.g., lymphatic aggregate regions), segment target regions, or provide image analysis of target regions, such as a two-dimensional CNN ("2DCNN"), a dynamic time warping ("DTW") technique, a hidden Markov model ("HMM"), or the like, or a combination of one or more of such techniques, such as a CNN-HMM or an MCNN (multiscale convolutional neural network). The computing environment 200 may use the same or different types of predictive models trained to segment non-target regions, segment target regions, or provide image analysis of target regions. For example, the computing environment 200 may include a first predictive model (e.g., a U-Net) for segmenting non-target regions (e.g., lymphatic aggregate regions, necrotic regions, or any other suitable regions).The computing environment 200 may also include a second predictive model (e.g., a 2DCNN) for segmenting a target region (e.g., a region of tumor cells). The computing environment 200 may also include a third model (e.g., a CNN) for image analysis of the target region. The computing environment 200 may also include a fourth model (e.g., an HMM) for diagnosing a disease for treatment or prognosis of a subject, such as a patient. In other examples consistent with the present disclosure, still other types of predictive models may be implemented. Furthermore, multiple models may be used to classify different cell types and regions.

[0058] In various embodiments, each predictive model 215a-n corresponding to a classifier subsystem 210a-n can be based on the global model 112, 114 provided by the server 110. In various embodiments, each predictive model 215a-n corresponding to a classifier subsystem 210a-n is further trained separately based on one or more sets of input image elements 220a-n. In some embodiments, each of the input image elements 220a-n includes image data from one or more scanned slides. Each of the input image elements 220a-n can correspond to image data from a single specimen and / or day on which the underlying image data corresponding to the image was collected. The image data can include any information about the image as well as the imaging platform on which the image was generated. For example, a tissue section may need to be stained by application of a staining assay containing one or more different biomarkers associated with a chromogenic stain for brightfield imaging or a fluorophore for fluorescent imaging. The staining assay can use chromogenic stains, organic fluorophores, quantum dots, or organic fluorophores for brightfield imaging, together with quantum dots for fluorescent imaging, or any other combination of stains, biomarkers, and observation or imaging devices. Furthermore, typical tissue sections are processed in an automated staining / assay platform that applies the staining assay to the tissue section, resulting in a stained sample. Various commercially available products suitable for use as staining / assay platforms exist on the market, one example being the VENTANA SYMPHONY product from the assignee, Ventana Medical Systems, Inc. The stained tissue section can be fed into an imaging system, for example, on a microscope or a whole slide scanner with a microscope and / or imaging components, one example being the VENTANA iScan Coreo product from the assignee, Ventana Medical Systems, Inc. Multiple tissue slides can be scanned using an equivalent multiple slide scanner system.Additional information provided by the imaging system can include any information regarding the staining platform, including the concentration of chemicals used for staining, the reaction time of the chemicals applied to the tissue in the stain, and / or the pre-analysis conditions of the tissue, such as the age of the tissue, fixation method, duration, how the sections were embedded, cut, etc.

[0059] The input image elements 220a-n can include one or more training input image elements 220a-d, validation input image elements 220e-g, and unlabeled input image elements 220h-n. It should be understood that the input image elements 220a-n corresponding to the training, validation, and unlabeled groups do not need to be accessed simultaneously. For example, a set of training and validation input image elements 220a-n may be initially accessed and used to further train the predictive model 215, and the unlabeled input image elements may be subsequently accessed or received (e.g., at one or more subsequent times) and used by the further trained predictive model 215 to provide a desired output (e.g., segmentation of non-target regions). In some examples, the predictive models 215a-n are trained using supervised training, with each of the training input image elements 220a-d and optional validation input image elements 220e-g being associated with one or more labels 225 that identify non-target regions, the "correct" interpretation of target regions, and the identity of various biological materials and structures within the training input image elements 220a-d and validation input image elements 220e-g. The labels may alternatively or additionally be used to classify the corresponding training input image elements 220a-d and validation input image elements 220e-g or pixels therein with respect to the presence and / or interpretation of staining associated with normal or abnormal biological structures (e.g., tumor cells). In certain examples, alternatively or additionally, the labels may be used to classify the corresponding training input image elements 220a-d and validation input image elements 220e-g at a time corresponding to when the underlying image was captured or at a subsequent time point (e.g., this is a predetermined period of time following the time the image was captured).

[0060] In some embodiments, the classifier subsystems 210a-n include a feature extractor 230, a parameter data store 235, a classifier 240, and a trainer 245, which are collectively used to train the predictive model 215 based on training data (e.g., training input image elements 220a-d) and optimize parameters of the predictive model 215 during supervised or unsupervised training. In some examples, the training process involves iterative operations to find a set of parameters for the predictive model 215 that minimizes a loss function for the predictive model 215. Each iteration may include finding a set of parameters for the predictive model 215 such that the value of the loss function using the set of parameters is less than the value of the loss function using a different set of parameters in the previous iteration. The loss function may be constructed to measure the difference between the output predicted using the predictive model 215 and the labels 225 contained in the training data. Once the set of parameters is identified, the predictive model 215 is trained and can be utilized for segmentation and / or prediction as designed.

[0061] In some embodiments, the classifier subsystems 210a-n access training data from the training input image elements 220a-d at an input layer. The feature extractor 230 can preprocess the training data to extract relevant features (e.g., edges, color, texture, or any other suitable relevant features) detected in specific portions of the training input image elements 220a-d. The classifier 240 can receive the extracted features and, according to weights associated with a set of hidden layers in one or more predictive models 215, convert the features into one or more output metrics that segment non-target or target regions, provide image analysis, provide a diagnosis of disease for treatment or prognosis of a subject, such as a patient, or a combination thereof. The trainer 245 can use the training data corresponding to the training input image elements 220a-d to train the feature extractor 230 and / or the classifier 240 by facilitating learning of one or more parameters. For example, the trainer 245 can use backpropagation techniques to facilitate learning of weights associated with a set of hidden layers of the predictive models 215 used by the classifier 240. Backpropagation may cumulatively update hidden layer parameters using, for example, a stochastic gradient descent (SGD) algorithm. The learned parameters may include, for example, weights, biases, and / or other hidden layer-related parameters, which may be stored in parameter data store 235.

[0062] Individual or ensembles of trained predictive models can be deployed to process unlabeled input image elements 220h-n to segment non-target or target regions, provide image analysis, provide a diagnosis of disease for treatment or prognosis of a subject, such as a patient, or a combination thereof. More specifically, a trained version of feature extractor 230 can generate feature representations of the unlabeled input image elements that can then be processed by a trained version of classifier 240. In some embodiments, image features can be extracted from the unlabeled input image elements 220h-n based on one or more convolutional blocks, convolutional layers, residual blocks, or pyramid layers that leverage an extension of predictive model 215 within classifier subsystems 210a-n. The features can be organized into a feature representation, such as a feature vector for the image. Predictive model 215 can be trained to learn feature types based on classification and subsequent adjustment of parameters in hidden layers, including fully connected layers, of predictive model 215.

[0063] In some embodiments, the image features extracted by the convolution block, convolution layer, residual block, or pyramid layer include feature maps, which are matrices of values ​​representing one or more portions of the specimen slide on which one or more image processing operations (e.g., edge detection, image resolution sharpening) have been performed. These feature maps can be flattened for processing by a fully connected layer of the predictive model 215, which outputs a non-target region mask, a target region mask, or one or more metrics corresponding to current or future predictions about the specimen slide. For example, input image elements can be fed to an input layer of the predictive model 215. The input layer can include nodes corresponding to particular pixels. A first hidden layer can include a set of hidden nodes, each of which is connected to multiple input layer nodes. Nodes in subsequent hidden layers can similarly be configured to receive information corresponding to multiple pixels. Thus, the hidden layers can be configured to learn to detect features that extend across multiple pixels. Each of the one or more hidden layers can include a convolution block, convolution layer, residual block, or pyramid layer. The predictive model 215 may further include one or more fully connected layers (e.g., a softmax layer).

[0064] At least some of the training input image elements 220a-d, validation input image elements 220e-g, and / or unlabeled input image elements 220h-n may include or be derived from data obtained directly or indirectly from sources that may, but need not, be elements of the analysis system 205. In some embodiments, the computing environment 200 includes an imaging device 250 that images a sample to obtain image data, such as a multichannel image (e.g., a multichannel fluorescence or brightfield image) having several (e.g., between 10 and 16) channels. The imaging device 250 includes, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), an optical system (e.g., one or more lenses, a sensor-focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) image sensor, etc.), photographic film, etc. In digital embodiments, the image capture device may include multiple lenses that cooperate to demonstrate on-the-fly focusing. An image sensor, e.g., a CCD sensor, can capture a digital image of the specimen. In some embodiments, imaging device 250 is a bright-field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. Imaging device 250 can capture images using invisible electromagnetic radiation (e.g., UV light) or other imaging techniques. For example, imaging device 250 can include a microscope and a camera configured to capture images magnified by the microscope. The image data received by image analysis system 205 can be identical to and / or derived from the raw image data captured by imaging device 250.

[0065] In some examples, the labels 225 associated with the training input image elements 220a-d and / or the validation input image elements 220e-g may be received or derived from data received from one or more provider systems 255, each of which may be associated with (e.g.,) a doctor, nurse, hospital, pharmacist, etc. associated with a particular subject. The received data may include (e.g.,) one or more medical records corresponding to a particular subject. The medical records may indicate (e.g.,) a specialist's diagnosis or characterization indicating whether the subject had a tumor and / or the stage of progression of the subject's tumor (e.g., along a standard scale and / or by identifying a metric, such as total metabolic tumor burden (TMTV)) for a time period corresponding to the time when one or more input image elements associated with the subject were collected or a defined period thereafter. The received data may further include pixels of the location of the tumor or tumor cells within one or more input image elements associated with the subject. Thus, the medical records may include or be used to identify one or more labels for each training / validation input image element 220a-g. The medical record may further indicate one or more treatments (e.g., medications) that the subject was receiving and the duration for which the subject was receiving each of the treatments. In some examples, the images or scans that are input to the one or more classifier subsystems are received from a provider system 255. For example, the provider system 255 may receive images from the imaging device 250 and then transmit the images or scans (e.g., along with a subject identifier and one or more labels) to the analysis system 205.

[0066] In some embodiments, data received or collected by one or more of the imaging devices 250 may be aggregated with data received or collected by one or more of the provider systems 255. For example, the analysis system 205 can identify corresponding or identical identifiers for subjects and / or time periods to associate image data received from the imaging devices 250 with label data received from the provider systems 255. The analysis system 205 can further use metadata or automated image analysis to process the data and determine which classifier subsystem to provide particular data components to. For example, the image data received from the imaging devices 250 can correspond to an entire slide or multiple regions of a slide or tissue. The metadata, automated alignment, and / or image processing can indicate, for each image, which region of the slide or tissue the image corresponds to, the type of stain used, the concentration of the stain used, the laboratory where the stain was performed, a timestamp, the type of scanner used, or any other appropriate data as would be understood by one skilled in the art. The automated alignment and / or image processing can include detecting whether the image has image characteristics corresponding to the slide substrate or biological structures and / or shapes associated with particular cells, such as white blood cells. The label-associated data received from the provider system 255 may be slide-specific, region-specific, or subject-specific. If the label-associated data is slide-specific or region-specific, metadata or automated analysis (e.g., using natural language processing or text analysis) can be used to identify which region the particular label-associated data corresponds to. If the label-associated data is subject-specific, the same label data (for a given subject) can be provided to each classifier subsystem 210a-n during training.

[0067] In some embodiments, computing environment 200 may further include a user device 260 that can be associated with a user requesting and / or coordinating the execution of one or more iterations of analysis system 205 (e.g., each iteration corresponds to one execution of a model and / or one generation of a model's output). A user may correspond to a physician, a researcher (e.g., associated with a clinical trial), a patient, a medical professional, etc. Accordingly, it will be understood that in some examples, provider system 255 may include and / or function as user device 260. Each iteration may be associated with a particular subject (e.g., a person) that may (but need not) be different from the user. A request for an iteration may include and / or be accompanied by information about the particular subject (e.g., the subject's name or other identifier, such as a de-identified patient identifier). A request for an iteration may include identifiers of one or more other systems that collect data, such as input image data, corresponding to the subject. In some examples, a communication from user device 260 corresponds to a request to perform an iteration for each subject represented in a particular set of subjects, including an identifier for each of the set.

[0068] Upon receiving the request, analysis system 205 can transmit a request for unlabeled input image elements (e.g., including a subject identifier) ​​to one or more corresponding imaging systems 250 and / or provider systems 255. Trained predictive model 215 can then process the unlabeled input image elements to segment non-target or target regions, provide image analysis, provide a diagnosis of disease for treatment or prognosis of a subject, such as a patient, or any combination thereof. The results for each identified subject can include or be based on the segmentation and / or one or more output metrics from trained predictive model 215 developed by classifier subsystems 110a-n. For example, the segmentation and / or one or more output metrics can include or be based on outputs generated by one or more fully connected layers of a CNN. In some examples, such outputs may be further processed using (for example) a softmax function. Additionally, the outputs and / or further processed outputs can then be aggregated using aggregation techniques (e.g., random forest aggregation) to generate one or more subject-specific metrics. One or more results (e.g., including the plane-specific output and / or one or more subject-specific outputs and / or processed versions thereof) can be transmitted to and / or utilized by user device 260. In some examples, some or all of the communication between analysis system 205 and user device 260 occurs via a website. It will be appreciated that CNN system 205 can gate access to results, data, and / or processing resources based on authentication analysis.

[0069] Although not explicitly shown, it will be understood that computing environment 200 can further include a developer device associated with a developer. Communications from the developer device can indicate what type of input image elements should be used for each predictive model 215 in analysis system 205, the number of neural networks to be used, the configuration of each neural network including the number of hidden layers and hyperparameters, as well as how data requests should be formatted and / or what training data should be used (e.g., and how to access the training data).

[0070] 3 illustrates an exemplary schematic diagram 300 depicting a model architecture (e.g., a portion of the analysis system 205 described with respect to FIG. 2) for non-target region segmentation according to various embodiments. The model architecture may include a pre-processing stage 305 comprising an image acquisition module 310 for generating or acquiring input images including single image data (e.g., images each having a single stain) and / or multiplexed image data (e.g., images having multiple stains), an optional image annotation module 315 for electronically annotating portions of the input images for further analysis, such as portions indicating tumor or immune cell regions, and an optional unmixing module 320 for generating image channel images corresponding to one or more stain channels present in the multiplexed image. The model architecture may further include a processing stage 325 comprising an image analysis module 330 for detecting and / or classifying biological materials or structures, including cells or nuclei (e.g., tumor cells, stromal cells, lymphocytes, etc.), based on features within the input images (e.g., within hematoxylin and eosin-stained images, biomarker images, or unmixed image channel images).

[0071] The model architecture may further comprise a post-processing stage 335 comprising an optional scoring module 340 for deriving an expression prediction and / or score for each biomarker in each of the identified regions or biological structures, and an optional metric generation module 345 for deriving a metric describing the variability between the derived expression predictions and / or scores in different regions or biological structures, and optionally providing a diagnosis of disease for treatment or prognosis of a subject, such as a patient. The model architecture may further comprise a segmentation and masking module 350 for segmenting regions or biological structures, such as lymphocyte aggregates or clusters of tumor cells, in the input image and generating a mask based on the segmented regions or biological structures, and an optional registration module 355 for mapping identified regions or biological structures (e.g., tumor cells or immune cells) from a first image or first set of images in the input image to at least one additional image or multiple additional images. The segmentation and masking module 350 and the optional registration module 355 may be implemented within the pre-processing stage 305, the processing stage 325, the post-processing stage 335, or any combination thereof.

[0072] In some embodiments, the image acquisition module 310 generates or acquires images or image data of a biological sample having one or more stains (e.g., the images may be single or multiple images). In some embodiments, the generated or acquired images are RGB images or multispectral images. In some embodiments, the generated or acquired images are stored in a memory device. The images or image data (used interchangeably herein) can be generated or acquired, for example, in real time, using an imaging device (e.g., the imaging device 250 described with respect to FIG. 2). In some embodiments, the images are generated or acquired from a microscope or other instrument capable of capturing image data of a microscope slide holding a specimen, as described herein. In some embodiments, the images are generated or acquired using a 2D scanner, such as one capable of scanning image tiles. Alternatively, the images may be images previously generated (e.g., scanned) and stored in a memory device (or, in that case, retrieved from a server via a communications network).

[0073] In some embodiments, the image acquisition module 310 is used to select a portion of the biological sample from which one or more images or image data are to be acquired. For example, the image acquisition module 310 can receive an identified region of interest or field of view (FOV). In some embodiments, the region of interest is identified by a user of the system of the present disclosure or another system communicatively coupled to the system of the present disclosure. Alternatively, in other embodiments, the image acquisition module 305 retrieves the location or identification of the region or interest from a storage / memory device. In some embodiments, the image acquisition module 310 automatically generates the field of view or region of interest (ROI), for example, via the methods described in PCT / EP2015 / 062015, the contents of which are incorporated herein in their entirety for all purposes. In some embodiments, the ROI is automatically determined by the image acquisition module 305 based on some predetermined criteria or characteristics within or of the image (e.g., for a biological sample stained with three or more stains, identifying a region of the image containing only two stains). In some examples, the image acquisition module 310 outputs the ROI.

[0074] In some embodiments, the image acquisition module 310 generates or acquires at least two images as input. In some embodiments, the images generated or acquired as input are obtained from successive tissue sections, e.g., serial sections from the same tissue sample. Generally, the at least two images received as input each include a signal corresponding to a stain (including a chromogen, a fluorophore, a quantum dot, etc.). In some embodiments, one of the images is stained with at least one primary stain (e.g., hematoxylin or eosin (H&E)), and another of the images is stained with at least one IHC assay or in-situ hybridization (ISH) assay to identify specific biomarkers. In some embodiments, one of the images is stained with both hematoxylin and eosin, and another of the images is stained with at least one IHC assay or ISH assay to identify specific biomarkers. In some embodiments, the input images are multiplexed images, e.g., stained for multiple different markers in a multiplexed assay according to methods known to those of skill in the art.

[0075] In some embodiments, the generated or acquired image is optionally annotated by a user (e.g., a medical professional such as a pathologist) for image analysis using the image annotation module 315. In some embodiments, the user identifies portions (e.g., subregions) of the image suitable for further analysis. The target or non-target regions (e.g., tumor or immune regions) annotated to generate a slide score may be either the entire tissue region or a designated set of regions on the digital slide. For example, in some embodiments, the identified portion represents a tumor region overexpressing a specific biomarker, such as a specific IHC marker. In other embodiments, the user, medical professional, or pathologist may annotate lymphocyte aggregate regions within the digital slide. In some embodiments, the annotated representative field can be selected by the pathologist to reflect the biomarker expression that the pathologist will use for overall slide interpretation. Annotations may be drawn using annotation tools provided in the viewer application (e.g., VENTANA VIRTUOSO software), and annotations may be drawn at any magnification or resolution. Alternatively or additionally, image analysis operations can be used to automatically detect target and non-target or other regions using automated image analysis operations such as segmentation, thresholding, edge detection, and an automatically generated field of view (FOV—a portion of an image having a predetermined size and / or shape) based on the detected regions. In some embodiments, user annotations can be utilized to further train one or more of the models.

[0076] In some embodiments, the generated or acquired image may be a multiplex image, i.e., the received image is an image of a biological sample stained with two or more stains. In these embodiments, before further processing, each multiplex image is first unmixed into its constituent channels, such as using the unmixing module 320, with each unmixed channel corresponding to a particular stain or signal. In some embodiments, the unmixed image (often referred to as a "channel image" or "image channel image") can be used as input for each module described herein. For example, a model architecture can be implemented to assess inter-marker heterogeneity (an indicator of the amount of protein expression heterogeneity of biomarkers in a sample) determined using a first H&E image, a second multiplex image stained for multiple differentiation marker clusters (e.g., CD3, CD8), and multiple single images each stained for a particular biomarker (e.g., ER, PR, Ki67, etc.). In this example, the multiplex image is first unmixed into its constituent channel images, which can be used together with the H&E image and the multiple single images to determine inter-marker heterogeneity.

[0077] Following image acquisition and / or unmixing, the input image or unmixed image channel images are processed by image analysis algorithms provided by the image analysis module 330 to identify and classify cells and / or nuclei. The procedures and algorithms described herein can be adapted to identify and classify various types of cells or cell nuclei based on features within the input image, including identifying and classifying tumor cells, non-tumor cells, stromal cells, lymphocytes, non-target staining, etc. Those skilled in the art should understand that the nuclei, cytoplasm, and membranes of cells have different characteristics, and that different stained tissue samples may exhibit different biological characteristics. Specifically, those skilled in the art should understand that certain cell surface receptors may have staining patterns that are membrane-localized or cytoplasm-localized. Thus, a "membrane" staining pattern is analytically distinct from a "cytoplasmic" staining pattern. Similarly, a "cytoplasmic" staining pattern is analytically distinct from a "nuclear" staining pattern. Each of these distinct staining patterns can be used as a feature to identify cells and / or nuclei. For example, stromal cells can be strongly stained by FAP, whereas tumor epithelial cells can be strongly stained by EpCAM, while cytokeratin can be stained by panCK. Thus, by utilizing different stains, different cell types can be identified and distinguished during image analysis, providing a classification solution.

[0078] A method for identifying, classifying, and / or scoring nuclei, cell membranes, and cell cytoplasm in an image of a biological sample having one or more stains is described in U.S. Patent No. 7,760,927 (the "'927 patent"), the contents of which are incorporated herein in their entirety for all purposes. For example, the '927 patent describes an automated method for simultaneously identifying multiple pixels in an input image of biological tissue stained with a biomarker, including: considering a first color plane of multiple pixels in the foreground of the input image for simultaneous identification of cytoplasmic and cell membrane pixels, the first color plane being processed to remove background portions of the input image and to remove counterstained components of the input image; determining a threshold level between the cytoplasmic and cell membrane pixels in the foreground of the digital image; and simultaneously determining, using the determined threshold level, whether a selected pixel is a cytoplasmic pixel, a cell membrane pixel, or a transition pixel in the digital image, together with the selected pixel and its eight neighboring pixels from the foreground. In some embodiments, tumor nuclei are automatically identified by first identifying candidate nuclei and then automatically distinguishing between tumor nuclei and non-tumor nuclei. Many methods for identifying candidate nuclei in tissue images are known in the art. For example, automatic candidate nuclei detection can be performed by applying Parvin's radial symmetry-based method, such as in the hematoxylin image channel or biomarker image channel after unmixing (see Parvin, Bahram et al., "Iterative voting for inference of structural saliency and characterization of subcellular events," Image Processing, IEEE Transactions on 16.3 (2007): 615-623, the disclosure of which is incorporated herein by reference in its entirety).

[0079] For example, in some embodiments, an image acquired as input is processed to detect nuclear centers (seeds) and / or segment nuclei, etc. For example, instructions can be provided and executed to detect nuclear centers based on radial symmetry voting using the technique of Parvin (supra). In some embodiments, nuclei are detected using radial symmetry to detect nuclear centers, and then the nuclei are classified based on the intensity of staining around the cell center. In some embodiments, a radial symmetry-based nuclei detection operation is used, as described in commonly-owned, co-pending patent application WO2014140085, the contents of which are incorporated herein in their entirety for all purposes. For example, image magnitudes can be calculated within the image, and one or more votes at each pixel are accumulated by adding the sum of magnitudes within a selected region. Mean-shift clustering can be used to find local centers within the region, which represent the location of the actual nuclei. Radial symmetry voting-based nuclei detection is performed on color image intensity data and explicitly utilizes a priori domain knowledge that nuclei are elliptical blobs with varying sizes and eccentricities. To achieve this, image gradient information, along with the color intensity of the input image, is used in radial symmetry voting, combined with an adaptive segmentation process, to accurately detect and locate cell nuclei. As used herein, "gradient" refers to a pixel intensity gradient calculated for a particular pixel by, for example, taking into account the intensity value gradients of a set of pixels surrounding the particular pixel. Each gradient can have a specific "direction" relative to a coordinate system whose x- and y-axes are defined by two orthogonal edges of the digital image. For example, nucleus seed detection involves defining a seed as a point assumed to be inside a cell nucleus and serving as a starting point for locating the cell nucleus. The first step is to detect seed points associated with each cell nucleus using a highly robust approach based on radial symmetry to detect elliptical blobs, which are structures resembling cell nuclei. The radial symmetry approach operates on gradient images using a kernel-based voting procedure.The voting response matrix is ​​created by processing each pixel accumulating votes through a voting kernel. The kernel is based on the gradient direction calculated at that particular pixel and the expected range of minimum and maximum kernel sizes and voting kernel angles (typically in the range [p / 4, p / 8]). In the resulting voting space, local maxima with vote values ​​higher than a predefined threshold are saved as seed points. Irrelevant seeds can be discarded later during subsequent segmentation or classification processes. Other methods are described in U.S. Patent Application Publication No. 2017 / 0140246, the disclosure of which is incorporated herein by reference.

[0080] After candidate nuclei are identified, they can be further analyzed to distinguish tumor nuclei from other candidate nuclei. Other candidate nuclei can be further classified (e.g., by distinguishing lymphocyte nuclei and stromal nuclei). In some embodiments, as further described herein, a trained supervised classifier is applied to identify tumor nuclei. For example, a trained supervised classifier is trained on nuclear features to identify tumor nuclei and then applied to classify nucleus candidates in the test image as either tumor nuclei or non-tumor nuclei. In some cases, the trained supervised classifier may be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the trained supervised classifier used to identify tumor nuclei is a random forest classifier. For example, the random forest classifier can be trained by: (i) creating a training set of tumor nuclei and non-tumor nuclei, (ii) extracting features of each nucleus, and (iii) training the random forest classifier to distinguish between tumor nuclei and non-tumor nuclei based on the extracted features. The trained random forest classifier can then be applied to classify nuclei in the test image into tumor and non-tumor nuclei. Optionally, the random forest classifier may be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte and stromal nuclei.

[0081] Nuclei can be identified using other techniques known to those skilled in the art. For example, image dimensions can be calculated from one specific image channel of an FI&E or IHC image, and each pixel around a specified dimension can be assigned a vote based on the sum of the dimensions within the area around the pixel. Alternatively, a mean-shift clustering operation can be performed to find a local center within the voting image that represents the actual location of the nucleus. In other embodiments, nuclei segmentation can be used to segment entire nuclei based on currently known nuclear centers through morphological operations and local thresholding. In yet other embodiments, model-based segmentation can be utilized to detect nuclei (i.e., learning a shape model of the nucleus from a training dataset and using it as prior knowledge to segment nuclei in test images).

[0082] In some embodiments, the nuclei are then segmented using a threshold calculated individually for each nucleus. For example, Otsu's method can be used to segment the region around the identified nuclei, since the pixel intensity in the nuclear region is expected to vary. As will be understood by those skilled in the art, Otsu's method is used to determine the optimal threshold by minimizing intra-class variance and is known to those skilled in the art. More specifically, Otsu's method is used to automatically perform clustering-based image thresholding, or the reduction of gray-level images to binary images. This algorithm assumes that the image contains two classes of pixels (foreground and background pixels) that follow a bimodal histogram. Next, an optimal threshold is calculated to separate the two classes so that their combined spread (intra-class variance) is minimized or equal (since the sum of pairwise squared distances is constant) so that the inter-class variance is maximized.

[0083] In some embodiments, the system and method further include automatically analyzing the spectral and / or shape characteristics of the identified nuclei in the image to identify nuclei of non-tumor cells. For example, blobs can be identified in the first digital image of the first step. As used herein, a "blob" can refer to, for example, a region of a digital image in which some characteristic, such as intensity or gray value, is constant or varies within a specified range of values. All pixels within a blob can be considered to be similar to one another in some sense. For example, blobs can be identified using differential methods based on the derivative of a function of position on the digital image and methods based on local extrema. A nuclear blob is a blob whose pixels and / or its contour shape indicate that the blob is likely generated by nuclei stained with the first stain. For example, the radial symmetry of the blob can be evaluated to determine whether the blob should be identified as a nuclear blob or any other structure, such as a staining artifact. For example, if a blob has an elongated shape and is not radially symmetric, the blob can be identified as a staining artifact rather than a nuclear blob. Depending on the embodiment, a blob identified as a "nuclear blob" can represent a set of pixels that can be identified as a candidate nucleus and further analyzed to determine whether the nuclear blob represents a nucleus. In some embodiments, any type of nuclear blob can be used directly as an "identified nucleus." In some embodiments, filtering operations are applied to identified nuclei or nuclear blobs to identify nuclei that do not belong to biomarker-positive tumor cells, and to remove the identified non-tumor nuclei from the list of already identified nuclei or not add the nuclei to the list of identified nuclei from scratch. For example, additional spectral and / or shape features of the identified nuclear blob can be analyzed to determine whether the nucleus or nuclear blob is a tumor cell nucleus. For example, the nuclei of lymphocytes are larger than the nuclei of other tissue cells, such as lung cells.If tumor cells originate from lung tissue, lymphocyte nuclei are identified by identifying all nuclear blobs with a minimum size or diameter significantly larger than the average size or diameter of normal lung cell nuclei. Identified nuclear blobs associated with lymphocyte nuclei can be removed (i.e., "filtered out") from the set of previously identified nuclei. Removing nuclei from non-tumor cells can improve the accuracy of the method. Depending on the biomarker, non-tumor cells may also express the biomarker to some extent and thus generate intensity signals in the first digital image that are not derived from tumor cells. Identifying and filtering nuclei that do not belong to tumor cells from the set of previously identified nuclei can improve the accuracy of identifying biomarker-positive tumor cells. These and other methods are described in U.S. Patent Application Publication No. 2017 / 0103521, the contents of which are incorporated herein in their entirety for all purposes. In some embodiments, once a seed is detected, a locally adaptive thresholding method can be used to create a blob around the detected center. In some embodiments, other methods can be incorporated, such as a marker-based watershed algorithm that can also be used to identify nuclear blobs around the detected nuclear centers. These and other methods are described in PCT / EP2016 / 051906, published as WO 2016 / 120442, the contents of which are incorporated herein in their entirety for all purposes.

[0084] In some embodiments, various marker expression scores are calculated for each stain or biomarker in each cell cluster in each image (either a single image from multiple images or an unmixed image channel image) using scoring module 340. Scoring module 340, in some embodiments, utilizes data acquired during cell detection and classification by image analysis module 330. For example, image analysis module 330 can include a series of image analysis algorithms and can be used to determine the presence of one or more of nuclei, cell walls, tumor cells, or other structures within identified cell clusters, as described herein. In some embodiments, the derived staining intensity values ​​and counts of specific nuclei for each field of view can be used by scoring module 340 to determine various marker expression scores, such as percent positive or H-score. Scoring methods are described in further detail in commonly assigned, co-pending applications WO 2014 / 102130, filed December 19, 2013, entitled "Image analysis for breast cancer prognosis," and WO 2014 / 140085, filed March 12, 2014, entitled "Tissue object-based machine learning system for automated scoring of digital whole slides," the contents of each of which are incorporated herein by reference in their entirety. For example, automated image analysis algorithms within image analysis module 330 can be used to interpret each of a series of IFIC slides to detect tumor nuclei that are positively and negatively stained for specific biomarkers, such as Ki67, ER, PR, and FIER2. Based on the detected positive and negative tumor nuclei, various slide-level scores, such as percent marker positivity, H-scores, etc., can be calculated using scoring module 340.

[0085] In some embodiments, the expression score is an H-score, which is used to assess the percentage of tumor cells with cell membrane staining graded as "weak," "moderate," or "strong." Grades are summed to give an overall maximum score of 300 and a cutoff point of 100 to distinguish between "positive" and "negative." For example, membrane staining intensity (0, 1+, 2+, or 3+) is determined for each cell in a fixed field (or each cell in a tumor or cell cluster, as used herein). The H-score can be based simply on the predominant staining intensity, or more complex, it can include the sum of individual H-scores for each intensity level observed. In other embodiments, the expression score is an Allred score. The Allred score is a scoring system that examines the percentage of cells that test positive for hormone receptors and how well the receptor appears after staining (this is called "intensity"). In other embodiments, the expression score is a percent positive. In the context of scoring breast cancer samples stained for PR and Ki-67 biomarkers, PR and Ki-67 slides versus PR and Ki-67 slides, the percent positive in a single slide is calculated as follows (e.g., the total number of positively stained cell nuclei (e.g., malignant cells) in each field of a digital image of a slide are added and divided by the total number of positively and negatively stained nuclei from each field of the digital image): percent positive = number of positively stained cells / (number of positively stained cells + number of negatively stained cells). In other embodiments, the expression score is an IHC combined score, which is a prognostic score based on a number of IHC markers, where the number of markers is greater than one. IHC4 is one such score based on four measured IHC markers in breast cancer samples: ER, HER2, Ki-67, and PR (e.g., Cuzick et al., J. Clin. Oncol. 29:4273-8, 2011, and Barton et al., Br. J. Cancer 1-6, April 24, 2012, both incorporated herein by reference).

[0086] Following image analysis and determination of expression scores for each marker within each identified or mapped cluster, metrics can be derived from the various identified clusters and biological structures using the metric generation module 345. In some examples, morphological metrics can be calculated by applying various image analysis algorithms to pixels contained in or surrounding a nuclear blob or seed. In some embodiments, morphological metrics include area, minor and major axis lengths, perimeter, radius, solidity, etc. At the cellular level, such metrics can be used to classify nuclei as belonging to healthy or diseased cells. At the tissue level, statistics of these features across the tissue are utilized to classify whether the tissue is diseased or not. In some examples, appearance metrics can be calculated for a particular nucleus by comparing pixel intensity values ​​of pixels contained in or surrounding the nuclear blob or seed used to identify the nucleus, whereby the pixel intensities compared are derived from different image channels (e.g., a background channel, a channel for biomarker staining, etc.). In some embodiments, metrics derived from appearance features are calculated from percentile values ​​(e.g., 10th, 50th, and 95th percentiles) of pixel intensities and gradient magnitudes calculated from different image channels. For example, first, the number P (X=10, 50, 95) of X percentile values ​​of pixel values ​​in each of the image channels (e.g., three channels: HTX, DAB, and luminance) of multiple ICs within a nuclear blob representing a nucleus of interest is identified. Calculating appearance feature metrics can be advantageous because the derived metrics can describe characteristics of the nuclear region and the membrane region surrounding the nucleus.

[0087] In some examples, background metrics can be calculated that indicate the appearance and / or presence of staining in the cytoplasm and cell membrane features of cells, including nuclei, whose background features are extracted from the image. Background features and corresponding metrics can be calculated for nuclei and corresponding cells depicted in a digital image, for example, by identifying nuclear blobs or seeds representing nuclei and analyzing pixel regions (e.g., a 20-pixel ribbon approximately 9 microns thick around the nuclear blob boundary) directly adjacent to the set of identified cells, thus capturing the appearance and presence of staining in the cytoplasm and membrane of the cells using this nucleus along with regions directly adjacent to the cell. In some examples, color metrics can be derived from colors including the color ratio R / (R+G+B) or color principal components. In other embodiments, color metrics derived from colors include local statistics (mean / median / variance / standard deviation) of each color and / or color intensity correlation within a local image window. In some examples, intensity metrics can be derived from a group of neighboring cells with a specific characteristic value that is set between the dark and white tones of the gray cells represented in the image. Correlations of color features can define instances of size classes, and thus determine cells whose intensity is affected by the cluster of surrounding dark cells.

[0088] In some examples, other features may be considered and used as the basis for calculating metrics, such as texture or spatial features. As another example, expression scoring can be utilized as a predictive measure or to guide treatment. For example, in the context of breast cancer and ER and PR biomarkers, a sample testing positive can guide the decision to provide hormone therapy during the treatment process. Those skilled in the art will also understand that not all clusters within a biological sample may have the same score for any particular marker. Being able to determine a heterogeneity score or metric describing the variability between clusters can provide additional guidance for making informed treatment decisions. In some embodiments, heterogeneity is determined to measure how different clusters compare to each other. Heterogeneity can be measured, for example, by a variability metric that describes how protein expression levels differ between various identified and mapped clusters, as described in International Publication No. 2019110567, the contents of which are incorporated herein in their entirety for all purposes. In some embodiments, heterogeneity is measured among all identified clusters. In other embodiments, heterogeneity is measured only among a subset of the identified clusters (eg, clusters that meet certain predetermined criteria).

[0089] In some embodiments, images received as input can be segmented and masked by the segmentation and masking module 350. For example, a trained convolutional neural network architecture or model can be used to segment non-target and / or target regions, and then the image can be masked for analysis before, during, or after inputting the image into an image analysis algorithm. In some embodiments, the input image is masked so that only tissue regions are present in the image. In some embodiments, a tissue region mask is generated to mask non-tissue regions from tissue regions. In some embodiments, a tissue region mask can be created by identifying tissue regions and excluding background regions (e.g., regions of the whole slide image corresponding to glass without a sample, such as when only white light from the imaging source is present).

[0090] In some embodiments, a segmentation technique is used to generate a tissue region mask image by masking tissue regions from non-tissue regions in the input image. In some embodiments, image segmentation techniques are utilized to distinguish between digitized tissue data and slides in an image, tissue corresponding to the foreground, and slides corresponding to the background. In some embodiments, the segmentation and masking module 350 calculates an area of ​​interest (AOI) in the whole slide image to detect all tissue regions within the AOI while limiting the amount of background non-tissue regions analyzed. A wide range of image segmentation techniques (e.g., HSV color-based image segmentation, Lab image segmentation, mean-shift color image segmentation, region growing, level set methods, fast marching methods, etc.) can be used to determine, for example, boundaries between tissue data and non-tissue or background data. Based at least in part on the segmentation, the segmentation and masking module 350 can generate a tissue foreground mask that can be used to identify portions of the digitized slide data that correspond to tissue data. Alternatively, the component can generate a background mask that can be used to identify portions of the digitized slide data that do not correspond to tissue data.

[0091] This identification can be enabled by image analysis operations such as edge detection. A tissue region mask can be used to remove non-tissue background noise, e.g., non-tissue regions, in an image. In some embodiments, generating a tissue region mask comprises one or more of the following operations (but is not limited to): calculating the intensity of a low-resolution input image, generating an intensity image, applying a standard deviation filter to the intensity image, generating a filtered intensity image, and applying a threshold to the filtered intensity image such that pixels with an intensity above a given threshold are set to 1 and pixels below the threshold are set to zero, generating a tissue region mask. Additional information and examples regarding generating tissue region masks are disclosed in PCT / EP / 2015 / 062015, entitled "An Image Processing Method and System for Analyzing a Multi-Channel Image Obtained from a Biological Tissue Sample Being Stained by Multiple Stains," the contents of which are incorporated herein in their entirety for all purposes.

[0092] In addition to masking non-tissue regions from tissue regions, the segmentation and masking module 350 can also mask other regions of interest, such as non-target regions or portions of tissue identified as belonging to a particular tissue type (e.g., lymphoid aggregate regions), or target regions or portions of tissue identified as belonging to a particular tissue type (e.g., suspicious tumor regions), as needed. In various embodiments, non-target region segmentation, such as lymphoid aggregate region segmentation, is performed by a CNN model (e.g., the CNN model associated with the classifier subsystem 210a described with reference to FIG. 2). In some embodiments, the CNN model is a two-dimensional segmentation model. For example, the CNN model may be a U-Net with residual blocks, dilation, and depthwise convolutions. Preprocessed or processed image data (e.g., two-dimensional regions or full slide images) can be used as input to the U-Net. The U-Net includes a reduction path supplemented with an augmentation path, where pooling operations in successive layers in the augmentation path are replaced with upsampling operators. Thus, these successive layers increase the resolution of the output. Based at least in part on the segmentation, the U-Net can generate a non-target area foreground mask that can be used to identify portions of the digitized slide data that correspond to the non-target area data. Alternatively, the component can generate a background mask that can be used to identify portions of the digitized slide data that do not correspond to the non-target area data. The output of the U-Net can be a foreground non-target area mask that represents the locations of non-target areas present in the underlying image, or a background non-target area mask that represents portions of the digitized slide data that do not correspond to the non-target area data (e.g., target areas).

[0093] In some embodiments, the registration module 355 and registration process are used to map biological material or structures, such as tumor cells or cell clusters, identified in one or more images to one or more additional images. Registration is the process of transforming different datasets, here images, or cell clusters within images, into a single coordinate system. More specifically, registration is the process of aligning two or more images and generally involves designating one image as a reference (also called a reference image or fixed image) and applying a geometric transformation to the other images so that they are aligned with the reference. The geometric transformation maps locations in one image to new locations in another image. Determining the correct geometric transformation parameters is key to the image registration process. In some embodiments, image registration is performed using the methods described in International Publication No. WO 2015 / 049233, filed September 30, 2014, entitled "Line-Based Image Registration and Cross-Image Annotation Devices, Systems and Methods," the contents of which are incorporated herein in their entirety for all purposes. WO 2015 / 049233 describes a registration process that includes a coarse registration process used alone or in combination with a fine registration process. In some embodiments, the coarse registration process can include selecting digital images for registration, generating a foreground image mask from each of the selected digital images, and matching tissue structures between the resulting foreground images. In a further embodiment, generating the foreground image mask includes generating a soft-weighted foreground image from a whole slide image of a stained tissue section and applying an OTSU threshold to the soft-weighted foreground image to generate a binary soft-weighted image mask.In yet another embodiment, generating the foreground image mask includes generating a binary soft-weighted image mask from a whole slide image of the stained tissue section, separately generating a gradient amplitude image mask from the same whole slide image, applying an OTSU threshold to the gradient image mask to generate the binary gradient amplitude image mask, and combining the binary soft-weighted image mask and the binary gradient amplitude image mask using a binary OR operation to generate the foreground image mask. As used herein, a "gradient" refers to a pixel intensity gradient calculated for a particular pixel, for example, by taking into account the intensity value gradient of a set of pixels surrounding the pixel. Each gradient can have a specific "direction" relative to a coordinate system whose x- and y-axes are defined by two orthogonal edges of the digital image. A "gradient orientation feature" can be a data value indicating the orientation of the gradient within the coordinate system.

[0094] In some embodiments, matching the tissue structures includes calculating line-based features from the respective boundaries of the resulting foreground image masks, calculating a global transformation parameter between a first set of line features on the first foreground image mask and a second set of line features on the second foreground image mask, and globally aligning the first and second images based on the transformation parameter. In yet another embodiment, the coarse alignment process includes mapping the selected digital images to a common grid based on the global transformation parameter, the grid encompassing the selected digital images. In some embodiments, the fine alignment process includes identifying a first subregion of a first digital image within the set of aligned digital images, identifying a second subregion of a second digital image within the set of aligned digital images, the second subregion being larger than the first subregion and the first subregion being substantially located within the second subregion on the common grid, and calculating an optimized position for the first subregion within the second subregion.

[0095] FIG. 4 depicts examples of staining variations across different H&E slide images 410, 420, 430, 440. In various examples, the H&E slides may vary in color and brightness. For example, different pathology laboratories and / or pathologists may choose to stain tissue samples based on individual preferences, different staining processes, and / or different staining / scanning equipment. Furthermore, the H&E slide images may be of different types of tissue (e.g., tumor, stroma, and necrosis) and / or different organs (e.g., liver, prostate, breast, etc.). Therefore, the global models 112, 114 should be appropriately trained so that they are general enough to still operate accurately despite variations in color, tissue, and organ, or multiple models can be utilized.

[0096] FIG. 5 illustrates a process for training a predictive model according to various embodiments.

[0097] The training process begins at block 500, where multiple tile images of a specimen are accessed. One or more of the multiple tile images include one or more tile image annotations (e.g., for identifying regions having tumor cells, for segmenting non-target and target regions, or any other suitable annotations). At block 510, the one or more tile images may be divided into image patches (e.g., 256 pixels by 256 pixels in size). At block 520, a predictive model, such as a two-dimensional segmentation model, is trained using the one or more tile images or image patches. In some examples, the two-dimensional segmentation model is a modified U-Net model including a reduction path and an expansion path, each of which has up to 256 channels, and one or more layers of the reduction path implement spatial dropout. The training may include performing iterative operations to find a set of parameters for the predictive model that minimizes a loss function of the predictive model. Each iteration may include finding a set of parameters for the predictive model such that the value of the loss function using the set of parameters is smaller than the value of the loss function using another set of parameters in the previous iteration. The loss function is configured to measure the difference between the output predicted using the predictive model and the annotations included in one or more tile images or image patches. In some examples, training further includes adjusting the learning rate of the modified U-Net by reducing the learning rate according to a predetermined schedule. The predetermined schedule may be a step-decay schedule that decreases the learning rate by a predetermined factor every predetermined number of epochs to optimize the loss function. In a particular example, the loss function is a binary cross-entropy loss function. At block 530, the further trained predictive model can be provided to a centralized server after a number of iterations, a length of time, or after the model has been modified beyond a threshold amount. For example, the further trained predictive model can be deployed for execution in an FL image analysis environment, as described with respect to FIGS. 2 and 3.

[0098] FIG. 6 illustrates a process for a round of FL training of a predictive model according to various embodiments.

[0099] The FL process for a round of training begins at block 600, where each of the client devices is provided with one or more global models for use in classification. Each of the client devices can access local data that can be used to further train the provided global model. One or more tile images from the local data include one or more tile image annotations (e.g., for identifying regions with tumor cells, for segmenting non-target and target regions, or any other suitable annotations). As described above, the one or more tile images can be divided into image patches. At block 610, a predictive model (e.g., a global model) is further trained on one or more tile images or image patches. At block 620, after the local training data is exhausted, the further trained predictive models are provided to a centralized server. At block 630, the server can receive one or more further trained models and aggregate weights from those models into the global model. The weights can be aggregated by performing an average, a weighted average, or other suitable method for combining weights as would be understood by one of ordinary skill in the art. For example, in some embodiments, weights may be incorporated into the global model based on a weighted average based on the number of training rounds performed (e.g., slides analyzed).

[0100] FIG. 7 shows results generated after multiple rounds of FL training of a predictive model according to various embodiments.

[0101] The improved accuracy provided by multiple training rounds can be visualized. For example, an H&E image 700 can be used to validate the training of the FL system. Ground truth 710 can be provided for comparison with the model's output. In this example, the image is colored blue to indicate tumors and purple for all other tissues. Exemplary results 720 using a model trained using centralized data are also provided. In this example, six rounds of classification and training are performed, and the resulting classifications 730 produced by each round are shown. After each round of FL, the global model is further trained on one or more client systems, and the results converge to the ground truth 710.

[0102] FIG. 8 illustrates a process for a round of FL training of a predictive model according to various embodiments.

[0103] In various embodiments, the FL process for a training round begins at block 800, where each of the client devices is provided with one or more global models for use in classification. As described above, each of the client devices can access local data that can be used to further train the provided global model, where one or more tile images from the local data include annotations (e.g., to identify regions with tumor cells, to segment non-target and target regions, or any other suitable annotations). Furthermore, the local data may include metadata that further describes the local data. For example, the metadata may include information about how the sample was prepared (e.g., the stain applied, the stain concentration, and / or any other relevant information related to sample preparation), the equipment used (e.g., the staining equipment, the scanning equipment, etc.), and additional patient information. In block 810, the metadata may be generated to determine whether data compensation or normalization needs to be administered. For example, certain scanning equipment may introduce artifacts that require compensation. In another example, some stain concentrations may produce overly light or dark coloration that can be compensated for. Thus, in block 820, the system can compensate for data imbalances using the metadata or other information. At block 830, the model is further trained on one or more tile images or image patches, and the updated model is provided to a centralized server, which updates the global model. At block 840, the updated global model is tested using a validation dataset to verify model improvement. Once the global model is improved, the changes can be saved. At block 850, the server can distribute the updated model to each client device.

[0104] FIG. 9 illustrates a process for receiving an updated model from a client according to various embodiments.

[0105] In various embodiments, the centralized server receives the updated model and metadata from the client device. As described above, in block 910, the system can evaluate metadata associated with the local training data. In various embodiments, the system may be configured to have multiple global classifiers selected according to various metadata. For example, separate classifiers can be used for locations utilizing specific instruments or staining techniques. Thus, in block 920, the system can be configured to determine whether the updated classifier should be used to update one of the multiple global models or whether a new global model should be added. In block 930, the received updated model is normalized and used to update one of the global models. In block 940, the newly updated model is validated using a validation dataset. In block 950, it is determined that a new global model needs to be added. This validates the received updated model. Next, in block 960, the validated model is added to the multiple global models. In block 970, the updated model is distributed to the appropriate client device.

[0106] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0107] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of any features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0108] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0109] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. A computer-implemented method for using a federated learning classifier, comprising: distributing, by a centralized server, a global model configured to classify pathology images to a plurality of client devices; receiving, by the centralized server, an updated model from at least one of the plurality of client devices, the updated model being further trained on the at least one of the plurality of client devices using a plurality of slide images and a plurality of corresponding annotations; aggregating, by the centralized server, the updated model with the global model to generate an updated global model; and delivering the updated global model to at least one of the plurality of client devices.

2. 2. The computer-implemented method of claim 1 , wherein aggregating the updated model with the global model to generate an updated global model comprises performing an averaging of at least one weight of the global model with at least one weight of the updated model.

3. 3. The computer-implemented method of claim 2, wherein performing the averaging comprises performing a weighted average of the at least one weight of the updated model and the at least one weight of the global model according to a number of the plurality of slide images used to further train the updated model and a total number of images used to train the global model.

4. The computer-implemented method of claim 1 , wherein the annotations are provided by a user observing an output of the global model on a slide image, the annotations comprising modifications to the output produced by the global model.

5. 2. The computer-implemented method of claim 1, further comprising receiving, by the centralized server, metadata associated with the plurality of slide images, and wherein aggregating further comprises normalizing the further trained model according to the metadata.

6. The computer-implemented method of claim 1 , further comprising validating, by the centralized server, performance improvements of the updated global model relative to the global model using a validation dataset.

7. 1. A computer-implemented method for using a federated learning classifier by a client device, comprising: receiving a global model configured to classify pathology images from a centralized server; receiving a stained tissue image, the stained tissue image being divided into image patches; performing image analysis on the image patches using the global model; and training the global model using the image patches and at least one corresponding user annotation to generate an updated model, wherein the at least one corresponding user annotation comprises a correction to the classification generated by the global model; transmitting the updated model to the centralized server; receiving an updated global model; and validating the performance improvement of the updated global using a client-specific validation data set.

8. The computer-implemented method of claim 7 , wherein the correction of the classification produced by the global model comprises a reclassification of at least one of cell type, tissue type, or tissue boundary.

9. The computer-implemented method of claim 7 , wherein the updated model does not include individual patient information.

10. generating metadata about the plurality of images; The computer-implemented method of claim 7 , further comprising providing the metadata to the centralized server.

11. 11. The computer-implemented method of claim 10, wherein the metadata includes at least one of the area of ​​the slide or tissue to which the image corresponds, the type of staining performed, the intensity of the staining, and the equipment used for staining or scanning.

12. 8. The computer-implemented method of claim 7, wherein sending the updated model is performed after a threshold number of iterations, an amount of time, or after the model has changed more than a threshold amount.

13. 1. A computer-implemented method for using a federated learning classifier in digital pathology, comprising: distributing the global model to a plurality of client devices by a centralized server; training, by a client device from the plurality of client devices, the global model using a plurality of images of a specimen to generate at least one further trained model, wherein one or more images of the plurality of images include at least one annotation; providing, by the client device, the further trained model to the centralized server; aggregating, by the centralized server, the further trained models with the global model to generate an updated global model; and distributing the updated global model to the plurality of client devices.

14. generating, by the client device, metadata associated with the plurality of images; 14. The computer-implemented method of claim 13, further comprising: providing, by the client device, the metadata to the centralized server, wherein aggregating, by the centralized server, the further trained models with the global model to generate an updated global model further comprises normalizing the further trained models according to the metadata.

15. 15. The computer-implemented method of claim 14, wherein the metadata includes at least one of the area of ​​the slide or tissue to which the image corresponds, the type of staining performed, the intensity of the staining, and the equipment used for staining or scanning.

16. The computer-implemented method of claim 13 , further comprising validating, by the centralized server, performance of the updated global model against the global model using a validation dataset.

17. The computer-implemented method of claim 16 , further comprising: rolling back the updates to the global model if the performance of the updated global model is poorer than the global model.

18. 14. The computer-implemented method of claim 13, wherein aggregating the updated model with the global model to generate an updated global model comprises performing an averaging of at least one weight of the global model with at least one weight of the updated model.

19. 20. The computer-implemented method of claim 18, wherein performing the averaging comprises performing a weighted average of the at least one weight of the updated model and the at least one weight of the global model according to a number of the plurality of slide images used to further train the updated model and a total number of images used to train the global model.

20. 15. The computer-implemented method of claim 14, wherein sending the updated model is performed after a threshold number of iterations, an amount of time, or after the model has changed more than a threshold amount.

Images