Systems and methods for processing electronic images for computational detection methods

Abstract

To provide systems and methods for processing electronic images for computational detection methods.SOLUTION: Systems and methods are disclosed for receiving one or more electronic slide images associated with a tissue specimen that is associated with a patient and / or medical case, partitioning a first slide image of the one or more electronic slide images into a plurality of tiles, detecting a plurality of tissue regions of the first slide image and / or a plurality of tiles to generate a tissue mask, determining whether any of the plurality of tiles corresponds to non-tissue, and removing any of the plurality of tiles that are determined to be non-tissue.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 966,716, filed January 28, 2020, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] Various embodiments of the present disclosure generally relate to creating a predictive model for predicting markers for prepared tissue samples by processing electronic images. More specifically, certain embodiments of the present disclosure relate to systems and methods for predicting, identifying, or detecting diagnostic information for prepared tissue samples. The present disclosure further provides systems and methods for creating a predictive model that predicts markers from first-look slides. [Background technology]

[0003] The performance of machine learning and deep learning models for histopathology can be limited by the quantity and quality of annotated examples used to train these models. Large-scale experiments on supervised image classification problems have shown that model performance continues to improve through approximately 50 million training examples. Manually annotating this amount of data can be prohibitively expensive, both in time and cost, and can present serious limitations to ensuring that systems can be implemented at a clinically relevant level and generalized across institutions.

[0004] The foregoing general description and the following detailed description are exemplary and explanatory only and are not limitations of the present disclosure. The background provided herein is generally for the purpose of providing a context for the present disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims in this application, and is not admitted to be prior art or an indication of prior art by inclusion in this section. Summary of the Invention [Means for solving the problem]

[0005] According to one aspect of the present disclosure, a system and method for deploying weakly supervised multi-label and multi-task learning for computational biomarker detection in digital pathology is disclosed.

[0006] A computer-implemented method for processing electronic images corresponding to a sample includes receiving one or more digital images associated with a tissue sample; receiving one or more electronic slide images associated with the tissue sample, the tissue sample being associated with a patient and / or medical case; partitioning a first slide image of the one or more electronic slide images into a plurality of tiles; detecting a plurality of tissue regions in the first slide image and / or the plurality of tiles and generating a tissue mask; determining whether any of the plurality of tiles correspond to non-tissue; removing any of the plurality of tiles determined to be non-tissue; determining a prediction regarding at least one marker for the one or more electronic slide images using a machine learning predictive model, the machine learning predictive model having been generated by processing a plurality of training images; and outputting a prediction of the trained machine learning predictive model.

[0007] A system for processing electronic images corresponding to a sample includes a memory that stores instructions and at least one processor that executes the instructions to perform a process including receiving one or more digital images associated with a tissue sample, receiving one or more electronic slide images associated with the tissue sample, the tissue sample being associated with a patient and / or medical case, partitioning a first slide image of the one or more electronic slide images into a plurality of tiles, detecting a plurality of tissue regions in the first slide image and / or the plurality of tiles and generating a tissue mask, determining whether any of the plurality of tiles correspond to non-tissue, removing any of the plurality of tiles that are determined to be non-tissue, determining a prediction regarding at least one marker for the one or more electronic slide images using a machine learning predictive model, the machine learning predictive model having been generated by processing a plurality of training images, and outputting a prediction of the trained machine learning predictive model.

[0008] A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for processing electronic images corresponding to a sample, the method including: receiving one or more digital images associated with a tissue sample; receiving one or more electronic slide images associated with the tissue sample, the tissue sample being associated with a patient and / or a medical case; partitioning a first of the one or more electronic slide images into a plurality of tiles; detecting a plurality of tissue regions in the first slide image and / or the plurality of tiles and generating a tissue mask; determining whether any of the plurality of tiles correspond to non-tissue; removing any of the plurality of tiles determined to be non-tissue; determining a prediction regarding at least one marker for the one or more electronic slide images using a machine learning predictive model, the machine learning predictive model having been generated by processing a plurality of training images; and outputting a prediction of the trained machine learning predictive model.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limitations of the disclosed embodiments as claimed. The present invention provides, for example, the following. (Item 1) 1. A computer-implemented method for processing an electronic slide image corresponding to a tissue sample, the method comprising: receiving one or more electronic slide images associated with a tissue sample, the tissue sample being associated with a patient and / or a medical case; partitioning a first slide image of the one or more electronic slide images into a plurality of tiles; Detecting a plurality of tissue regions in the first slide image and / or a plurality of tiles and generating a tissue mask; determining whether any of the plurality of tiles corresponds to a non-tissue; removing any of the plurality of tiles determined to be non-tissue; determining a prediction for at least one marker for the one or more electronic slide images using a machine learning prediction model, wherein the machine learning prediction model was generated by processing a plurality of training images; and outputting predictions of the trained machine learning predictive model; 11. A computer-implemented method comprising: (Item 2) Item 10. The computer-implemented method of item 1, wherein the plurality of tiles determined to be non-tissue are further determined to be background of the tissue sample. (Item 3) Item 10. The computer-implemented method of item 1, wherein detecting the plurality of tissue regions comprises segmenting the tissue regions from a background of the one or more electronic slide images. (Item 4) generating a tissue mask in response to segmenting the tissue region from the background, the segmentation using thresholding based on color / intensity and / or texture features; Item 4. The computer-implemented method of item 3, further comprising: (Item 5) Item 10. The computer-implemented method of item 1, wherein the plurality of training images comprises a plurality of electronic slide images and a plurality of target markers. (Item 6) Processing the plurality of training images includes: receiving a collection of digital images associated with at least one training tissue sample, the collection of digital images comprising a plurality of training electronic slide images; receiving a plurality of synoptic annotations comprising one or more labels for each of the plurality of training electronic slide images; partitioning one of the plurality of training electronic slide images into a plurality of training tiles for the plurality of training electronic slide images; Segmenting at least one tissue region from a background of the one or more electronic slide images to create a training tissue mask; removing at least one of the tiles detected as non-tissue; training the machine learning prediction model in a weakly supervised manner to infer at least one multi-label tile-level prediction using at least one label from the plurality of synoptic annotations; Item 1. The computer-implemented method of item 1, comprising: (Item 7) Item 7. The computer-implemented method of item 6, wherein training the machine learning predictive model under weak supervision includes using at least one of multiple-instance learning (MIL), multiple-instance multi-label learning (MIMLL), self-supervised learning, and unsupervised clustering. (Item 8) Processing the plurality of training images to generate the machine learning predictive model further comprises: receiving a plurality of predictions or a plurality of vectors of at least one feature from a weakly supervised tile-level learning module for the plurality of training tiles; training a machine learning model taking as input the predictions or the vectors of the at least one feature from the weakly supervised tile-level learning module for the training tiles; using the plurality of training tiles to predict a plurality of labels for a slide or patient sample; Item 7. The computer-implemented method of item 6, comprising: (Item 9) Item 9. The computer-implemented method of item 8, wherein at least one of the plurality of labels is binary, categorical, ordinal, or real-valued. (Item 10) 9. The computer-implemented method of claim 8, wherein training the machine learning model and taking as input the plurality of predictions or the plurality of vectors of the at least one feature from the weakly supervised tile-level learning module for the plurality of training tiles comprises a plurality of image features. (Item 11) Item 10. The computer-implemented method of item 1, wherein the trained machine learning predictive model predicts at least one label using at least one first-look slide. (Item 12) 1. A system for processing electronic slide images corresponding to a tissue sample, the system comprising: at least one memory for storing instructions; at least one processor, said at least one processor executing said instructions; receiving one or more electronic slide images associated with a tissue sample, the tissue sample being associated with a patient and / or a medical case; partitioning a first slide image of the one or more electronic slide images into a plurality of tiles; Detecting a plurality of tissue regions in the first slide image and / or a plurality of tiles and generating a tissue mask; determining whether any of the plurality of tiles corresponds to a non-tissue; removing any of the plurality of tiles determined to be non-tissue; determining a prediction for at least one marker for the one or more electronic slide images using a machine learning prediction model, wherein the machine learning prediction model was generated by processing a plurality of training images; and Outputting predictions from the trained machine learning predictive model at least one processor configured to perform operations including A system comprising: (Item 13) Item 13. The system of item 12, wherein the plurality of tiles determined to be non-tissue are further determined to be background of the tissue sample. (Item 14) Item 13. The system of item 12, wherein detecting the plurality of tissue regions comprises segmenting the tissue regions from a background of the one or more electronic slide images. (Item 15) generating a tissue mask in response to segmenting the tissue region from the background, the segmentation using thresholding based on color / intensity and / or texture features; Item 15. The system of item 14, further comprising: (Item 16) Item 13. The system of item 12, wherein the plurality of training images comprises a plurality of electronic slide images and a plurality of target markers. (Item 17) Processing the plurality of training images includes: receiving a collection of digital images associated with at least one training tissue sample, the collection of digital images comprising a plurality of training electronic slide images; receiving a plurality of synoptic annotations comprising one or more labels for each of the plurality of training electronic slide images; partitioning one of the plurality of training electronic slide images into a plurality of training tiles for the plurality of training electronic slide images; Segmenting at least one tissue region from a background of the one or more electronic slide images to create a training tissue mask; removing at least one of the tiles detected as non-tissue; training the machine learning prediction model in a weakly supervised manner to infer at least one multi-label tile-level prediction using at least one label from the plurality of synoptic annotations; Item 13. The system according to item 12, comprising: (Item 18) Item 18. The system of item 17, wherein training the machine learning predictive model under weak supervision includes using at least one of MIL, MIMLL, self-supervised learning, and unsupervised clustering. (Item 19) Processing the plurality of training images to generate the machine learning predictive model further comprises: receiving a plurality of predictions or a plurality of vectors of at least one feature from a weakly supervised tile-level learning module for the plurality of training tiles; training a machine learning model taking as input the predictions or the vectors of the at least one feature from the weakly supervised tile-level learning module for the training tiles; using the plurality of training tiles to predict a plurality of labels for a slide or patient sample; Item 18. The system according to item 17, comprising: (Item 20) A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform a method for processing electronic slide images corresponding to a tissue sample, the method comprising: receiving one or more electronic slide images associated with a tissue sample, the tissue sample being associated with a patient and / or a medical case; partitioning a first slide image of the one or more electronic slide images into a plurality of tiles; Detecting a plurality of tissue regions in the first slide image and / or a plurality of tiles and generating a tissue mask; determining whether any of the plurality of tiles corresponds to a non-tissue; removing any of the plurality of tiles determined to be non-tissue; determining a prediction for at least one marker for the one or more electronic slide images using a machine learning prediction model, wherein the machine learning prediction model was generated by processing a plurality of training images; and Outputting predictions from the trained machine learning predictive model 1. A non-transitory computer-readable medium comprising: [Brief explanation of the drawings]

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0011] [Figure 1A] FIG. 1A illustrates an example block diagram of a system and network for creating a predictive model, according to an example embodiment of the present disclosure.

[0012] [Figure 1B] FIG. 1B illustrates an example block diagram of a predictive model platform according to an example embodiment of the present disclosure.

[0013] [Figure 1C] FIG. 1C illustrates an exemplary block diagram of a slide analysis tool according to an exemplary embodiment of the present disclosure.

[0014] [Figure 2A] FIG. 2A is a flowchart illustrating an example method for using a predictive model created by a trained machine learning system, according to one or more example embodiments of the present disclosure.

[0015] [Figure 2B] FIG. 2B is a flowchart illustrating an example method for training a weakly supervised tile-level learning module within a trained machine learning system, according to one or more example embodiments of the present disclosure.

[0016] [Figure 2C] FIG. 2C is a flowchart illustrating an example method for training a weakly supervised aggregation module in a trained machine learning system, according to one or more example embodiments of the present disclosure.

[0017] [Figure 3] FIG. 3 is a flowchart illustrating an exemplary method for training and using a machine learning system to simultaneously detect and grade prostate cancer, according to one or more exemplary embodiments of the present disclosure.

[0018] [Figure 4] FIG. 4 is a flowchart illustrating an exemplary method for training and using a machine learning system for tumor quantification in prostate needle biopsies, according to one or more exemplary embodiments of the present disclosure.

[0019] [Figure 5] FIG. 5 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict cancer subtypes, according to one or more exemplary embodiments of the present disclosure.

[0020] [Figure 6] FIG. 6 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict surgical margins, according to one or more exemplary embodiments of the present disclosure.

[0021] [Figure 7] FIG. 7 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict bladder cancer biomarkers, according to one or more exemplary embodiments of the present disclosure.

[0022] [Figure 8]FIG. 8 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a pan-cancer diagnosis, according to one or more exemplary embodiments of the present disclosure.

[0023] [Figure 9] FIG. 9 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict organ toxicity, according to one or more exemplary embodiments of the present disclosure.

[0024] [Figure 10] FIG. 10 illustrates an exemplary connected components algorithm according to an embodiment of the present disclosure.

[0025] [Figure 11] FIG. 11 depicts an example system that can implement the techniques presented herein. DETAILED DESCRIPTION OF THE INVENTION

[0026] Description of the embodiment Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0027] The systems, devices, and methods disclosed herein are described in detail, by way of example, with reference to the Figures. The examples discussed herein are examples only and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be construed as essential for any particular implementation of any of these devices, systems, or methods, unless specifically designated as essential.

[0028] Also, with respect to any method described, whether the method is described in conjunction with a flow diagram or not, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in the execution of the method does not imply that these steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

[0029] As used herein, the term "exemplary" is used in the sense of "an example," as opposed to "ideal." Furthermore, the terms "a" and "an," as used herein, do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced item.

[0030] Pathology refers to the study of disease and its causes and effects. More specifically, pathology refers to the performance of tests and analyses used to diagnose disease. For example, a tissue sample may be placed on a slide to be viewed under a microscope by a pathologist (e.g., a medical doctor, a specialist who analyzes tissue samples and determines whether any abnormalities are present). That is, a pathology specimen may be cut into multiple sections, stained, and prepared as a slide for the pathologist to examine and render a diagnosis. When diagnostic uncertainty is found on a slide, the pathologist may prescribe additional sections, stains, or other tests to gather more information from the tissue. A technician may then create a new slide that may contain additional information for the pathologist to use in making a diagnosis. This process of creating additional slides can be time-consuming, not only because it may involve taking a block of tissue, cutting it, creating a new slide, and then staining the slide, but also because it may be bulky for multiple instructions. This can significantly delay the final diagnosis rendered by the pathologist. Additionally, even after a delay, there may still be no guarantee that the new slides will have enough information to render a diagnosis.

[0031] Pathologists may evaluate cancer and other disease pathology slides in isolation. This disclosure presents an integrated workflow for improving cancer and other disease diagnosis. The workflow may, for example, integrate slide evaluation, tasks, image analysis and cancer detection artificial intelligence (AI), annotation, consultation, and recommendations within one workstation. In particular, this disclosure describes various exemplary user interfaces available within the workflow, as well as AI tools that may be integrated into the workflow to facilitate and improve the pathologist's work.

[0032] For example, a computer can be used to analyze images of tissue samples and quickly identify whether additional information may be needed for a particular tissue sample and / or highlight areas that a pathologist should examine more closely. Thus, the process of acquiring additional stained slides and tests can be done automatically before being reviewed by a pathologist. When paired with automated slide segmentation and staining machinery, this can provide a fully automated slide preparation pipeline. This automation has the following advantages: (1) minimizing the amount of time wasted by a pathologist determining insufficient slides to make a diagnosis; (2) minimizing the (average total) time from sample acquisition to diagnosis by avoiding additional time between when additional tests are ordered and when they are generated; (3) reducing the amount of time and material wasted per resection by allowing resections to be performed while the tissue block (e.g., pathology sample) is on the cutting table; (4) reducing the amount of tissue material wasted / discarded during slide preparation; (5) reducing the cost of slide preparation by partially or fully automating the procedure; (6) enabling automated, customized cutting and staining of slides, which may result in more representative / informative slides from the sample; (7) reducing the overhead of requiring additional tests for the pathologist, thereby allowing a larger number of slides to be generated per tissue block, contributing to a more informative / precise diagnosis; and / or (8) identifying or verifying the correct nature (e.g., with respect to sample type) of digital pathology images.

[0033] The process of using computers to assist pathologists is known as computational pathology. Computing methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but is not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. The use of computational pathology can save lives by helping pathologists improve their diagnostic accuracy, reliability, efficiency, and accessibility. For example, computational pathology may be used to assist in detecting slides that are suspicious for cancer, thereby allowing pathologists to check and confirm their initial assessment before rendering a final diagnosis.

[0034] As described above, the disclosed computational pathology process and device may provide an integrated platform that enables a fully automated process, including data capture, processing, and viewing of digital pathology images via a web browser or other user interface, while also integrating with a laboratory information system (LIS). Additionally, clinical information may be aggregated using cloud-based data analysis of patient data. Data may originate from hospitals, clinics, field researchers, etc., and may be analyzed by machine learning, computer vision, natural language processing, and / or statistical algorithms to provide real-time monitoring and prediction of health patterns at multiple levels of geographic specificity.

[0035] Histopathology refers to the study of specimens mounted on slides. For example, a digital pathology image may consist of a digitized image of a microscope slide containing a specimen (e.g., a smear). One method a pathologist may use to analyze the image on a slide is to identify nuclei and classify them as normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying nuclei, histological stains may be used to visualize cells. Many dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method in which hematoxylin stains cell nuclei blue, eosin stains cytoplasm and extracellular matrix pink, and other tissue regions take on variations of these colors. However, in many cases, histological preparations with H&E staining do not provide sufficient information for pathologists to visually identify biomarkers that may aid in diagnosis or guide treatment. In this situation, techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH), or fluorescence in situ hybridization (FISH) may be used. IHC and immunofluorescence involve the use of antibodies, for example, to bind to specific antigens within tissues and allow visual detection of cells expressing specific proteins of interest, which may reveal biomarkers that are not reliably identifiable to trained pathologists based on analysis of H&E-stained slides. ISH and FISH may be employed to assess the number of gene copies or the abundance of specific RNA molecules, depending on the type of probe employed (e.g., DNA probes for gene copy number and RNA probes for assessing RNA expression). If these methods also fail to provide sufficient information to detect some biomarkers, genetic testing of tissue may be used to confirm whether the biomarker is present (e.g., overexpression of a specific protein or gene product in the tumor, amplification of a given gene in the cancer).

[0036] Digitized images may be prepared to represent stained microscope slides, allowing pathologists to manually inspect the images on the slides and estimate the number of stained abnormal cells within the image. However, this process can be time-consuming and lead to errors in identifying abnormalities because some abnormalities are difficult to detect. Computerized processes and devices may be used to assist pathologists in detecting abnormalities that may otherwise be difficult to detect. For example, AI may be used to predict biomarkers (such as overexpression, amplification, or specific gene mutations of proteins and / or gene products) from prominent regions within digital images of tissue stained using H&E and other dye-based methods. The images of tissue may be whole slide images (WSIs), images of tissue cores in microarrays, or selected areas of interest within tissue sections. Using staining methods such as H&E, these biomarkers may be difficult for humans to visually detect or quantify without the aid of additional testing. Using AI to infer these biomarkers from digital images of tissue has the potential to improve patient treatment while also being faster and less expensive.

[0037] The detected biomarkers or images alone can then be used to recommend specific cancer drugs or drug combination therapies to be used to treat the patient, and AI can identify drugs or drug combinations that are unlikely to be successful by correlating the detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendations of immunotherapy drugs to target a patient's specific cancer. Furthermore, this can be used to enable personalized cancer treatment for specific subsets of patients and / or rarer cancer types.

[0038] As described above, the disclosed computational pathology process and device may provide an integrated platform, enabling a fully automated process including data capture, processing, and viewing of digital pathology images via a web browser or other user interface while integrating with a laboratory information system (LIS). Additionally, clinical information may be aggregated using cloud-based data analytics of patient data. Data may originate from hospitals, clinics, field researchers, etc., and may be analyzed by machine learning, computer vision, natural language processing, and / or statistical algorithms to provide real-time monitoring and prediction of health patterns at multiple levels of geographic specificity.

[0039] While the digital pathology images described above may be stored with tags and / or labels relating to the nature of the sample or the digital pathology image, such tags / labels may be incomplete. Thus, the systems and methods disclosed herein predict at least one label from a collection of digital images.

[0040] The performance of machine learning and deep learning models for histopathology can be limited by the volume and the quantity and quality of annotated examples used to train these models. Large-scale experiments on supervised image classification problems have shown that model performance continues to improve through approximately 50 million training examples. However, most clinically relevant tasks in pathology involve much more than classification. When a pathologist renders a diagnosis, the diagnosis may take the form of a report containing many heterogeneous, interrelated fields and involving an entire slide or set of slides. In oncology, these fields may include the presence of cancer, cancer grade, tumor quantification, cancer grade group, the presence of various features important for cancer staging, etc. In preclinical drug research animal studies, these fields may include the presence of toxicity, the severity of toxicity, and the type of toxicity. Procuring the necessary annotations for training most supervised deep learning models may involve a pathologist labeling individual pixels, tiles (e.g., one or more relatively small rectangular regions within a slide image), or regions of interest (e.g., polygons) from a slide image with appropriate annotations. A different set of training annotations may be used for each field in the report. Furthermore, a typical digital pathology slide may contain approximately 10 gigapixels or more than 100,000 tiles. Manually annotating this amount of data may be prohibitively expensive, both in time and cost, and may present serious limitations to ensuring that the system can be implemented at a clinically relevant level and generalized across institutions. Therefore, there is a desire to generate training data that can be used for histopathology.

[0041] Embodiments of the present disclosure may overcome the above limitations. In particular, embodiments disclosed herein may use weak supervision, in which a deep learning model may be trained directly from a pathologist's diagnosis rather than using additional labels for each pixel or tile in a digital image. The machine learning or deep learning model may, in some embodiments, comprise a machine learning algorithm. While one technique may determine binary cancer detection, the techniques discussed herein further disclose how deep learning systems can be trained in a weakly supervised multi-label and multi-task setting to perform, for example, malignancy assessment, subtype classification, simultaneous inference of multiple disease attributes, and more. This allows the system to be trained directly from diagnostic reports or test results without the need for extensive annotation, reducing the number of required training labels by five orders of magnitude or more.

[0042] The disclosed systems and methods can automatically predict sample or image properties without relying on stored tags or labels. Furthermore, systems and methods are disclosed for quickly and correctly identifying and / or matching sample types in digital pathology images or any information related to the digital pathology images, without necessarily accessing an LIS or similar information database. One embodiment of the present disclosure may include a system that is trained to identify various properties of digital pathology images based on a dataset of previous digital pathology images. The trained system can provide a classification for the sample depicted in the digital pathology image. The classification can help provide a treatment or diagnostic prediction for the patient associated with the sample.

[0043] The present disclosure includes one or more embodiments of a slide analysis tool. Input to the tool may include a digital pathology image and any associated additional inputs. Output of the tool may include global and / or local information about the sample. The sample may include a biopsy or surgical resection sample.

[0044] FIG. 1A illustrates a block diagram of a system and network for determining sample property or image property information for digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure.

[0045] 1A illustrates an electronic network 120 that may be connected to servers at a hospital, laboratory, and / or doctor's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, etc., may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to exemplary embodiments of the present application, the electronic network 120 may also be connected to a server system 110, which may include a processing device configured to implement a disease detection platform 100, according to exemplary embodiments of the present disclosure, including a slide analysis tool 101 for determining sample property or image property information for digital pathology images using machine learning to classify samples.

[0046] The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may create or otherwise acquire one or more patient images of cytology samples, histopathology samples, slides of cytology samples, digitized images of histopathology sample slides, or any combination thereof. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsies, or cytology information. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may transmit the digitized slide images and / or patient-specific information to the server system 110 via the electronic network 120. The server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The server system 110 may also include a processing device for processing the images and data stored in the one or more storage devices 109. The server system 110 may further include one or more machine learning tools or capabilities. For example, the processing device may include machine learning tools for the disease detection platform 100, according to one embodiment. Alternatively, or in addition, the present disclosure (or portions of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0047] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review slide images. In a hospital setting, tissue type information may be stored within the laboratory information system 125. However, the correct tissue classification information is not always paired with the image content. Additionally, even if an LIS is used to access the sample type for digital pathology images, this label may be incorrect due to the fact that many components of the LIS are manually entered and may leave a large margin of error. According to exemplary embodiments of the present disclosure, the sample type may be identified without the need to access the laboratory information system 125, or possibly to correct the laboratory information system 125. For example, a third party may be given anonymized access to image content without the corresponding sample type label stored within the LIS. Additionally, access to LIS content may be limited due to its sensitive content.

[0048] 1B illustrates an exemplary block diagram of a disease detection platform 100 for determining sample property or image property information for digital pathology images using machine learning. For example, the disease detection platform 100 may include a slide analysis tool 101, a data capture tool 102, a slide capture tool 103, a slide scanner 104, a slide manager 105, a storage device 106, and a viewing application tool 108.

[0049] Slide analysis tool 101, as described below, refers to a process and system for processing digital images and using machine learning to analyze slides associated with tissue samples, according to an exemplary embodiment.

[0050] Data capture tools 102 refer to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and devices used to classify and process the digital pathology images, according to an exemplary embodiment.

[0051] Slide capture tool 103 refers to a process and system for scanning pathology images and converting them into digital form, according to an exemplary embodiment. Slides may be scanned using slide scanner 104, and slide manager 105 may process the images on the slides into digitized pathology images and store the digitized images in storage device 106.

[0052] The viewing application tool 108 refers to processes and systems for providing a user (e.g., a pathologist) with sample property or image property information related to a digital pathology image, according to an exemplary embodiment. The information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0053] The slide analysis tool 101 and its components may each transmit and / or receive digitized slide images and / or patient information to and / or from the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 via the electronic network 120. Additionally, the server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the slide analysis tool 101, the data capture tool 102, the slide capture tool 103, the slide scanner 104, the slide manager 105, and the viewing application tool 108. The server system 110 may also include a processing device for processing images and data stored in the storage device. The server system 110 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively, or in addition, the present disclosure (or portions of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0054] Any of the above devices, tools, and modules may be located on devices that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0055] 1C illustrates an exemplary block diagram of a slide analysis tool 101, according to an exemplary embodiment of the present disclosure. The slide analysis tool 101 may include a training image platform 131 and / or a target image platform 135.

[0056] According to one embodiment, the training image platform 131 may create or receive training images that are used to train the machine learning system to effectively analyze and classify digital pathology images. For example, the training images may be received from any one or combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The images used for training may be derived from real sources (e.g., humans, animals, etc.) or synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains, such as (but not limited to) H&E, hematoxylin only, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices, such as micro-CT.

[0057] The training image capture module 132 may create or receive a dataset comprising one or more training images corresponding to one or both of images of human tissue and graphically rendered images. For example, the training images may be received from any one or combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. This dataset may be maintained on a digital storage device. The quality score determiner module 133 may identify quality control (QC) issues (e.g., imperfections) with the training images at a global or local level that may significantly affect the usefulness of the digital pathology image. For example, the quality score determiner module may use information about the image as a whole, such as the sample type, the overall quality of the sample sections, the overall quality of the glass pathology slide itself, or tissue morphological characteristics, to determine an overall quality score for the image. The treatment identification module 134 may analyze images of tissue to determine digital pathology images that have a treatment effect (e.g., post-treatment) and images that do not have a treatment effect (e.g., pre-treatment). Identifying whether a digital pathology image has a treatment effect is useful because previous treatment effects in tissue can affect the morphology of the tissue itself. Most LISs do not explicitly track this characteristic, and therefore, classifying sample types with previous treatment effects may be desirable.

[0058] According to one embodiment, the target image platform 135 may include a target image capture module 136, a sample detection module 137, and an output interface 138. The target image platform 135 may receive a target image and apply a machine learning model to the received target image to determine characteristics of the target sample. For example, the target image may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The target image capture module 136 may receive a target image corresponding to the target sample. The sample detection module 137 may apply a machine learning model to the target image to determine characteristics of the target sample. For example, the sample detection module 137 may detect a sample type of the target sample. The sample detection module 137 may also apply the machine learning model to the target image to determine a quality score for the target image. Furthermore, the sample detection module 137 may apply the machine learning model to the target sample to determine whether the target sample is pre-treatment or post-treatment.

[0059] The output interface 138 may be used to output the target image and information about the target sample (eg, to a screen, monitor, storage device, web browser, etc.).

[0060] 2A is a flowchart illustrating an exemplary method for using a predictive model created by a trained machine learning system, according to one or more exemplary embodiments of the present disclosure. For example, exemplary method 200 (steps 202-210) may be performed by slide analysis tool 101 automatically or in response to a request from a user.

[0061] According to one embodiment, an exemplary method 200 for using a predictive model may include one or more of the following steps: In step 202, the method may include receiving one or more digital images associated with the tissue sample, the one or more digital images comprising a plurality of slide images. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), or any other suitable storage device.

[0062] In step 204, the method may include partitioning one of the plurality of slide images into a collection of tiles for the plurality of slide images.

[0063] In step 206, the method may include detecting a plurality of tissue regions from the background of one of the plurality of slide images, creating a tissue mask, and removing at least one tile from the set of tiles that is detected as non-tissue. The non-tissue tile may comprise the background of the slide image. This may be accomplished in a variety of ways, including thresholding-based methods based on color, color intensity, texture features, or Otsu's binarization method, followed by invocation of a connected component algorithm, segmentation algorithms such as k-means, graph cuts, masked region convolutional neural network (Mask R-CNN), or any other suitable method.

[0064] In step 208, the method may include using a machine learning system to determine predictions regarding markers for a plurality of slide images corresponding to a patient or medical case, the machine learning system having been generated by processing a plurality of training examples to create a predictive model. The training examples may comprise one or more digital slide images and a set of a plurality of target markers.

[0065] In step 210, the method may include outputting a predictive model of the training machine learning system that predicts at least one indicator from at least one slide that was not used to train the machine learning system, and outputting the prediction to an electronic storage device.

[0066] 2B is a flowchart illustrating an example method for training a weakly supervised tile-level learning module within a trained machine learning system according to one or more exemplary embodiments of the present disclosure. The weakly supervised learning module may use the slide-level training indicators to train a model and make tile-level predictions. For example, example method 220 (steps 222-230) may be performed by slide analysis tool 101 automatically or in response to a request from a user.

[0067] According to one embodiment, an exemplary method 220 for using a predictive model may include one or more of the following steps: In step 222, the method may include receiving a collection of digital images associated with training tissue samples into a digital storage device, the collection of digital images comprising a plurality of training slide images. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), or any other suitable storage device.

[0068] At step 224, the method may include receiving a plurality of synoptic annotations for each of a plurality of training slide images, the annotations comprising one or more indicators, which may be binary, multi-level binary, categorical, ordinal, or real-valued.

[0069] In step 226, the method may include partitioning one of the plurality of training slide images into a set of training tiles for the plurality of training slide images.

[0070] At step 228, the method may include detecting at least one tissue region from the background of the plurality of training slide images to create a training tissue mask, and removing at least one training tile from the set of training tiles that is detected as non-tissue. This may be accomplished in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected component algorithm, or any other suitable method.

[0071] In step 230, the method may include training a weakly supervised predictive model using at least one synoptic landmark to infer at least one multi-marker tile-level prediction. There may be four general approaches for training a model in a weakly supervised setting, although any suitable approach for training a model may be used. 1. Multi-instance learning (MIL) can be used to train tile-level prediction models for binary or categorical labels by learning to identify tiles containing target labels on slides. This identification may be accomplished by finding salient tiles (e.g., the highest-scoring tiles based on synoptic annotations or labels received in each training iteration) and using these tiles to update a classifier using the received synoptic training labels associated with each salient tile. For example, a classifier may be trained to identify cancer based on a collection of overlapping tiles. As salient tiles are determined, the synoptic labels may be used to update the tile-level labels. The tile-level labels and classifier may then determine or provide a label for a group of tiles. MIL may also be used to train machine learning models and extract diagnostic features for other downstream tasks, such as cancer grading, cancer subclassification, and biomarker detection. 2. Multi-instance multi-label learning (MIMLL) may be a tile-level predictive model that comprises a generalization of MIL that treats each slide as a set of tiles that may be associated with multiple labels, rather than just a single binary label as in MIL. These slide labels may come from a pathologist's diagnostic report, genetic testing, immunological testing, or other measurements / assays. A MIMLL model may be trained to select tiles that correspond to each of the synoptic training labels that belong to one or more sets of slides. This embodiment may involve MIMLL training a neural network (e.g., a convolutional neural network (CNN), a capsule network, etc.) by iterating the following steps: a. For each label to be predicted, a scoring function is used to select the most relevant set of tiles. The scoring function may be organized to rank multiple tiles simultaneously. For example, with multiple binary labels, a CNN may be run on each tile attempting to predict each of the multiple binary labels from all tiles in the set of slides, and the tile with an output closest to 1 for one or more of the labels may be selected. b. The selected tile is used to update the weights of the CNN model for its associated label assignment. Each label may have its own output layer in the model. Similar to the MIL model, the MIMLL model may also be used to extract diagnostic features for other downstream tasks. 3. Self-supervised learning can use supervised learning to create an initial tile-based classifier using a small amount of tile-level training data. This initial classifier can be used to bootstrap the full training process by alternating: a. Use predictions from the current tile-based model to reassign tile labels within the training set. b. Update the model for each tile for the latest sign assignments. 4. Unsupervised clustering can learn to group similar instances together without the use of target markers. Sliding tiles can be treated as instances, and the number of groups can be either predefined or automatically learned by the algorithm. Such clustering algorithms can include, but are not limited to, the following methods: a. Expectation Maximization (EM) b. Upper bound maximization (MM) cK-Nearest Neighbor (KNN) d. Hierarchical clustering e. Agglomerative clustering The resulting model may be used to extract diagnostic features for use by the slide-level prediction module.

[0072] 2C is a flowchart illustrating an exemplary method for training a weakly supervised aggregation module within a trained machine learning system, according to one or more exemplary embodiments of the present disclosure. For example, exemplary method 240 (steps 242-244) may be performed by slide analysis tool 101 automatically or in response to a request from a user.

[0073] According to one embodiment, an exemplary method 240 for training a weakly supervised aggregation module may include one or more of the following steps: In step 242, the method may include receiving a plurality of predictions or a plurality of vectors of at least one feature from the weakly supervised tile-level learning module for the set of training tiles.

[0074] In step 244, the method may include training a machine learning model that takes as input multiple predictions or multiple vectors of at least one feature from the weakly supervised tile-level learning module for the collection of tiles. The aggregation module may train a multi-task slide-level aggregation model that takes the tile-level input and generates final predictions for the tiles and / or slide images input into the system. The general form of the model may consist of multiple outputs (e.g., multi-task learning), where each indicator may be binary, categorical, ordinal, or real-valued. The tile-level input may include any type of image feature, including, but not limited to: a. Output from a weakly supervised model (e.g., feature vector or embedding) b.CNN features c. Scale Invariant Feature Transform (SIFT) d. Speed-Up Robust Features (SURF) e. Rotation Invariant Feature Transform (RIFT) f.Directed FAST and Rotating BRIEF (ORB) The multi-tasking slide-level aggregation model of the aggregation module may take many forms, including but not limited to: a. A fully connected neural network trained using a set of multiple output tasks b.CNN c. Fully convolutional neural network d. Recurrent neural networks (RNNs), including gated recurrent unit (GRU) and long short-term memory (LSTM) networks e. Graph Neural Networks f. Transformer Network g. Random Forest, Boost Forest, XGBoost, etc.

[0075] 3 is a flowchart illustrating an exemplary method for training and using a machine learning system to simultaneously detect and grade prostate cancer according to one or more exemplary embodiments of the present disclosure. Cancer grade may measure the differentiation of cancer cells from normal tissue and may be assessed both at the local level by examining cell morphology and at a slide-level summary containing the relative amount of malignancy. Grade may be performed as part of a pathologist's diagnostic report for common cancers such as prostate, kidney, and breast. Exemplary methods 300 and 320 may be used to simultaneously train and use a machine learning system to detect and grade prostate cancer.

[0076] According to one embodiment, exemplary methods 300 and 320 may include one or more of the following steps: In step 301, the method may include receiving one or more digital images of the stained prostate tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0077] In step 303, the method may include receiving at least one indicator related to the one or more digital images, the at least one indicator including an indication of the presence of cancer and a cancer grade. The cancer grade may comprise a primary and a secondary Gleason grade.

[0078] In step 305, the method may include partitioning each of the one or more digital images into a set of tiles.

[0079] In step 307, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. Detecting tissue regions and removing non-tissue tiles may be performed by a thresholding method based on color, color intensity, texture features, Otsu's thresholding method, etc., followed by invoking a connected components algorithm. The thresholding may provide an indication of tissue versus non-tissue regions for one or more pixels of each received slide image based on the thresholding method. The connected components algorithm may detect interconnected image regions or pixels to detect tissue versus non-tissue regions across the entire image region, slide image, or slide. Detecting tissue regions and removing non-tissue tiles may also be performed by a segmentation algorithm, such as k-means, graph cut, or Mask R-CNN.

[0080] In step 309, the method may include training a machine learning model to predict the presence or absence of cancer and the aggressiveness of the cancer for one or more digital images. Training may be accomplished in a variety of ways, including but not limited to: a. Using the MIMLL model as disclosed above, a CNN is trained to predict primary, secondary, and / or tertiary grades, e.g., by treating each slide as a set of tiles associated with multiple labels, selecting slides corresponding to synoptic training labels, scoring each tile according to its association with the label, and updating the CNN model's weights for the associated label assignments. The trained CNN may then extract embeddings from each tile in the set of slides and train a multi-task aggregator (e.g., the aggregation model disclosed above) to predict the presence of cancer, cancer Gleason grade group, and / or primary, secondary, and tertiary grade for each tile or slide. Alternatively, the predictions output from each tile may be used and aggregated using a hand-designed post-processing method, e.g., by having each tile vote for each grade and taking a majority vote. b. Use the MIL model to classify each tile as cancerous or benign, incorporating grade assessment labels for "pure" cases where primary / secondary / tertiary grades are the same grade. Use supervised learning to train a tile-level classifier with the incorporated labels. Use self-supervised learning as disclosed in the weakly supervised learning module above to refine the model. c. Features / embeddings are extracted from each tile and then a multi-task aggregator (e.g., the aggregation model disclosed above) is used to predict the presence of cancer, the cancer Gleason grade group, and / or primary, secondary, and tertiary grades. Embeddings may be from pre-trained CNNs, random features, features from unsupervised clustering models, SIFT, ORB, etc.

[0081] In step 321, the method may include receiving one or more digital images of the stained prostate sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, RAM, etc.

[0082] In step 323, the method may include partitioning one or more digital images into a set of tiles.

[0083] In step 325, the method may include detecting at least one tissue region from the background of the digital image to create a tissue mask and removing at least one tile that is non-tissue. The detecting step may be accomplished in a variety of ways, including, but not limited to, based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected component algorithm, a segmentation algorithm such as k-means, graph cut, Mask R-CNN, or any other suitable method.

[0084] In step 327, the method may include applying the trained machine learning model to the set of tiles to predict the presence of cancer and cancer grade, which may comprise a cancer Gleason grade group, and / or primary, secondary, and tertiary grade groups.

[0085] In step 329, the method may include outputting the prediction, for example to an electronic storage device.

[0086] 4 is a flowchart illustrating an exemplary method for training and using a machine learning system for tumor quantification in prostate needle biopsies according to one or more exemplary embodiments of the present disclosure. Tumor quantification for prostate needle biopsies may consist of estimating the total and relative amount of cancer for each cancer grade (e.g., Gleason grade). Tumor quantification may play an important role in understanding the composition and severity of prostate cancer and may be a common element of pathology diagnostic reports. Quantifying tumor size may traditionally be performed manually using a physical ruler on a glass slide. Such manual quantification may suffer from both inaccuracy and consistency. Exemplary methods 400 and 420 may be used to train and use a machine learning system to quantify tumors in prostate needle biopsies.

[0087] According to one embodiment, exemplary methods 400 and 420 may include one or more of the following steps: In step 401, the method may include receiving one or more digital images of the stained prostate tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0088] In step 403, the method may include receiving at least one real-valued tumor quantification indicator for each of the one or more digital images, the at least one real-valued tumor quantification indicator including an indication of primary and secondary grades. The indicator may also include a respective volume, a respective length, and a respective size of the tumor in the one or more digital images.

[0089] In step 405, the method may include partitioning each of the one or more digital images into a set of tiles.

[0090] In step 407, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0091] In step 409, the method may include training a machine learning model and outputting a cancer grading prediction, as described in exemplary method 300. Tumor quantification estimates may be estimated in a number of ways, including but not limited to: a. Count the number of malignant tiles and geometrically estimate their volume and ratio to the volume of benign tissue. b. Train a model using a slide-level grading module, for example, as described in exemplary method 300. This model may take as input tile-level diagnostic features from a machine learning cancer grading prediction model (e.g., a model trained in exemplary method 300) and output each tumor quantification measurement using a real-valued regression model.

[0092] In step 421, the method may include receiving one or more digital images of the stained prostate sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0093] In step 423, the method may include partitioning the one or more digital images into a set of tiles.

[0094] In step 425, the method may include detecting at least one tissue region from the background of the digital image to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected component algorithm, or any other suitable method.

[0095] In step 427, the method may include applying the trained machine learning model to the set of tiles to calculate a tumor quantification prediction. The prediction may be output to an electronic storage device. The tumor quantification may be in the form of a size measurement or a percentage.

[0096] In step 429, the method may include outputting the prediction to an electronic storage device.

[0097] 5 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict cancer subtypes, according to one or more exemplary embodiments of the present disclosure. Many cancers have multiple subtypes. For example, in breast cancer, it can be determined whether the cancer is invasive, whether it is lobular or ductal, and whether various other attributes such as calcification are present. This method of predicting cancer subtypes may include predicting multiple, non-exclusive categories, which may involve the use of multi-label learning.

[0098] According to one embodiment, exemplary methods 500 and 520 may include one or more of the following steps: In step 501, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0099] In step 503, the method may include receiving, for one or more digital images, a plurality of labels, which may be labels and / or biomarkers of the tissue sample. In a breast cancer sample, the relevant biomarkers may be the presence of calcifications, the presence or absence of cancer, ductal carcinoma in situ (DCIS), invasive ductal carcinoma (IDC), inflammatory breast cancer (IBC), Paget's disease of the breast, angiosarcoma, phyllodes tumor, invasive lobular carcinoma, lobular carcinoma in situ, and various forms of atypical cancer. The labels are not necessarily mutually exclusive, and multiple subtypes may be observed simultaneously.

[0100] In step 505, the method may include partitioning each of the one or more digital images into a set of tiles.

[0101] In step 507, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0102] In step 509, the method may include training a machine learning model to predict cancer morphology and / or subtype for each tile and / or slide. Training the machine learning model may be performed using the MIMLL model disclosed above. The trained subtype prediction machine learning model may be refined using a slide-level prediction model (e.g., an aggregate model) as disclosed above. The slide-level prediction model may take as input the tile-level subtype predictions from the MIMLL model and output slide-level predictions indicating the presence of each cancer subtype.

[0103] In step 521, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0104] In step 523, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected component algorithm, or any other suitable method.

[0105] In step 525, the method may include partitioning the one or more digital images into a set of tiles and discarding any tiles that do not contain tissue.

[0106] In step 527, the method may include calculating a cancer subtype prediction from the set of tiles and outputting the prediction to an electronic storage device.

[0107] 6 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict surgical margins according to one or more exemplary embodiments of the present disclosure. When a tumor is surgically removed from a patient, it may be important to assess whether the tumor has been completely removed by analyzing the margin of tissue surrounding the tumor. The width of this margin and the identification of any cancerous tissue in the margin can play an important role in determining how the patient can be treated. Training a model and predicting margin width and composition may take the form of multi-label multi-task learning.

[0108] According to one embodiment, exemplary methods 600 and 620 may include one or more of the following steps: In step 601, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0109] In step 603, the method may include receiving a plurality of indicators for one or more digital images, the plurality of indicators indicating the tumor margin and whether the margin is positive (e.g., tumor cells are found at the margin), negative (e.g., the margin is completely free of cancer), or borderline (e.g., not definitively positive or negative).

[0110] In step 605, the method may include partitioning each of the one or more digital images into a set of tiles.

[0111] In step 607, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0112] In step 609, the method may include training a machine learning model to predict cancer detection, presence, or grade, as disclosed above.

[0113] In step 621, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0114] In step 623, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0115] In step 625, the method may include partitioning each of the one or more digital images into a set of tiles.

[0116] In step 627, the method may include calculating a surgical margin, tumor margin size, or tumor composition prediction from the tiles. The method may also include outputting the prediction to an electronic storage device.

[0117] 7 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict bladder cancer biomarkers according to one or more exemplary embodiments of the present disclosure. Bladder cancer is one of the most common cancers in the world. If bladder cancer is detected, the pathologist may also determine whether the muscularis propria is present on any of the slides on which bladder cancer was detected. The muscularis propria is a layer of smooth muscle cells that forms a significant portion of the bladder wall. Detecting the presence or absence of the muscularis propria is a key step in determining whether bladder cancer is invasive. The embodiment performs both cancer detection and muscularis propria detection, but can be extended to any number of binary classification tasks.

[0118] According to one embodiment, exemplary methods 700 and 720 may include one or more of the following steps: In step 701, one or more digital images associated with a tissue sample are received into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0119] In step 703, the method may include receiving a plurality of markers for one or more digital images, the plurality of markers indicating the presence or absence of cancer or the presence or absence of the muscularis propria.

[0120] In step 705, the method may include partitioning each of the one or more digital images into a set of tiles.

[0121] In step 707, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0122] In step 709, the method may include training a machine learning model by training a MIMLL model, for example, using a weakly supervised learning module (as disclosed above), and aggregating output scores across multiple tiles to indicate the presence or absence of cancer or the presence or absence of muscularis propria. Alternatively, the aggregated model may be trained to predict multi-labels for each image, tile, or slide using embeddings from each tile.

[0123] In step 721, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0124] In step 723, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0125] In step 725, the method may include partitioning each of the one or more digital images into a set of tiles.

[0126] In step 727, the method may include calculating a muscularis propria prediction or an invasive cancer prediction from the set of tiles. The method may also include outputting the prediction to an electronic storage device.

[0127] 8 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict pan-cancer diagnoses according to one or more exemplary embodiments of the present disclosure. While machine learning has been successfully used to create good models for predicting cancer in common cancer types, prediction for rare cancers is a challenge because there may not be much training data. Another challenge is predicting where cancer originated when metastatic, which is sometimes impossible to determine. Understanding the progenitor cancer can help guide cancer treatment. Embodiments enable pan-cancer prediction and progenitor cancer prediction using a single machine learning model. By training on many tissue types, the method can achieve an understanding of tissue morphology so that it can effectively generalize to rare cancer types, even when very little data may be available.

[0128] According to one embodiment, exemplary methods 800 and 820 may include one or more of the following steps: In step 801, one or more digital images associated with a tissue sample are received into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0129] In step 803, the method may include receiving a plurality of data indicative of the type of tissue shown in each of the digital images received for the patient.

[0130] In step 805, the method may include receiving a set of binary indicators for each digital image, indicating the presence or absence of cancer.

[0131] In step 807, the method may include partitioning each of the one or more digital images into a set of tiles.

[0132] In step 809, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0133] In step 811, the method may include organizing at least one pan-cancer prediction output for the patient into a binary list. One element of the list may indicate the presence of any cancer, and other elements in the list may indicate the presence of each specific cancer type. For example, a prostate cancer sample may have a positive indicator for general cancers, a positive indicator for a prostate indicator for prostate cancer, and negative indicators for all other outputs corresponding to other tissues (e.g., lung, breast, etc.). A patient for whom all slides are benign may have a label list containing all negative indicators.

[0134] In step 813, the method may include training a machine learning model to predict a binary vector for the patient. The machine learning model may comprise a MIMLL model as described above, and the weakly supervised learning module may train the MIMLL model. In addition, the method may include aggregating the pan-cancer prediction output of MIMLL across various tiles using an aggregation model (as disclosed above). Alternatively, the aggregation model may be trained to predict the pan-cancer predictive indicator(s) using embeddings from each tile.

[0135] In step 821, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0136] In step 823, the method may include receiving a plurality of data indicative of the type of tissue shown in each of the digital images received for the patient.

[0137] In step 825, the method may include partitioning each of the one or more digital images into a set of tiles.

[0138] In step 827, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0139] In step 829, the method may include calculating a pan-cancer prediction using the trained machine learning model. The machine learning model may comprise a trained MIMLL model and / or an aggregate model (as disclosed above). Exemplary outputs may include, but are not limited to, the following: a. Pan-cancer prediction: The cancer presence output can be used to determine the presence of cancer regardless of tissue type, even for tissue types not observed during training. This can be useful for rare cancers, for which there may not be enough data available to train a machine learning model. b. Protocancer Prediction: The cancer subtype outputs can be used to predict the origin of metastatic cancer by identifying the maximum subtype output. If one of the cancer outputs for the subtype is sufficiently higher than the type of tissue input into the system, this can indicate to a pathologist that the output is protocancer. For example, if a bladder tissue sample is found to have cancer by a machine learning model but has a prostate cancer subtype output, this can indicate to a pathologist that the cancer found in the bladder is likely metastatic prostate cancer, instead of cancer that originated within the bladder.

[0140] In step 831, the method may include saving the prediction to an electronic storage device.

[0141] 9 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict organ toxicity according to one or more exemplary embodiments of the present disclosure. In preclinical animal studies for drug development, pathologists determine whether any toxicity is present, the form of toxicity, and / or the organs in which toxicity may be found. Embodiments allow these predictions to be performed automatically. A challenge with preclinical studies is that slides may contain multiple organs in order to conserve glass during preparation.

[0142] According to one embodiment, exemplary methods 900 and 920 may include one or more of the following steps: In step 901, one or more digital images associated with a tissue sample are received into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0143] In step 903, the method may include receiving a plurality of binary indicators indicating the presence or absence of toxicity and / or the type or severity of toxicity.

[0144] In step 905, the method may include receiving the presence or absence and / or type or severity of toxicity for at least one organ.

[0145] In step 907, the method may include partitioning each of the one or more digital images into a set of tiles.

[0146] In step 909, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected components algorithm, or any other suitable method.

[0147] In step 911, the method may include organizing at least one toxicity prediction output for the patient into a binary list, where one element of the list may indicate the presence or type of any toxicity found on the slide, and other elements in the list may indicate the presence / type of toxicity in each organ.

[0148] In step 913, the method may include training a machine learning model to predict a binary vector for the patient. The machine learning model may comprise a MIMLL model as described above, and the weakly supervised learning module may train the MIMLL model. In addition, the method may include aggregating the toxicity prediction output of MIMLL across various tiles using an aggregation model (as disclosed above). Alternatively, the aggregation model may be trained to predict a toxicity predictive signature using embeddings from each tile.

[0149] In step 921, the method may include receiving one or more digital images associated with the tissue sample into a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0150] In step 923, the method may include partitioning each of the one or more digital images into a set of tiles.

[0151] In step 925, the method may include detecting at least one tissue region from the background of each of the one or more digital images to create a tissue mask and removing at least one tile that is non-tissue. Further processing may begin without the non-tissue tile. This may be achieved in a variety of ways, including, but not limited to, thresholding methods, segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc., based on color, color intensity, texture features, Otsu's thresholding method, or any other suitable method, followed by invocation of a connected component algorithm, or any other suitable method.

[0152] In step 927, the method may include calculating a toxicity prediction using the trained machine learning model. The machine learning model may comprise a trained MIMLL model and / or an aggregate model (as disclosed above). Exemplary outputs may include, but are not limited to, the following: a. Toxicity Present: The Toxicity Present output can be used to determine the presence and / or severity of toxicity across the entire slide, regardless of tissue type. b. Organ Toxicity Prediction: The organ toxicity output can be used to determine organs where toxicity may be found.

[0153] In step 929, the method may include storing the toxicity prediction in an electronic storage device.

[0154] FIG. 10 illustrates an exemplary connected component algorithm according to an embodiment of the present disclosure. The connected component algorithm may aggregate features across image regions. For example, thresholding may result in a binary (e.g., black and white) image. The connected component algorithm or model may identify various regions within the image at the pixel level, e.g., three regions (green, red, and brown). Each pixel may use connected components to belong to a tile and component (green, red, or brown) in a specific implementation. Aggregation may occur in many ways, including majority voting (e.g., for all tiles in the green component vote, resulting in the color green having a value of 1) or a learned aggregator (e.g., a vector of features may be extracted from each tile and, for each component, input to a component aggregator module process; thus, tiles in the green component may be fed into the component aggregator module, which may generate a severity score). The CNN may output either a prediction for the tile (e.g., a number), a feature vector for the tile describing its visual properties, or both.

[0155] 11, device 1100 may include a central processing unit (CPU) 1120. CPU 1120 may be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As will be understood by those skilled in the art, CPU 1120 may also be a single processor in a multi-core / multi-processor system, such as a system operating alone or within a cluster of computing devices operating within a cluster or server farm. CPU 1120 may be connected to a data communications infrastructure 1110, for example, a bus, a message queue, a network, or a multi-core message passing scheme.

[0156] The device 1100 may also include a main memory 1140, e.g., random access memory (RAM), and may also include a secondary memory 1130. The secondary memory 1130, e.g., read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. The removable storage drive, in this embodiment, reads from and / or writes to a removable storage unit in a well-known manner. The removable storage device may comprise a floppy disk, magnetic tape, optical disk, etc., which is read by and written to the removable storage drive. As will be appreciated by those skilled in the art, such a removable storage unit generally includes a computer-usable storage medium having computer software and / or data stored therein.

[0157] In alternative implementations thereof, secondary memory 1130 may include similar means for allowing computer programs or other instructions to be loaded into device 1100. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 1100.

[0158] Device 1100 may also include a communications interface (“COM”) 1160. Communications interface 1160 allows software and data to be transferred between device 1100 and external devices. Communications interface 1160 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 1160 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1160. These signals may be provided to communications interface 1160 over a communications path in device 1100, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0159] The hardware elements, operating systems, and programming languages ​​of such devices are conventional in nature and are assumed to be sufficiently familiar to those skilled in the art. Device 1100 may also include input and output ports 1150 to connect to input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of one computer hardware platform.

[0160] Throughout this disclosure, references to components or modules generally refer to items that may be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0161] The tools, modules, and functions described above may be implemented by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which may provide non-transitory storage for software programming from time to time.

[0162] The software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, the communication may allow the software to be loaded from one computer or processor into another. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0163] The foregoing general description is exemplary and explanatory only and is not a limitation of the present disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only.

Claims

[Claim 1] The invention described in this specification.

Images