System and method for processing electronic images for a computational detection method

Abstract

Systems and methods for receiving one or more electronic slide images associated with a tissue specimen associated with a patient and / or medical case, dividing a first slide image of the one or more electronic slide images into a plurality of tiles, detecting a plurality of tissue regions of the first slide image and / or plurality of tiles to generate a tissue mask, determining whether any tile of the plurality of tiles corresponds to non-tissue, removing any tile of the plurality of tiles determined to be non-tissue, determining a prediction for at least one label of the one or more electronic slide images using a machine learning prediction model, the machine learning prediction model generated by processing a plurality of training images, and outputting the prediction of the trained machine learning prediction model.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 966,716, filed January 28, 2020, the entire disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Various embodiments of this disclosure generally relate to creating predictive models to predict labels on prepared tissue specimens by processing electronic images. More specifically, particular embodiments of this disclosure relate to systems and methods for predicting, identifying, or detecting diagnostic information about prepared tissue specimens. This disclosure further provides systems and methods for creating predictive models that predict labels from invisible slides. Background Technology

[0004] The performance of machine learning and deep learning models used in histopathology can be limited by the quantity and quality of the annotated examples used to train these models. Large-scale experiments on supervised image classification problems have shown that model performance can improve continuously with up to 50 million training examples. Manually annotating such large amounts of data can be extremely costly in terms of time and resources, and can be a serious limitation in ensuring that the system is implemented at a clinically relevant level and can be promoted across institutions.

[0005] The foregoing general description and the following detailed description are both exemplary and illustrative only, and are not restrictive of this disclosure. The background description provided herein is for the purpose of presenting the overall context of this disclosure. Unless otherwise stated herein, the materials described in this section are not prior art to the claims of this application, and are not admitted as prior art or suggestions of prior art by virtue of their inclusion in this section. Summary of the Invention

[0006] According to certain aspects of this disclosure, systems and methods for developing weakly supervised multi-label and multi-task learning for computational biomarker detection in digital pathology are disclosed.

[0007] A computer-implemented method for processing electronic images corresponding to a specimen includes: receiving one or more digital images associated with a tissue specimen; receiving one or more electronic slide images associated with a tissue specimen, the tissue specimen being associated with a patient and / or medical case; dividing a first slide image of the one or more electronic slide images into a plurality of tiles; detecting a plurality of tissue regions of the first slide image and / or the plurality of tiles to generate a tissue mask; determining whether any tile in the plurality of tiles corresponds to non-tissue; removing any tile in the plurality of tiles determined to be non-tissue; determining a prediction for at least one label of the one or more electronic slide images using a machine learning prediction model, the machine learning prediction model being generated by processing a plurality of training images; and outputting the prediction of the trained machine learning prediction model.

[0008] A system for processing electronic images corresponding to a specimen includes: a memory storing instructions; and at least one processor that executes the instructions to implement a process comprising the steps of: receiving one or more digital images associated with a tissue specimen; receiving one or more electronic slide images associated with a tissue specimen, the tissue specimen being associated with a patient and / or medical case; dividing a first slide image of the one or more electronic slide images into a plurality of patches; detecting a plurality of tissue regions of the first slide image and / or the plurality of patches to generate a tissue mask; determining whether any patch in the plurality of patches corresponds to non-tissue; removing any patch in the plurality of patches that is determined to be non-tissue; determining a prediction for at least one label of the one or more electronic slide images using a machine learning prediction model generated by processing a plurality of training images; and outputting the prediction of the trained machine learning prediction model.

[0009] A non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform a method for processing electronic images corresponding to a specimen, the method comprising: receiving one or more digital images associated with a tissue specimen; receiving one or more electronic slide images associated with a tissue specimen, the tissue specimen being associated with a patient and / or medical case; dividing a first slide image of the one or more electronic slide images into a plurality of patches; detecting a plurality of tissue regions of the first slide image and / or the plurality of patches to generate a tissue mask; determining whether any patch in the plurality of patches corresponds to non-tissue; removing any patch in the plurality of patches determined to be non-tissue; determining a prediction for at least one label of the one or more electronic slide images using a machine learning prediction model, the machine learning prediction model being generated by processing a plurality of training images; and outputting the prediction of the trained machine learning prediction model.

[0010] It should be understood that both the foregoing general description and the following detailed description are merely exemplary and illustrative, and are not restrictive on the disclosed embodiments as claimed. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments:

[0012] Figure 1A An exemplary block diagram of a system and network for creating a predictive model according to an exemplary embodiment of the present disclosure is illustrated;

[0013] Figure 1B An exemplary block diagram of a prediction model platform according to an exemplary embodiment of the present disclosure is illustrated;

[0014] Figure 1C An exemplary block diagram of a wafer analysis tool according to an exemplary embodiment of the present disclosure is illustrated;

[0015] Figure 2A This is a flowchart illustrating an exemplary method for using a predictive model created by a trained machine learning system, according to one or more exemplary embodiments of this disclosure.

[0016] Figure 2B This is a flowchart illustrating an exemplary method for training a weakly supervised tile-level learning module in a trained machine learning system, according to one or more exemplary embodiments of the present disclosure.

[0017] Figure 2C This is a flowchart illustrating an exemplary method for training a weakly supervised aggregation module in a trained machine learning system according to one or more exemplary embodiments of the present disclosure;

[0018] Figure 3 This is a flowchart illustrating an exemplary method for training and using a machine learning system to simultaneously detect and grade prostate cancer, according to one or more exemplary embodiments of this disclosure;

[0019] Figure 4 This is a flowchart illustrating an exemplary method for training and using a machine learning system to quantify tumors in a prostate biopsy, according to one or more exemplary embodiments of the present disclosure.

[0020] Figure 5 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict cancer subtypes according to one or more exemplary embodiments of this disclosure;

[0021] Figure 6This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict surgical margins according to one or more exemplary embodiments of the present disclosure;

[0022] Figure 7 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict bladder cancer biomarkers according to one or more exemplary embodiments of this disclosure;

[0023] Figure 8 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict pan-cancer diagnoses according to one or more exemplary embodiments of the present disclosure;

[0024] Figure 9 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict organ toxicity according to one or more exemplary embodiments of this disclosure;

[0025] Figure 10 An exemplary connected component algorithm according to an embodiment of the present disclosure is illustrated;

[0026] Figure 11 An exemplary system that can perform the techniques presented herein is described. Detailed Implementation

[0027] Reference will now be made in detail to exemplary embodiments of this disclosure, examples of which are illustrated in the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0028] The systems, apparatuses, and methods disclosed herein are described in detail by way of example and with reference to the accompanying drawings. The examples discussed herein are merely illustrative and are provided to aid in the explanation of the apparatuses, apparatuses, systems, and methods described herein. Unless specifically designated as mandatory, the features or components shown in the accompanying drawings or discussed below should not be considered mandatory for any particular implementation of any of these apparatuses, systems, or methods.

[0029] Furthermore, for any method described, whether or not it is described in conjunction with a flowchart, it should be understood that, unless the context otherwise specifies or requires, any explicit or implicit ordering of the steps performed in the execution of the method does not imply that these steps must be performed in the presented order, but may be performed in a different order or in parallel.

[0030] As used herein, the term “exemplary” is used in the sense of “example” rather than “ideal”, and the terms “a” and “an” in this document do not indicate a limitation on quantity, but rather indicate the presence of one or more of the items referenced.

[0031] Pathology refers to the study of disease, as well as its causes and effects. More specifically, pathology refers to the performance of tests and analyses used to diagnose disease. For example, tissue samples can be placed on slides for a pathologist (e.g., a physician who is an expert in analyzing tissue samples to determine the presence of any abnormalities) to examine under a microscope. That is, pathological specimens can be cut into multiple sections, stained, and prepared into slides for a pathologist to examine and give a diagnosis. When the diagnosis on a slide is uncertain, the pathologist can arrange additional cutting levels, staining, or other tests to gather more information from the tissue. Then, one or more technicians can create new slides, which may contain additional information used by the pathologist in making a diagnosis. The process of creating additional slides can be time-consuming, not only because it may involve retrieving the tissue block, cutting it to make a new slide, and then staining the slide, but also because it may be processed in batches for multiple orders. This can significantly delay the final diagnosis given by the pathologist. Furthermore, even after the delay, there may still be no guarantee that the new slides will have sufficient information to give a diagnosis.

[0032] Pathologists can evaluate pathology slides for cancer and other diseases independently. This disclosure presents a comprehensive workflow for improving the diagnosis of cancer and other diseases. This workflow can integrate, for example, slide evaluation, tasks, image analysis, and artificial intelligence (AI) for cancer detection, annotation, consultation, and recommendations within a single workstation. Specifically, this disclosure describes various exemplary user interfaces available in the workflow, as well as AI tools that can be integrated into the workflow to accelerate and improve the work of pathologists.

[0033] For example, computers can be used to analyze images of tissue samples to quickly identify whether additional information about a particular tissue sample is needed, and / or to highlight areas that the pathologist should examine more closely. Thus, the process of obtaining additional stained slides and testing can be automated before they are reviewed by the pathologist. When paired with an automated slide splitting and staining machine, this can provide a fully automated slide preparation pipeline. This automation has at least the following benefits: (1) minimizing the amount of time wasted by pathologists in determining that slides are insufficient for diagnosis; (2) minimizing the (average total) time from specimen collection to diagnosis by avoiding the additional time between scheduling and generating additional tests; (3) reducing the amount of time and material wasted per recut by allowing recutting while the tissue block (e.g., pathology specimen) is in the cutting table; (4) reducing the amount of tissue material wasted / discarded during slide preparation; (5) reducing the cost of slide preparation by partially or fully automating the procedure; (6) allowing automated, customized cutting and staining of slides, which will result in more representative / informative slides from the sample; (7) allowing more slides to be generated per tissue block, contributing to more informed / accurate diagnoses by reducing the overhead of requesting additional tests from pathologists; and / or (8) identifying or verifying the correct attributes of digital pathology images (e.g., regarding specimen type), etc.

[0034] The process of using computers to assist pathologists is called computational pathology. The computational methods used in computational pathology can include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI can include, but is not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. By using computational pathology, lives can be saved by helping pathologists improve the accuracy, reliability, efficiency, and accessibility of diagnoses. For example, computational pathology can be used to help examine slides suspected of containing cancer, allowing pathologists to review and confirm their initial assessments before giving a final diagnosis.

[0035] As described above, the computational pathology process and apparatus of this disclosure can provide an integrated platform that allows for a fully automated process, including data acquisition, processing, and viewing of digital pathology images via a web browser or other user interface, while integrating with a laboratory information system (LIS). Furthermore, cloud-based data analytics on patient data can be used to aggregate clinical information. Data can originate from hospitals, clinics, field researchers, etc., and can be analyzed using machine learning, computer vision, natural language processing, and / or statistical algorithms to enable real-time monitoring and prediction of health patterns at multiple geographic-specific levels.

[0036] Histopathology refers to the study of specimens already placed on slides. For example, digital pathology images can consist of digital images of microscope slides containing specimens (e.g., smears). One method pathologists can use to analyze images on slides is to identify cell nuclei and classify them as normal (e.g., benign) or abnormal (e.g., malignant). To help pathologists identify and classify cell nuclei, histological staining can be used to make cells visible. Many dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method in which hematoxylin stains cell nuclei blue, eosin stains the cytoplasm and extracellular matrix pink, and other tissue areas exhibit these color variations. However, in many cases, H&E-stained histological preparations do not provide pathologists with sufficient information to visually identify biomarkers that can aid in diagnosis or guide treatment. In such cases, techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH), or fluorescence in situ hybridization (FISH) can be used. For example, IHC and immunofluorescence involve using antibodies that bind to specific antigens in the tissue, enabling visual detection of cells expressing specific proteins of interest. This can reveal biomarkers that trained pathologists cannot reliably identify based on analysis of H&E-stained slides. Depending on the type of probe used (e.g., DNA probes for gene copy number and RNA probes for assessing RNA expression), ISH and FISH can be used to assess gene copy number or the abundance of specific RNA molecules. If these methods also fail to provide sufficient information to detect some biomarkers, tissue genetic testing can be used to confirm the presence of biomarkers (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a given gene in cancer).

[0037] Digital images can be prepared to show stained microscope slides, allowing pathologists to manually observe the images on the slides and estimate the number of abnormally stained cells in the images. However, this process can be time-consuming and prone to errors in identifying abnormalities, as some are difficult to detect. Computational processes and devices can be used to assist pathologists in detecting abnormalities that would otherwise be difficult to detect. For example, AI can be used to predict biomarkers (such as overexpression of protein and / or gene products, amplification or mutation of specific genes) from salient regions within digital images of tissues stained using H&E and other dye-based methods. The tissue image can be a whole slide image (WSI), an image of the tissue core within a microarray, or an image of a selected region of interest within a tissue section. Using staining methods such as H&E, these biomarkers may be difficult for humans to visually detect or quantify without the aid of additional testing. Using AI to infer these biomarkers from digital images of tissues has the potential to improve patient care, while also being faster and cheaper.

[0038] The detected biomarkers or images can then be used individually to recommend specific cancer drugs or drug combinations for treating a patient, and AI can identify which drugs or drug combinations are unlikely to be successful by associating the detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendations of immunotherapies for a patient's specific cancer. Furthermore, this can enable personalized cancer treatment for specific subsets of patients and / or rare cancer types.

[0039] As described above, the computational pathology process and apparatus of this disclosure can provide an integrated platform that allows for a fully automated process, including data acquisition, processing, and viewing of digital pathology images via a web browser or other user interface, while integrating with a laboratory information system (LIS). Furthermore, cloud-based data analytics of patient data can be used to aggregate clinical information. Data can originate from hospitals, clinics, field researchers, etc., and can be analyzed using machine learning, computer vision, natural language processing, and / or statistical algorithms to enable real-time monitoring and prediction of health patterns at multiple geographic-specific levels.

[0040] The aforementioned digital pathology images may be stored together with tags and / or labels regarding the attributes of the specimen or the digital pathology image, and such tags / labels may be incomplete. Therefore, the system and method disclosed herein predict at least one label from a collection of digital images.

[0041] The performance of machine learning and deep learning models used in histopathology can be limited by the quantity and quality of the annotated examples used to train these models. Large-scale experiments on supervised image classification problems have shown that model performance continues to improve, reaching the order of 50 million training examples. However, most clinically relevant tasks in pathology require more than just classification. When a pathologist gives a diagnosis, it can take the form of a report containing many different kinds of relevant domains and relating to the entire slide or a set of slides. In oncology, these domains can include the presence of cancer, cancer grade, tumor quantification, cancer grade groups, the presence of various features important for cancer staging, etc. In preclinical drug research animal studies, these domains can include the presence of toxicity, the severity of toxicity, and the type of toxicity. Obtaining the necessary annotations to train most supervised deep learning models can involve pathologists labeling individual pixels, patches (e.g., one or more relatively small rectangular areas in a slide image), or regions of interest (e.g., polygons) from slide images with appropriate annotations. Different sets of training annotations can be used for each domain in the report. Furthermore, a typical digital pathology slide can contain tens of gigapixels, or more than 100,000 tiles. Manually annotating such a large amount of data can be extremely expensive in terms of both time and cost, and may be a serious limitation in ensuring the system's clinical relevance and cross-institutional deployment. Therefore, there is a desire to generate training data that can be used for histopathology.

[0042] The embodiments disclosed herein overcome the limitations described above. Specifically, the embodiments disclosed herein may use weak supervision, where the deep learning model can be trained directly from a pathologist's diagnosis, rather than utilizing additional labels for each pixel or patch in a digital image. In some embodiments, the machine learning or deep learning model may include machine learning algorithms. One technique can determine binary cancer detection; however, the techniques discussed herein further disclose, for example, how deep learning systems can be trained in weakly supervised multi-label and multi-task settings to simultaneously perform grading, subtyping, inferring multiple disease attributes, etc. This allows the system to be trained directly from diagnostic reports or test results without requiring extensive annotations, thereby reducing the number of required training labels by five orders of magnitude or more.

[0043] The disclosed systems and methods can automatically predict specimen or image characteristics without relying on stored markers or labels. Furthermore, systems and methods for quickly and accurately identifying and / or verifying specimen types or any information associated with digital pathology images without having to access a LIS or similar information database are disclosed. One embodiment of this disclosure may include a system trained to identify various characteristics of digital pathology images based on a dataset of previous digital pathology images. The trained system can provide classification for specimens shown in digital pathology images. This classification helps to provide treatment or diagnostic predictions for patients(s) associated with the specimen.

[0044] This disclosure includes one or more embodiments of a slide analysis tool. The input to the tool may include digital pathology images and any associated additional input. The output of the tool may include global and / or local information about the specimen. The specimen may include a biopsy or surgically removed specimen.

[0045] Figure 1A The diagram illustrates a system and network for using machine learning to determine specimen or image characteristic information related to one or more digital pathology images, according to exemplary embodiments of the present disclosure.

[0046] Specifically, Figure 1A The illustration depicts an electronic network 120 that can be connected to servers such as those in hospitals, laboratories, and / or doctors' offices. For example, physician servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and / or laboratory information systems 125 can each be connected to the electronic network 120, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of this application, the electronic network 120 can also be connected to a server system 110, which may include processing equipment configured to implement a disease detection platform 100, which includes a slide analysis tool 101 for determining specimen characteristic or image characteristic information related to one or more digital pathology images according to an exemplary embodiment of this disclosure, and for classifying the specimens using machine learning.

[0047] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may create or otherwise obtain digital images or any combination thereof of one or more cytological specimens, one or more histopathological specimens, one or more slides of cytological specimens, one or more slides of histopathological specimens, or other images of one or more patients. Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytological information, etc. Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may transmit digitized slide images and / or patient-specific information to server system 110 via electronic network 120. Server system 110 may include one or more storage devices 109 for storing images and data received from at least one of physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. Server system 110 may also include processing means for processing the images and data stored in the one or more storage devices 109. Server system 110 may further include one or more machine learning tools or capabilities. For example, according to one embodiment, the processing means may include machine learning tools for disease detection platform 100. Alternatively or additionally, this disclosure (or part of the systems and methods of this disclosure) may be executed on a local processing device (e.g., a laptop computer).

[0048] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to examine slides. In a hospital setting, tissue type information may be stored in laboratory information system 125. However, correct tissue classification information is not always paired with image content. Furthermore, even when using LIS to access specimen types of digital pathology images, the labeling may be incorrect due to the fact that many components of LIS may be manually entered, leaving a large margin of error. According to exemplary embodiments of this disclosure, specimen types may be identified without requiring access to library information system 125, or may be identified to potentially correct library information system 125. For example, a third party may anonymously access image content in LIS that does not have a corresponding specimen type label stored. Furthermore, access to LIS content may be restricted due to its sensitive content.

[0049] Figure 1B An exemplary block diagram of a disease detection platform 100 is illustrated, which is used to determine specimen or image characteristic information related to one or more digital pathology images using machine learning. For example, the disease detection platform 100 may include a slide analysis tool 101, a data acquisition tool 102, a slide capture tool 103, a slide scanner 104, a slide manager 105, a memory 106, and a viewing application tool 108.

[0050] As described below, slide analysis tool 101 refers to a process and system, according to an exemplary embodiment, for processing digital images associated with tissue specimens and using machine learning to analyze slides.

[0051] Data acquisition tool 102 refers to the process and system, according to an exemplary embodiment, for facilitating the transfer of digital pathology images to various tools, modules, components, and devices for classifying and processing digital pathology images.

[0052] According to an exemplary embodiment, slide intake tool 103 refers to a process and system for scanning pathological images and converting them into digital form. Slides can be scanned using slide scanner 104, and slide manager 105 can process the images on the slides into digital pathological images and store the digital images in memory 106.

[0053] According to an exemplary embodiment, viewing application tool 108 refers to a process and system for providing a user (e.g., a pathologist) with information about specimen characteristics or image characteristics related to one or more digital pathology images. The information can be provided through various output interfaces, such as screens, monitors, storage devices, and / or web browsers.

[0054] Each of the slide analysis tool 101 and its components can transmit and / or receive digitized slide images and / or patient information to and from the server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 via electronic network 120. Furthermore, the server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the slide analysis tool 101, data acquisition tool 102, slide ingestion tool 103, slide scanner 104, slide manager 105, and viewing application tool 108. The server system 110 may also include processing equipment for processing the images and data stored in the storage devices. The server system 110 may further include one or more machine learning tools or capabilities, for example, due to the processing equipment. Alternatively or additionally, this disclosure (or part of the systems and methods of this disclosure) may be executed on a local processing device (e.g., a laptop computer).

[0055] Any of the aforementioned devices, tools, and modules may be located on a device that can be connected to electronic network 120 via one or more computers, servers, and / or handheld mobile devices, such as the Internet or cloud service providers.

[0056] Figure 1C An exemplary block diagram of a slice analysis tool 101 according to an exemplary embodiment of the present disclosure is illustrated. The slice analysis tool 101 may include a training image platform 131 and / or a target image platform 135.

[0057] According to one embodiment, the training image platform 131 can create or receive training images for training a machine learning system to effectively analyze and classify digital pathology images. For example, training images can be received from any one or any combination of server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. Images used for training can be from real-world sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images can include (a) digital slides stained with various staining agents, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, etc.; and / or (b) digital tissue samples from 3D imaging devices, such as microCT.

[0058] The training image acquisition module 132 can create or receive a dataset comprising one or more training images corresponding to any one or both of images of human tissue and graphically rendered images. For example, training images can be received from any or any combination of server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. This dataset can be stored on a digital storage device. The quality score determiner module 133 can identify quality control (QC) problems (e.g., defects) in the training images at a global or local level, problems that may significantly affect the usability of the digital pathology images. For example, the quality score determiner module can use information about the entire image, such as specimen type, overall quality of specimen cutting, overall quality of the glass pathology slide itself, or tissue morphological characteristics, and determine an overall quality score for the image. The treatment identification module 134 can analyze images of tissue and determine which digital pathology images have therapeutic effects (e.g., post-treatment) and which images do not (e.g., pre-treatment). It is useful to identify whether digital pathology images have therapeutic effects, as prior treatment effects in tissue can affect the morphology of the tissue itself. Most digital inoculosurgical systems (LIS) do not explicitly maintain this characteristic, and therefore it may be desirable to classify specimen types with prior treatment effects.

[0059] According to one embodiment, the target image platform 135 may include a target image input module 136, a specimen detection module 137, and an output interface 138. The target image platform 135 can receive target images and apply a machine learning model to the received target images to determine the characteristics of the target specimen. For example, target images can be received from any one or any combination of server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The target image acquisition module 136 can receive target images corresponding to target specimens. The specimen detection module 137 can apply a machine learning model to the target images to determine the characteristics of the target specimens. For example, the specimen detection module 137 can detect the specimen type of the target specimen. The specimen detection module 137 can also apply a machine learning model to the target images to determine the quality score of the target images. Furthermore, the specimen detection module 137 can apply a machine learning model to the target specimens to determine whether the target specimen is pre-treatment or post-treatment.

[0060] Output interface 138 can be used to output information about the target image and the target specimen (e.g., output to a screen, monitor, storage device, web browser, etc.).

[0061] Figure 2AThis is a flowchart illustrating an exemplary method of using a predictive model created by a trained machine learning system according to one or more exemplary embodiments of the present disclosure. For example, exemplary method 200 (steps 202-210) may be performed automatically by wafer analysis tool 101 or in response to a request from a user.

[0062] According to one embodiment, an exemplary method 200 for using a predictive model may include one or more of the following steps. In step 202, the method may include receiving one or more digital images associated with a tissue specimen, wherein the one or more digital images include a plurality of slide images. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), or any other suitable storage device.

[0063] In step 204, the method may include dividing one of the plurality of slide images into a set of tiles of the plurality of slide images.

[0064] In step 206, the method may include detecting multiple tissue regions from the background of one of the plurality of slide images to create a tissue mask, and removing at least one patch from the set of patches that is detected as non-tissue. The non-tissue patches may include the background of the slide images. This can be achieved in various ways, including: threshold-based methods based on color, color intensity, texture features, or the Otsu method, followed by a connected component algorithm; segmentation algorithms such as k-means, graph cutting, mask region convolutional neural networks (mask R-CNN); or any other suitable method.

[0065] In step 208, the method may include using a machine learning system to determine predictions for labels corresponding to multiple slide images of a patient or medical case, said machine learning system being generated by processing multiple training examples to create a predictive model. The training examples may include one or more sets of digital slide images and multiple target labels.

[0066] In step 210, the method may include outputting a predictive model for training a machine learning system, the predictive model predicting at least one label from at least one chip never used to train the machine learning system, and outputting the prediction to an electronic storage device.

[0067] Figure 2B This is a flowchart illustrating an exemplary method for training a weakly supervised tile-level learning module in a trained machine learning system according to one or more exemplary embodiments of the present disclosure. The weakly supervised learning module can be trained using slice-level training labels to perform tile-level predictions. For example, exemplary method 220 (steps 222-230) can be performed automatically by slice analysis tool 101 or in response to a request from a user.

[0068] According to one embodiment, an exemplary method 220 for using a predictive model may include one or more of the following steps. In step 222, the method may include receiving a set of digital images associated with training tissue specimens into a digital storage device, wherein the set of digital images includes a plurality of training slide images. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), or any other suitable storage device.

[0069] In step 224, the method may include receiving a plurality of summary annotations, the summary annotations including one or more labels for each of a plurality of training slice images. The labels may be binary, multilevel binary, categorical, ordinal, or real-valued.

[0070] In step 226, the method may include dividing one of a plurality of training slice images into a set of training patches of the plurality of training slice images.

[0071] In step 228, the method may include detecting at least one tissue region from the background of a plurality of training slice images to create a training tissue mask, and removing at least one training patch from the set of training patches that is detected as non-tissue. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0072] In step 230, the method may include training a prediction model under weak supervision to infer at least one multi-label tile-level prediction using at least one summary label. Under weak supervision, there are four general methods for training the model, but any suitable method can be used.

[0073] 1. Multi-Instance Learning (MIL)It can be used to train a tile-level prediction model with binary or categorical labels by learning to identify tiles containing target labels for slides. This identification can be accomplished by: finding salient tiles (e.g., the highest-scoring tile based on the summary annotations or labels received at each training iteration), and using these tiles to update a classifier trained with the received summary labels associated with each salient tile. For example, the classifier can be trained to identify cancer based on a set of overlapping tiles. When salient tiles are determined, the summary labels can be used to update the tile-level labels. The tile-level labels and the classifier can then determine or provide labels for the set of tiles. MIL can also be used to train machine learning models to extract diagnostic features for other downstream tasks such as cancer grading, cancer subtyping, biomarker detection, etc.

[0074] 2. Multi-Instance Multi-Label Learning (MIMLL) This could be a generalized tile-level prediction model including MIL, which treats each slide as a set of tiles that can be associated with multiple labels, rather than just a single binary label as in MIL. These slide labels might come from pathologist reports, genetic tests, immune tests, or other measurements / tests. A MIMLL model can be trained to select tiles corresponding to each of a set of summary training labels belonging to one or more slides. This embodiment could involve MIMLL training a neural network (e.g., a convolutional neural network (CNN), capsule network, etc.) by iteratively following these steps:

[0075] For each label to be predicted, a scoring function is used to select the most relevant set of tiles. The scoring function can be formulated to rank multiple tiles simultaneously. For example, in the case of multiple binary labels, a CNN can be run on each tile, which attempts to predict each of the multiple binary labels from each tile in the tile set, and can select the tile with the output closest to 1 for one or more labels.

[0076] The weights of the CNN model are updated using the associated labels of the selected tiles. Each label has its own output layer in the model.

[0077] Similar to the MIL model, the MIMLL model can also be used to extract diagnostic features for other downstream tasks.

[0078] 3. Self-supervised learning A small amount of tile-level training data can be used to create an initial tile-based classifier using supervised learning. This initial classifier can then be used to guide the entire training process by alternating the following steps:

[0079] Use predictions from the current tile-based model to reassign tile labels in the training set.

[0080] The model for each tile is updated based on the latest label assignment.

[0081] 4. Unsupervised clustering It can learn to group similar instances together without using target labels. Map tiles can be considered instances, and the number of groups can be pre-specified or learned automatically by the algorithm. Such clustering algorithms can include, but are not limited to, the following methods:

[0082] Expectation Maximization (EM)

[0083] Optimization Maximization (MM)

[0084] k-Nearest Neighbors (KNN)

[0085] Hierarchical clustering

[0086] Agglomeration and Clustering

[0087] The resulting model can be used to extract diagnostic features to be used by the slice-level prediction module.

[0088] Figure 2C This is a flowchart illustrating an exemplary method for training a weakly supervised aggregation module in a trained machine learning system according to one or more exemplary embodiments of the present disclosure. For example, exemplary method 240 (steps 242-244) may be performed automatically by the slice analysis tool 101 or in response to a request from a user.

[0089] According to one embodiment, an exemplary method 240 for training a weakly supervised aggregation module may include one or more of the following steps. In step 242, the method may include receiving multiple predictions or multiple vectors of at least one feature from a weakly supervised tile-level learning module used to train a set of tiles.

[0090] In step 244, the method may include training a machine learning model to take as input multiple predictions or multiple vectors of at least one feature from a weakly supervised tile-level learning module for a set of tiles. The aggregation module may train a multi-task tile-level aggregation model to obtain tile-level input and produce a final prediction for tiles and / or tile images input to the system. The general form of the model may consist of multiple outputs (e.g., multi-task learning), and each label may be binary, categorical, ordinal, or real-valued. Tile-level inputs may include any type of image feature, including but not limited to:

[0091] Outputs from a weakly supervised model (e.g., feature vectors or embeddings).

[0092] CNN features

[0093] Scale-Invariant Feature Transform (SIFT)

[0094] Speed-up Robust Feature (SURF)

[0095] Rotation-Invariant Feature Transform (RIFT)

[0096] Oriented FAST and Rotating BRIEF (ORB)

[0097] The multi-task slice-level aggregation model of the aggregation module can take various forms, including but not limited to:

[0098] Fully connected neural network trained using multiple output task groups

[0099] CNN

[0100] Fully convolutional neural networks

[0101] Recurrent Neural Networks (RNNs) include Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks.

[0102] Graphical Neural Networks

[0103] Converter Network

[0104] Random Forest, Boosted Forest, XGBoost, etc.

[0105] Figure 3 This is a flowchart illustrating an exemplary method for training and using a machine learning system to simultaneously detect and grade prostate cancer, according to one or more exemplary embodiments of this disclosure. Cancer grading can measure the differentiation of cancer cells from normal tissue and can be assessed at a local level by examining cell morphology and at a slide-level summary containing relative amounts of grading. Grading can be performed as part of a pathologist's diagnostic report for common cancers such as prostate cancer, kidney cancer, and breast cancer. Exemplary methods 300 and 320 can be used to train and use a machine learning system to simultaneously detect and grade prostate cancer.

[0106] According to one embodiment, exemplary methods 300 and 320 may include one or more of the following steps. In step 301, the method may include receiving one or more digital images of a stained prostate tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0107] In step 303, the method may include receiving at least one label from the one or more digital images, wherein the at least one label contains an indication of the presence and grade of cancer. The cancer grade may include primary and secondary Gleason classifications.

[0108] In step 305, the method may include dividing each of one or more digital images into a set of tiles.

[0109] In step 307, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. Detecting tissue regions and removing non-tissue patches can be implemented using thresholding methods based on color, color intensity, texture features, the Otsu method, etc., followed by running a connected component algorithm. Based on the thresholding method, thresholding can provide a label of tissue relative to non-tissue regions for one or more pixels of each received slice image. The connected component algorithm can detect image regions or pixels that are connected to each other to detect tissue relative to non-tissue regions across the entire image region, slice image, or slice. Detecting tissue regions and removing non-tissue patches can also be implemented using segmentation algorithms such as k-means, graph cutting, mask R-CNN, etc.

[0110] In step 309, the method may include training a machine learning model to predict the presence and grade of cancer for one or more digital images. Training can be performed in a variety of ways, including but not limited to:

[0111] Using a MIMLL model to train a CNN to predict primary, secondary, and / or tertiary classifications, as disclosed above, is performed, for example, via the following steps: treating each slice as a set of tiles associated with multiple labels, selecting slices corresponding to the summary training labels, scoring each slice by its relevance to the labels, and updating the weights of the CNN model with respect to the associated labels. The trained CNN can then be used to extract embeddings from each slice in the slice set to train a multi-task aggregator (e.g., the previously disclosed aggregation model) to predict the presence of cancer, cancer Gleason classification groups, and / or the primary, secondary, and tertiary classifications of each slice or slice. Alternatively, the predicted outputs from each slice can be used, and aggregation can be performed using manually designed post-processing methods, such as having each slice vote for each classification and taking a majority vote.

[0112] Using the MIL model, each tile is classified as either cancerous or benign, and a grading label is passed to "pure" cases where the primary / secondary / tertiary levels are of the same grading. A tile-level classifier is trained using supervised learning with the passed labels. The model is then refined using self-supervised learning methods as presented in the weakly supervised learning module above.

[0113] Features / embeds are extracted from each tile, and then a multi-task aggregator (e.g., the aggregation model disclosed above) is used to predict the presence of cancer, cancer Gleason grading groups, and / or primary, secondary, and tertiary gradings. Embeddings can come from pre-trained CNNs, random features, features from unsupervised clustering models, SIFT, ORB, etc.

[0114] In step 321, the method may include receiving one or more digital images of the stained prostate specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, RAM, etc.

[0115] In step 323, the method may include dividing the one or more digital images into a set of tiles.

[0116] In step 325, the method may include detecting at least one tissue region from the background of the digital image to create a tissue mask, and removing at least one non-tissue patch. Detection may be implemented in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0117] In step 327, the method may include applying a trained machine learning model to a set of tiles to predict the presence and grade of cancer. The cancer grade may include Gleason classification groups and / or primary, secondary, and tertiary classification groups.

[0118] In step 329, the method may include sending the predicted output to, for example, an electronic storage device.

[0119] Figure 4 This is a flowchart illustrating an exemplary method for training and using a machine learning system to quantify tumors in a prostate biopsy, according to one or more exemplary embodiments of this disclosure. Tumor quantification in a prostate biopsy may consist of estimating the total and relative volume of cancer for each cancer grade (e.g., Gleason grade). Tumor quantification can play a significant role in understanding the composition and severity of prostate cancer, and it is likely a common element in pathological diagnostic reports. Traditionally, tumor size can be quantified manually using a physical ruler on a glass slide. Manual quantification in this manner can suffer from both inaccuracy and inconsistency. Exemplary methods 400 and 420 can be used to train and use a machine learning system to quantify tumors in a prostate biopsy.

[0120] According to one embodiment, exemplary methods 400 and 420 may include one or more of the following steps. In step 401, the method may include receiving one or more digital images of a stained prostate tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0121] In step 403, the method may include receiving at least one real-valued tumor quantification label for each of one or more digital images, wherein the at least one real-valued tumor quantification label contains indications of primary and secondary grading. The label may also include the corresponding volume, length, and size of the tumor in the one or more digital images.

[0122] In step 405, the method may include dividing each of one or more digital images into a set of tiles.

[0123] In step 407, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0124] In step 409, the method may include training a machine learning model to output a cancer grade prediction, as described in exemplary method 300. Tumor quantification estimation can be performed in a variety of ways, including but not limited to:

[0125] The number of graded patches is counted, and their volume and ratio relative to the volume of benign tissue are estimated geometrically.

[0126] The model is trained using a slice-level grading module, for example, as described in Exemplary Method 300. This model can take slice-level diagnostic features from a machine learning cancer grading prediction model (e.g., the model trained in Exemplary Method 300) as input and output a quantitative measure for each tumor using a real-valued regression model.

[0127] In step 421, the method may include receiving one or more digital images of the stained prostate specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0128] In step 423, the method may include dividing the one or more digital images into a set of tiles.

[0129] In step 425, the method may include detecting at least one tissue region from the background of the digital image to create a tissue mask, and removing at least one non-tissue patch. This may be implemented in various ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0130] In step 427, the method may include applying a trained machine learning model to a set of tiles to compute a tumor quantification prediction. This prediction may be output to an electronic storage device. The tumor quantification may be in the form of a size metric or a percentage.

[0131] In step 429, the method may include sending the predicted output to an electronic storage device.

[0132] Figure 5 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict cancer subtypes according to one or more exemplary embodiments of this disclosure. Many cancers have multiple subtypes. For example, in breast cancer, it can be determined whether the cancer is invasive, lobular, or ductal, and whether various other properties, such as calcification, are present. Such methods for predicting cancer subtypes may include predictions of multiple non-exclusive categories, which may involve the use of multi-label learning.

[0133] According to one embodiment, exemplary methods 500 and 520 may include one or more of the following steps. In step 501, the method may include receiving one or more digital images associated with a tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0134] In step 503, the method may include receiving multiple tags of the one or more digital images, wherein the multiple tags and / or biomarkers of the tissue specimen. In breast cancer specimens, relevant biomarkers may be the presence or absence of calcification, cancer, ductal carcinoma in situ (DCIS), invasive ductal carcinoma (IDC), inflammatory breast cancer (IBC), Paget's disease of the breast, angiosarcoma, phyllodes tumor, invasive lobular carcinoma, lobular carcinoma in situ, and various forms of atypia. Tags are not necessarily mutually exclusive, and multiple subtypes may be observed simultaneously.

[0135] In step 505, the method may include dividing each of one or more digital images into a set of tiles.

[0136] In step 507, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0137] In step 509, the method may include training a machine learning model to predict the form and / or subtype of cancer for each tile and / or slide. Training the machine learning model can be accomplished using the MIMLL model disclosed above. The trained subtype prediction machine learning model can be refined using a slide-level prediction model (e.g., an aggregation model) as described above. The slide-level prediction model can take the tile-level subtype prediction from the MIMLL model as input and output a slide-level prediction indicating the presence of each cancer subtype.

[0138] In step 521, the method may include receiving one or more digital images associated with the tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0139] In step 523, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0140] In step 525, the method may include dividing the one or more digital images into a set of tiles and discarding any tiles that do not contain any organization.

[0141] In step 527, the method may include calculating a cancer subtype prediction from a set of tiles and outputting the prediction to an electronic storage device.

[0142] Figure 6This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict surgical margins according to one or more exemplary embodiments of the present disclosure. When a tumor is surgically removed from a patient, it may be important to assess whether the tumor has been completely removed by analyzing the margins of the tissue surrounding the tumor. The width of the margin and the identification of any cancerous tissue within the margin may play an important role in determining how to treat the patient. Training a model to predict margin width and composition can take the form of multi-label, multi-task learning.

[0143] According to one embodiment, exemplary methods 600 and 620 may include one or more of the following steps. In step 601, the method may include receiving one or more digital images associated with a tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0144] In step 603, the method may include receiving multiple tags of one or more digital images, wherein the multiple tags indicate tumor margins and whether the margins are positive (e.g., tumor cells are found in the margins), negative (e.g., no cancer is found in the margins), or close to positive (e.g., not absolutely positive or negative).

[0145] In step 605, the method may include dividing each of one or more digital images into a set of tiles.

[0146] In step 607, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0147] In step 609, the method may include training a machine learning model to predict cancer detection, presence, or grading, as disclosed above.

[0148] In step 621, the method may include receiving one or more digital images associated with the tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0149] In step 623, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0150] In step 625, the method may include dividing each of one or more digital images into a set of tiles.

[0151] In step 627, the method may include calculating predicted surgical margins, tumor margin sizes, or tumor components from the tiles. The method may also include outputting the predictions to an electronic storage device.

[0152] Figure 7 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict bladder cancer biomarkers according to one or more exemplary embodiments of this disclosure. Bladder cancer is one of the most common cancers in the world. If bladder cancer is detected, a pathologist can also determine the presence of the muscularis propria on any slide in which bladder cancer is detected. The muscularis propria is a layer of smooth muscle cells that forms an important part of the bladder wall. Detecting the presence or absence of the muscularis propria is an important step toward determining whether the bladder cancer is aggressive. This embodiment performs both cancer detection and muscularis propria detection, but can be extended to any number of binary classification tasks.

[0153] According to one embodiment, exemplary methods 700 and 720 may include one or more of the following steps. In step 701, one or more digital images associated with a tissue specimen are received into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0154] In step 703, the method may include receiving multiple labels of one or more digital images, wherein the multiple labels indicate the presence or absence of cancer or the presence / absence of the muscularis propria.

[0155] In step 705, the method may include dividing each of one or more digital images into a set of tiles.

[0156] In step 707, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0157] In step 709, the method may include training a machine learning model, for example, training a MIMLL model using a weakly supervised learning module (as disclosed above), and aggregating output scores that indicate the presence / absence of cancer or the presence / absence of the muscularis propria across multiple tiles. Alternatively, embeddings from each tile may be used to train the aggregation model to predict multiple labels for each image, tile, or slice.

[0158] In step 721, the method may include receiving one or more digital images associated with a tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0159] In step 723, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0160] In step 725, the method may include dividing each of one or more digital images into a set of tiles.

[0161] In step 727, the method may include calculating a muscle layer intrinsic prediction or an invasive cancer prediction from a set of tiles. The method may also include outputting the prediction to an electronic storage device.

[0162] Figure 8This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict pan-cancer diagnoses according to one or more exemplary embodiments of the present disclosure. While machine learning has been successfully used to create good models for predicting common cancer types, prediction for rare cancers is challenging because training data may be scarce. Another challenge is predicting the origin of cancer metastasis, which is sometimes impossible. Understanding the tissue of origin can help guide cancer treatment. This embodiment allows for pan-cancer prediction and prediction of the cancer of origin using a single machine learning model. By training on many tissue types, the method achieves an understanding of tissue morphology, enabling it to effectively generalize to rare cancer types where little data is available.

[0163] According to one embodiment, exemplary methods 800 and 820 may include one or more of the following steps. In step 801, one or more digital images associated with a tissue specimen are received into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0164] In step 803, the method may include receiving multiple data indicating the tissue type shown in each of the digital images received for the patient.

[0165] In step 805, the method may include receiving a set of binary labels for each digital image, which indicate the presence or absence of cancer.

[0166] In step 807, the method may include dividing each of one or more digital images into a set of tiles.

[0167] In step 809, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0168] In step 811, the method may include organizing at least one pan-cancer prediction output from the patient into a binary list. One element in the list may indicate the presence of any cancer, and other elements in the list may indicate the presence of each specific cancer type. For example, a prostate cancer specimen may have positive indicators for general cancers, positive indicators for prostate indicators for prostate cancer, and negative indicators for all other outputs corresponding to other tissues (e.g., lungs, breasts, etc.). For patients whose slides are all benign, a label list containing all negative indicators may be available.

[0169] In step 813, the method may include training a machine learning model to predict the patient's binary vector. The machine learning model may include a MIMLL model as described above, wherein a weakly supervised learning module can train the MIMLL model. Furthermore, the method may include using an aggregation model (as disclosed above) across various tiles to aggregate the pan-cancer prediction output of the MIMLL. Alternatively, the aggregation model may be trained to predict (multiple) pan-cancer prediction labels using embeddings from each tile.

[0170] In step 821, the method may include receiving one or more digital images associated with a tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0171] In step 823, the method may include receiving multiple data indicating the tissue type shown in each of the digital images received for the patient.

[0172] In step 825, the method may include dividing each of one or more digital images into a set of tiles.

[0173] In step 827, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0174] In step 829, the method may include using a trained machine learning model to compute a pan-cancer prediction. The machine learning model may include a trained MIMLL model and / or an aggregation model (as disclosed above). Exemplary outputs may include, but are not limited to, the following:

[0175] Pan-cancer prediction: The presence of cancer can be determined using one or more cancer presence outputs, regardless of tissue type, even for tissue types not observed during training. This can be helpful for rare cancers where there may not be enough data available to train a machine learning model.

[0176] Origin Cancer Prediction: The output of (one or more) cancer subtypes can be used to predict the origin of metastatic cancer by identifying the largest subtype output. If one of the cancer subtype outputs is sufficiently higher than the tissue type input to the system, this can indicate to the pathologist that the output is the origin cancer. For example, if a bladder tissue specimen is found to have cancer by (one or more) machine learning models, but the output is a prostate cancer subtype, this can indicate to the pathologist that the cancer found in the bladder is likely metastatic prostate cancer, rather than cancer originating from the bladder.

[0177] In step 831, the method may include saving the prediction to an electronic storage device.

[0178] Figure 9 This is a flowchart illustrating an exemplary method for training and using a machine learning system to predict organ toxicity according to one or more exemplary embodiments of this disclosure. In preclinical animal studies for drug development, pathologists determine the presence of any toxicity, the form of toxicity, and / or organs where toxicity may be found. This embodiment enables the automation of these predictions. One challenge in preclinical work is that slides can contain multiple organs to save glass during preparation.

[0179] According to one embodiment, exemplary methods 900 and 920 may include one or more of the following steps. In step 901, one or more digital images associated with a tissue specimen are received into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0180] In step 903, the method may include receiving multiple binary tags indicating the presence or absence of toxicity and / or the type or severity of toxicity.

[0181] In step 905, the method may include receiving the presence or absence of toxicity in at least one organ and / or its type or severity.

[0182] In step 907, the method may include dividing each of one or more digital images into a set of tiles.

[0183] In step 909, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by running a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0184] In step 911, the method may include organizing at least one toxicity prediction output from the patient into a binary list. One element in the list may indicate the presence or type of any toxicity found on the slide, and other elements in the list may indicate the presence / type of toxicity in each organ.

[0185] In step 913, the method may include training a machine learning model to predict the patient's binary vector. This machine learning model may include a MIMLL model as described above, wherein a weakly supervised learning module can train the MIMLL model. Furthermore, the method may include using an aggregation model (as disclosed above) to aggregate the toxicity prediction outputs of the MIMLL across individual tiles. Alternatively, the aggregation model may be trained to predict toxicity prediction labels using embeddings from each tile.

[0186] In step 921, the method may include receiving one or more digital images associated with the tissue specimen into a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, random access memory (RAM), etc.

[0187] In step 923, the method may include dividing each of one or more digital images into a set of tiles.

[0188] In step 925, the method may include detecting at least one tissue region from the background of each of one or more digital images to create a tissue mask, and removing at least one non-tissue patch. Further processing may begin without any non-tissue patches. This can be achieved in a variety of ways, including but not limited to: thresholding methods based on color, color intensity, texture features, the Otsu method, or any other suitable method, followed by a connected component algorithm; and segmentation algorithms such as k-means, graph cutting, mask R-CNN, or any other suitable method.

[0189] In step 927, the method may include using a trained machine learning model to compute a toxicity prediction. The machine learning model may include a trained MIMLL model and / or an aggregate model (as disclosed above). Exemplary outputs may include, but are not limited to, the following:

[0190] Toxicity Presence: The toxicity presence output can be used to determine the presence and / or severity of toxicity, regardless of the tissue type across the entire slide.

[0191] Organ toxicity prediction: Organ toxicity output can be used to determine in which organ toxicity may be found.

[0192] In step 929, the method may include saving the toxicity prediction to an electronic storage device.

[0193] Figure 10 An exemplary connected component algorithm according to an embodiment of this disclosure is illustrated. Connected component algorithms can aggregate features across image regions. For example, thresholding can produce binary (e.g., black and white) images. Connected component algorithms or models can identify various regions in an image, such as three regions at the pixel level (green, red, brown). In a particular implementation using connected components, each pixel can belong to a patch and a component (green, red, or brown). Aggregation can occur in various ways, including majority voting (e.g., all patches voting for the green component result in green having a value of 1) or a learned aggregator (e.g., where feature vectors can be extracted from each patch and input to a component aggregator module running for each component, so patches in the green component are fed into a component aggregator module that can produce hierarchical numbers). CNNs can output predictions (e.g., numbers) of patches, feature vectors describing the visual characteristics of patches, or both.

[0194] like Figure 11 As shown, device 1100 may include a central processing unit (CPU) 1120. CPU 1120 can be any type of processor device, including, for example, any type of dedicated or general-purpose microprocessor device. As those skilled in the art will appreciate, CPU 1120 may also be a single processor in a multi-core / multi-processor system (such a system operating independently), or in a cluster of computing devices operating in a cluster or server farm. CPU 1120 may be connected to data communication infrastructure 1110, such as a bus, message queue, network, or multi-core messaging scheme.

[0195] Device 1100 may also include main memory 1140, such as random access memory (RAM), and may also include auxiliary memory 1130. Auxiliary memory 1130, such as read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may include, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, or flash memory. The removable storage drive in this example reads from and / or writes to the removable storage unit in a well-known manner. Removable storage units may include floppy disks, magnetic tapes, optical disks, etc., which are read from and written to by the removable storage drive. As will be appreciated by those skilled in the art, such removable storage units typically include computer-usable storage media in which computer software and / or data are stored.

[0196] In an alternative implementation, auxiliary memory 1130 may include similar components for allowing computer programs or other instructions to be loaded into device 1100. Examples of such components may include program cassette tapes and cassette tape interfaces (such as those found in video game devices), removable storage chips (such as EPROMs or PROMs) and associated sockets, as well as other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 1100.

[0197] Device 1100 may also include a communication interface (“COM”) 1160. Communication interface 1160 allows software and data to be transferred between device 1100 and external devices. Communication interface 1160 may include a modem, network interface (such as an Ethernet card), communication port, or PCMCIA slot and card. Software and data transmitted via communication interface 1160 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by communication interface 1160. These signals may be provided to communication interface 1160 via communication paths of device 1100, which may be implemented using, for example, wires or cables, fiber optic cables, telephone lines, cellular telephone links, RF links, or other communication channels.

[0198] The hardware components, operating system, and programming language of such equipment are essentially conventional and are assumed to be sufficiently familiar to those skilled in the art. Device 1100 may also include input and output ports 1150 for connection to input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. Of course, various server functions can be implemented in a distributed manner on multiple similar platforms to distribute the workload. Alternatively, the server can be implemented through appropriate programming of a single computer hardware platform.

[0199] Throughout this disclosure, references to components or modules generally refer to items that can be logically combined to perform a function or a set of related functions. Similar reference numerals are generally intended to refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0200] The aforementioned tools, modules, and functions can be executed by one or more processors. "Storage" media can include any or all of the tangible memory of a computer or processor or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time.

[0201] Software can communicate via the Internet, cloud service providers, or other telecommunications networks. For example, communication enables the loading of software from one computer or processor onto another. As used herein, unless limited to non-transitory tangible "storage" media, terms such as "computer or machine-readable medium" refer to any medium involved in providing instructions to the processor for execution.

[0202] The foregoing general description is merely exemplary and explanatory, and not intended to limit this disclosure. Other embodiments of the invention will be apparent to those skilled in the art in light of the specification and practice of the invention disclosed herein. The specification and examples are considered exemplary only.

Claims

1. A computer-implemented method for processing electronic slide images corresponding to tissue specimens, the method comprising: Receive one or more electronic slide images associated with a tissue specimen, said tissue specimen being associated with a patient and / or medical case; The one or more electronic slide images are divided into multiple image blocks; Detect at least one tissue region from the background of one or more electronic slide images to generate a tissue mask; Determine whether any of the plurality of tiles corresponds to a non-organization; Remove any tile from the plurality of tiles that is determined to be non-organizational; A prediction is determined for at least one label of the one or more electron slide images, the prediction being determined by inputting the one or more electron slide images in which patches identified as disorganized have been removed into a machine learning prediction model, the machine learning prediction model being generated by processing a plurality of training images, the processing of the plurality of training images including: Receive a set of digital images associated with at least one training tissue specimen, wherein the set of digital images includes a plurality of training electronic slide images; Receive multiple summary annotations, the multiple summary annotations including one or more labels for each of multiple training electron slide images; One of the plurality of training electronic slide images is divided into multiple training patches of the plurality of training electronic slide images; At least one tissue region is segmented from the background of the one or more electron slide images to create a training tissue mask; Remove at least one of the multiple tiles detected as non-organized; and Train the machine learning prediction model under weak supervision to infer at least one multi-label tile-level prediction using at least one label from the plurality of summary annotations; and Output the predictions of the trained machine learning prediction model.

2. The computer-implemented method of claim 1, wherein the plurality of patches identified as non-organic are further identified as the background of the tissue specimen.

3. The computer-implemented method of claim 1, wherein detecting the plurality of tissue regions includes segmenting the tissue regions from the background of the one or more electron slide images.

4. The computer-implemented method according to claim 3, further comprising: When segmenting tissue regions from the background, a tissue mask is generated, and the segmentation is performed using thresholding based on color / intensity and / or texture features.

5. The computer-implemented method according to claim 1, wherein, The multiple training images include multiple electronic slide images and multiple target labels.

6. The computer-implemented method of claim 1, wherein training the machine learning prediction model under weak supervision comprises using at least one of multi-instance learning (MIL), multi-instance multi-label learning (MIMLL), self-supervised learning, and unsupervised clustering.

7. The computer-implemented method of claim 1, wherein processing the plurality of training images to generate the machine learning prediction model further comprises: Receives multiple predictions or multiple vectors for at least one feature from a weakly supervised tile-level learning module used for multiple training tiles; Train a machine learning model to take multiple predictions or multiple vectors from at least one feature from a weakly supervised tile-level learning module used for multiple training tiles as input. and The multiple training tiles are used to predict multiple labels on slides or patient specimens.

8. The computer-implemented method of claim 7, wherein at least one of the plurality of labels is binary, categorical, or sequential, or real-valued.

9. The computer-implemented method of claim 7, wherein the machine learning model is trained to take the plurality of predictions or the plurality of vectors of at least one feature from the weakly supervised tile-level learning module for the plurality of training tiles as input including the plurality of image features.

10. The computer-implemented method of claim 1, wherein the trained machine learning prediction model uses at least one unseen slice to predict at least one label.

11. A system for processing electronic slide images corresponding to tissue specimens, the system comprising: At least one memory for storing instructions; and At least one processor is configured to execute the instructions to perform an operation, the operation including: Receive one or more electronic slide images associated with a tissue specimen, said tissue specimen being associated with a patient and / or medical case; The one or more electronic slide images are divided into multiple image blocks; Detect at least one tissue region from the background of one or more electron slide images to generate a tissue mask; Determine whether any of the plurality of tiles corresponds to a non-organization; Remove any tile from the plurality of tiles that is determined to be non-organizational; A prediction is determined for at least one label of the one or more electron slide images, the prediction being determined by inputting the one or more electron slide images in which patches identified as disorganized have been removed into a machine learning prediction model, the machine learning prediction model being generated by processing a plurality of training images, the processing of the plurality of training images including: Receive a set of digital images associated with at least one training tissue specimen, wherein the set of digital images includes a plurality of training electronic slide images; Receive multiple summary annotations, the multiple summary annotations including one or more labels for each of multiple training electron slide images; One of the plurality of training electronic slide images is divided into multiple training patches of the plurality of training electronic slide images; At least one tissue region is segmented from the background of the one or more electron slide images to create a training tissue mask; Remove at least one of the multiple tiles detected as non-organized; and Train the machine learning prediction model under weak supervision to infer at least one multi-label tile-level prediction using at least one label from the plurality of summary annotations; and Output the predictions of the trained machine learning prediction model.

12. The system of claim 11, wherein the plurality of patches identified as non-tissue are further identified as the background of a tissue specimen.

13. The system of claim 11, wherein detecting the plurality of tissue regions includes segmenting the tissue regions from the background of the one or more electron slide images.

14. The system of claim 13, further comprising: When segmenting tissue regions from the background, a tissue mask is generated, and the segmentation is performed using thresholding based on color / intensity and / or texture features.

15. The system of claim 11, wherein the plurality of training images comprises a plurality of electronic slide images and a plurality of target labels.

16. The system of claim 11, wherein training the machine learning prediction model under weak supervision comprises using at least one of MIL, MIMLL, self-supervised learning, and unsupervised clustering.

17. The system of claim 11, wherein processing the plurality of training images to generate the machine learning prediction model further comprises: Receives multiple predictions or multiple vectors for at least one feature from a weakly supervised tile-level learning module used for multiple training tiles; Train a machine learning model to take multiple predictions or multiple vectors from at least one feature from a weakly supervised tile-level learning module used for multiple training tiles as input. and The multiple training tiles are used to predict multiple labels on slides or patient specimens.

18. A non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform a method for processing an electronic slide image corresponding to a tissue specimen, the method comprising: Receive one or more electronic slide images associated with a tissue specimen, said tissue specimen being associated with a patient and / or medical case; The one or more electronic slide images are divided into multiple image blocks; Detect at least one tissue region from the background of one or more electron slide images to generate a tissue mask; Determine whether any of the plurality of tiles corresponds to a non-organization; Remove any tile from the plurality of tiles that is determined to be non-organizational; A prediction is determined for at least one label of one or more electron slide images, the prediction being determined by inputting the one or more electron slide images in which patches identified as disorganized have been removed into a machine learning prediction model, the machine learning prediction model being generated by processing a plurality of training images, the processing of the plurality of training images including: Receive a set of digital images associated with at least one training tissue specimen, wherein the set of digital images includes a plurality of training electronic slide images; Receive multiple summary annotations, the multiple summary annotations including one or more labels for each of multiple training electron slide images; One of the plurality of training electronic slide images is divided into multiple training patches of the plurality of training electronic slide images; At least one tissue region is segmented from the background of the one or more electron slide images to create a training tissue mask; Remove at least one of the multiple tiles detected as non-organized; and Train the machine learning prediction model under weak supervision to infer at least one multi-label tile-level prediction using at least one label from the plurality of summary annotations; and Output the predictions of the trained machine learning prediction model.

Images