System and method for processing electronic images to identify abnormal morphology - Patents.com

Abstract

A system and method for identifying morphology present in a digital whole slide image includes receiving one or more digital whole slide images associated with a patient, determining a plurality of foreground tiles in the one or more digital whole slide images associated with the patient, determining whether each foreground tile of the plurality of foreground tiles contains a known morphology or an unknown morphology using a trained machine learning model, and upon determining that one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, where the clustering algorithm associates each of the one or more tiles with a cluster of unknown morphology, and predicting at least one outcome for the patient based on the associated cluster of unknown morphology.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 290,708, filed December 17, 2021, which is incorporated by reference in its entirety.

[0002] Various techniques of this disclosure relate generally to electronic image processing. More specifically, certain techniques of this disclosure relate to systems and methods for processing electronic images to determine rare or unknown morphology. [Background technology]

[0003] introduction Tissue types, whether typical or atypical, display characteristic histologic patterns overall. Pathologists studying these patterns have found some that occur with some frequency, especially in cancer histology, and some of these patterns are associated with clinical outcomes. Many patterns have already been established, and new morphological patterns have been discovered over time. New or rare morphological patterns are often only discovered when a pathologist happens to notice a pattern that occurs across a small fraction of patients. However, finding new patterns in this way is difficult, unstructured, inefficient, and dependent on a significant number of chances when a pathologist notices a pattern.

[0004] The background discussion provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the material described in this section is not, and is not admitted to be, prior art or suggestion of prior art to the claims in this application by inclusion in this section. Summary of the Invention [Means for solving the problem]

[0005] According to certain aspects of the disclosure, methods and systems for processing electronic images to identify morphology are disclosed. Each of the aspects disclosed herein may include one or more of the features described in connection with any of the other disclosed aspects.

[0006] A method of identifying morphology present in a digital whole slide image may be described, the method may include receiving one or more digital whole slide images associated with a patient, determining a plurality of foreground tiles in the one or more digital whole slide images associated with the patient, determining whether each foreground tile of the plurality of foreground tiles contains a known morphology or an unknown morphology using a trained machine learning model, and upon determining that one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, where the clustering algorithm associates each of the one or more tiles with a cluster of unknown morphology, and predicting at least one outcome for the patient based on the associated cluster of unknown morphology.

[0007] A system for identifying morphology present in a digital medical image may be described. The system may include a processor and a memory coupled to the processor having instructions stored thereon, which when executed by the processor, cause the processor to perform operations. The operations may include receiving one or more digital whole slide images associated with a patient, determining a plurality of foreground tiles in the one or more digital whole slide images associated with the patient, determining whether each foreground tile of the plurality of foreground tiles contains a known morphology or an unknown morphology using a machine learning model, and upon determining that the one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, where the clustering algorithm associates each of the one or more tiles with a cluster of unknown morphology, and predicting at least one outcome for the patient based on the associated cluster of unknown morphology.

[0008] A non-transitory computer-readable medium may be described that may store instructions that, when executed by a processor, cause the processor to perform operations to identify morphology present in a digital medical image. The operations may include receiving one or more digital whole slide images associated with a patient, determining a plurality of foreground tiles in the one or more digital whole slide images associated with the patient, using a machine learning model to determine whether each foreground tile of the plurality of foreground tiles contains a known morphology or an unknown morphology, and upon determining that one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, where the clustering algorithm associates each of the one or more tiles with a cluster of unknown morphology, and predicting at least one outcome for the patient based on the associated cluster of unknown morphology.

[0009] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed.

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary technologies and, together with the description, serve to explain the principles of the disclosed technology. [Brief description of the drawings]

[0011] [Figure 1A] 1 illustrates an exemplary system for processing electronic images to identify morphology in accordance with one or more techniques.

[0012] [Figure 1B] 1 illustrates an exemplary system for characterizing a digital whole slide image in accordance with one or more techniques.

[0013] [Figure 1C] 1 illustrates an exemplary system for analyzing digital whole slide images in accordance with one or more techniques.

[0014] [Diagram 2] 1 depicts a flowchart of an exemplary method for processing electronic images to identify morphology in accordance with one or more techniques.

[0015] [Diagram 3] 1 depicts a flowchart of an example method for training a machine learning system to determine known and unknown patterns in accordance with one or more techniques.

[0016] [Figure 4] 1 depicts a diagram of predicting outcome using rare and unknown morphology according to one or more techniques.

[0017] [Diagram 5]1 depicts a flowchart of an exemplary method for clustering digital medical images to predict normal and abnormal morphology using pathologist annotations and active learning, in accordance with one or more techniques.

[0018] [Figure 6] 1 depicts a flowchart of an example method for training a machine learning system to predict normal and abnormal morphology, in accordance with one or more techniques.

[0019] [Figure 7] 1 depicts a schematic of filtering abnormal foreground tiles for faster training of downstream genomics H&E method tasks, according to one or more techniques.

[0020] [Figure 8] 1 depicts a diagram of using cluster labels to train a classifier when label logic is constrained, according to one or more techniques.

[0021] [Figure 9] 1 depicts a flowchart of an example method for processing electronic images to identify morphology using supervised learning with strong annotations, in accordance with one or more techniques.

[0022] [Figure 10] 1 depicts a flowchart of an exemplary method for training a machine learning system using supervised learning and strong annotations, according to one or more embodiments.

[0023] [Figure 11] 1 depicts a simplified functional block diagram of a computer in accordance with one or more technologies. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Reference will now be made to the exemplary technology of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0025] The systems, devices, and methods disclosed herein are described in detail by way of example and with reference to the drawings. The examples discussed herein are merely examples and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be considered essential to any particular embodiment of any of those devices, systems, or methods, unless specifically designated as essential.

[0026] Also, with respect to any method described, whether or not the method is described with reference to a flowchart, it should be understood that any explicit or implicit order of steps performed in executing the method does not imply that the steps must be performed in the order presented, but rather, may be performed in a different order or in parallel, unless otherwise specified or required by context.

[0027] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Furthermore, the words "a" and "an" do not denote a limitation of quantity herein, but rather denote the presence of one or more of the referenced item.

[0028] As mentioned above, finding new tissue patterns is difficult, unstructured, inefficient, and relies on a significant number of chances for a pathologist to notice a pattern. This disclosure describes discovering rare or unknown patterns for review by a pathologist or by an artificial intelligence (AI) system, and correlating those patterns with patient data, e.g., patient outcomes. Thus, certain techniques described herein may help improve diagnosis, treatment, and / or clinical prognosis for disease.

[0029] In addition, it may be difficult to obtain data about rare cancers or know which rare cancer data may be needed to train a classifier. Therefore, certain techniques described herein may help identify rare cancers by identifying anomalies in the data (e.g., using whole slide images (WSI)). This identification may be performed by various unsupervised learning methods, as labels for normal and abnormal tissues may not be included in the dataset.

[0030] Furthermore, pathologists may only be able to annotate classifications for slides based on their training. Similarly, some artificial intelligence (AI) systems may generally only be trained by supervised learning to detect or classify known findings in clinical settings. Thus, an advantage of some techniques may be the identification of rare and unknown morphologies in a repository of histopathological data so that they can be reviewed by, for example, a pathologist or biologist for unknown or abnormal patterns and / or to gain better insight into the disease. Some techniques, such as clustering, may discover those unknown or abnormal patterns. Some techniques may also be used to accelerate supervised learning or for other applications.

[0031] One technique may use AI to classify each region of a slide based on abnormalities within each local region of the slide. Various exemplary techniques are described below.

[0032] FIG. 1A illustrates a block diagram of a system and network for processing electronic images to identify morphology according to an exemplary embodiment of the present disclosure. In particular, FIG. 1A illustrates an electronic network 120 that may be connected to servers in a hospital, laboratory, and / or doctor's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a laboratory server 124, and / or a laboratory information system (LIS) 125, etc., may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present disclosure, the electronic network 120 may also be connected to a server system 110, which may include a processing device 100 configured to implement a tissue characterization platform 101, which is configured to process the image to generate a plurality of foreground tiles. The server system 110 may also include an image analysis tool 102, as described below, which may be configured to use one or more machine learning systems to classify and / or cluster foreground tiles based on the extracted vectors of one or more features.

[0033] The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125 may also generate or otherwise acquire medical images of variable modalities. For example, digital medical images may be created or acquired including one or more patients' whole slide image(s), cytological specimen(s), histopathological specimen(s), slide(s) of cytological specimen(s), digitized images of slide(s) of histopathological specimen(s), or any combination thereof. Additionally or alternatively, images of other modality types may be created or acquired including magnetic resonance imaging (MRI), computed tomography (CT), x-ray, nuclear medicine imaging, or ultrasound. The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy information or cytology information, etc. The server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125 may transmit medical images and / or patient-specific information in digital or electronic format to the server system 110 over the electronic network 120.

[0034] Physician Server 121, Hospital Server 122, Clinical Trial Server 123, Laboratory Server 124, and / or LIS 125 refer to systems used to view medical images of variable modalities, including digital whole slide images. Medical images can be utilized by both medical professionals (e.g., pathologists, physicians, etc.) and such AI systems for training purposes to improve accuracy in detecting morphology, among other tasks. Higher availability of image data presenting a particular morphology enhances the ability of medical professionals and AI systems to learn given the increased variability in presentation in the image data. However, rare conditions or diseases often do not have a large amount of associated image data, which necessarily limits the morphology data that can be learned from. For example, the determination of rare morphology can be made difficult due to low levels of diagnosis and the resulting low amount of data (e.g., if doctors are not familiar with a particular rare disease, they may not collect the requisite data to analyze the morphology).

[0035] At least some of the medical images may include training images used to train an AI system and / or medical professional to diagnose a condition or predict a morphology. In some examples, some of the training images may be reserved and used as test images to evaluate the accuracy of the diagnostic system. Some of the medical images may present a context (e.g., in one or more foreground tiles with unknown morphology), while others of the medical images may include reference images that may not contain or present a context and / or cluster (e.g., in one or more foreground tiles in adipose tissue). Medical images stored for use as training images may be stored in association with labels indicating the data type of the medical image, including any morphology present, for use in training. Those labels may be sourced from the LIS 125. The server system 110 may also include a processing device to process the images and data stored in the storage device 109.

[0036] The server system 110 may include one or more storage devices 109 that store data, e.g., digital whole slide images and data received from at least one of the processing device 100, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125. In some examples, the storage device 109 may include one or more data stores for storing data, e.g., cloud-based storage, hard disks, and / or random access memory (RAM). As discussed below, the storage device 109 may store one or more trained machine learning systems, machine learning system outputs, and the like. For example, cluster assignments for multiple foreground tiles generated by the image analysis tool 102 may be stored in the storage device 109.

[0037] Server system 110 may include one or more machine learning tool(s) or capabilities. For example, processing device 100 may execute one or more machine learning systems utilized by image analysis tool 102, according to one embodiment. In some techniques, tissue characterization platform 101 may include image analysis tool 102. Alternatively or in addition, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing device (e.g., a laptop).

[0038] According to an exemplary embodiment of the present disclosure, a tissue characterization platform 101 may be implemented to generate multiple foreground tiles in a digital medical image. This implementation may also normalize the magnification across the multiple foreground tiles. FIG. 1B illustrates an exemplary block diagram of the tissue characterization platform 101. The tissue characterization platform 101 may include an image analysis tool 102, a data ingestion tool 103, a tile determination tool 104, and a viewing application tool 108.

[0039] Image analysis tools 102, described in more detail below, refer to processes and systems that identify and predict one or more attributes of a whole slide image. Machine learning may be used to predict clinical outcomes, such as survival rates, based on one or more morphologies, such as new or unknown morphologies, present in a whole slide image. According to an exemplary technique, machine learning may further be used to determine tissue-specific morphologies present in a digital whole slide image. According to another exemplary technique, image analysis tools 102 may also be used to cluster one or more foreground tiles based on anomaly scores to streamline tissue analysis.

[0040] The data ingestion tools 103, according to exemplary embodiments, may facilitate the transfer of the digital whole slide image to various tools, modules, components, and devices used to process the whole slide image. In some examples, if the digital whole slide image is adjusted utilizing one or more features of the tissue characterization platform 101, such as the image magnifier 105, only the adjusted digital whole slide image may be transferred. In other examples, both the original digital whole slide image and the adjusted digital whole slide image may be transferred. The data ingestion tools 103 may convert the digital whole slide image to one or more preferred formats, such as JPG or JPEG2000.

[0041] Digital whole slide images, such as those transferred by data ingestion tool 103, may be received by tile determination tool 104. Tile determination tool 104 may include image magnifier 105 and foreground module 106. The images may be analyzed using image magnifier 105, which may define a magnification level for extraction of foreground tiles. For example, multiple digital whole slide images may be normalized to a given magnification via image magnifier 105.

[0042] The foreground module 106 may generate a number of foreground tiles in each digital whole slide image. The foreground module 106 may generate a number of foreground tiles (e.g., regions in the digital whole slide image that have tissue) and / or background tiles (e.g., regions in the digital whole slide image that do not have tissue or have artifacts) using one or more methods described herein. The foreground module 106 may use an artifact detector system to detect tiles that have artifacts, e.g., tiles that contain blur, thick fragments, noise, etc. The background tiles, along with tiles determined to contain artifacts, may be removed from the digital whole slide image or marked as irrelevant for later processing. The foreground tiles may be any suitable shape, e.g., square, and may be stored, e.g., in storage device 109.

[0043] The tissue characterization platform 101 may also provide graphical user interface (GUI) control elements (e.g., a slide bar) for display along with the entire slide image through the user interface of the viewing application tool 108. The viewing application tool 108 may enable user-inputted annotation of slides for tissue type, known-unknown classification, and normal-abnormal classification, among other similar examples, as described in more detail below.

[0044] The viewing application tool 108 may also provide a user interface to a user (e.g., a pathologist) that displays the digital whole slide image throughout the various stages of adjustment. For example, the viewing application tool 108 may display the digital whole slide image unprocessed by the data ingestion tool 103 and / or foreground tiles output from the viewing application tool 108. The user interface may also include GUI control elements of the image analysis tool 102 that may be interacted with to annotate the whole slide image and / or foreground tiles, according to the exemplary techniques. Information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0045] 1C illustrates a block diagram of an image analysis tool 102 in accordance with an exemplary technique. The image analysis tool 102 may include a training image platform 131 and / or a target image platform 135. This embodiment may increase the amount of medical image data associated with rare presentations, e.g., rare cancers, that are available for training machine learning systems and / or medical professionals.

[0046] According to one technique, the training image platform 131 may include multiple software modules, such as a training data ingestion module 132, a training classifier module 133, and a training clustering module 134. According to one embodiment, the training image platform 131 may be configured to create or receive training images used to train one or more machine learning systems to identify unknown and / or abnormal morphologies. Exemplary machine learning systems are discussed in detail below. In some examples, the medical images may be one or more direct outputs of the machine learning systems. In other examples, one or more outputs of the machine learning systems may be used as inputs to further processing to enable identification of unknown and / or abnormal morphologies. The training images may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125. The images used for training may originate from real sources (e.g., humans, animals, etc.) or may originate from synthetic sources (e.g., graphics simulators, graphics rendering engines, 3D models, etc.). In other examples, a third party may train one or more of the machine learning systems and provide the trained machine learning system(s) to the server system 110 for storage (e.g., in storage 109) and / or execution by the image analysis tool 102.

[0047] The training data capture module 132 may create or receive one or more data sets including training images, known morphologies and / or patterns, normal and / or abnormal morphologies and / or patterns, clinical data, tile or slide annotations, etc. The training images may include digital whole slide images, such as digitized histology or cytology slides stained with various stains, including, but not limited to, hematoxylin and eosin, hematoxylin alone, toluidine blue, alcian blue, Giemsa, trichrome, acid-fast, Nissl stains, etc. The training images may be segmented into one or more foreground tiles, for example, by the tissue characterization platform 101. The known morphologies may include various shapes of cancer, normal tissue, etc. The clinical data may include overall survival data, progression-free survival with corresponding censored data, drug treatment outcome data, etc. The tile or slide annotations may include determinations by a medical professional, for example, a pathologist, regarding morphological patterns displayed within the slide. For example, a pathologist may review one or more foreground tiles to determine whether the tile contains unknown or abnormal morphology and / or patterns, and those determinations are annotated for that tile. In some examples, the subsets of the training dataset may overlap between or among various training images, known morphology and / or patterns, clinical data, and / or tile or slide annotations. The dataset may be stored in a digital storage device (e.g., one of the storage devices 109).

[0048] The training classifier module 133 may use at least the WSI and known tissue type datasets as inputs to generate one or more machine learning systems capable of predicting classifications for a plurality of foreground tiles, e.g., known or unknown morphologies and / or normal or abnormal morphologies. The training classifier module 133 may be configured to receive one or more training datasets, including, but not limited to, WSI, known morphologies, normal morphologies, tiles, and / or slide annotations. The training classifier module 133 may receive the training datasets from the training data ingestion module 132, the tissue characterization platform 101, the storage device 109, etc. As discussed in more detail below, the training classifier module 133 may use supervised training, semi-supervised training, or unsupervised training to generate machine learning systems configured to analyze the training datasets. For example, the training classifier module 133 may include open set classifiers (e.g., open set recognition methods), such as, but not limited to, Tempered Mixup, Open Data Inventory (ODIN), and / or One-Class Support Vector Machine (SVM). The training classifier module 133 may be configured to output a trained classifier and / or one or more outputs. The trained classifier may be stored in one or more storage systems, such as a storage system internal to the training image platform 131 or a storage system in the storage device 109.

[0049] The training clustering module 134 may use at least the dataset corresponding to the known and / or unknown vectors of features as input to generate one or more trained machine learning systems (e.g., clustering algorithms) capable of extracting vectors of features from foreground tiles and, for example, clustering the foreground tiles based on the extracted vectors. As discussed in more detail below, the training clustering module 134 may use supervised training, semi-supervised training, or unsupervised training to generate machine learning systems configured to analyze the training dataset. For example, the training clustering module 134 may include clustering algorithms such as, but not limited to, Mixture Model, K-Means, agglomerative clustering, and the like. In some examples, a clustering algorithm may be generated for each of the different tissue types to learn the corresponding unknown classifications. For example, a clustering algorithm may be trained for large intestine tissue morphology and another clustering algorithm may be trained for prostate tissue morphology. In other examples, one clustering algorithm may be generated that has the ability to learn unknown classifications for more than one tissue type. The trained clustering algorithm may be stored in one or more storage systems, for example, a storage system internal to the training image platform 131 or a storage system in the storage device 109 .

[0050] According to one technique, the target image platform 135 may include software modules such as a target data capture module 136, a classifier module 137, a clustering module 138, and an output interface 140. The target image platform 135 may receive a target dataset at the target data capture module 136. The target dataset may include, for example, digital WSI, foreground tiles, known morphological annotations, etc. The target dataset may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125.

[0051] The classifier module 137 may receive a trained classifier from, for example, the training classifier module 133 or the storage device 109, and / or may receive a target dataset from other aspects of the environments 100c, 100b, and 100a, such as from the storage device 109 and / or the target data ingestion module 136. As discussed herein, the classifier module 137 may include open set classifiers (e.g., open set recognition methods), such as, but not limited to, Tempered Mixup, Open Data Inventory (ODIN), and / or One-Class Support Vector Machine (SVM). The classifier module 137 may be configured to output foreground tiles classified into known categories (e.g., various known types of cancer, normal tissue, etc.) and / or unknown categories (e.g., unrecognized and atypical patterns). The classifier module 137 may add foreground tiles with an “unknown” classification (unknown foreground tiles) to a database of other foreground tiles with an “unknown” classification, e.g., the storage device 109. The classifier module 137 may discard foreground tiles with a “known” classification (known foreground tiles) or may store them in a database, for example, in the storage device 109.

[0052] The clustering module 138 may receive a trained clustering algorithm, for example, from the training cluster module 134 or the storage 109, and / or may receive a target dataset from other aspects of the environments 100c, 100b, and / or 100a, such as from the storage 109 and / or the classifier module 137. As discussed herein, the trained clustering algorithm may include, for example, Mixture Model, K-Means, agglomerative clustering, and / or the like. The clustering module 138 may be configured to receive classified foreground tiles, e.g., unknown foreground tiles, as input. The clustering module 138 may be configured to extract one or more vectors of features from the one or more unknown foreground tiles and / or to cluster the one or more foreground tiles based on the one or more extracted vectors of features. The vectors of features may encode information of the tiles, e.g., tiles that encompass one or more morphologies, and / or may describe the tiles, e.g., to map the tiles to regular morphological patterns. The clustering module 138 may be configured to output clustered foreground tiles based on the extracted vectors of features. The clustering module 138 may also be configured to correlate outcomes based on the clusters, as described in more detail below. The output cluster assignments may be annotated on the respective foreground tiles and / or stored in a database, e.g., storage device 109.

[0053] An output interface 140 may be used to output the foreground tile clusters (e.g., to a screen, monitor, storage device, web browser, etc.). A user, e.g., a pathologist, may interact with the output interface 140 to annotate the foreground tiles, e.g., to determine normal and abnormal regions.

[0054] According to one embodiment, the target image platform 135 may receive a request for identification of one or more unknown features and may execute one or more of the machine learning systems trained by the training image platform 131 to generate one or more clusters of medical images having unknown morphology. For example, the request may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the LIS 125. In another example, the request may be generated automatically by the image analysis tool 102 in response to detecting a number of medical images stored in the storage device 109 (e.g., a number of medical images determined to have unknown morphology).

[0055] FIG. 2 depicts a flow chart of an exemplary method of processing electronic images to identify unknown or rare morphologies according to one or more techniques. This embodiment may analyze a digital WSI to detect unknown or rare morphologies and / or predict outcomes. In step 202, one or more digital whole slide images associated with a patient may be received, for example, at tissue characterization platform 101. In step 204, a number of foreground tiles within one or more digital WSIs associated with the patient may be determined, for example, by foreground module 106. The one or more foreground tiles may be determined by any suitable means, for example, by using thresholding based on the variance of pixels and / or voxels within the tile, by using the Otsu method, thresholding based on minimizing within-class intensity variance, thresholding based on maximizing between-class intensity variance, and / or by comparing tile pixel and / or voxel values ​​to a reference foreground distribution.

[0056] In step 206, one or more foreground tiles may be classified based on known and unknown morphology, for example, by classifier module 137. Classifier module 137 may classify the foreground tiles by open set recognition methods (e.g., Tempered Mixup, Open Data Inventory (ODIN), and / or One-Class Support Vector Machine (SVM)), by supervised learning methods (e.g., CNN-based methods or logistic regression), and / or any other suitable means. Training classifier module 137 is described in more detail below. Classifier module 133 may annotate which foreground tiles contain known morphology or patterns and which foreground tiles contain unknown morphology or patterns. In some techniques, the foreground tiles may be labeled based on known patterns within annotated known pattern categories. For example, foreground tiles with known morphology may be annotated with the known morphology. In some techniques, digital WSI tiles with artifacts and / or irrelevant tissue regions may be discarded by a trained machine learning system, for example, using an artifact detector system or manual annotation. Artifacts may include blur, thick fragments, etc., and irrelevant tissue regions may include fat, background, etc.

[0057] In step 208, the unknown foreground tiles may be provided to a clustering algorithm, such as clustering module 138. The one or more unknown foreground tiles may be clustered based on at least one vector of features. As described herein, the at least one vector of features may be extracted by clustering module 138 by any suitable means, including, for example, manually designed features, pre-trained CNN embeddings using supervised learning, pre-trained CNN embeddings using self-supervised learning, and / or pre-trained transform neural network features. The manually designed features may include, for example, Scale-Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB), Rapid parameter Inference on gravitational wave sources via Iterative Fitting (RIFT), and / or speed Up Robust Features (SURF). Clustering may be performed using any suitable method, for example, Mixture Model, K-Means clustering, agglomerative clustering, and / or Expectation-Maximization Algorithm clustering.

[0058] In step 210, one or more outcomes may be predicted based on the unknown foreground tile clusters. For example, the predicted outcomes may include, but are not limited to, overall survival data, progression-free survival with corresponding censored data, drug treatment outcome data, data related to recurrence, presence or severity of disease, treatment success rate, etc. The system may determine that an unknown pattern is strongly correlated with certain patient metadata (e.g., an unknown pattern may be correlated with cancer recurrence). In other words, a technique may link a cluster of unknown morphological patterns to a clinical outcome. This may improve the accuracy of disease detection, classification, or treatment. In some techniques, the clustered foreground tiles and / or outcomes may be output, for example, to a GUI.

[0059] In some techniques, the system may then process slide images of actual tissue samples (e.g., from a laboratory). The system may then output information or scores based on the processing. For example, the information or scores may confirm or negate another morphological pattern analysis (e.g., Gleason score) and may indicate diagnostic accuracy or disease severity, and / or likelihood of treatment success or recurrence, etc., based on correlation between unknown patterns and patient data. In this way, some techniques described herein may identify rare morphologies without explicitly having information about whether they exist a priori. In addition, some techniques may correlate rare morphologies to outcome data. Furthermore, some techniques may facilitate the use of unknown morphological tissue regions for other clinically relevant tasks.

[0060] 3 depicts a flowchart of an exemplary method for training one or more machine learning systems of FIG. 2. In step 302, one or more digital whole slide images associated with a patient may be received, e.g., from the LIS 125. In step 304, known morphology data corresponding to the one or more digital whole slide images associated with the patient may be received. The known morphology data may include recorded morphologies of various types of cancer, normal tissue, etc. In step 306, a magnification level may be determined for each of the one or more digital WSIs, and the magnification level may be normalized, e.g., by the image magnifier 105.

[0061] In step 308, a number of foreground tiles in one or more digital WSIs associated with the patient may be determined. As discussed herein, the digital WSI may be divided, for example, into square tiles based on the foreground and background of the slide. Foreground tiles may be determined by thresholding based on the variance of pixels in the tile to identify whether they are foreground, by using the Otsu method, by comparing tile pixel values ​​to a reference foreground distribution, etc. Foreground tiles may be isolated for further analysis or may be stored (e.g., in storage 109). Background tiles and / or other irrelevant tiles, e.g., tiles with artifacts, may be discarded.

[0062] In step 310, a machine learning system may be trained to determine whether each foreground tile of the plurality of foreground tiles contains a known or unknown morphology. In some techniques, the model may be trained using supervised or semi-supervised training, although any suitable approach for training the model may be used. As described herein, the model may be trained using foreground tiles annotated based on known morphology present in the foreground tile and / or filtered to remove tiles having artifacts and / or extraneous tissue regions.

[0063] In some examples, a supervised machine learning system may be trained using strong annotations (e.g., digital WSI labeled with known morphology). In such examples, the supervised machine learning system may include a multimodal deep neural network, a graph neural network, a transformer neural network, a convolutional neural network (CNN), a multi-layer perceptron (MLP), a support vector machine (SVM), a nearest neighbor algorithmic model, or a random forest algorithmic model, among other similar examples. To enable learning, one or more digital WSIs, multiple foreground tiles, and / or known morphology associated with the digital WSIs and / or foreground tiles may be provided as input to the machine learning system. One or more vectors of features may be extracted from each foreground tile, and each foreground tile may be run using a range of techniques, such as manually designed features (e.g., scale-invariant feature transform (SIFT), oriented fast and rotated brief (ORB), radiation-invariant feature transform (RIFT), speeded up robust feature (SURF), etc., descriptors), pre-trained convolutional neural network (CNN) embeddings using supervised learning, pre-trained CNN embeddings using self-supervised learning techniques, pre-trained transformer neural network features, etc.

[0064] The machine learning system may output a foreground tile having a predicted known morphology and a predicted unknown morphology. The predicted unknown morphology may be compared to the corresponding annotated known morphology and / or unknown morphology to determine losses or errors. The corresponding known morphology may correspond to the foreground tile and may be part of a strong annotation of the training gene sequencing data indicative of known cancer tissue morphology in the foreground tile. The machine learning system may be corrected or altered based on the errors (e.g., weights and / or biases associated with one or more nodes and / or layers may be adjusted) to improve the accuracy of the machine learning system. This process may be repeated for each of the received training digital WSIs, or at least until the determined losses or errors are below a predefined threshold. In some examples, a portion of the training digital WSIs may be retained and used to further validate or test the machine learning system.

[0065] In other examples, a supervised machine learning system may be trained using Multiple Instance Learning (MIL) and weak annotations (e.g., labels at the tile or region level). For example, when MIL is used, the machine learning system receives a set of "bags," each of which includes multiple "instances." In particular, each of the training foreground tiles may be described as a "bag," and the extracted vectors of features from each training foreground tile may be the "instances" included in the "bag." Weak annotations may be associated with the "bag." For example, if at least one of the vectors of features included in the training foreground tile indicates a given unknown morphology, the training foreground tile may be labeled as positive for the unknown morphology. To learn, the machine learning system may identify at least one vector of features common across the training foreground tiles labeled as positive for the unknown morphology. Once trained, the machine learning model may be configured to generate an output for each of the one or more foreground tiles classified based on at least one unknown morphology.

[0066] Upon determining in step 312 that one or more foreground tiles contain unknown morphology, a clustering algorithm may be trained to cluster multiple vectors of features associated with the one or more unknown foreground tiles. Any suitable approach to training a model may be used, for example, unsupervised learning. Unsupervised learning may be configured to learn to group together similar vectors of features associated with the foreground tiles without the use of target labels. The vectors of features may be treated as instances, and the number of groupings may either be pre-defined or automatically learned by an algorithm. Such clustering algorithms may include, but are not limited to, Tempered Mixup, ODIN, OpenMax, One-class SVM, Expectation Maximization (EM), Majorization Maximization (MM), K-Nearest Neighbors (KNN), hierarchical clustering, and / or agglomerative clustering. The resulting trained clustering algorithm may be used by the clustering module 138 to cluster the foreground tiles based on the extracted vectors of features.

[0067] As discussed herein, patient outcomes can be predicted directly from digital WSI. In some embodiments, patient outcomes can be predicted from unknown morphologies using a binary model, such as that depicted in FIG. 4. This can circumvent existing tests that may attempt to predict outcomes solely from known morphologies, e.g., Gleason pattern, cribriform pattern, well-differentiated pattern. Environment 400 of FIG. 4 represents an exemplary diagram for determining a correlation between the presence of one or more unknown morphologies in at least one foreground tile and outcome data for each unknown morphology.

[0068] One or more foreground tiles 402 (derived from one or more digital whole slide images using one or more methods described herein) may be input to an open set classifier 401. Open set classification may provide a classifier, e.g., classifier module 137 and / or open set classifier 401, with the ability to classify data that does not fit within another classification into an unknown classification. In aspects where many similar looking rare morphologies of a particular tissue type are determined, the open set classifier 401 may be configured to determine correlations between rare morphologies and patient outcomes, e.g., 5-year survival rates. The open set classifier 401 may determine tissue regions predicted to have unknown patterns (step 403).

[0069] One or more known foreground tiles 404 may be discarded or stored, for example, in storage 109. One or more unknown foreground tiles 405 may be stored, for example, in storage 109, and / or clustered (step 407) into one or more clusters 406 based on one or more vectors of features. The one or more vectors of features may be extracted by one or more methods described herein, for example, by manually designed features and / or by pre-trained CNN embeddings. The one or more clusters of unknown foreground tiles 405 may be used to predict an outcome based on similarity between the one or more clusters of unknown foreground tiles 405 (step 408).

[0070] In some techniques, a machine learning system 409, e.g., a binary model CNN, may be configured to predict outcomes from one or more clusters of similar unknown morphologies determined in step 408. The outcomes of patients with assigned patterns may be correlated with each cluster to determine if any patterns occur, which may indicate that the pattern may be associated with a good or bad outcome compared to known morphologies. The machine learning system 409 may be trained using any of the methods described herein, e.g., semi-supervised training.

[0071] Methods for identifying morphology in a digital WSI may include techniques in addition to those described above. In some techniques, it may be beneficial to classify and / or cluster foreground tiles based on normal and abnormal morphology. These techniques may further improve the quality and speed of analysis of large numbers of foreground tiles. The following figures and diagrams illustrate some aspects of these example techniques.

[0072] 5 depicts a flowchart of a method for using a clustering method coupled with pathologist annotations and active learning according to one or more techniques. At step 502, one or more digital WSIs associated with a patient may be received. At step 504, a number of foreground tiles may be determined within one or more digital whole slide images associated with the patient, as discussed herein. For example, the foreground tiles may be determined using the Otsu method.

[0073] In step 506, the trained machine learning model may be used to determine whether each foreground tile of the plurality of foreground tiles contains a normal or abnormal morphology. The machine learning model may be trained using methods described in more detail below. The trained machine learning model may include manually designed features (e.g., SIFT, ORB, RIFT, SURF, etc., Descriptor), pre-trained CNN embeddings using supervised learning, pre-trained CNN embeddings using self-supervised learning, pre-trained Transform Neural Network features, etc. As described herein, the trained machine learning model may be configured to extract at least one vector of features from one or more foreground tiles and based on the at least one vector of features, predict whether the foreground tile contains a normal morphology (normal foreground tile) or whether the foreground tile contains an abnormal morphology (abnormal foreground tile). The normal foreground tiles may be discarded or stored, for example, in storage device 109. The abnormal foreground tiles may be stored or further analyzed.

[0074] Upon determining the one or more anomalous foreground tiles in step 508, the one or more anomalous foreground tiles may be provided to a clustering algorithm. The clustering algorithm may be trained using methods described in more detail below. The clustering algorithm may include any suitable algorithm, such as a Mixture Model, K-Means, EM algorithm approach, etc. The clustering algorithm may cluster the one or more anomalous foreground tiles based on one or more vectors of features extracted from the anomalous foreground tiles.

[0075] Based on the dysmorphology clusters and / or feature vectors, at least one outcome may be predicted in step 510. The outcome may include patient prognosis, number of years to live, likelihood of response to drugs, likelihood of recurrence, likelihood of metastasis, survival rate, types of effective drugs, types of effective treatments, 5-year survival rate, as discussed herein. The one or more predicted outcomes may be output to a GUI, e.g., output interface 140.

[0076]

[0023] Figure 6 depicts a flowchart of an example method for training one or more machine learning systems of Figure 5 in accordance with some techniques. At step 602, one or more digital whole slide images associated with a patient may be received. At step 604, normal morphology data and abnormal morphology data, each of which corresponds to one or more digital WSIs associated with the patient, may be received. The normal morphology data may include patterns that are common across data points, e.g., common across a population of digital whole slide images.

[0077] In step 606, a number of foreground tiles within one or more digital WSIs associated with the patient may be determined. A magnification level may be determined and normalized for each of the one or more digital WSIs, for example, by image magnifier 105, as discussed herein. The digital WSI may be segmented based on the foreground and background of the slide, as discussed herein. Foreground tiles may be determined using methods described herein, for example, by comparing tile pixel values ​​to a reference foreground distribution. The foreground tiles may be isolated for further analysis or stored (e.g., in storage 109). Background tiles and / or other irrelevant tiles, for example tiles containing artifacts, may be discarded.

[0078] In step 608, a clustering algorithm may be trained to cluster multiple vectors of features associated with one or more normal foreground tiles and / or one or more abnormal foreground tiles. Any suitable approach to training a model may be used, for example, unsupervised clustering. Unsupervised clustering may learn to group together similar vectors of features associated with foreground tiles without the use of target labels. The vectors of features may be treated as instances, and the number of groupings may either be pre-defined or automatically learned by an algorithm. Such clustering algorithms may include, but are not limited to, Tempered Mixup, ODIN, OpenMax, One-class SVM, Expectation Maximization (EM), Majorization Maximization (MM), K-Nearest Neighbors (KNN), hierarchical clustering, and / or agglomerative clustering. The resulting trained clustering algorithm may be used by the clustering module 138 to cluster the foreground tiles based on the extracted vectors of features into one or more clusters based on normal or abnormal morphology.

[0079] In step 610, a medical professional may review the output clusters. In some techniques, the foreground tile clusters may be visualized so that the tiles can be analyzed by, for example, a pathologist. Using their training, the pathologist may use their objective understanding of normal and abnormal morphology to review the clusters output by the clustering algorithm, for example, to review whether one or more abnormal clusters contain non-rare cancer information. The pathologist may annotate the contents of the clusters and feed that data to the clustering algorithm. Based on the pathologist's input, the clustering algorithm may adjust its labeling. Pathologist review may occur during training as well as during use of the trained machine learning system.

[0080] In step 612, a machine learning system may be trained to determine whether each foreground tile in a cluster of foreground tiles contains a normal or abnormal morphology. In some techniques, the machine learning system may be trained using a supervised training method as discussed herein, although any suitable approach to training the system may be used. The machine learning system may be trained using the foreground tiles that have been clustered based on one or more vectors of features and / or using pathologist annotations as described herein. The clusters output in step 610 may be input to the machine learning system, for example, as labels of normal or abnormal morphology.

[0081] In some examples, the supervised machine learning system may be trained using strong annotations (e.g., clustered foreground tiles annotated by a pathologist). In such examples, the supervised machine learning system may include a multimodal deep neural network, a logistic regression, a transform neural network, a convolutional neural network (CNN), a multi-layer perceptron (MLP), a support vector machine (SVM), a nearest neighbor algorithm model, or a random forest algorithm model, among other similar examples. To enable learning, one or more digital WSIs, a plurality of foreground tiles, one or more clusters of the foreground tiles, and / or normal morphologies associated with the digital WSIs, the foreground tiles, and / or the clusters may be provided as inputs to the machine learning system. The machine learning system may then output predicted abnormal morphologies within the foreground tiles and / or within the one or more clusters. The predicted abnormal morphologies may be compared to corresponding annotated normal morphologies and / or abnormal morphologies to determine losses or errors. The corresponding normal morphologies may correspond to the foreground tiles and may be part of the strong annotations of the training clusters indicative of normal tissue aspects of the foreground tiles. The machine learning system may be modified or altered based on the errors (e.g., weights and / or biases associated with one or more nodes and / or layers may be adjusted) to improve the accuracy of the system. This process may be repeated for each of the received training clusters, or at least until the determined loss or error falls below a predefined threshold. In some examples, some of the training clusters may be retained and used to further validate or test the machine learning system.

[0082] In another example, a supervised machine learning system may be trained using Multiple Instance Learning (MIL) and weak annotations (e.g., labels at the tile or region level). For example, when MIL is used, the machine learning system receives a set of "bags," each of which includes multiple "instances." In particular, each of the training foreground tiles may be described as a "bag," and the extracted vectors of features from each training foreground tile may be the "instances" included in the "bag." Weak annotations may be associated with the "bag." For example, if at least one of the vectors of features included in the training foreground tile indicates a given anomalous morphology, then the training cluster may be labeled as positive for the anomalous morphology. To learn, the machine learning system may identify at least one vector of features common across the training foreground tiles labeled as positive for the anomalous morphology. Once trained, the machine learning model may be configured to generate an output for each of the one or more foreground tiles classified based on at least one anomalous morphology. Steps 608, 610, and 612 may be repeated as necessary to increase the accuracy of the clustering algorithm, e.g., to reduce errors to a certain threshold.

[0083] In some aspects, the amount of data, e.g., digital WSI, to be analyzed by one or more systems may make the analysis process laborious and slow. For example, a single digital WSI may contain thousands of foreground tiles, each of which is annotated by a pathologist. In this example, annotation may take an inordinate amount of time, thereby slowing down training and compromising output data. However, if the foreground tiles are first clustered based on predicted abnormalities, as described in Figures 5 and 6, faster training times and better output may result. Figure 7 depicts an exemplary diagram of this process.

[0084] As depicted in the environment 700 of FIG. 7, one or more foreground tiles 702 (derived from one or more digital WSIs by the methods described herein) may be input to a CNN trained on a particular tissue type (tissue-specific CNN) 704. The tissue-specific CNN 704 may extract one or more embeddings 708, e.g., tissue-specific features, from the foreground tiles. The one or more embeddings 708 may be used to assign each foreground tile to a cluster (step 705). Each cluster may be an interpretation of the abnormality of a given region. The one or more clusters 706 may be reviewed by a medical professional, e.g., a pathologist (step 710). A pathologist review may be performed to assess the accuracy of the clusters. For example, the training data may not include labels for all normal and / or abnormal morphological patterns, and the pathologist may input information to the system that one or more of the normal and / or abnormal morphological patterns are in fact normal and / or abnormal morphological patterns. In step 712, tiles from the abnormal clusters may be used for downstream detection or classification tasks, for example to predict tissue-specific, rare morphology. In some techniques, a digital whole slide image may be output with normal foreground tiles removed or annotated on tiles 714. Some techniques may use the cluster assignments as labels to train a supervised model, and the process may be repeated.

[0085] For label generation, e.g., cluster labels, many reports from many sites may need to be reviewed by a pathologist to understand which features need to be identified and to standardize the data, so it may be difficult and time-consuming to generate labels for training a model for a particular tissue type. In some cases, the report itself may not contain all the information present in the slide, or different options may exist. This may result in the need for several iterations to build a sufficient system. A system that can detect abnormal tissue with limited supervision may facilitate faster iterations without label selection or label generation, which may facilitate faster turnaround for downstream biomarker tasks.

[0086] An exemplary diagram of using cluster labels to train a classifier when label logic is constrained is depicted in FIG. 8. As depicted in environment 800, one or more foreground tiles 802 (derived from one or more digital WSIs by methods described herein) may be input to a CNN, e.g., a self-supervised CNN 804. The self-supervised CNN 804 may extract one or more embeddings 808 from the foreground tiles that may be used to separate different abnormal tissues into clusters (step 806). One or more clusters 811 may be reviewed by a medical professional, e.g., a pathologist (step 812). A pathologist review may be performed to confirm abnormal and normal regions and / or morphologies. In step 813, tiles from abnormal clusters may be used for downstream detection or classification tasks, e.g., to predict tissue-specific, rare morphologies. In some techniques, algorithms may be used to remove or eliminate normal foreground tiles (see 814). In step 816, a semi-supervised or fully supervised CNN may be trained to predict regions of interest for different tissue types. The predicted regions of interest output from the tissue-specific CNN 818 may be filtered for downstream H&E-based biomarker tasks, step 820. The tissue-specific CNN 818 may be trained using any suitable method.

[0087] In some techniques, tissue type may be provided to the system as a strong annotation, which may increase the accuracy and sensitivity for detecting morphology of a given tissue type. FIG. 9 depicts a flowchart of an exemplary method using unsupervised learning with strong annotations, according to one or more techniques. In step 902, one or more whole slide images associated with a patient may be received. In some techniques, tissue types indicated in a laboratory information system or other database may also be received. In step 904, a number of foreground tiles within one or more digital WSIs associated with the patient may be determined using any suitable method as discussed herein.

[0088] In step 906, the trained machine learning system may determine whether each foreground tile of the plurality of foreground tiles contains a normal or abnormal morphology based on the binarized abnormality score. The presence of normal and / or abnormal morphology may be determined based on a tensor or vector of features. One or more tensors or vectors of features may be extracted using one or more techniques described herein, such as manually designed features, pre-trained CNN embeddings using supervised learning, pre-trained CNN embeddings using self-supervised learning, pre-trained transform neural network features, original unaltered pixel patches, etc. The trained machine learning system, such as the classifier module 137, may be trained using the techniques described in FIG. 10. The trained classifier may output an abnormality score based on the binarized abnormality threshold. In some aspects the trained classifier and / or the binarized output from the trained classifier may be stored, for example, in the storage device 109. In some techniques, the abnormality score may be used to filter the foreground tiles for downstream tasks, for example, to screen for rare cancer morphologies.

[0089] FIG. 10 illustrates a flowchart of an exemplary method of training the machine learning system described in FIG. 9. In step 1002, one or more digital WSIs associated with a patient may be received. The one or more digital WSIs may be annotated based on the type of tissue present at the location of the slide, and may be defined, for example, using a polygon, a pixel mask, or the like. In step 1004, normal morphology data and abnormal morphology data corresponding to one or more foreground tiles in the one or more digital WSIs may be received. In step 1006, a plurality of foreground tiles having one or more digital WSIs associated with the patient may be determined. The plurality of foreground tiles may be determined based on the annotations from step 1002. As described herein, any suitable method may be used to determine the foreground tiles, such as, but not limited to, thresholding based on the variance of pixels in the tiles to identify whether they are foreground.

[0090] In step 1008, the machine learning system may be trained to determine whether each foreground tile contains a normal or abnormal morphology based on the binarized anomaly score. The binarized anomaly score may be determined, for example, by classifying a tensor or vector of features extracted from the foreground tile using a classifier (such as classifier module 137). The classifier may be trained using supervised learning to determine an anomaly score threshold based on the tensor or vector of features. The anomaly score threshold may be used to determine whether the anomaly score may be truly abnormal, so that a binary classification can be performed on the tissue region. The trained classifier may be stored, for example, in storage device 109.

[0091] In some examples, a supervised machine learning system may be trained using strong annotations (e.g., a digital WSI annotated with the type(s) of tissue(s) present in the WSI). In such examples, the supervised machine learning system may include a convolutional neural network (CNN), a multi-layer perceptron (MLP), a support vector machine (SVM), a nearest neighbor algorithm model, or a random forest algorithm model, among other similar examples. To enable learning, one or more annotated digital WSIs, normal morphologies associated with the annotated digital WSIs, and / or abnormal morphologies associated with the annotated digital WSIs may be provided as inputs to the machine learning system. The machine learning system may then output a predicted anomaly score for each of the foreground tiles. The predicted anomaly score may be compared to the corresponding binarized normal and / or abnormal morphologies to determine loss or error. The corresponding normal morphology may correspond to the foreground tile and may be part of a strong annotation of the training annotated digital WSI that indicates normal tissue aspects of the foreground tile. The machine learning system may be modified or altered based on the errors (e.g., weights and / or biases associated with one or more nodes and / or layers may be adjusted) to improve the accuracy of the system. This process may be repeated for each of the received training annotated digital WSI, or at least until the determined loss or error falls below a predefined threshold. In some examples, a portion of the training annotated digital WSI may be retained and used to further validate or test the machine learning system.

[0092] FIG. 11 illustrates an example system or device 1100 that may perform the techniques presented herein. The device 1100 may include a central processing unit (CPU) 1120. The CPU 1120 may be any type of processor device, including, for example, any type of special purpose or general purpose microprocessor device. As will be appreciated by those skilled in the art, the CPU 1120 may also be a single processor in a multi-core / multi-processor system, in such a system operating alone or in a cluster of computing devices, in a cluster or in a server farm. The CPU 1120 may be connected to a data communications infrastructure 1110, for example, a bus, a message queue, a network, or a message passing scheme.

[0093] The device 1100 may also include a main memory 1140, e.g., a random access memory (RAM), and may also include a secondary memory 1130. The secondary memory 1130, e.g., a read-only memory (ROM), may be, e.g., a hard disk drive or a removable storage drive. Such a removable storage drive may include, e.g., a floppy disk drive, a magnetic tape drive, an optical disk drive, or a flash memory. The removable storage drive in this example reads from and / or writes to a removable storage unit in a well-known manner. The removable storage device may include a floppy disk, a magnetic tape, an optical disk, or the like, which is read from and written to by the removable storage drive. As will be appreciated by those skilled in the art, such a removable storage unit generally includes a computer usable storage medium having computer software and / or data stored thereon.

[0094] In alternative implementations, secondary memory 1130 may include similar means for allowing computer programs or other instructions to be loaded into device 1100. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from the removable storage units to device 1100.

[0095] Device 1100 may also include a communications interface ("COM") 1160. Communications interface 1160 allows software and data to be transferred between device 1100 and external devices. Communications interface 1160 may include a modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card, or the like. The software and data transferred via communications interface 1160 may be in the form of signals, which may be electrical, electromagnetic, optical, or other signals capable of being received by communications interface 1160. Those signals may be provided to communications interface 860 over a communications path of device 1100, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0096] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature and are presumed to be fully familiar to those skilled in the art. The device 1100 may also include input and output ports 1150 for connecting with input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, the various server functions may be implemented in a distributed fashion on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by suitable programming of one computer hardware platform.

[0097] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. The components and / or modules may be implemented in software, hardware, or a combination of software and / or hardware.

[0098] The tools, modules, and / or functions described above may be executed by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or associated modules such as various semiconductor memories, tape drives, disk drives, etc., which may provide non-transitory storage at any time for software programming.

[0099] The software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, the communication may enable loading of the software from one computer or processor to another. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0100] The foregoing general description is exemplary and explanatory only and is not intended to limit the present disclosure. Other embodiments of the present invention will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. It is intended that the specification and examples be considered as exemplary only.

Claims

1. 1. A method for identifying morphologies present in a digital whole slide image, comprising: receiving one or more digital whole slide images associated with a patient; determining a plurality of foreground tiles within the one or more digital whole slide images associated with a patient; determining whether each foreground tile of the plurality of foreground tiles contains a known feature or an unknown feature using a trained machine learning model; upon determining that one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, the clustering algorithm associating each of the one or more tiles with a cluster of unknown morphology; predicting at least one outcome for the patient based on the clusters of associated unknown morphologies; and The method comprising:

2. 2. The method of claim 1 , wherein the one or more foreground tiles are determined by thresholding based on pixel variance, by thresholding based on minimizing within-class intensity variance, by thresholding based on maximizing between-class intensity variance, and / or by comparing foreground tile pixel values ​​to a reference foreground distribution.

3. The method of claim 1 , wherein determining a number of foreground tiles in the one or more digital whole slide images associated with a patient further comprises normalizing the digital whole slide images for a magnification level.

4. The method of claim 1 , wherein whether each foreground tile of the plurality of foreground tiles contains a known feature or an unknown feature is determined using an open set classifier.

5. The method of claim 1 , wherein the clustering algorithm uses a Mixture Model, a K-Means Model, an Agglomerative Clustering, and / or an Expectation-Maximization Algorithm approach.

6. Providing the one or more foreground tiles to the clustering algorithm comprises: determining a vector of features for each foreground tile having unknown morphology, the clustering algorithm clustering a plurality of vectors associated with foreground tiles of unknown tissue morphology; The method of claim 1.

7. 7. The method of claim 6, wherein the clustering algorithm may extract the plurality of vectors using manually designed features, pre-trained Convolutional Neural Network (CNN) embeddings using supervised learning, pre-trained CNN embeddings using self-supervised learning, or pre-trained Transformer Neural Network features.

8. Predicting at least one outcome for the patient includes: receiving patient data associated with the cluster of unknown morphology; determining outcome data using the received patient data; and The method of claim 1 further comprising:

9. 2. The method of claim 1, wherein the at least one outcome comprises at least one of a patient prognosis, the patient prognosis comprising years of survival, likelihood of response to a drug, likelihood of recurrence, likelihood of metastasis, survival rate, types of effective drugs, types of effective treatment, and 5-year survival rate.

10. The method of claim 1 , wherein the at least one outcome is predicted using a binary model based on the presence of unknown tiles.

11. The method of claim 1 , further comprising visualizing the one or more foreground tiles that are assigned to a cluster to be analyzed by a medical professional.

12. 10. The method of claim 1, further comprising correlating a patient outcome with the one or more clusters to predict a prognosis based on the one or more clusters.

13. 1. A system for identifying morphologies present in a digital medical image, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, the instructions comprising: receiving one or more digital whole slide images associated with a patient; determining a plurality of foreground tiles within the one or more digital whole slide images associated with a patient; determining whether each foreground tile of the plurality of foreground tiles contains a known feature or an unknown feature using a trained machine learning model; upon determining that one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, the clustering algorithm associating each of the one or more tiles with a cluster of unknown morphology; predicting at least one outcome for the patient based on the clusters of associated unknown morphologies; and The system performs operations including:

14. 14. The system of claim 13, wherein the one or more foreground tiles are determined by thresholding based on pixel variance, by thresholding based on minimizing within-class intensity variance, by thresholding based on maximizing between-class intensity variance, and / or by comparing foreground tile pixel values ​​to a reference foreground distribution.

15. 14. The system of claim 13, wherein determining a number of foreground tiles in the one or more digital whole slide images associated with a patient further comprises normalizing the digital whole slide images for a magnification level.

16. Providing the one or more foreground tiles to the clustering algorithm comprises: determining a vector of features for each foreground tile having unknown morphology, the clustering algorithm clustering a plurality of vectors associated with foreground tiles of unknown tissue morphology; The system of claim 13.

17. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations to identify morphologies present in a digital medical image, the operations comprising: receiving one or more digital whole slide images associated with a patient; determining a plurality of foreground tiles within the one or more digital whole slide images associated with a patient; determining whether each foreground tile of the plurality of foreground tiles contains a known feature or an unknown feature using a machine learning model; upon determining that one or more foreground tiles contain an unknown morphology, providing the one or more foreground tiles having the unknown morphology to a clustering algorithm, the clustering algorithm associating each of the one or more tiles with a cluster of unknown morphology; predicting at least one outcome for the patient based on the clusters of associated unknown morphologies; and The non-transitory computer-readable medium.

18. 20. The non-transitory computer-readable medium of claim 17, wherein the one or more foreground tiles are determined by thresholding based on pixel variance, by thresholding based on minimizing within-class intensity variance, by thresholding based on maximizing between-class intensity variance, and / or by comparing foreground tile pixel values ​​to a reference foreground distribution.

19. 20. The non-transitory computer-readable medium of claim 17, wherein determining a number of foreground tiles in the one or more digital whole slide images associated with a patient further comprises normalizing the digital whole slide images for a magnification level.

20. Providing the one or more foreground tiles to the clustering algorithm comprises: determining a vector of features for each foreground tile having unknown morphology, the clustering algorithm clustering a plurality of vectors associated with foreground tiles of unknown tissue morphology; 20. The non-transitory computer-readable medium of claim 17.

Images