Systems and methods for processing slide images and inferring biomarkers - Patents.com

Abstract

Disclosed are systems and methods for receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine the expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present within the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; and outputting the determined expression level, category, and / or presence of the biomarker within the region of interest.

Description

[Technical field]

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 897,734, filed September 9, 2019, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE Various embodiments of the present disclosure relate generally to image-based prediction of biomarkers and related image processing methods. More specifically, certain embodiments of the present disclosure relate to systems and methods for predicting one or more biomarkers based on processing of images of tissue samples. [Background technology]

[0003] (background) Histological stains may be used in pathology to visualize cells. Many dye-based staining systems have been developed. However, the methods developed may not provide enough information for a pathologist to visually identify a biomarker that may aid in diagnosis or guide treatment. Techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH), or fluorescence in situ hybridization (FISH) may be used. If these methods cannot provide enough information to detect a biomarker, genetic testing of tissue may be used to confirm whether a biomarker is present (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a given gene in a cancer, etc.). IHC is more expensive than dyes such as hematoxylin and eosin (H&E). However, genetic testing is even more costly and may not be available in many clinics and hospitals.

[0004] There is a desire for a method of biomarker detection that can avoid costly IHC techniques and / or genetic testing. The disclosed embodiments can use artificial intelligence (AI) to predict biomarkers (e.g., overexpression of proteins and / or gene products, amplification and / or mutations of specific genes) from prominent regions in digital images of tissue stained using H&E and / or other dye-based methods.

[0005] The foregoing general description and the following detailed description are exemplary and explanatory only and are not limitations of the present disclosure. The background provided herein is generally for the purpose of providing a context for the present disclosure. Unless otherwise indicated herein, the matter described in this section is not, by inclusion in this section, made prior art to the claims of this application, nor is it an admission of prior art or an indication of prior art. Summary of the Invention [Problem to be solved by the invention]

[0006] (summary) According to one aspect of the present disclosure, a system and method for predicting one or more biomarkers from image analysis of a tissue sample is disclosed. [Means for solving the problem]

[0007] A method for analyzing an image corresponding to a sample includes receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine an expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present within the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; and outputting the determined expression level, category, and / or presence of the biomarker within the region of interest.

[0008] A system for analyzing an image corresponding to a sample includes at least one memory storing instructions; and at least one processor that executes the instructions and performs a process including receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine an expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present in the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; and outputting the determined expression level, category, and / or presence of the biomarker within the region of interest.

[0009] A non-transitory computer readable medium stores instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image corresponding to a sample, the method including: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine an expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present within the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; and outputting the determined expression level, category, and / or presence of the biomarker within the region of interest.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments as claimed. The present invention provides, for example, the following: (Item 1) 1. A computer-implemented method for analyzing an image corresponding to a sample, the method comprising: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine an expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present within the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; outputting the determined expression level, category, and / or presence of said biomarkers within said region of interest; 4. A computer-implemented method comprising: (Item 2) 2. The computer-implemented method of claim 1, wherein identifying an area of ​​interest in the target sample includes dividing the target electronic image into one or more sub-areas and determining whether an area of ​​interest is within any of the one or more sub-areas. (Item 3) Identifying a region of interest in the target sample comprises: Segmenting the target electronic image into one or more sub-regions using at least one of: creating tiles of the target electronic image; segmenting the target electronic image based on edges and / or contrast; segmenting the target electronic image using color difference; and / or supervised decision making; determining whether a region of interest is within any of the one or more sub-regions; 2. The computer-implemented method of claim 1, comprising: (Item 4) Dividing the target electronic image into one or more sub-regions by creating tiles of the target electronic image; determining an expression level, category, and / or presence of biomarkers within a region of interest by determining a first expression score for each tile of the target electronic image and aggregating the first expression scores for each tile to determine a second expression score for the portion of the target sample; 2. The computer-implemented method of claim 1, further comprising: (Item 5) 2. The computer-implemented method of claim 1, wherein determining the expression level, category, and / or presence of a biomarker in the region of interest comprises determining a probability that there is a presence of the biomarker in the region of interest. (Item 6) Determining the expression level, category, and / or presence of a biomarker in the region of interest includes: clustering expression levels into one or more diagnostic categories, said one or more diagnostic categories representing a scale of presence of said biomarkers within said region of interest; identifying a diagnostic category from among the one or more diagnostic categories and classifying the biomarkers within the region of interest; 2. The computer-implemented method of claim 1, comprising: (Item 7) 2. The computer-implemented method of claim 1, wherein outputting the determined expression level, category, and / or presence of the biomarker within the region of interest includes displaying a visual indicator regarding the expression level, category, and / or presence of the biomarker, and displaying one or more treatment options based on the expression level, category, and / or presence of the biomarker. (Item 8) 1. A system for analyzing an image corresponding to a sample, the system comprising: at least one memory storing instructions; At least one processor and Equipped with The at least one processor executes the instructions to: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine an expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present within the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; outputting the determined expression level, category, and / or presence of said biomarkers within said region of interest; The system is configured to perform operations including: (Item 9) 9. The system of claim 8, wherein identifying an area of ​​interest in the target sample includes dividing the target electronic image into one or more sub-areas and determining whether an area of ​​interest is within any of the one or more sub-areas. (Item 10) Identifying a region of interest in the target sample comprises: Segmenting the target electronic image into one or more sub-regions using at least one of: creating tiles of the target electronic image; segmenting the target electronic image based on edges and / or contrast; segmenting the target electronic image using color difference; and / or supervised decision making; determining whether a region of interest is within any of the one or more sub-regions; 9. The system according to item 8, comprising: (Item 11) Dividing the target electronic image into one or more sub-regions by creating tiles of the target electronic image; determining an expression level, category, and / or presence of biomarkers within a region of interest by determining a first expression score for each tile of the target electronic image and aggregating the first expression scores for each tile to determine a second expression score for the portion of the target sample; 9. The system of claim 8, further comprising: (Item 12) 9. The system of claim 8, wherein determining the expression level, category, and / or presence of a biomarker in the region of interest comprises determining a probability that there is a presence of the biomarker in the region of interest. (Item 13) Determining the expression level, category, and / or presence of a biomarker in the region of interest includes: clustering expression levels into one or more diagnostic categories, said one or more diagnostic categories representing a scale of presence of said biomarkers within said region of interest; identifying a diagnostic category from among the one or more diagnostic categories and classifying the biomarkers within the region of interest; 9. The system according to item 8, comprising: (Item 14) 9. The system of claim 8, wherein outputting the determined expression level, category, and / or presence of the biomarker in the region of interest includes displaying a visual indicator regarding the expression level, category, and / or presence of the biomarker, and displaying one or more treatment options based on the expression level, category, and / or presence of the biomarker. (Item 15) A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image corresponding to a sample, the method comprising: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to identify a region of interest in the target sample and determine an expression level, category, and / or presence of a biomarker within the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images and predicting whether a region of interest is present within the target electronic image, the training images comprising images of human tissue and / or algorithmically generated images; outputting the determined expression level, category, and / or presence of said biomarkers within said region of interest; 16. A non-transitory computer readable medium comprising: (Item 16) 20. The non-transitory computer-readable medium of claim 15, wherein identifying an area of ​​interest in the target sample comprises dividing the target electronic image into one or more sub-areas and determining whether an area of ​​interest is within any of the one or more sub-areas. (Item 17) Identifying a region of interest in the target sample comprises: Segmenting the target electronic image into one or more sub-regions using at least one of: creating tiles of the target electronic image; segmenting the target electronic image based on edges and / or contrast; segmenting the target electronic image using color difference; and / or supervised decision making; determining whether a region of interest is within any of the one or more sub-regions; Item 16. The non-transitory computer-readable medium of item 15, comprising: (Item 18) Dividing the target electronic image into one or more sub-regions by creating tiles of the target electronic image; determining an expression level, category, and / or presence of biomarkers within a region of interest by determining a first expression score for each tile of the target electronic image and aggregating the first expression scores for each tile to determine a second expression score for the portion of the target sample; Item 16. The non-transitory computer-readable medium of item 15, further comprising: (Item 19) 16. The non-transitory computer readable medium of claim 15, wherein determining the expression level, category, and / or presence of a biomarker in the region of interest comprises determining a probability that there is a presence of the biomarker in the region of interest. (Item 20) Determining the expression level, category, and / or presence of a biomarker in the region of interest includes: clustering expression levels into one or more diagnostic categories, said one or more diagnostic categories representing a scale of presence of said biomarkers within said region of interest; identifying a diagnostic category from among the one or more diagnostic categories and classifying the biomarkers within the region of interest; Item 16. The non-transitory computer-readable medium of item 15, comprising: [Brief description of the drawings]

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0012] [Figure 1A] FIG. 1A illustrates an example block diagram of a system and network for predicting one or more biomarkers in a digital pathology image, according to an example embodiment of the present disclosure.

[0013] [Figure 1B] FIG. 1B illustrates an example block diagram of a disease detection platform 100 according to an example embodiment of the present disclosure.

[0014] [Figure 1C] FIG. 1C illustrates an example block diagram of a biomarker tool 101 according to an example embodiment of the present disclosure.

[0015] [Figure 2A] 2A and 2B are flowcharts illustrating an example method for predicting one or more biomarkers in a digital pathology image using machine learning, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 2B] 2A and 2B are flowcharts illustrating an example method for predicting one or more biomarkers in a digital pathology image using machine learning, in accordance with one or more exemplary embodiments of the present disclosure.

[0016] [Figure 3A] FIG. 3A is a flow chart illustrating an exemplary method for predicting estrogen receptor (ER), progesterone receptor (PR), and / or human epidermal growth factor receptor 2 (HER2, which may be known as ERBB2) biomarkers, according to an exemplary embodiment of the present disclosure.

[0017] [Figure 3B] FIG. 3B illustrates an exemplary system and method for training to discriminate between HER2, ER, and PR overexpression levels, according to an exemplary embodiment of the present disclosure.

[0018] [Figure 4A] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4B]4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4C] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4D] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4E] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4F] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4G] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4H] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4I] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4J] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4K]4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4L] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4M] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4N] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4O] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4P] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4Q] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4R] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4S] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4T]4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure. [Figure 4U] 4A-4U illustrate an example system and method for predicting HER2 biomarkers by analyzing H&E images, according to one or more example embodiments of the present disclosure.

[0019] [Diagram 5] FIG. 5 is a flow chart illustrating an exemplary method for predicting epidermal growth factor receptor (EGFR) biomarkers, according to an exemplary embodiment of the present disclosure.

[0020] [Figure 6] FIG. 6 is a flow chart illustrating an exemplary method for predicting DNA mismatch repair (MMR) deficiency biomarkers, according to an exemplary embodiment of the present disclosure.

[0021] [Figure 7] FIG. 7 is a flow chart illustrating an exemplary method for predicting APOBEC mutagenesis biomarkers, according to an exemplary embodiment of the present disclosure.

[0022] [Figure 8] FIG. 8 is a flow chart illustrating an exemplary method for predicting DNA polymerase ε (POLE) mutagenesis biomarkers, according to an exemplary embodiment of the present disclosure.

[0023] [Figure 9] FIG. 9 is a flow chart illustrating an exemplary method for predicting CDK4 / 6 inhibitor biomarkers, according to an exemplary embodiment of the present disclosure.

[0024] [Figure 10]FIG. 10 is a flow chart illustrating an exemplary method for predicting neurotrophin receptor tyrosine kinase (NTRK) aberrant biomarkers, according to an exemplary embodiment of the present disclosure.

[0025] [Figure 11] FIG. 11 is a flow chart illustrating an exemplary method for predicting immune response biomarkers, according to an exemplary embodiment of the present disclosure.

[0026] [Figure 12] FIG. 12 is a flow chart illustrating an exemplary method for predicting homologous recombination DNA repair deficiency (HRD) biomarkers, according to an exemplary embodiment of the present disclosure.

[0027] [Figure 13] FIG. 13 is a flow chart illustrating an exemplary method for predicting SWItch / Sucrose Non-Fermentable (SWI / SNF) biomarkers according to an exemplary embodiment of the present disclosure.

[0028] [Figure 14] FIG. 14 depicts an example system that can implement the techniques presented herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] (Description of the embodiment) Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0030] The systems, devices, and methods disclosed herein are described in detail, by way of example, with reference to the figures. The examples discussed herein are examples only, and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be taken as essential for any particular implementation of any of these devices, systems, or methods, unless specifically designated as essential.

[0031] Also, with respect to any method described, whether or not the method is described in conjunction with a flow diagram, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in the execution of the method does not imply that the steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

[0032] As used herein, the term "exemplary" is used in the sense of "an example" rather than "ideal." Moreover, the terms "a" and "an" do not denote a limitation of quantity herein, but rather denote the presence of one or more of the referenced item.

[0033] Pathology refers to the study of disease. More specifically, pathology refers to performing tests and analyses used to diagnose disease. For example, tissue samples may be placed on slides to be viewed under a microscope by a pathologist (e.g., a medical doctor who is an expert who analyzes tissue samples and determines if any abnormalities are present). That is, pathology samples may be cut into multiple pieces, stained, and prepared as slides for the pathologist to examine and give a diagnosis. When the diagnostic findings on the slide are uncertain, the pathologist may order additional cutting levels, stains, or other tests to gather more information from the tissue. A technician may then create a new slide, which may contain additional information for the pathologist to use in making a diagnosis. This process of creating additional slides may be time consuming, not only because it may involve retrieving a block of tissue, cutting it, making new slides, and then staining the slides, but also because it may be batched for multiple orders. This may significantly delay the final diagnosis that the pathologist gives. Additionally, even after a delay, there may still be no assurance that the new slide will have sufficient information to render a diagnosis.

[0034] Pathologists may independently evaluate cancer and other disease pathology slides. Workflows may integrate, for example, slide evaluation, tasks, image analysis, and cancer detection artificial intelligence (AI), annotation, consultation, and recommendations in one workstation.

[0035] For example, a computer may be used to analyze images of tissue samples and quickly identify whether additional information about a particular tissue sample may be needed and / or highlight areas that may possibly be examined more closely by a pathologist. Thus, the process of acquiring additional stained slides and tests may be done automatically before being reviewed by a pathologist. When paired with an automated slide sectioning and staining machine, this may provide a fully automated slide preparation pipeline. This automation has the advantages of at least (1) minimizing the amount of time wasted by a pathologist determining that a slide is insufficient to make a diagnosis; (2) minimizing the (average total) time from specimen acquisition to diagnosis by avoiding additional time between the time additional tests are ordered and the time they are produced; (3) reducing the amount of time and amount of material wasted per re-cut by allowing re-cutting to be performed while the tissue block (e.g., pathology sample) is in the cutting desk; (4) reducing the amount of tissue material wasted / discarded during slide preparation; (5) reducing the cost of slide preparation by partially or fully automating the procedure; (6) enabling automated customized cutting and staining of slides that may result in more representative / informative slides from the sample; (7) allowing a larger number of slides to be generated per tissue block by reducing the overhead of requiring additional tests on the pathologist, contributing to a more informative / precise diagnosis; and / or (8) identifying or collating the correct nature (e.g., with respect to sample type) of a digital pathology image, etc.

[0036] The process of using computers to assist pathologists is known as computational pathology. Computing methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but are not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. By using computational pathology, lives may be saved by helping pathologists improve their diagnostic accuracy, reliability, efficiency, and accessibility. For example, computational pathology may be used to help detect slides that are suspicious for cancer, thereby allowing pathologists to check and confirm their initial assessment before giving a final diagnosis.

[0037] Histopathology refers to the study of a sample that is placed on a slide. For example, a digital pathology image may consist of a digitized image of a microscope slide containing a sample (e.g., a smear). One method a pathologist may use is to analyze the image on the slide to identify the nuclei and classify whether the nuclei are normal (e.g., benign) or abnormal (e.g., malignant). To assist the pathologist in identifying and classifying the nuclei, histological stains may be used to visualize the cells. Dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method in which hematoxylin stains cell nuclei blue, eosin stains cytoplasm and extracellular matrix pink, and other tissue regions take on variations of these colors. IHC and immunofluorescence, for example, involve the use of antibodies that bind to specific antigens in tissues and allow visual detection of cells expressing specific proteins of interest, which may reveal biomarkers that are not reliably identifiable to a trained pathologist based on the analysis of H&E stained slides. ISH and FISH may be employed to assess the number of copies of a gene or the abundance of a specific RNA molecule, depending on the type of probe employed (e.g., DNA probes for gene copy number and RNA probes for assessment of RNA expression).

[0038] Digitized images may be prepared to represent stained microscope slides, which may allow a pathologist to manually view the images on the slides and estimate the number of stained abnormal cells within the images. However, this process may be time consuming and may lead to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computer processes and devices may be used to assist the pathologist in detecting abnormalities that may otherwise be difficult to detect.

[0039] Only the detected biomarkers and / or images can then be used to recommend a specific cancer drug or drug combination therapy to be used to treat the patient, where the AI ​​can identify drugs and / or drug combinations that are less likely to be successful by correlating the detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendation of immunotherapy drugs to target the patient's specific cancer. Furthermore, this can be used to enable personalized cancer treatment for specific subsets of patients and / or rarer cancer types.

[0040] As explained above, the present disclosure can use AI to predict biomarkers (e.g., overexpression of proteins and / or gene products, amplification or mutation of specific genes) from prominent regions in digital images of tissue stained using H&E and other dye-based methods. The images of tissue can be whole slide images (WSIs), images of tissue cores in microarrays, and / or selected areas of interest in tissue sections. Using staining methods such as H&E, biomarkers can be difficult to visually detect or quantify without additional testing. Using AI to infer these biomarkers from digital images of tissue can improve patient treatment while making it faster and less expensive.

[0041] The AI ​​of the present disclosure may simultaneously infer one or more biomarkers from the same digital image of a pathology sample with H&E stained histological sections (e.g., whole tissue sections, microarray cores, and / or areas of interest within a tissue preparation). For example, given an H&E stained whole slide digital image of a breast cancer sample, the AI ​​of the present disclosure may infer the sample's HER2 status, ER status, PR status, inflammatory infiltrate (and its composition), and resistance or response to specific therapies, such as hormonal therapy, anti-HER2 agents, CDK4 / 6 inhibitors, immune checkpoint inhibitors, and chimeric antigen receptor T (CART-T) cell-based therapy. This may mean that a comprehensive battery of tests using IHC and other techniques may be avoided, since biomarkers may be inferred from H&E images alone. The detected biomarkers or images alone can then be used to recommend a specific breast cancer drug or drug combination therapy to be used to treat the patient, and the AI ​​can identify drugs or drug combinations that are less likely to be successful by correlating the detected biomarkers with a database of treatment options. This may be used to facilitate automatic recommendation of immunotherapy drugs to target the patient's specific cancer. The methods described above may be useful to enable personalized cancer treatment for specific subsets of patients and / or rarer cancer types.

[0042] The present exemplary embodiment may include salient region detection to identify regions of an image for which biomarkers may be identified. For example, biomarkers of diagnostic relevance may be inferred from cancerous tissue, while other tissues may be less relevant to the identification of the biomarker. Salient region detection may allow for better sample complexity such that machine learning models and / or systems may be effectively trained to distinguish biomarkers of interest from relevant tissues (e.g., cancerous tissues) with less relevant tissues excluded from the analysis.

[0043] According to one or more exemplary embodiments, biomarker detection may be inexpensive since the biomarkers may be detected using only H&E, thus allowing the biomarkers to be detected in a reproducible and conclusive manner. When scoring IHC, immunofluorescence, ISH, and FISH, variability may exist between pathologists, which may undermine both treatment recommendations and / or drug research.

[0044] FIG. 1A illustrates a block diagram of a system and network for predicting one or more biomarkers in digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure.

[0045] 1A illustrates an electronic network 120 that may be connected to servers in a hospital, laboratory, and / or doctor's office, etc. For example, a doctor server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, etc. may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present application, the electronic network 120 may also be connected to a server system 110, which may include a processing device configured to implement a disease detection platform 100, according to an exemplary embodiment of the present disclosure, which includes a biomarker tool 101 for predicting one or more biomarkers in a digital pathology image using machine learning.

[0046] The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may create or otherwise obtain images of one or more patient cytology samples, histopathology samples, slides of cytology samples, digitized images of histopathology sample slides, or any combination thereof. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsy or cytology information, etc. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may transmit the digitized slide images and / or patient-specific information to the server system 110 via the electronic network 120. The server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The server system 110 may also include a processing device for processing the images and data stored in the storage device 109. The server system 110 may further include one or more machine learning tools or capabilities. For example, the processing device may include machine learning tools for the disease detection platform 100 according to one embodiment. Alternatively or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0047] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review images of slides.

[0048] FIG. 1B illustrates an example block diagram of a disease detection platform 100 for predicting one or more biomarkers in digital pathology images using machine learning.

[0049] 1B depicts components of a disease detection platform 100, according to one embodiment. For example, the disease detection platform 100 may include a biomarker tool 101, a data collection tool 102, a salient region detection tool 103, a biomarker prediction tool 104, a storage device 106, a viewing application tool 108, a slide capture tool 111, a slide scanner 112, and / or a slide manager 113.

[0050] The biomarker tool 101 as described below refers to a process and system for predicting one or more biomarkers in digital pathology images using machine learning, according to an example embodiment.

[0051] Data collection tools 102 refer to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and devices used to predict one or more biomarkers in the pathology images, according to an exemplary embodiment.

[0052] The salient region detection tool 103 may identify salient regions of one or more digital images to be analyzed. This detection may be performed manually by a human or automatically using AI. The entire image or specific image regions may be considered salient. Image regions that are salient for biomarker detection, e.g., regions with tumors, may occupy a fraction of the entire image. Regions of interest may be defined by human experts using image segmentation masks, bounding boxes, or polygons. Alternatively, or in addition, AI may provide a complete end-to-end solution in identifying locations. Salient region identification may enable downstream AI systems to learn how to detect biomarkers from less annotated data and make more accurate predictions. Exemplary embodiments may include (1) strongly supervised methods that precisely identify where biomarkers may be found, and / or (2) weakly supervised methods that may not provide precise locations. During AI training, a strongly supervised system may receive as input images and locations of salient regions that may potentially represent biomarkers. These locations may be defined using pixel-level landmarks, bounding box-based landmarks, polygon-based landmarks, and / or using corresponding images in which saliency has been identified (e.g., using IHC). A weakly supervised system may receive as input an image or multiple images and the presence / absence of salient regions. The exact locations of salient locations in one or more images may not be defined when training a weakly supervised system.

[0053] The biomarker prediction tool 104 may use machine learning and / or computer vision to predict and / or infer the presence of biomarkers. The predictions may be output to an electronic storage device. A notification or visual indicator may be sent / displayed to the user to alert the user to the presence or absence of one or more of the biomarkers.

[0054] Slide capture tool 111 refers to a process and system for scanning pathology images and converting them into digital form, according to an exemplary embodiment. Slides may be scanned using slide scanner 112, and slide manager 113 may process the images on the slides into digitized pathology images and store the digitized images in storage device 106.

[0055] The viewing application tool 108 refers to processes and systems for providing a user (e.g., a pathologist) with sample property or image property information related to a digital pathology image, according to an example embodiment. The information may be provided through a variety of output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0056] The biomarker tool 101 and its components may each transmit and / or receive digitized slide images and / or patient information to and / or from the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 via the network 120. Additionally, the server system 110 may include a storage device for storing images and data received from at least one of the biomarker tool 101, the data collection tool 102, the slide capture tool 111, the slide scanner 112, the slide manager 113, and the viewing application tool 108. The server system 110 may also include a processing device for processing the images and data stored in the storage device. The server system 110 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0057] Any of the above devices, tools, and modules may be located on devices that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0058] 1C illustrates an example block diagram of a biomarker tool 101, according to an example embodiment of the present disclosure. The biomarker tool 101 may include a data collection tool 102, a salient region detection tool 103, and / or a biomarker prediction tool 104.

[0059] The salient region detection tool 103 may include a training image capture module 132 , a salient region identifier module 133 , a target image capture module 134 , and / or a salient region prediction module 135 .

[0060] The training image capture module 132 may receive one or more digital images of a pathology sample (e.g., histology, cytology, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and may receive an indication of the presence or absence of a salient region (e.g., disease present at a location within the image) for the one or more images. For example, the training image capture module 132 may divide the one or more digital images into sub-regions. The one or more sub-regions may be determined for saliency. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0061] The salient region identifier module 133 may train a machine learning algorithm that takes as input a digital image of a pathology sample and predicts whether a salient region is present, including, but not limited to, (1) weak supervision (using multiple instance learning (MIL) using weak labels of a digital image or collection of images to train a machine learning system (e.g., multi-layer perceptron (MLP), convolutional neural network (CNN), graph neural network, support vector machine (SVM), random forest, etc.), where the labels may correspond to the presence or absence of a salient region, which may represent an associated biomarker); (2) bounding box or polygon based supervision (using bounding boxes or polygons to define sub-regions of the digital image that are salient for detection of the presence or absence of a biomarker, to train a machine learning system (e.g., region-based CNN (R-CNN), Many methods may be used to learn the salient regions, including (3) pixel-level labeling (e.g., semantic or instance parcellation) (the pixel-level labeling is used to train a machine learning system (e.g., Mask R-CNN, U-NET, fully convolutional neural network) where individual pixels are identified as being salient for detection of biomarkers), and / or (4) the use of a corresponding but different digital image that identifies salient tissue regions (a digital image of the tissue that highlights salient regions (e.g., cancer identified using IHC) may be aligned with the input digital image). For example, a digital image of an H&E image may be aligned / matched with an IHC image that identifies salient tissue (e.g., cancerous tissue where biomarkers should be found), and the IHC may be used to determine salient pixels based on image color characteristics.

[0062] The target image capture module 134 may receive one or more digital images of a pathology sample (e.g., histology, cytology, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). The one or more digital images may be divided into sub-regions and the saliency of the one or more sub-regions (e.g., cancerous tissue for which biomarkers are to be identified) may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0063] The salient region prediction module 135 may apply trained machine learning algorithms to the images / sub-regions to predict regions of the image that are salient and may potentially exhibit biomarkers of interest (e.g., cancerous tissue). If salient regions exist, they are identified and flagged as such. Salient regions may be detected using a variety of methods, including, but not limited to, (1) running a machine learning system on the image sub-regions to generate predictions for one or more sub-regions, and / or (2) using machine learning visualization tools, e.g., class activation maps, GradCAM, etc., to create detailed heat maps by then extracting relevant regions.

[0064] The biomarker prediction tool 104 may include a training image capture module 136 , a salient region identifier module 137 , a target image capture module 138 , an expression level prediction module 139 , and / or an output interface 140 .

[0065] The training image capture module 136 may receive one or more digital images of a pathology sample (e.g., histology, cytology, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and may receive the level (e.g., binary or ordinal value) of a biomarker present for the one or more images. For example, the one or more digital images may be divided into sub-regions. The one or more sub-regions may be determined for their saliency. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0066] The salient region identifier module 137 may use an AI-based system and / or manual annotations from experts to identify salient regions that may be associated with a biomarker of interest. Machine learning algorithms may be trained to predict expression levels of one or more biomarkers from (salient) image regions. Expression levels may be represented as binary numbers, ordinal numbers, real numbers, etc. The techniques presented herein may be implemented in multiple ways, including, but not limited to, CNNs, MIL-trained CNNs, recurrent neural networks (RNNs), long short-term memory RNNs (LSTMs), gated recurrent units RNNs (GRUs), graph convolutional networks, support vector machines, and / or random forests.

[0067] The targeted image capture module 138 may receive one or more digital images of a pathology sample (e.g., histology, cytology, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and receive locations of salient areas, which may be automatically identified using AI and / or manually defined by an expert.

[0068] The expression level prediction module 139 may apply machine learning algorithms to provide a prediction of whether a biomarker is present.

[0069] The output interface 140 may output a prediction of whether a biomarker is present to an electronic storage device. For example, the output interface 140 may display a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) to the presence of a biomarker.

[0070] 2A is a flow chart illustrating an exemplary method of a tool for predicting one or more biomarkers, according to an exemplary embodiment of the present disclosure. For example, the exemplary method 200 (e.g., steps 202-208) may be performed by the biomarker tool 101 automatically or in response to a request from a user (e.g., a physician, pathologist, technician, etc.).

[0071] According to one embodiment, an exemplary method 200 for predicting biomarkers may include one or more of the following steps: In step 202, the method may include receiving a target electronic image of a slide corresponding to a target sample, the target sample comprising a patient tissue sample. For example, the target electronic image may be received from any one or any combination of server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125.

[0072] In step 204, the method may include applying a machine learning system to the target electronic image to identify regions of interest (e.g., salient regions) of the target sample. In step 206, the method may include applying a machine learning system to the target electronic image to determine expression levels of biomarkers in the regions of interest. In step 208, the method may include outputting the determined expression levels of the biomarkers.

[0073] The machine learning system may be generated by processing multiple training images and predicting whether a region of interest is present in a target electronic image, where the training images may include images of human tissue and / or algorithmically generated images. The machine learning system may be implemented using machine learning methods for classification and regression. The training inputs may include real or synthetic images. The training inputs may or may not be augmented (e.g., adding noise or creating variants of the inputs by inversion / distortion). Exemplary machine learning systems may include, without limitation, any one or any combination of neural networks, convolutional neural networks, random forests, logistic regression, and nearest neighbor methods. Convolutional neural networks can directly learn the image feature representations required to discriminate characteristics, which can work very well when there is a large amount of data to train on per sample, while other methods can be used in conjunction with either traditional computer vision features, such as scale invariant feature transform (SIFT) and / or speed up robust features (SURF), and / or learned embeddings (e.g., descriptors) generated by trained convolutional neural networks, which can provide advantages when there is only a small amount of data to train on. Training images may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. This data set may be kept on a digital storage device. Images used for training may come from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as (but not limited to) H&E, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices such as microCT.

[0074] As illustrated in FIG. 2B, according to one embodiment, the exemplary methods 220 and 240 for predicting biomarkers may include one or more of the following steps. In step 222, during a training phase, the method may include a data collection process, which may include receiving one or more digital images of a pathology sample (e.g., histology, cytology, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). To train the machine learning system, the one or more images may be paired with information about the biomarkers (e.g., from genetic testing, from IHC results analyzed by a pathologist, clinician annotations, etc.). The one or more digital images may be divided into sub-regions, and the one or more sub-regions may be determined for their saliency. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by the machine learning system, and / or edge boxes, etc.

[0075] In step 224, during the training phase, the method may include training a machine learning system to detect and / or identify salient regions of one or more digital image inputs of a pathology sample using AI-based methods and / or manual identification. The entire image or specific image regions may be considered as salient. An image region that is salient to biomarker detection, e.g., a tumor, may occupy a certain fraction of the entire image. Regions of interest may be defined by a human expert using image segmentation masks, bounding boxes, or polygons. Alternatively, the AI ​​may provide a complete end-to-end solution in identifying locations. Salient region identification may enable downstream AI systems to learn how to detect biomarkers from less annotated data and make more accurate predictions.

[0076] In step 226, the method may include training a machine learning system to predict expression levels of one or more biomarkers from the salient regions. The expression levels may be represented as binary numbers, ordinal numbers, and / or real numbers, etc. The training method may be implemented in multiple ways. For example, according to one embodiment, the algorithm may be implemented by any one or any combination of the following: (1) neural network methods, e.g., machine learning algorithms and / or architectures such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); (2) training methodologies such as multiple instance learning, reinforcement learning, activation learning, etc.; (3) long short-term memory RNNs (LSTMs); (4) gated recurrent units RNNs (GRUs); (5) graph convolutional networks; (6) support vector machines; and / or (7) random forests.

[0077] According to one or more embodiments, any of the above algorithms, architectures, methodologies, attributes, and / or features may be combined with any or all of the other algorithms, architectures, methodologies, attributes, and / or features. For example, any of the machine learning algorithms and / or architectures (e.g., neural network methods, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.) may be trained using any of the training methodologies (e.g., multiple instance learning, reinforcement learning, activation learning, etc.).

[0078] The following explanations of terms are merely illustrative and are not intended to limit the terms in any way.

[0079] The labels may refer to information about the inputs to a machine learning algorithm that the algorithm attempts to predict.

[0080] For a given image size of N×M, the segmentation may be another image of size N×M that assigns to each pixel in the original image a number that describes the class or type of that pixel. For example, in WSI, elements in the mask may categorize each pixel in the input image as belonging to, for example, background, tissue, and / or unknown classes.

[0081] Slide-level information may refer to information about a slide in general, but not necessarily to the specific location of that information within the slide.

[0082] A heuristic may refer to a logical rule or function that deterministically produces an output given an input, e.g., output 1 if prediction that the slide contains a biomarker, output 0 if not.

[0083] Embedding may refer to a conceptual high-dimensional numerical representation of low-dimensional data. For example, if a WSI is passed through a CNN training to classify tissue types and / or predict biomarkers, the numbers on the last layer of the network may provide a set (e.g., on the order of thousands) of numbers that contain information about the slide (e.g., information about the type of tissue).

[0084] A slide-level prediction may refer to a concrete prediction about the slide as a whole. For example, a slide-level prediction may be that the slide contains one or more biomarkers.

[0085] A classifier may refer to a model that is trained to take input data and associate a category with it.

[0086] According to one or more embodiments, the machine learning system may be trained in different ways. For example, the training of the machine learning system may be performed by any one or any combination of supervised training, semi-supervised training, unsupervised training, classifier training, mixed training, and / or uncertainty estimation. The type of training used may depend on the amount of data, the type of data, and / or the quality of the data. Table 1 below describes a non-limiting list of some types of training and corresponding features. [Table 1]

[0087] Supervised training may be used in conjunction with small amounts of data to provide a seed for a machine learning system, where the machine learning system may look for specific items (e.g., biomarkers) and quantify the amount of the specific items present in a slide.

[0088] According to one embodiment, an exemplary fully supervised training may take as input the WSI and may include an indication of the segmentation. A pipeline for fully supervised training may include (1) 1, (2) 1, heuristic, (3) 1, 4, heuristic, (4) 1, 4, 5, heuristic, and / or (5) 1, 5, heuristic. An advantage of fully supervised training may be that (1) it may require fewer slides and / or (2) the output is interpretable because the area of ​​the image that contributed to the diagnosis may be known. A disadvantage of using fully supervised training may be that it may require a large amount of segmentation, which may be difficult to obtain.

[0089] According to one embodiment, an exemplary semi-supervised (e.g., weakly supervised) training may take as input the WSI and may include indicators of slide-level information. A pipeline for semi-supervised training may include (1) 2, (2) 2, heuristic, (3) 2,4, heuristic, (4) 2,4,5, heuristic, and / or (5) 2,5, heuristic. An advantage of using semi-supervised training may be that the output is interpretable because (1) the type of potentially required indicator may be present in many hospital records and (2) the area of ​​the image that contributed most to the diagnosis may be known. A disadvantage of using semi-supervised training is that it may be difficult to train. For example, the model may need to use training schemes such as multiple instance learning, activation learning, and / or distributed training to take into account the fact that there is limited information about the locations in the slide where the information that should lead to a decision is located.

[0090] According to one embodiment, an exemplary unsupervised training may take the WSI as an input and may not require any labels. A pipeline for unsupervised training may include (1) 3,4, and / or (2) 3,4, heuristics. An advantage of unsupervised training may be that it does not require any labels. A disadvantage of using unsupervised training may be that (1) it may be difficult to train. For example, using training schemes such as multiple instance learning, activation learning, and / or distributed training, one may need to take into account the fact that there is limited information about where in the slide there is information that should lead to a decision, (2) it may require additional slides, and / or (3) it may not be very interpretable because it may output predictions and probabilities without explaining why the predictions were made.

[0091] According to one embodiment, the exemplary mixed training may include training any of the exemplary pipelines described above for fully supervised training, semi-supervised training, and / or unsupervised training, and then use the resulting model as an initial point for any of the training methods. The advantages of mixed training may be that (1) it may require less data, (2) it may have improved performance, and / or (3) it may allow mixing of different levels of labels (e.g., segmentation, slide-level information, no information). The disadvantages of mixed training may be that (1) training may be more complex and / or expensive, and / or (2) it may require more code, which may increase the number and / or complexity of potential bugs.

[0092] According to one embodiment, the exemplary uncertainty estimation may include training any of the exemplary pipelines described above for fully supervised, semi-supervised, and / or unsupervised training for any task involving slide data with the uncertainty estimation at the end of the pipeline. Additionally, heuristics or classifiers may be used to predict the expression level of a biomarker based on the amount of uncertainty in the prediction of the test. The advantage of the uncertainty estimation may be that it is robust to out-of-distribution data. For example, when presented with unfamiliar data, it may still correctly predict uncertainty. The disadvantage of the uncertainty estimation may be that (1) more data may be required, (2) it may have poor overall performance, and / or (3) it may not be very interpretable since the model may not necessarily identify the extent to which a slide or slide embedding is abnormal.

[0093] According to one embodiment, ensemble training may include simultaneously launching models generated by any of the exemplary pipelines described above and combining the outputs by heuristics or classifiers to generate robust and accurate results. The advantages of ensemble training may be that (1) it is robust to out-of-distribution data, and / or (2) it may combine advantages and disadvantages of other models, resulting in minimization of disadvantages (e.g., a supervised training model combined with an uncertainty estimation model, and a heuristic using a supervised model when the incoming data is in-distribution and an uncertainty model when the data is out-of-distribution, etc.). The disadvantages of ensemble training may be that (1) it may be more complex, and / or (2) it may be expensive to train and launch.

[0094] The training techniques discussed herein may also be phased, with images with more annotations being used initially for training, which may allow for more effective subsequent training using less supervised, etc. slides with fewer annotations.

[0095] Training may begin using the most fully annotated slides compared to all training slide images that may be used. For example, training may begin using supervised learning. A first set of slide images may be received or determined with associated annotations. Each slide may have marked and / or masked regions and may include information such as identifying salient regions and predicting biomer expression levels. The first set of slides may be provided to a training algorithm, e.g., a CNN, which may determine correlations between the first set of slides and their associated annotations.

[0096] After training with the first set of images is completed, a second set of slide images may be received or determined with fewer annotations than the first set, e.g., with partial annotations. In one embodiment, the annotations may only indicate that the slide has a salient area and may not specify the disease or its location that may be found, etc. The second set of slide images may be trained using a different training algorithm than the first, e.g., multiple instance learning. The first set of training data may be used to partially train the system, making the second training step more effective in generating accurate algorithms.

[0097] In this manner, training may proceed in any number of stages based on the quality and type of training slide images using any number of algorithms. These techniques may be utilized in situations where multiple sets of training images are received, which may be of varying quality, annotation levels, and / or annotation types.

[0098] According to one embodiment illustrated in FIG. 2B, an exemplary method 240 for using the tool may include one or more of the following steps: According to one embodiment, an exemplary method 240 for predicting one or more biomarkers using a biomarker tool may include one or more of the following steps: In step 242, the method may include receiving one or more digital images of a pathology sample (e.g., histology, cytology, etc.) in a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and receiving locations of salient regions using AI-based methods and / or manual identification. In step 244, the method may include applying a salient region detector machine learning algorithm (e.g., method 220) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and to identify salient image regions that may correspond to cancerous tissue. Additionally, the method may include applying a machine learning biomarker detection algorithm (e.g., method 220) to the image and determining a prediction of the expression level of the biomarker.

[0099] At step 246, the method may include outputting the prediction to an electronic storage device. The method may include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) to the presence of one or more biomarkers and / or the expression level of one or more biomarkers.

[0100] As illustrated in FIG. 3A, according to one embodiment, exemplary methods 300 and 320 for predicting estrogen receptor (ER), progesterone receptor (PR), and / or human epidermal growth factor receptor 2 (HER2, which may be known as ERBB2) are described below.

[0101] IHC can be used by pathologists to identify the expression of biomarkers in tumor tissue. For breast cancer, biomarkers can include ER, PR, and / or HER2. In addition, patients with ER-positive / HER2-negative disease can request further assessment of the tumor based on gene signatures (e.g., Oncotype DX, Mammaprint, Endopredict, Breast Cancer Index, and / or ProSigna ROR). If one of these tests predicts a low risk of metastasis, hormone therapy alone can be considered; otherwise, a more aggressive treatment plan involving both chemotherapy and / or hormone therapy can be recommended.

[0102] Although H&E may be part of a pathologist's typical workflow, it may be difficult for the pathologist to visually distinguish the presence of biomarkers from H&E staining. Even IHC may be ambiguous and additional genetic testing of tissue may be required. For example, when IHC is used for HER2, the pathologist's reading may be inconclusive, leading to a follow-up test using FISH to determine whether the cells have extra copies of the HER2 gene. IHC is expensive, but FISH is more expensive. However, FISH allows the presence of HER2 gene amplification and overexpression to be determined conclusively. Determining the expression of ER, PR, and / or HER2 protein overexpression and / or gene amplification may be useful for prescribing medical treatment to breast cancer patients. The techniques presented herein may be used to predict the results of ER, PR, and / or HER2 overexpression and gene signatures from digital images stained with H&E. This may allow for the identification of patients who may be eligible for endocrine, CDK4 / 6-inhibitor therapy, anti-HER2 therapy, and / or chemotherapy, as well as the definition of patients with "triple-negative" disease without the need for three separate IHC tests (one for ER, one for PR, and one for HER2) and / or gene signatures.

[0103] The techniques presented herein may be used to detect pre-malignant and malignant breast tissues and metastatic breast cancers at other sites. In addition, the techniques presented herein may be used to detect overexpression of these biomarkers in other tissues (e.g., HER2 amplification is also routinely assayed in gastric cancer tissues, and HER2 is known to be amplified and overexpressed in some forms of ovarian, lung, and / or uterine cancer). The techniques presented herein may be used to allow definitive quantification of these biomarkers.

[0104] Exemplary methods may utilize a salient region detection tool 103 to identify tissue regions that may be suspicious of cancer. This may greatly reduce sample complexity for machine learning tasks, allowing biomarkers to be more efficiently learned by a biomarker prediction tool 104, e.g., a CNN. The techniques presented herein may be used to predict gene signature outcomes or to identify patients for testing with validated gene signatures (e.g., Oncotype DX Breast Recurrence Score Test and / or Mammaprint Test). The techniques presented herein may allow eligibility for the Mammaprint Test or eligibility for the Oncotype DX Breast Recurrence Score Test to be assessed, in part, directly from H&E images of breast tumors, without the need for IHC or FISH.

[0105] Exemplary methods 300 and 320 may include one or more of the following steps. In step 301, during a training phase, the method may include receiving one or more digital images of an H&E stained tissue sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, for the one or more images. The one or more digital images may be divided into sub-regions and the salience of the one or more sub-regions may be determined. The image may include a cancer recurrence score generated from a genomic assay. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0106] In step 303, the method may include training a machine learning algorithm that takes as input a digital image of an H&E stained pathology sample and predicts, using AI-based methods and / or manual identification, whether a salient region is present, e.g., whether cancer is likely present. The machine learning algorithm may be performed in a weakly supervised or strongly supervised manner.

[0107] In step 305, the method may include training a machine learning algorithm to predict expression levels of one or more of the ER, PR, and / or HER2 biomarkers based on salient regions of the digital image of the pathology sample and the received biomarker / score information. The method may include receiving an indication per slide of expression of one or more of the biomarkers, e.g., ER, PR, and / or HER2. The level of biomarker expression may be identified using IHC, FISH, or some additional orthogonal technique. The level of expression may be a numerical, ordinal, or binary scale. The indication may be assigned to the entire image and / or image sub-regions, e.g., the image may be divided into tiles and each tile may be assigned a HER2 overexpression level. The indication may include categorical data, e.g., "low risk" or "high risk." For example, the indication may include an Oncotype DX result. Such a result may include a continuous score with predefined or user-determined thresholds for low, medium, and / or high risk.

[0108] According to one embodiment, an exemplary method 320 for predicting ER, PR, and / or HER2 biomarkers using a biomarker tool may include one or more of the following steps: In step 321, the method may include receiving one or more digital images of an H&E stained pathology sample (e.g., a breast cancer pathology sample) into a digital storage device (e.g., a hard drive, a network drive, cloud storage, RAM, etc.). In step 323, the method may include applying a salient region detector machine learning algorithm (e.g., method 300) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 300) to the image and determining a prediction of the expression level of the biomarker. The method may include grouping the expression levels into diagnostic categories. For example, HER2 may be graded using IHC on a scale of 0, 1+, 1+-2+, 2+, and / or 3+. Using a probabilistic ordinal regression model, the probability of various combinations may be calculated, for example, the probability of a score greater than zero. This may be useful because the efficacy of some drugs depends on the level of expression.

[0109] At step 325, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) to the expression levels of one or more biomarkers and / or recommending a treatment that is potentially effective for the cancer given the presence of a given biomarker.

[0110] FIG. 3B illustrates a system and method for training to identify HER2, ER, and / or PR overexpression levels. For example, as shown on the left side of FIG. 3B, a machine learning system may be fed a digital whole slide image of breast tissue, where some of the tissue may possibly be cancerous. A salient tissue detector may filter the image and identify one or more tissue regions that may have cancer, which are regions of interest for identifying HER2, ER, and / or PR biomarkers that may help guide treatment. Less relevant tissue may be removed using the identified salient regions, and the trained AI may infer the expression levels of each of the HER2, ER, and / or PR biomarkers.

[0111] 4A-4U illustrate an exemplary system and method for predicting HER2 levels from H&E WSIs, which may possibly have cancer, according to an exemplary embodiment of the present disclosure. For example, the input may include all H&E images from the breast cancer part. An exemplary output may be a HER2 score for expression levels. For example, HER2 may be graded on a scale of 0, 1+, 1+ to 2+, 2+, and / or 3+ using IHC. Anti-HER2 therapy may be administered based on HER2 expression levels, which may derive from the likelihood that HER2 is a clonal driver of growth and thus the tumor may be sensitive to an anti-HER2 antibody, e.g., trastuzumab. However, based on the development of anti-HER2 antibody-drug conjugates (ADCs), which may generate a potent bystander effect, administering anti-HER2 antibodies based on HER2 expression levels may not be the only effective method. Some patients with "low HER2" (e.g., with some HER2 expression but not enough for the tumor to be considered positive by current clinical guidelines) may benefit from anti-HER2 ADCs. However, it may be difficult to define which patients biomarkers would benefit from these ADCs. Exemplary embodiments of the present disclosure may include AI-based analysis of histological and IHC samples of patients enrolled in clinical trials testing ADCs, whose outcome data may be available to characterize a collection of morphological features that distinguish between (1) HER2-high and HER2-low, (2) HER2-high patients who responded to the ADC, and / or (3) HER2-low patients who responded to anti-HER2 ADCs. The features employed by the AI ​​algorithms predicting resistance to these agents may be supported by a specific repertoire of genetic and / or epigenetic modifications. Exemplary clinical trial data may include digital images of tumors and / or longitudinal follow-up information from patients included within anti-HER2 ADC studies.

[0112] FIG. 4A illustrates an exemplary training method for predicting HER2 scores from H&E as a 4-fold ordinal regression model. For example, a digital image may be divided into sub-regions and tiles of images may be created. HER2 may be graded using IHC on a scale of 0, 1+, 1+ to 2+, 2+, and / or 3+. The method may include training a CNN to predict HER2 scores. The HER2 scores may be from IHC, however, the input may include a set of all H&E images from a part, which may allow for quicker iteration than using only weak supervision and may provide the system with more HER2 score information to train.

[0113] FIG. 4B illustrates an exemplary test method for predicting HER2 score. For example, a digital image may be divided into sub-regions and tiles of images may be created. HER2 may be graded using IHC on a scale of 0, 1+, 1+ to 2+, 2+, and / or 3+. The method may include aggregating the tile predictions into part-level HER2 predictions.

[0114] FIG. 4C illustrates an exemplary method for detecting tiles in which HER2 biomarkers are present. For example, a digital image may be divided into sub-regions and tiles of images may be created. The method may include using classifier training with CNN to predict the probability that cancer may be present in the tiles. The tiles may be ranked based on probability and used to train a machine learning system as to which tiles may show invasive cancer and benign. As an example, tiles with a probability of cancer above 0.96 may be used to distinguish invasive cancer vs. benign. The AUC number refers to the area under the receiver operating characteristic curve value for predicting HER2 score. The AP number refers to the average precision, which is the area under the precision-recall curve.

[0115] FIG. 4D illustrates an exemplary training method for detecting tiles in which HER2 biomarkers are present. For example, a digital image may be divided into sub-regions and tiles of the image may be created. The method may include using CNN training to predict a 4-fold HER2 score prediction (e.g., (P(HER2t=0|Xt), (P(HER2t=0|Xt), (P(HER2t=0|Xt), (P(HER2t=0|Xt))).

[0116] FIG. 4E illustrates an exemplary evaluation method for detecting the presence of HER2 biomarkers at the part level. For example, a digital image may be divided into sub-regions and tiles of images may be created. The method may include using CNN training to predict the probability of the presence of HER2 biomarkers at a level of 0, 1, 2, or 3 at the tile level. A part level score may be calculated and cancer tiles from the part may be fed into the CNN to obtain a 4-fold HER2 score prediction. Predictions may be aggregated across all tiles to perform classification.

[0117] FIG. 4F illustrates an exemplary data set associated with an exemplary training set and test set for detecting parts and / or tiles in which HER2 biomarkers are present. For the data in FIG. 4F, non-invasive tiles have been removed. Thus, the data set includes tiles that are classified as invasive carcinomas by the detection model. The right side of FIG. 4F illustrates that HER2 biomarkers at levels of 0 and 1+ may be considered negative, scores of 3+ may be considered positive, and scores of 1+-2+ and 2+ may be considered indeterminate and removed from the data set.

[0118] Figure 4G illustrates an exemplary normalized confusion matrix for an exemplary data set. Three tasks can be performed on the data: (1) multi-way classification; (2) detecting HER2 above zero from H&E; and / or detecting HER2- vs. HER2+ from H&E. The numbers in the exemplary matrix represent predicted vs. true labeling for HER2 levels of 0, 1, 2, and / or 3. The legend on the right highlights the accuracy level for each prediction.

[0119] FIG. 4H shows the results of a comparison of HER2 levels 0 versus HER2 levels 1, 2, and 3 (e.g., score=max t( P(HER2>0|X t 1 illustrates an example graph charting false positive rate vs. true positive rate for . The baseline MIL model shows an AUC of 0.674. The ordinal regression model shows an AUC of 0.970. The softmax model shows an AUC of 0.871.

[0120] FIG. 4I shows the AUC for the MIL, softmax, and / or ordinal regression models along with the mean mean of HER2 level 0 versus HER2 levels 1, 2, and / or 3 (e.g., score=max t (P(HER2>0|X t 4I illustrates another example graph for the ordinal regression model at 0.95 sensitivity, 0.98 sensitivity, and 0.669 sensitivity.

[0121] Figure 4J shows HER2 negative (0 / 1) vs. HER2 positive (3) (e.g., score = max t (P(HER2=3|X t 1 illustrates an example graph charting false positive rate vs. true positive rate for HER2 score 1 (n = 1). The dataset excludes a HER2 score of 2. The baseline MIL model shows an AUC of 0.804. The ordinal regression model shows an AUC of 0.978. The softmax model shows an AUC of 0.939.

[0122] Figure 4K shows the AUC for the MIL, softmax, and / or ordinal regression models for HER2-negative (0 / 1) vs. HER2-positive (3) (e.g., score = max t (P(HER2=3|X t 4I illustrates another example graph for the ordinal regression model at 0.95 sensitivity, 0.98 sensitivity, and 0.669 sensitivity.

[0123] 4L illustrates an example visualization for H&E tiles that are correctly predicted as HER2 level 0. For example, H&E tiles with heat maps 401, 402, 403, 404 were correctly predicted by the biomarker tool as having HER2 level 0.

[0124] FIG. 4M illustrates an exemplary IHC staining of the same part of the tile illustrated in FIG. 4L.

[0125] 4N illustrates an example visualization for H&E tiles predicted as HER2 level 3. For example, H&E tiles with heat maps 405, 406, and 408 were correctly predicted by the biomarker tool as having HER2 level 3. H&E tile 407 was predicted as having HER2 level 1.

[0126] FIG. 4O illustrates an exemplary IHC staining of the same part of the tile illustrated in FIG. 4N.

[0127] 4P illustrates an example visualization for H&E tiles that are correctly predicted as HER2 level 1. For example, H&E tiles with heat maps 409, 410, 411, 412 were correctly predicted by the biomarker tool as having HER2 level 1.

[0128] FIG. 4Q illustrates an exemplary IHC staining of the same part of the tile illustrated in FIG. 4P.

[0129] 4R illustrates an example visualization for H&E tiles that are incorrectly predicted as HER2 level 2. For example, H&E tiles with heat maps 413 and 414 were incorrectly predicted as HER2 level 2, and H&E tiles 415 and 416 were predicted as HER2 level 0.

[0130] FIG. 4S illustrates an exemplary IHC staining of the same part of the tile illustrated in FIG. 4R.

[0131] FIG. 4T illustrates an exemplary H&E heat map in which a slide was correctly classified as HER2+.

[0132] FIG. 4U illustrates an exemplary IHC staining in which the slide was correctly classified as HER2+.

[0133] As illustrated in FIG. 5, according to one embodiment, exemplary methods 500 and 520 for predicting epidermal growth factor receptor (EGFR) biomarkers are described below.

[0134] Lung cancers with EGFR activating gene mutations may be treatable with specialized drugs that can improve outcomes. Approximately 15% of people with lung cancer in the United States have EGFR mutations, but it is estimated that up to 50% of people with lung cancer of Southeast Asian descent have targetable alterations that affect this gene. EGFR mutations can be determined by several sequencing methods. The techniques presented herein can define the presence of EGFR activating mutations and / or EGFR mutations that cause resistance to anti-EGFR agents (e.g., EGFR T790m mutations) from H&E stained lung tissue samples.

[0135] Exemplary methods 500 and 520 for predicting EGFR biomarkers may include one or more of the following steps: In step 501, during a training phase, the method may include receiving one or more digital images of H&E stained lung tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and / or receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, for the one or more images. The one or more digital images may be divided into sub-regions and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0136] In step 503, the method may include training a machine learning algorithm that takes as input digital image pathology of the H&E stained specimen and uses AI-based methods and / or manual identification to predict whether a salient region is present, e.g., whether cancer is likely present. The machine learning algorithm may be performed in a weakly supervised or strongly supervised manner.

[0137] In step 505, the method may include training a machine learning algorithm to predict expression levels of one or more of the EGFR biomarkers based on salient regions of the digital image of the pathology sample and the received biomarker / score information. The method may include receiving an indication per slide of the presence of one or more of the EGFR mutants. The presence of the biomarkers may be identified using a validated sequencing method. The presence of the mutants may be reported as categorical variables. The variant allele fraction of the mutants and the cancer cell fraction (e.g., the bioinformatically inferred percentage of cancer cells in the sample that have a given mutant) may be reported on a numerical, ordinal, or binary scale. The indications may be assigned to the entire image and / or to image sub-regions.

[0138] According to one embodiment, an exemplary method 520 for predicting EGFR biomarkers using a biomarker tool may include one or more of the following steps: In step 521, the method may include receiving one or more digital images of an H&E stained pathology sample (e.g., a breast cancer pathology sample) into a digital storage device (e.g., a hard drive, a network drive, cloud storage, RAM, etc.). In step 523, the method may include applying a salient region detector machine learning algorithm (e.g., method 500 (e.g., a deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is likely present in the received image and to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 500) to the image and determining a prediction of the presence of an EGFR mutation. The method may include assigning the presence of an EGFR mutation to a diagnostic category.

[0139] In step 525, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0140] As illustrated in FIG. 6, according to one embodiment, exemplary methods 600 and 620 for predicting DNA mismatch repair (MMR) deficiency biomarkers are described below.

[0141] MMR genes can help identify metastatic cancer patients who may benefit from immunotherapy. All patients with metastatic disease may be candidates for immunotherapy, but screening for MMR may be important because the presence of MMR deficiency and / or high levels of microsatellite instability (MSI-H) predicts response to immunotherapy on a tissue-independent basis. In one embodiment, MMR deficiency and / or MSI-H may be detected from H&E images without the need for IHC for MMR protein level assessment and / or PCR or massively parallel sequencing for MSI-H assessment.

[0142] Exemplary methods 600 and 620 for predicting DNA mismatch repair (MMR) deficiency biomarkers may include one or more of the following steps. In step 601, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or an indication of the type of tissue within the tissue sample, for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0143] In step 603, the method may include training a machine learning algorithm that takes as input digital image pathology of the H&E stained specimen and uses AI-based methods and / or manual identification to predict whether a salient region is present, e.g., whether cancer is likely present. The machine learning algorithm may be performed in a weakly supervised or strongly supervised manner.

[0144] In step 605, the method may include training a machine learning algorithm to predict expression levels of one or more of the MMR deficiency biomarkers based on salient regions of the digital image of the pathology sample and the received biomarker / score information. The method may include receiving an indication per slide of the MMR deficiency level. The indication may be obtained via the presence of MSI-H as defined by IHC and / or PCR or massively parallel sequencing. The level of expression may be a numerical, ordinal, or binary scale. The indication may be assigned to the entire image and / or to image sub-regions.

[0145] According to an embodiment, an exemplary method 620 for predicting MMR-deficiency biomarkers using a biomarker tool may include one or more of the following steps: In step 621, the method may include receiving one or more digital images of an H&E stained pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). In step 623, the method may include applying a salient region detector machine learning algorithm (e.g., method 600 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 600) to the image and determining a prediction of expression levels of MMR proteins, MMR deficiency, and / or MSI-H. The method may include assigning expression levels of MMR proteins, MMR deficiency, and / or MSI-H to a diagnostic category.

[0146] In step 625, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0147] As illustrated in FIG. 7, according to one embodiment, exemplary methods 700 and 720 for predicting APOBEC mutagenesis biomarkers are described below.

[0148] APOBEC cytidine deaminase can induce mutant clusters in human tumors.APOBEC mutagenesis process can be widely observed across cancer types, for example, in primary bladder cancer, post-treatment ER-positive breast cancer, and / or lung cancer.APOBEC mutagenesis can be identified in cancer through tumor sequencing analysis.The techniques presented herein can be used to detect APOBEC mutagenesis.

[0149] Exemplary methods 700 and 720 for predicting APOBEC mutagenesis biomarkers may include one or more of the following steps: In step 701, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or tissue sample type for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0150] In step 703, the method may include training a machine learning algorithm that takes as input a digital image of a pathology sample and uses AI-based methods and / or manual identification to predict whether a salient region is present, e.g., whether cancer is likely present, based on the tissue sample type. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner.

[0151] In step 705, the method may include training a machine learning algorithm to predict expression levels of one or more of the APOBEC mutagenesis biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information. The method may include receiving an indication per slide of the APOBEC mutagenesis biomarker levels. The levels may be a numeric, ordinal, or binary scale. The indications may be assigned to the entire image and / or to image sub-regions.

[0152] According to one embodiment, an exemplary method 720 for predicting APOBEC mutagenesis biomarkers using a biomarker tool may include one or more of the following steps: In step 721, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). In step 723, the method may include applying a salient region detector machine learning algorithm (e.g., method 700 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is likely present in the received image and identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 700) to the image and determining a prediction of the APOBEC mutagenesis biomarker level. The method may include assigning the APOBEC mutagenesis biomarker level to a diagnostic category.

[0153] In step 725, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0154] As illustrated in FIG. 8, according to one embodiment, exemplary methods 800 and 820 for predicting DNA polymerase ε (POLE) mutagenesis biomarkers are described below.

[0155] Defective DNA POLE proofreading can lead to an ultramutator phenotype in cancer, characterized by widespread somatic single nucleotide polymorphism mutations. POLE mutations and their associated ultramutator phenotypes can be predictive of response to immune checkpoint inhibitors. The techniques presented herein can be used to detect POLE mutagenesis in cancer, and the detected POLE mutagenesis can be used as a biomarker of response to immune checkpoint inhibitors.

[0156] Exemplary methods 800 and 820 for predicting POLE mutagenesis biomarkers may include one or more of the following steps: In step 801, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or receiving an indication of tissue sample type for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0157] In step 803, the method may include training a machine learning algorithm that takes as input a digital image of the pathology sample and uses AI-based methods and / or manual identification to predict whether a significant region is present, e.g., whether cancer is likely present, based on the indication. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner.

[0158] In step 805, the method may include training a machine learning algorithm to predict expression levels of one or more of the POLE mutagenesis biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information. The method may include receiving an indication of the presence (e.g., categorical) of a POLE mutant and its variant allele (e.g., numerical value) and cancer cell fraction (e.g., numerical value), and the presence (e.g., categorical) and abundance (e.g., numerical value) for the POLE mutagenesis process. The level of expression may be numerical, ordinal, or binary scale. The indication may be assigned to the entire image and / or to image sub-regions.

[0159] According to one embodiment, an exemplary method 820 for predicting POLE mutagenesis biomarkers using a biomarker tool may include one or more of the following steps: In step 821, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). In step 823, the method may include applying a salient region detector machine learning algorithm (e.g., method 800 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and / or to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 800) to the image to determine the presence of POLE mutants, their variant allele fractions and cancer cell fractions, and a prediction of the level of POLE mutagenesis. The method may comprise assigning the presence of POLE mutations, their variant allele fractions and cancer cell fractions, and the level of POLE mutagenesis to a diagnostic category.

[0160] In step 825, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0161] As illustrated in FIG. 9, according to one embodiment, exemplary methods 900 and 920 for predicting CDK4 / 6 inhibitor biomarkers are described below.

[0162] CDK4 / 6 inhibitors may be a class of drugs that target CDK4 and CDK6 enzymes. CDK4 / 6 enzymes may be useful in cell division, and drugs that act as CDK4 / 6 inhibitors may be used to treat cancer, including ER-positive breast cancer. Determining whether a cancer is resistant to CDK4 / 6 inhibitors may help guide treatment so that unnecessary drugs are not given to patients. The techniques presented herein may be used to detect whether a tumor known to be ER-positive is resistant to CDK4 / 6 inhibitors. ER positivity may be performed using techniques such as IHC and / or may be determined using another method described herein.

[0163] Exemplary methods 900 and 920 for predicting CDK4 / 6 inhibitor biomarkers may include one or more of the following steps. In step 901, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), identifying significant image regions corresponding to cancerous tissue using AI-based methods and / or manual identification, and / or receiving an indication on one or more images of CDK4 / 6 inhibitor biomarker levels. The level of expression may be a numerical, ordinal, or binary scale. Indications may be assigned to the entire image and / or to image sub-regions. The significant regions may be identified as having cancer and / or as estrogen receptor (ER) positive.

[0164] In step 903, the method may include training a machine learning algorithm that takes as input a digital image of the pathology sample and uses AI-based methods and / or manual identification to predict whether a significant region is present, e.g., whether cancer is likely present, based on the indication. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner.

[0165] In step 905, the method may include training a machine learning algorithm to predict expression levels of one or more of the CDK4 / 6 inhibitor biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information.

[0166] According to one embodiment, an exemplary method 920 for predicting CDK4 / 6 inhibitor biomarkers using a biomarker tool may include one or more of the following steps: In step 921, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). In step 923, the method may include applying a salient region detector machine learning algorithm (e.g., method 900) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and / or to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 900) to the image and determining a prediction of whether the tumor is resistant to a CDK4 / 6 inhibitor. The method may include assigning a level of resistance to a CDK4 / 6 inhibitor to a diagnostic category.

[0167] In step 925, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0168] As illustrated in FIG. 10, according to one embodiment, exemplary methods 1000 and 1020 for predicting neurotrophin receptor tyrosine kinase (NTRK) abnormal biomarkers are described below.

[0169] Many cancers have NTRK abnormalities, which may include fusion genes. These alterations can be found in a significant fraction of prostate, pancreas, breast, liver, lung, and / or skin cancers. NTRK fusions can be found in both pediatric and adult cancers. Specific drug treatments have been approved for patients whose tumors have fusion genes involving NTRK1, NTRK2, or NTRK3. Fusion genes can be rare, and screening for these alterations can be cumbersome and very laborious and / or time-consuming. The techniques presented herein can be used to identify cancers that have NTRK1, NTRK2, or NTRK3 fusion genes.

[0170] Exemplary methods 1000 and 1020 for predicting neurotrophin receptor tyrosine kinase (NTRK) abnormal biomarkers may include one or more of the following steps. In step 1001, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or receiving an indication of tissue sample type for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0171] In step 1003, the method may include training a machine learning algorithm that takes as input a digital image of a pathology sample and uses AI-based methods and / or manual identification to predict whether a salient region is present, e.g., whether cancer is likely present. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner. The machine learning algorithm may be trained for a specific type of cancer, or may be a pan-cancer model, allowing it to work on rare cancers in which the amount of training data is limited.

[0172] In step 1005, the method may include training a machine learning algorithm to predict expression levels of one or more of the NTRK aberrant biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information. The method may include receiving an indication for each slide of the presence of an NTRK1, NTRK2, or NTRK3 fusion gene. The presence of the fusion may be indicated on a categorical scale (e.g., present vs. absent). The indication may be assigned to the entire image and / or to image sub-regions.

[0173] According to one embodiment, an exemplary method 1020 for predicting NTRK abnormal biomarkers using a biomarker tool may include one or more of the following steps. In step 1021, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and / or receiving an indication of a tissue sample type. In step 1023, the method may include applying a salient region detector machine learning algorithm (e.g., method 1000 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and / or to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 1000) to the image and determining a prediction of the presence of an NTRK1, NTRK2, or NTRK3 fusion gene. The method may include assigning the presence of an NTRK1, NTRK2, or NTRK3 fusion gene to a diagnostic category.

[0174] In step 1025, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0175] As illustrated in FIG. 11, according to one embodiment, exemplary methods 1100 and 1120 for predicting immune response biomarkers are described below.

[0176] Recognition of tumor cells by the immune system for destruction may require a set of conditions that may be utilized in several biomarkers for assessment of the potential efficacy of immunotherapy, including antibodies against PD1, PDL1, and / or CD28, among others. Some of these biomarkers may include several somatic mutations (e.g., tumor mutant load), IHC for markers including PDL1 and / or PD1, gene expression signatures for the level of inflammation within the microenvironment, etc. The techniques presented herein may be used to identify cancers with the potential to be recognized by the immune system and susceptible to checkpoint therapy.

[0177] Exemplary methods 1100 and 1120 for predicting immune response biomarkers may include one or more of the following steps: In step 1101, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or receiving an indication of tissue sample type for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0178] In step 1103, the method may include training a machine learning algorithm that takes as input a digital image of a pathology sample and uses AI-based methods and / or manual identification to predict whether a salient region is present, e.g., whether cancer is likely present. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner. The machine learning algorithm may be trained for a specific type of cancer, or may be a pan-cancer model, allowing it to work on rare cancers in which the amount of training data is limited.

[0179] In step 1105, the method may include training a machine learning algorithm to predict expression levels of one or more of the immune response biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information. The method may include receiving an indication for each slide of sensitivity to checkpoint inhibitors or tumor mutant burden or inflamed tumor microenvironment or PDL1 / PD1 positivity. The presence of these may be indicated on a categorical scale (e.g., present vs. absent). The indication may be assigned to the entire image and / or to image sub-regions.

[0180] According to one embodiment, an exemplary method 1120 for predicting immune response biomarkers using a biomarker tool may include one or more of the following steps: In step 1121, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.) and / or receiving an indication of a tissue sample type. In step 1123, the method may include applying a salient region detector machine learning algorithm (e.g., method 1100 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and / or to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 1100) to the image and determining a prediction of the presence of an immune response biomarker. The method may include assigning the presence of the immune response biomarker to a diagnostic category.

[0181] In step 1125, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0182] As illustrated in FIG. 12, according to one embodiment, exemplary methods 1200 and 1220 for predicting homologous recombination DNA repair deficiency (HRD) biomarkers are described below.

[0183] Homologous recombination can be used by cells to repair harmful DNA breaks, especially double-strand breaks. HRD can be strongly associated with cancer, especially breast, ovarian, pancreatic, and / or prostate cancer. Mutations affecting homologous recombination DNA repair genes, such as BRCA1, BRCA2, PALB2, RAD51C, and / or RAD51D, can not only increase the risk of developing breast, ovarian, pancreatic, and / or prostate cancer, but can also predict the response to platinum-based chemotherapy and poly(ADP) ribose polymerase (PARP) inhibitors. Sequencing analysis of constitutive DNA for the detection of mutations affecting homologous recombination DNA repair genes can be important for the use of PARP inhibitors in the treatment of patients with breast and / or ovarian cancer. In addition, sequencing analysis of tumor-derived DNA can provide evidence of HRD, since the lack of this specific modality of DNA repair can leave a "scar" in the genome.

[0184] Sequencing-based methods for identifying HRD genomic scars may be available, but their clinical utility may be questionable.In addition, the inactivation of some DNA repair genes, such as ATM and CHEK2, does not result in HRD genomic scars, but may predict sensitivity to PARP inhibitors and / or other agents that target specific DNA repair defects.The techniques presented herein may define the presence of inactivation of homologous recombination DNA repair-related genes, DNA damage response genes, and / or HRD in human tumors based on AI analysis of H&E stained fragments.This embodiment may provide a biomarker for the use of PARP inhibitors and / or other agents that target tumors with HRD.

[0185] Exemplary methods 1200 and 1220 for predicting homologous recombination DNA repair deficiency (HRD) biomarkers may include one or more of the following steps: In step 1201, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or tissue sample type for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0186] In step 1203, the method may include training a machine learning algorithm that takes as input a digital image of a pathology sample and uses AI-based methods and / or manual identification to predict whether a salient region is present, e.g., whether cancer is likely present, based on the tissue sample type. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner.

[0187] In step 1205, the method may include training a machine learning algorithm to predict expression levels of one or more of the HRD biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information. The method may include receiving an indication (e.g., categorical) of the presence of mutations affecting homologous recombination and / or DNA damage response related genes, whether these mutations are mono- or bi-allelic (e.g., categorical), their variant alleles (e.g., numerical values) and cancer cell fractions (e.g., numerical values). In addition, the presence (e.g., categorical) and abundance (e.g., numerical values) for HRD genomic scars may also be received. The indication may be assigned to the entire image and / or to image sub-regions.

[0188] According to one embodiment, an exemplary method 1220 for predicting HRD biomarkers using a biomarker tool may include one or more of the following steps: In step 1221, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). In step 1223, the method may include applying a salient region detector machine learning algorithm (e.g., method 1200 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and / or to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 1200) to the image to determine the presence of specific mutations, their variant allele fractions or cancer cell fractions, and a prediction of the HRD level of the tumor. The method may include assigning the presence of a specific mutation, its variant allele fraction or cancer cell fraction, and the HRD level of the tumor to a diagnostic category.

[0189] In step 1225, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0190] As illustrated in FIG. 13, according to one embodiment, exemplary methods 1300 and 1320 for predicting SWItch / Sucrose Non-Fermentable (SWI / SNF) biomarkers are described below.

[0191] SWI / SNF is a group of proteins that remodel the way DNA is packaged. SWI / SNF complexes can function as tumor suppressors, and SWI / SNF proteins can be absent in cancerous tissues. Alterations in the genes of SWI / SNF complexes can cause resistance to endocrine therapy in patients with ER-positive breast cancer. The techniques presented herein can predict the presence or absence of SWI / SNF subunits from digital images of pathological samples.

[0192] Exemplary methods 1300 and 1320 for predicting SWI / SNF biomarkers may include one or more of the following steps: In step 1301, during a training phase, the method may include receiving one or more digital images of H&E stained tissue samples into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.), receiving an indication of salient regions, e.g., the presence or absence of cancerous tissue, and / or receiving an indication of tissue sample type for the one or more images. The one or more digital images may be divided into sub-regions, and the salience of the one or more sub-regions may be determined. The regions may be defined in a variety of ways, including creating tiles of the image, edge / contrast based segmentation, segmentation via color difference, supervised determination by a machine learning system, and / or edge boxes, etc.

[0193] In step 1303, the method may include training a machine learning algorithm that takes as input a digital image of the pathology sample and uses AI-based methods and / or manual identification to predict whether a significant region is present, e.g., whether cancer is likely present, based on the indication. The machine learning algorithm may be implemented in a weakly supervised or strongly supervised manner.

[0194] In step 1305, the method may include training a machine learning algorithm to predict expression levels of one or more of the SWI / SNF complex biomarkers based on salient regions of the digital image of the pathology sample and / or the received biomarker / score information. The method may include receiving an indication of the presence of mutations (e.g., categorical) affecting genes of the SWI / SNF complex, whether these mutations are mono- or bi-allelic (e.g., categorical), their variant alleles (e.g., numerical values) and cancer cell fractions (e.g., numerical values). The indications may be assigned to the entire image and / or to image sub-regions.

[0195] According to one embodiment, an exemplary method 1320 for predicting SWI / SNF complex biomarkers using a biomarker tool may include one or more of the following steps: In step 1321, the method may include receiving one or more digital images of a pathology sample into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.). In step 1323, the method may include applying a salient region detector machine learning algorithm (e.g., method 1300 (e.g., deep neural network)) to the image and using AI-based methods and / or manual identification to predict whether cancer is possibly present in the received image and / or to identify salient image regions that may correspond to cancerous tissue. In addition, the method may include applying a machine learning biomarker detection algorithm (e.g., method 1300) to the image to determine the presence of mutations affecting genes of the SWI / SNF complex, whether these mutations are mono- or bi-allelic, and a prediction of their variant alleles and cancer cell fraction. The method may include the presence of mutations affecting genes of the SWI / SNF complex, whether these mutations are mono- or bi-allelic, and assigning the variant alleles and cancer cell fraction to a diagnostic category.

[0196] In step 1325, the method may include outputting the prediction to an electronic storage device. The method may also include displaying a visual indicator to alert a user (e.g., a pathologist, a histology technician, etc.) of the expression level of one or more biomarkers.

[0197] As shown in FIG. 14, the device 1400 may include a central processing unit (CPU) 1420. The CPU 1420 may be any type of processor device, including, for example, any type of special purpose or general purpose microprocessor device. As will be appreciated by those skilled in the art, the CPU 1420 may also be a single processor in a multi-core / multi-processor system, such systems operating alone or in a cluster of computing devices operating in a cluster or server farm. The CPU 1420 may be connected to a data communications infrastructure 1410, for example, a bus, a message queue, a network, or a multi-core message passing scheme.

[0198] The device 1400 may also include a main memory 1440, e.g., a random access memory (RAM), and may also include a secondary memory 1430. The secondary memory 1430, e.g., a read only memory (ROM), may be, e.g., a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, e.g., a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. The removable storage drive, in this embodiment, reads from and / or writes to a removable storage unit in a well-known manner. The removable storage unit may comprise a floppy disk, a magnetic tape, an optical disk, or the like, which is read and written by the removable storage drive. As will be appreciated by those skilled in the art, such a removable storage unit generally includes a computer usable storage medium having computer software and / or data stored therein.

[0199] In alternative implementations, secondary memory 1430 may include other similar means for allowing computer programs or other instructions to be loaded into device 1400. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 1400.

[0200] Device 1400 may also include a communications interface ("COM") 1460. Communications interface 1460 allows software and data to be transferred between device 1400 and external devices. Communications interface 1460 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 1460 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1460. These signals may be provided to communications interface 1460 over a communications path of device 1400, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0201] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature and are assumed to be well familiar to those skilled in the art. The device 1400 may also include input and output ports 1450 to connect with input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of one computer hardware platform.

[0202] Throughout this disclosure, references to components or modules generally refer to items that may be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0203] The tools, modules, and functions described above may be implemented by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which may provide optional non-transitory storage for software programming.

[0204] The software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, the communication may enable the software to be loaded from one computer or processor into another. As used herein, unless limited to non-transitory, the term tangible "storage" media, such as computer or machine "readable media," refers to any medium that participates in providing instructions to a processor for execution.

[0205] The foregoing general description is exemplary and explanatory only and is not a limitation of the present disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only.

Claims

1. 1. A computer-implemented method for analyzing an image corresponding to a sample, the computer-implemented method comprising: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image; applying a second machine learning algorithm that focuses on the one or more salient regions of the target electronic image to identify a region of interest of the target sample and determine an expression level of a biomarker in the region of interest, a category of a biomarker in the region of interest, and a presence of a biomarker in the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the second machine learning algorithm being generated by predicting whether a region of interest is present in the target electronic image by processing a plurality of training images, the plurality of training images comprising images of human tissue and / or algorithmically generated images, determining the expression level of the biomarker comprises dividing the one or more salient regions of the target electronic image into a plurality of tiles, and the second machine learning algorithm determining for each tile the expression level corresponding to a prediction of the presence of the biomarker in the region of interest; outputting the determined expression level of the biomarker in the region of interest, the determined category of the biomarker in the region of interest, and the determined presence of the biomarker in the region of interest; 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein identifying an area of ​​interest in the target sample comprises determining whether an area of ​​interest exists within any of the plurality of tiles.

3. Identifying a region of interest in the target sample comprises: Segmenting the target electronic image based on edges and / or contrast, segmenting the target electronic image using color difference, and / or supervised decision making; determining whether a region of interest is present within any of the plurality of tiles; The computer-implemented method of claim 1 , comprising:

4. The computer-implemented method comprises: determining an expression level of a biomarker in a region of interest, a category of a biomarker in a region of interest, and a presence of a biomarker in a region of interest by determining a first expression score for each tile of the target electronic image and determining a second expression score for the portion of the target sample by aggregating the first expression scores for each of the plurality of tiles. The computer-implemented method of claim 1 , further comprising:

5. Determining expression levels of biomarkers in the region of interest, categories of biomarkers in the region of interest, and presence of biomarkers in the region of interest includes: clustering the plurality of expression levels into one or more diagnostic categories, the one or more diagnostic categories representing a scale of presence of the biomarkers within the region of interest; classifying the biomarkers in the region of interest by identifying a diagnostic category from among the one or more diagnostic categories; The computer-implemented method of claim 1 , comprising:

6. 2. The computer-implemented method of claim 1, wherein outputting the determined expression level of the biomarker in the region of interest, the determined category of the biomarker in the region of interest, and the determined presence of the biomarker in the region of interest includes displaying a visual indicator related to the expression level of the biomarker, the category of the biomarker, and the presence of the biomarker, and displaying one or more treatment options based on the expression level of the biomarker, the category of the biomarker, and the presence of the biomarker.

7. 1. A system for analyzing an image corresponding to a sample, the system comprising: at least one memory storing instructions; At least one processor; Equipped with The at least one processor executes the instructions to: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image; applying a second machine learning algorithm that focuses on the one or more salient regions of the target electronic image to identify a region of interest of the target sample and determine an expression level of a biomarker in the region of interest, a category of a biomarker in the region of interest, and a presence of a biomarker in the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the second machine learning algorithm being generated by predicting whether a region of interest is present in the target electronic image by processing a plurality of training images, the plurality of training images comprising images of human tissue and / or algorithmically generated images, determining the expression level of the biomarker comprises dividing the one or more salient regions of the target electronic image into a plurality of tiles, and the second machine learning algorithm determining for each tile the expression level corresponding to a prediction of the presence of the biomarker in the region of interest; outputting the determined expression level of the biomarker in the region of interest, the determined category of the biomarker in the region of interest, and the determined presence of the biomarker in the region of interest; A system configured to perform operations including:

8. 8. The system of claim 7, wherein identifying an area of ​​interest in the target sample comprises determining whether an area of ​​interest exists within any of the plurality of tiles.

9. Identifying a region of interest in the target sample comprises: Segmenting the target electronic image based on edges and / or contrast, segmenting the target electronic image using color difference, and / or supervised decision making; determining whether a region of interest is present within any of the plurality of tiles; The system of claim 7 , comprising:

10. The operation comprises: determining an expression level of a biomarker in a region of interest, a category of a biomarker in a region of interest, and a presence of a biomarker in a region of interest by determining a first expression score for each tile of the target electronic image and determining a second expression score for the portion of the target sample by aggregating the first expression scores for each of the plurality of tiles. The system of claim 7 further comprising:

11. Determining expression levels of biomarkers in the region of interest, categories of biomarkers in the region of interest, and presence of biomarkers in the region of interest includes: clustering the plurality of expression levels into one or more diagnostic categories, the one or more diagnostic categories representing a scale of presence of the biomarkers within the region of interest; classifying the biomarkers in the region of interest by identifying a diagnostic category from among the one or more diagnostic categories; The system of claim 7 , comprising:

12. 8. The system of claim 7, wherein outputting the determined expression level of the biomarker in the region of interest, the determined category of the biomarker in the region of interest, and the determined presence of the biomarker in the region of interest includes displaying a visual indicator regarding the expression level of the biomarker, the category of the biomarker, and the presence of the biomarker, and displaying one or more treatment options based on the expression level of the biomarker, the category of the biomarker, and the presence of the biomarker.

13. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image corresponding to a sample; The method comprises: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image; applying a second machine learning algorithm that focuses on the one or more salient regions of the target electronic image to identify a region of interest of the target sample and determine an expression level of a biomarker in the region of interest, a category of a biomarker in the region of interest, and a presence of a biomarker in the region of interest, the biomarker comprising at least one of an epidermal growth factor receptor (EGFR) biomarker and / or a DNA mismatch repair (MMR) deficiency biomarker, the second machine learning algorithm being generated by predicting whether a region of interest is present in the target electronic image by processing a plurality of training images, the plurality of training images comprising images of human tissue and / or algorithmically generated images, determining the expression level of the biomarker comprises dividing the one or more salient regions of the target electronic image into a plurality of tiles, and the second machine learning algorithm determining for each tile the expression level corresponding to a prediction of the presence of the biomarker in the region of interest; outputting the determined expression level of the biomarker in the region of interest, the determined category of the biomarker in the region of interest, and the determined presence of the biomarker in the region of interest; A non-transitory computer readable medium comprising:

14. 14. The non-transitory computer readable medium of claim 13, wherein identifying an area of ​​interest in the target sample comprises determining whether an area of ​​interest is present within any of the plurality of tiles.

15. Identifying a region of interest in the target sample comprises: Segmenting the target electronic image based on edges and / or contrast, segmenting the target electronic image using color difference, and / or supervised decision making; determining whether a region of interest is present within any of the plurality of tiles; 14. The non-transitory computer readable medium of claim 13, comprising:

16. The method according to claim 1, determining an expression level of a biomarker in a region of interest, a category of a biomarker in a region of interest, and a presence of a biomarker in a region of interest by determining a first expression score for each tile of the target electronic image and determining a second expression score for the portion of the target sample by aggregating the first expression scores for each of the plurality of tiles.

14. The non-transitory computer readable medium of claim 13, further comprising:

17. Determining expression levels of biomarkers in the region of interest, categories of biomarkers in the region of interest, and presence of biomarkers in the region of interest includes: clustering the plurality of expression levels into one or more diagnostic categories, the one or more diagnostic categories representing a scale of presence of the biomarkers within the region of interest; classifying the biomarkers in the region of interest by identifying a diagnostic category from among the one or more diagnostic categories; 14. The non-transitory computer readable medium of claim 13, comprising:

Images