System and method for processing image of slide for digital pathology

Abstract

To provide a system and a method for processing an image of a slide for suitable digital pathology.SOLUTION: A system and a method include receiving a target electronic image corresponding to a target sample, the target sample including a tissue sample from a patient, and applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image. The machine learning system is generated by processing a plurality of training images and predicting at least one characteristic, the training images including an image of a human tissue and / or an algorithmically generated image, and for outputting, on the basis of at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the target electronic image identifying an area of interest.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 897,745, filed September 9, 2019, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE Various embodiments of the present disclosure relate generally to image-based sample analysis and related image processing methods. More specifically, certain embodiments of the present disclosure relate to systems and methods for identifying sample properties and providing integrated pathology workflows based on processing images of tissue samples. [Background technology]

[0003] (background) To use digital pathology images within a hospital or research environment, it may be important to identify and categorize the tissue type of the sample, the nature of the sample's acquisition (e.g., prostate needle biopsy, breast biopsy, mastectomy, etc.), and other relevant properties of the sample or image.

[0004] A method for providing an integrated pathology workflow based on the processing of images of tissue samples is desired. The following disclosure is directed to systems and methods for providing user interfaces and artificial intelligence (AI) tools that can be integrated into the workflow to facilitate and improve a pathologist's workflow.

[0005] The foregoing general description and the following detailed description are exemplary and explanatory only and are not limitations of the present disclosure. The background provided herein is generally for the purpose of providing a context for the present disclosure. Unless otherwise indicated herein, the matter described in this section is not, by inclusion in this section, made prior art to the claims of this application, nor is it an admission of or suggestion of prior art. Summary of the Invention [Problem to be solved by the invention]

[0006] (summary) According to an aspect of the present disclosure, a system and method are disclosed for identifying sample properties and providing an integrated pathology workflow based on processing images of tissue samples. [Means for solving the problem]

[0007] A computer-implemented method for analyzing an electronic image corresponding to a sample includes receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting the at least one characteristic, the training images comprising images of a human and / or algorithmically generated images; and outputting the target electronic image, identifying an area of ​​interest based on the at least one characteristic of the target sample and / or the at least one characteristic of the target electronic image.

[0008] A system for analyzing an electronic image corresponding to a sample includes a memory that stores instructions; and a processor that executes the instructions and performs a process including receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting the at least one characteristic, the training images comprising images of human tissue and / or algorithmically generated images; and outputting the target electronic image that identifies an area of ​​interest based on the at least one characteristic of the target sample and / or the at least one characteristic of the target electronic image.

[0009] A non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image corresponding to a sample, the method including: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting the at least one characteristic, the training images comprising images of a human and / or algorithmically generated images; and outputting the target electronic image, identifying an area of ​​interest based on the at least one characteristic of the target sample and / or the at least one characteristic of the target electronic image.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments as claimed. The present specification also provides, for example, the following items: (Item 1) 1. A computer-implemented method for analyzing an electronic image corresponding to a specimen, the method comprising: receiving a target electronic image corresponding to a target sample, the target sample comprising a tissue sample from a patient; applying a machine learning system to the target electronic image to determine at least one property of the target sample and / or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and / or algorithmically generated images; outputting the target electronic image identifying an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image; 11. A computer-implemented method comprising: (Item 2) Item 10. The computer-implemented method of item 1, wherein identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image. (Item 3) Item 10. The computer-implemented method of item 1, wherein identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains an anomaly. (Item 4) identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains an anomaly; Item 10. The computer-implemented method of item 1, wherein the heat map overlay is transparent or semi-transparent. (Item 5) displaying a magnification window over at least a portion of the target electronic image; presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image; Item 1. The computer-implemented method of item 1, further comprising: (Item 6) displaying a magnification window over at least a portion of the target electronic image; presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image; further comprising Item 10. The computer-implemented method of item 1, wherein the magnification window comprises a selectable icon for toggling a heatmap overlay on the magnified image. (Item 7) displaying a slide tray tool over the target electronic image that identifies an outline of the target sample; applying the machine learning system to the target electronic image to determine whether a portion of the target sample contains an anomaly; and in response to determining that the portion contains an anomaly, presenting an indicator of the anomaly within the slide tray tool; Item 1. The computer-implemented method of item 1, further comprising: (Item 8) Item 10. The computer-implemented method of item 1, further comprising displaying an annotation log comprising an indicator identifying the area of ​​interest and a consultation request associated with the area of ​​interest. (Item 9) receiving a second target electronic image corresponding to the target sample; determining a first portion of the target sample associated with the target electronic image; determining a second portion of the target sample associated with the second target electronic image; identifying whether the first portion and the second portion are the same or overlap; displaying a first representation of the target electronic image and a second representation of the second target electronic image in a predetermined proximity to one another in response to identifying the first portion and the second portion as the same or overlapping; determining whether there is an area of ​​interest associated with the target electronic image and / or an area of ​​interest associated with the second target electronic image; displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of ​​interest is associated with the target electronic image; displaying an indicator associated with a second representation of the second target electronic image in response to determining that the area of ​​interest is associated with the second target electronic image; and Item 1. The computer-implemented method of item 1, further comprising: (Item 10) 1. A system for analyzing an electronic image corresponding to a specimen, the system comprising: at least one memory for storing instructions; At least one processor and Equipped with the at least one processor executes the instructions; receiving a target electronic image corresponding to a target sample, the target sample comprising a tissue sample from a patient; applying a machine learning system to the target electronic image to determine at least one property of the target sample and / or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and / or algorithmically generated images; outputting the target electronic image identifying an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image; A system that implements a process including (Item 11) Item 11. The system of item 10, wherein identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image. (Item 12) Item 11. The system of item 10, wherein identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains an anomaly. (Item 13) identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains an anomaly; Item 11. The system of item 10, wherein the heatmap overlay is transparent or semi-transparent. (Item 14) displaying a magnification window over at least a portion of the target electronic image; presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image; Item 11. The system of item 10, further comprising: (Item 15) displaying a magnification window over at least a portion of the target electronic image; presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image; further comprising Item 11. The system of item 10, wherein the magnification window comprises a selectable icon for toggling a heatmap overlay on the magnified image. (Item 16) displaying a slide tray tool over the target electronic image that identifies an outline of the target sample; applying the machine learning system to the target electronic image to determine whether a portion of the target sample contains an anomaly; and in response to determining that the portion contains an anomaly, presenting an indicator of the anomaly within the slide tray tool; Item 11. The system of item 10, further comprising: (Item 17) 11. The system of claim 10, further comprising displaying an annotation log comprising an indicator identifying the area of ​​interest and a consultation request associated with the area of ​​interest. (Item 18) receiving a second target electronic image corresponding to the target sample; determining a first portion of the target sample associated with the target electronic image; determining a second portion of the target sample associated with the second target electronic image; identifying whether the first portion and the second portion are the same or overlap; displaying a first representation of the target electronic image and a second representation of the second target electronic image in a predetermined proximity to one another in response to identifying the first portion and the second portion as the same or overlapping; determining whether there is an area of ​​interest associated with the target electronic image and / or an area of ​​interest associated with the second target electronic image; displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of ​​interest is associated with the target electronic image; displaying an indicator associated with a second representation of the second target electronic image in response to determining that the area of ​​interest is associated with the second target electronic image; and Item 11. The system of item 10, further comprising: (Item 19) A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an electronic image corresponding to a sample, the method comprising: receiving a target electronic image corresponding to a target sample, the target sample comprising a tissue sample from a patient; applying a machine learning system to the target electronic image to determine at least one property of the target sample and / or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and / or algorithmically generated images; outputting the target electronic image identifying an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image; 1. A non-transitory computer-readable medium comprising: (Item 20) 20. The non-transitory computer-readable medium of claim 19, wherein identifying the area of ​​interest comprises displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains an anomaly. [Brief explanation of the drawings]

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0012] [Figure 1A] FIG. 1A illustrates an exemplary block diagram of a system and network for determining sample property or image property information for a digital pathology image, according to an exemplary embodiment of the present disclosure.

[0013] [Figure 1B] FIG. 1B illustrates an exemplary block diagram of a disease detection platform 100 according to an exemplary embodiment of the present disclosure.

[0014] [Figure 1C] FIG. 1C illustrates an example block diagram of a browsing application tool 101 according to an example embodiment of the present disclosure.

[0015] [Figure 1D] FIG. 1D illustrates an exemplary block diagram of a browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0016] [Figure 2] FIG. 2 is a flowchart illustrating an example method for identifying an area of ​​interest and outputting a target image, according to one or more example embodiments of the present disclosure.

[0017] [Figure 3] FIG. 3 illustrates an example output of the browsing application tool 101 according to an example embodiment of the present disclosure.

[0018] [Figure 4] FIG. 4 illustrates an example output of an overlay tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0019] [Figure 5]FIG. 5 illustrates an example output of a task list tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0020] [Figure 6] FIG. 6 illustrates an example output of a slide tray tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0021] [Figure 7] FIG. 7 illustrates an example output of a slide sharing tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0022] [Figure 8] FIG. 8 illustrates an example output of an annotation tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0023] [Figure 9] FIG. 9 illustrates an example output of an annotation tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0024] [Figure 10] FIG. 10 illustrates an example output of an annotation tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0025] [Figure 11] FIG. 11 illustrates an example output of a research tool of the browsing application tool 101 according to an example embodiment of the present disclosure.

[0026] [Figure 12] FIG. 12 illustrates an example output of a research tool of the browsing application tool 101 according to an example embodiment of the present disclosure.

[0027] [Figure 13]FIG. 13 illustrates an example workflow of the browsing application tool 101 according to an example embodiment of the present disclosure.

[0028] [Figure 14A] 14A, 14B, and 14C illustrate example outputs for a slide view of the viewing application tool 101 according to an example embodiment of the present disclosure. [Figure 14B] 14A, 14B, and 14C illustrate example outputs for a slide view of the viewing application tool 101 according to an example embodiment of the present disclosure. [Figure 14C] 14A, 14B, and 14C illustrate example outputs for a slide view of the viewing application tool 101 according to an example embodiment of the present disclosure.

[0029] [Figure 15-1] FIG. 15 illustrates an exemplary workflow for creating an account for a new user and requesting a consultation, according to an exemplary embodiment of the present disclosure. [Figure 15-2] FIG. 15 illustrates an exemplary workflow for creating an account for a new user and requesting a consultation, according to an exemplary embodiment of the present disclosure.

[0030] [Figure 16A] 16A and 16B illustrate example outputs for a pathology consultation dashboard according to an example embodiment of the present disclosure. [Figure 16B] 16A and 16B illustrate example outputs for a pathology consultation dashboard according to an example embodiment of the present disclosure.

[0031] [Figure 17] FIG. 17 illustrates an example output of a case view for the viewing application tool 101, according to an example embodiment of the present disclosure.

[0032] [Figure 18] FIG. 18 depicts an example system that can implement the techniques presented herein. DETAILED DESCRIPTION OF THE INVENTION

[0033] (Description of the embodiment) Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0034] The systems, devices, and methods disclosed herein are described in detail, by way of example, with reference to the Figures. The examples discussed herein are examples only and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be construed as essential for any particular implementation of any of these devices, systems, or methods, unless specifically designated as essential.

[0035] Also, with respect to any method described, whether or not the method is described in conjunction with a flow diagram, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in the execution of the method does not imply that the steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

[0036] As used herein, the term "exemplary" is used in the sense of "example," as opposed to "ideal." Furthermore, the terms "a" and "an," as used herein, do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced item.

[0037] Pathology refers to the study of disease. More specifically, pathology refers to the performance of tests and analyses used to diagnose disease. For example, a tissue sample may be placed on a slide to be viewed under a microscope by a pathologist (e.g., a medical doctor, an expert who analyzes tissue samples and determines whether any abnormalities are present). That is, a pathology specimen may be cut into multiple sections, stained, and prepared as a slide for the pathologist to examine and render a diagnosis. When the diagnostic findings on the slide are uncertain, the pathologist may order additional sections, stains, or other tests to gather further information from the tissue. A technician may then create a new slide, which may contain additional information for the pathologist to use in making a diagnosis. This process of creating additional slides can be time-consuming, not only because it may involve retrieving a block of tissue, cutting it, creating a new slide, and then staining the slide, but also because it may be batched for multiple orders. This can significantly delay the final diagnosis rendered by the pathologist. Additionally, even after the delay, there may still be no guarantee that the new slides will have enough information to render a diagnosis.

[0038] A computer can be used to analyze images of tissue samples and quickly identify whether additional information about a particular tissue sample may be needed and / or highlight areas that may need to be examined more closely by a pathologist. Thus, the process of obtaining additional stained slides and tests can be done automatically before being reviewed by a pathologist. When paired with automated slide sectioning and staining machinery, this can provide a fully automated slide preparation pipeline. This automation has the advantage of at least (1) minimizing the amount of time spent by a pathologist determining that a slide is insufficient to make a diagnosis; (2) minimizing the (average total) time from sample acquisition to diagnosis by avoiding additional time between the time additional tests are ordered and the time they are produced; (3) reducing the amount of time and amount of material wasted per re-cut by allowing re-cutting to occur while the tissue block (e.g., pathology sample) is in the cutting desk; (4) reducing the amount of tissue material required during slide preparation; (5) reducing the cost of slide preparation by partially or fully automating the procedure; (6) enabling automated, customized cutting and staining of slides that will result in more representative / informative slides from the sample; (7) reducing the overhead of requiring additional tests on the pathologist, allowing a larger number of slides to be generated per tissue block, contributing to a more informative / precise diagnosis; and / or (8) identifying or collating the correct characteristics (e.g., with respect to sample type) of digital pathology images, etc.

[0039] The process of using computers to assist pathologists is known as computational pathology. Computing methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but is not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. The use of computational pathology can save lives by helping pathologists improve their diagnostic accuracy, reliability, efficiency, and accessibility. For example, computational pathology may be used to assist in detecting slides that are suspicious for cancer, thereby allowing pathologists to check and confirm their initial assessment before rendering a final diagnosis.

[0040] Histopathology refers to the study of specimens mounted on slides. For example, a digital pathology image may consist of a digitized image of a microscope slide containing a specimen (e.g., a smear). One method a pathologist may use is to analyze the image on the slide to identify nuclei and classify them as normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying nuclei, histological stains may be used to visualize cells. Many dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method in which hematoxylin stains cell nuclei blue, eosin stains cytoplasm and extracellular matrix pink, and other tissue regions take on variations of these colors. However, in many cases, H&E-stained histological preparations do not provide sufficient information for pathologists to visually identify biomarkers that may aid diagnosis or guide treatment. In this situation, techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH), or fluorescence in situ hybridization (FISH) may be used. IHC and immunofluorescence involve the use of antibodies that bind to specific antigens within tissues and allow visual detection of cells expressing specific proteins of interest, which may reveal biomarkers that are not reliably identifiable to trained pathologists based on the analysis of H&E-stained slides. ISH and FISH may be employed to assess the number of gene copies or the abundance of specific RNA molecules, depending on the type of probe employed (e.g., DNA probes for gene copy number and RNA probes for assessing RNA expression). If these methods also fail to provide sufficient information to detect some biomarkers, genetic testing of tissue may be used to confirm whether the biomarker is present (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a given gene in a cancer).

[0041] Digitized images may be prepared to represent stained microscope slides, allowing pathologists to manually view the images on the slides and estimate the number of stained abnormal cells within the image. However, this process can be time-consuming and can lead to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computer processes and devices may be used to assist pathologists in detecting abnormalities that may otherwise be difficult to detect. For example, AI may be used to predict biomarkers (such as overexpression of proteins and / or gene products, amplification or mutation of specific genes, etc.) from prominent areas within digital images of tissue stained using H&E and other dye-based methods. The images of tissue may be whole slide images (WSIs), images of tissue cores within microarrays, or selected areas of interest within tissue sections. Using staining methods such as H&E, these biomarkers may be difficult for humans to visually detect or quantify without the aid of additional testing. Using AI to infer these biomarkers from digital images of tissue has the potential to improve patient treatment while also being faster and less expensive.

[0042] The detected biomarkers or images alone can then be used to recommend specific cancer drugs or drug combination therapies for treating the patient, and AI can identify drugs or drug combinations with a high likelihood of success by correlating the detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendations of immunotherapy drugs to target a patient's specific cancer. Furthermore, this can be used to enable personalized cancer treatment for specific subsets of patients and / or rarer cancer types.

[0043] In today's pathology field, providing systematic quality control ("QC") for pathology sample preparation and quality assurance ("QA") for diagnostic quality throughout the histopathology workflow can be challenging. Systematic quality assurance is challenging because it can require duplicated efforts by two pathologists and is resource- and time-intensive. Some methods for quality assurance include (1) a second review of initially diagnosed cancer cases, (2) periodic review of discordant or changed diagnoses by a quality assurance committee, and (3) random review of a subset of cases. These are non-comprehensive, primarily retrospective, and manual. Using automated and systematic QC and QA mechanisms, quality can be ensured on a case-by-case basis throughout the workflow. Laboratory quality control and digital pathology quality control can be critical to the successful collection, processing, diagnosis, and archiving of patient samples. Manual and sampling approaches to QC and QA offer substantial advantages. Systematic QC and QA have the potential to provide efficiencies and improve diagnostic quality.

[0044] As described above, the disclosed computational pathology process and device may integrate with laboratory information systems (LIS) to provide a unified platform, enabling a fully automated process, including data collection, processing, and viewing of digital pathology images via a web browser or other user interface. Furthermore, clinical information may be aggregated using cloud-based data analytics of patient data. Data may originate from hospitals, clinics, field researchers, etc., and may be analyzed with machine learning, computer vision, natural language processing, and / or statistical algorithms to provide real-time monitoring and prediction of health patterns at multiple levels of geographic specificity.

[0045] The present disclosure presents an integrated workflow for facilitating disease detection and / or cancer diagnosis by providing a workflow that integrates, for example, slide evaluation, tasks, image analysis and cancer detection AI, annotation, consultation, and recommendations in one workstation. The present disclosure describes various exemplary user interfaces available within the workflow, as well as AI tools that can be integrated into the workflow to facilitate and assist the pathologist's work.

[0046] FIG. 1A illustrates a block diagram of a system and network for providing a workflow for determining and outputting sample property or image property information for digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure.

[0047] 1A illustrates an electronic network 120 that may be connected to servers in a hospital, laboratory, and / or doctor's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, etc., may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to exemplary embodiments of the present application, the electronic network 120 may also be connected to a server system 127, which may include a processing device configured to implement the disease detection platform 100, according to exemplary embodiments of the present disclosure, including a viewing application tool 101 for using machine learning to determine and output sample property or image property information for digital pathology images.

[0048] The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may create or otherwise acquire images of one or more patient cytology samples, histopathology samples, slides of cytology samples, digitized images of histopathology sample slides, or any combination thereof. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsy or cytology information, etc. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 may transmit the digitized slide images and / or patient-specific information to the server system 127 via the electronic network 120. The server system 127 may include one or more storage devices 126 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The server system 127 may also include a processing device for processing the images and data stored in the storage device 126. The server system 127 may further include one or more machine learning tools or capabilities. For example, the processing device may include machine learning tools for the disease detection platform 100, according to one embodiment. Alternatively, or in addition, the present disclosure (or portions of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0049] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review slide images. In a hospital setting, tissue type information may be stored within LIS 125. However, the correct tissue classification information is not always paired with image content. Additionally, even when an LIS is used to access the sample type for digital pathology images, this indication may be incorrect due to the fact that many components of the LIS are manually entered and may leave a large margin of error. According to exemplary embodiments of the present disclosure, sample type and / or other sample information may be identified without requiring access to LIS 125, or potentially identified to the correct LIS 125. For example, a third party may be given anonymized access to image content without the corresponding sample type indication stored within the LIS. Additionally, access to LIS content may be limited due to its sensitive content.

[0050] FIG. 1B illustrates an exemplary block diagram of a disease detection platform 100 for using machine learning to determine and output sample property or image property information for digital pathology images.

[0051] 1B depicts components of a disease detection platform 100, according to one embodiment. For example, the disease detection platform 100 may include a viewing application tool 101, a data collection tool 102, a slide capture tool 103, a slide scanner 104, a slide manager 105, and / or a storage device 106.

[0052] The viewing application tool 101, as described below, may refer to processes and systems for providing a user (e.g., a pathologist) with sample property and / or image property information related to a digital pathology image, according to exemplary embodiments. The information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0053] Data harvesting tools 102 refer to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and devices used to classify and process the digital pathology images, according to an example embodiment.

[0054] Slide capture tool 103 refers to a process and system for scanning pathology images and converting them into digital form, according to an exemplary embodiment. Slides may be scanned using slide scanner 104, and slide manager 105 may process the images on the slides into digitized pathology images and store the digitized images in storage device 106.

[0055] The viewing application tool 101 and its components may each transmit and / or receive digitized slide images and / or patient information to and / or from a server system 127, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125 via a network 120. Additionally, the server system 127 may include a storage device for storing images and data received from at least one of the viewing application tool 101, the data collection tool 102, the slide capture tool 103, the slide scanner 104, and / or the slide manager 105. The server system 127 may also include a processing device for processing images and data stored in the storage device. The server system 127 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively, or in addition, the present disclosure (or portions of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0056] Any of the above devices, tools, and modules may be located on devices that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0057] FIG. 1C illustrates an exemplary block diagram of a viewing application tool 101 according to an exemplary embodiment of the present disclosure. The viewing application tool 101 may include a worklist tool 107, a slide tray tool 108, a slide share tool 109, and annotation tools 110, research tools 111, and / or overlay tools 112. The worklist tool 107 may provide an overview of the end-to-end workflow for slide viewing and / or case management. The slide tray tool 108 may organize the slides of a case into parts and provide high-level case information, including case number, demographic information, etc. The slide share tool 109 may provide users with the ability to share various slides and include and / or write brief notes about the nature of the shares. The annotation tools 110 may include a brush tool, an auto brush tool, a trajectory tool, a pinning tool, an arrow tool, a text field tool, a detailed area tool, a ROI tool, a prediction tool, a measurement tool, a multi-measurement tool, an annotation tool, and / or a screenshot tool. The investigation tool 111 may provide an investigation window featuring a magnified view of an area of ​​interest on the target image. The overlay tool may provide a heat map overlay on the magnified view of the area of ​​interest on the target image to identify the area of ​​interest on the tissue sample in the magnified view of the target image.

[0058] 1D illustrates an example block diagram of a viewing application tool 101 according to an example embodiment of the present disclosure. The viewing application tool 101 may include a training image platform 131 and / or a target image platform 135.

[0059] According to one embodiment, the training image platform 131 may include a training image capture module 132 and / or an image analysis module 133 .

[0060] According to one embodiment, training image platform 131 may create or receive training images that are used to train the machine learning system to effectively analyze and classify digital pathology images. For example, training images may be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. Images used for training may be derived from real sources (e.g., humans, animals, etc.) or synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains, such as (but not limited to) H&E, hematoxylin only, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices, such as micro-CT.

[0061] The training image capture module 132 may create or receive a dataset comprising one or more training images corresponding to one or both of images of human tissue and graphically / synthetically rendered images. For example, the training images may be received from any one or combination of the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The dataset may be maintained on a digital storage device. The image analysis module 133 may analyze the images to identify sample property and / or image property information (e.g., sample type, overall quality of the sample cut, overall quality of the glass pathology slide itself, and / or tissue morphology and structural characteristics).

[0062] According to one embodiment, the target image platform 135 may include a target image capture module 136, a sample detection module 137, and an output interface 138. The target image platform 135 may receive a target image and apply a machine learning system to the received target image to determine characteristics of the target sample. For example, the target image may be received from any one or any combination of the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The target image capture module 136 may receive a target image corresponding to the target sample. The sample detection module 137 may apply a machine learning system to the target image to determine characteristics of the target sample and / or characteristics of the target image. For example, the sample detection module 137 may detect a sample type of the target sample. Furthermore, the sample detection module 137 may apply a machine learning system to determine whether an area of ​​the sample includes one or more anomalies.

[0063] The output interface 138 may be used to output the target image and information about the target sample (eg, to a screen, monitor, storage device, web browser, etc.).

[0064] 2 is a flowchart illustrating an exemplary method of providing a viewing application tool 101 for identifying sample information and providing a user interface for viewing the sample information, according to an exemplary embodiment of the present disclosure. For example, exemplary method 200 (e.g., steps 202-206) may be performed by viewing application tool 101 automatically or in response to a request from a user (e.g., a physician, pathologist, technician, etc.).

[0065] According to one embodiment, an exemplary method 200 for identifying sample information and providing a user interface for viewing the sample information may include one or more of the following steps: In step 202, the method may include receiving a target image corresponding to the target sample, the target sample comprising a patient tissue sample. For example, the target image may be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125.

[0066] In step 204, the method may include applying a machine learning system to the target image to determine at least one characteristic of the target sample and / or at least one characteristic of the target image. Determining the characteristic of the target sample may include determining sample information for the target sample. For example, determining the characteristic may include determining whether an anomaly is present in the target sample.

[0067] The machine learning system may be generated by processing multiple training images and predicting at least one characteristic, where the training images may include images of human tissue and / or algorithmically / synthetically generated images. The machine learning system may be implemented using machine learning methods for classification and regression. The training inputs may include real or synthetic images. The training inputs may or may not be augmented (e.g., adding noise or creating variants of the inputs by inversion / distortion). Exemplary machine learning systems may include, but are not limited to, any one or any combination of neural networks, convolutional neural networks, random forests, logistic regression, and nearest neighbor methods. While convolutional neural networks can directly learn the image feature representations necessary to discriminate characteristics, which can work very well when there is a large amount of data to train on per sample, other methods can be used in conjunction with traditional computer vision features, such as speed-up robust features (SURF) or scale-invariant feature transform (SIFT), or learned embeddings (e.g., descriptors) generated by a trained convolutional neural network, which can provide advantages when there is only a small amount of data to train on. Training images may be received from any one or any combination of the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. This dataset may be kept on a digital storage device. Images used for training may be from real sources (e.g., humans, animals, etc.) or synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as (but not limited to) H&E, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices such as microCT.

[0068] In step 206, the method may include outputting a target image that identifies an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target image.

[0069] Different methods for implementing machine learning algorithms and / or architectures may include, but are not limited to, (1) CNN (Convolutional Neural Network), (2) MIL (Multiple Instance Learning), (3) RNN (Recurrent Neural Network), (4) feature aggregation via CNN, and / or (5) feature extraction followed by ensemble methods (e.g., Random Forest), linear / nonlinear classifiers (e.g., SVM (Support Vector Machine), MLP (Multilayer Perceptron), and / or dimensionality reduction techniques (e.g., PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), etc.). Exemplary features may include vector embeddings from CNN, single / multi-class output from CNN, and / or multidimensional output from CNN (e.g., mask overlay of the original image). CNNs may learn feature representations for classification tasks directly from pixels, which may lead to better diagnostic performance. When detailed annotations for regions or per-pixel labels are available, CNNs can be used to directly However, when the labeling is only at the global slide level or across a collection of slides within a group (which in pathology may be called a "part"), MIL may be used to train a CNN or another neural network classifier, with MIL learning image regions that are diagnostic for classification tasks, leading to the ability to learn without global annotations. RNNs may be used on features extracted from multiple image regions (e.g., tiles), which are then processed and predictions are made. Other machine learning methods, such as random forests, SVMs, and many others, may be used in conjunction with either features learned by a CNN, a CNN with MIL, or by using hand-crafted image features (e.g., SIFT or SURF) to perform classification tasks, but they may perform poorly when trained directly from pixels. These methods may perform poorly compared to CNN-based systems when there is a large amount of annotated training data available.Dimensionality reduction techniques can be used as a preprocessing step before using any of the mentioned classifiers, which can be useful when there is little data available.

[0070] According to one or more embodiments, any of the above algorithms, architectures, methodologies, attributes, and / or features may be combined with any or all of the other algorithms, architectures, methodologies, attributes, and / or features. For example, any of the machine learning algorithms and / or architectures (e.g., neural network methods, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.) may be trained using any of the training methodologies (e.g., multiple instance learning, reinforcement learning, activation learning, etc.).

[0071] The following explanations of terms are merely illustrative and are not intended to limit the terms in any way.

[0072] The indicators may refer to information about the inputs to the machine learning algorithm that the algorithm attempts to predict.

[0073] For a given image size of N×M, the segmentation may be another image of size N×M that assigns to each pixel in the original image a number that describes the class or type of that pixel. For example, in WSI, elements in the mask may categorize each pixel in the input image as belonging to, for example, background, tissue, and / or unknown classes.

[0074] Slide-level information may refer to information about a slide generally, but not necessarily to the specific location of that information within the slide.

[0075] A heuristic may refer to a logical rule or function that deterministically generates an output given an input. For example, a prediction that a slide has an anomaly outputs a 1 if present and a 0 if not.

[0076] Embedding may refer to a conceptual high-dimensional numerical representation of low-dimensional data. For example, if a WSI is passed through a CNN training to classify tissue types, the numbers on the last layer of the network may provide a series (e.g., on the order of thousands) of numbers containing information about the slide (e.g., information about the tissue type).

[0077] Slide-level predictions may refer to concrete predictions for the slide as a whole. For example, a slide-level prediction may be that the slide contains an anomaly. Additionally, slide-level predictions may refer to individual probability predictions across a defined set of classes.

[0078] A classifier may refer to a model and / or system that is trained to take input data and associate a category with it.

[0079] According to one or more embodiments, the machine learning model and / or system may be trained in different ways. For example, training of the machine learning model may be performed by any one or any combination of supervised training, semi-supervised training, unsupervised training, classifier training, hybrid training, and / or uncertainty estimation. The type of training used may depend on the amount, type, and / or quality of data. Table 1 below describes a non-limiting list of some types of training and corresponding features. [Table 1]

[0080] Supervised training may be used in conjunction with small amounts of data to provide seeds for machine learning models, which may look for specific items (e.g., bubbles, tissue folds, etc.), flag slides, and quantify the amount of the specific items present in the slide.

[0081] According to one embodiment, exemplary fully supervised training may take WSI as input and may include indicators of segmentation. A pipeline for fully supervised training may include (1) 1, (2) 1, heuristic, (3) 1, 4, heuristic, (4) 1, 4, 5, heuristic, and / or (5) 1, 5, heuristic. Advantages of fully supervised training may be that (1) it may require fewer slides and / or (2) the output is interpretable because the area of ​​the image that contributed to the diagnosis may be known. A disadvantage of using fully supervised training may be that it may require a large amount of segmentation, which may be difficult to obtain.

[0082] According to one embodiment, exemplary semi-supervised (e.g., weakly supervised) training may take WSI as input and may include indicators of slide-level information. A pipeline for semi-supervised training may include (1) 2, (2) 2, heuristic, (3) 2, 4, heuristic, (4) 2, 4, 5, heuristic, and / or (5) 2, 5, heuristic. An advantage of using semi-supervised training may be that the output is interpretable because (1) the type of indicator required may be present in many hospital records and (2) the area of ​​the image that most contributed to the diagnosis may be known. A disadvantage of using semi-supervised training is that it may be difficult to train. For example, the system may need to use training schemes such as multiple-instance learning, activation learning, and / or distributed training to account for the fact that there is limited information about the location in the slide where the information that should lead to a decision is located.

[0083] According to one embodiment, exemplary unsupervised training may take a WSI as input and may not require a label. A pipeline for unsupervised training may include (1) 3, 4, and / or (2) 3, 4, heuristics. An advantage of unsupervised training may be that it does not require any label. Disadvantages of using unsupervised training may be that (1) it may be difficult to train. For example, using training schemes such as multiple instance learning, activation learning, and / or distributed training may need to account for the fact that there is limited information about the location within a slide where the information that should lead to a decision is located; (2) it may require additional slides; and / or (3) it may not be very interpretable because it may output predictions and probabilities without explaining why the predictions were made.

[0084] According to one embodiment, exemplary mixed training may involve training any of the exemplary pipelines described above with respect to fully supervised training, semi-supervised training, and / or unsupervised training, and then using the resulting model as an initial point for any of the training methods. Advantages of mixed training may be that (1) it may require less data, (2) it may have improved performance, and / or (3) it may allow for the mixing of different levels of labels (e.g., segmentation, slide-level information, no information). Disadvantages of mixed training may be that (1) training may be more complex and / or expensive, and / or (2) it may require more code, which may increase the number and complexity of potential bugs.

[0085] According to one embodiment, exemplary uncertainty estimation may include training any of the exemplary pipelines described above with respect to fully supervised, semi-supervised, and / or unsupervised training for any task involving slide data using uncertainty estimation at the end of the pipeline. Additionally, heuristics or classifiers may be used to predict whether a slide has an anomaly based on the amount of uncertainty in the test prediction. An advantage of uncertainty estimation may be that it is robust to out-of-distribution data. For example, when presented with unfamiliar data, it may still correctly predict uncertainty. Disadvantages of uncertainty estimation may be that (1) more data may be required, (2) it may have poor overall performance, and / or (3) it may be less interpretable because the model may not necessarily identify the degree to which a slide or slide embedding is anomalous.

[0086] According to one embodiment, ensemble training may involve simultaneously launching models generated by any of the exemplary pipelines described above and combining the outputs via heuristics or classifiers to produce robust and accurate results. The advantages of ensemble training may be that it (1) is robust to out-of-distribution data and / or (2) may combine advantages and disadvantages of other models, resulting in minimization of disadvantages (e.g., a supervised training model combined with an uncertainty estimation model, and a heuristic using the supervised model when the incoming data is within the distribution and the uncertainty model when the data is out-of-distribution). The disadvantages of ensemble training may be that it (1) may be more complex and / or (2) may be expensive to train and launch.

[0087] The training techniques discussed herein may also be phased, with images with more annotations being used for training initially, which may allow for more effective subsequent training using less supervised, etc. slides with fewer annotations.

[0088] Training may begin using the most completely annotated slides compared to all available training slide images. For example, training may begin using supervised learning. A first set of slide images may be received or determined using associated annotations. Each slide may have marked and / or masked regions and may include information such as whether the slide has an anomaly. The first set of slides may be provided to a training algorithm, such as a CNN, which may determine correlations between the first set of slides and their associated annotations.

[0089] After training using the first set of images is completed, a second set of slide images may be received or determined using fewer annotations than the first set, e.g., partial annotations. In one embodiment, the annotations may only indicate that the slide has a diagnosis or quality issue associated with it, but may not specify the disease or its location where it may be found. The second set of slide images may be trained using a different training algorithm than the first, e.g., multiple instance learning. The first set of training data may be used to partially train the system, making the second training process more effective in generating accurate algorithms.

[0090] In this manner, training may proceed in any number of stages based on the quality and type of training slide images using any number of algorithms. These techniques may be utilized in situations where multiple sets of training images are received, which may be of varying quality, annotation level, and / or annotation type.

[0091] FIG. 3 illustrates an exemplary output 300 of the viewing application tool 101, according to an exemplary embodiment. As illustrated in FIG. 3, the viewing application tool 101 may include a navigation menu 301 for navigating to a work list view for the work list tool 107, a slide view for the slide tray tool 108 and slide share tool 109, an annotation tool 110, an investigation tool 111, and / or an overlay toggle for the overlay tool 112. The navigation menu 301 may also include a view mode input for adjusting the view (e.g., splitting the screen and viewing multiple slides and / or stains at once). A zoom menu 302 may be used to quickly adjust the zoom level of the target image 303. The target image may be displayed in any color. For example, a dark neutral color scheme may be used to display details of the target image 303. The viewing application tool 101 may include a positive / negative indicator for whether a possible disease is present at an (x,y) coordinate in the image. Additionally, the positive / negative indicators may be viewed and / or edited by the user.

[0092] Slides may be prioritized using prioritization features based on characteristics identified in the target image. Slides may be organized using a folder system, including default and custom systems. The viewing application tool 101 may include a case view, which may include results, notes, attachments, patient information, and status icons. Slides within a case may be organized by type and color-coded. The case sharing feature may be designed to be secure and reliable. Cases may be archived to provide necessary storage space. Functionality may exist for switching between masses within a single slide. The viewing application tool 101 may provide notifications for new messages and / or new slides. Status icons may indicate whether a case and / or slide is new and whether it has been shared and / or ordered. Functionality may exist for searching cases, patients, and / or projects. The viewing application tool 101 may include a patient queue and / or a consultation queue.

[0093] The viewing application tools 101 may include viewing options, which may include a viewing mode, window, rotation, scale bar, and / or overlay. Exemplary outputs of the viewing application tools 101 may include patient information, slide information (e.g., image overview), and / or file information (e.g., slide map). Viewing modes may include a default view and viewing modes for different ratios. The viewing application tools 101 may include a fluorescence mode, a magnifying glass view, a scale bar, pan and zoom functionality, image rotation functionality, focus of interest, and / or other slide actions (e.g., ordering a new stain). The viewing application may analyze the images and determine the patient's viability.

[0094] The viewing application tools 101 may include functionality for ordering slides and levels and for recommending slides. For example, the viewing application tools 101 may include options for selecting a slide type, submitting a new slide order, receiving a slide order confirmation, a slide ordering indicator, viewing a new slide, and / or comparing a new slide with a previous slide. The viewing application tools 101 may also include functionality for recommending levels and selecting levels.

[0095] The viewing application tool 101 may include functionality for searching for similar cases using the viewing application tool 101, according to an example embodiment. For example, areas of interest may be identified, the areas of interest may be rotated and quantified, and cases with similar areas of interest may be identified and linked to the patient case.

[0096] The viewing application tool 101 may include functionality for providing mitotic counts. For example, areas of interest may be identified, the areas of interest may be moved / rotated and quantified, and cases with similar areas of interest may be identified and linked to patient cases. The viewing application tool 101 may display the count results, including the number of mitoses and / or visual markers. Identified areas may be zoomed in, and the count results may be reviewed and / or edited by the user. Results may be shared across the system.

[0097] FIG. 4 illustrates an exemplary output 400 of the overlay tool 112 of the viewing application tool 101, according to an exemplary embodiment. The overlay tool 112 may provide a heatmap overlay 401 on the target image 303. The heatmap overlay may identify areas of interest on a tissue sample of the target image. For example, the heatmap overlay may identify areas in which an AI system predicts that an abnormality may be present in the tissue sample. The predictive heatmap overlay visualization may be toggled on and off using the navigation menu 301. For example, a user may select an overlay icon on the navigation menu and toggle the heatmap overlay on or off. The predictive heatmap interface may indicate to a user (e.g., a pathologist) one or more areas on the tissue that the user should investigate. The heatmap overlay may be transparent and / or semi-transparent, so that the user can see the underlying tissue while viewing the heatmap overlay. The heatmap overlay may include different colors and / or shading to indicate the severity of detected disease. Other types of overlays may also be available in context to the type of tissue being viewed. For example, Figure 4 illustrates a prostate biopsy. However, other diseases may require different visualizations or AI systems associated with them.

[0098] FIG. 5 illustrates an example output 500 of the worklist tool 107 of the viewing application tool 101, according to an exemplary embodiment. As illustrated in FIG. 5, the viewing application tool 101 may include a worklist 501 that can be opened by selecting a worklist icon in the navigation menu 301. The main navigation menu 301 may provide an overview of the end-to-end workflow for slide viewing and / or case management. For example, the overview may include a worklist 501 that can display all user (e.g., pathologist) cases. This may be accessed from the viewer user interface (UI) by opening a worklist panel. Within the worklist panel, a pathologist may view the patient's medical record number (MRN), patient ID number, patient name, suspected disease type, case status, surgery date, and select from various case actions, such as viewing case details, sharing the case, and / or searching.

[0099] FIG. 6 illustrates an example output 600 of the slide tray tool 108 of the viewing application tool 101, according to an exemplary embodiment. As illustrated in FIG. 6, a slide icon in the navigation menu 301 may open the slide tray 601 as a drawer / drop-down from the menu on the left of the screen. The menu may present high-level case information, including case number, demographic information, etc. The slides of a case may be organized into parts and represented in the slide tray 601 as parent tabs in a dark gray color. Each part may expand to display the slides within it. Thus, slides determined to be associated with the same part, which may be a predetermined area of ​​a sample or a predetermined sample area of ​​a patient, may be displayed within a predetermined proximity to each other. A red dot 602, which may be a different color or shape indicator, on a part may indicate that the AI ​​system has found a possible disease (e.g., cancer) at a location within that part. A red dot 603 on a slide may indicate that the AI ​​system has found a possible disease at a location on that slide. Those parts and slides where the AI ​​system finds possible diseases may be brought to the top of the list for immediate viewing by a user (e.g., a pathologist). This workflow may help pathologists identify diseases more quickly.

[0100] FIG. 7 illustrates an exemplary output 700 of the slide share tool 109 of the view application tool 101, according to an exemplary embodiment. From the slide panel, a user may select the “Share” button and select single, multiple, and / or all slides to share. The slide share panel 701 may be located to the right of the slide tray 601 and may allow a user to enter recipients for sharing, select various slides to include, and / or write a brief comment about the nature of the share. Once the send button is selected, the slides and brief commentary may be sent to the recipient, who may receive a notification and view these slides and / or commentary within their view application. Interactions between users may be captured in a log, described below.

[0101] FIG. 8 illustrates an example output 800 of the annotation tool 110 of the viewing application tool 101, according to an exemplary embodiment. For each annotation made, the annotation log 801 may capture the appearance of the actual annotation made, the type of annotation, any measurements, the annotation, the user who made the annotation, and / or the time of the annotation. The annotation log 801 may serve as a centralized view for consultation and information sharing to be reviewed within the context of the annotation or area of ​​interest. The user requesting the consultation (e.g., a pathologist) and the consulting pathologist may read annotations, make annotations, and / or have ongoing dialogue specific to each annotation, each accompanied by a timestamp. The user may also select a thumbnail in the annotation log 801 to view that area of ​​interest at a larger scale in the main viewer window.

[0102] 9 illustrates an example output of the annotation tool 110 of the viewing application tool 101, according to an example embodiment. For example, the annotation log illustrated in FIG. 9 includes a conversation between pathologists to discuss a slide. Pathologists may submit notes along with the annotated image for quick consultation about any areas of interest on the slide.

[0103] FIG. 10 illustrates an example output 1000 of the annotation tool 110 of the viewing application tools 101, according to an example embodiment. The viewing application may include various annotation tools, as illustrated in the annotation menu 1001. For example, the annotation tools may include a brush tool, an auto brush tool, a trajectory tool, a pinning tool, an arrow tool, a text field tool, a detailed area tool, a ROI tool, a prediction tool, a measurement tool, a multi-measurement tool, a lesion area drawing tool, a region of interest tool, an add tag tool, an annotation tool, and / or a screenshot tool. Annotations may be used for research purposes and / or in clinical settings. For example, a pathologist may make annotations, write annotations, and / or share them with colleagues to obtain second opinions. Annotations may also be used in a final diagnosis report as supporting evidence of a diagnosis (e.g., tumor is x length, has y number of mitotic counts, etc.).

[0104] FIG. 11 illustrates an example output 1100 of the investigation tool 111 of the viewing application tool 101, according to an exemplary embodiment. The investigation tool 111 may include an investigation window 1101 featuring a magnified view of an area of ​​interest for a target image. Based on user input, the investigation window may be dragged across an image to quickly examine the image. Thus, a pathologist may be able to quickly move across a slide in a manner similar to the speed at which a slide is currently moved within a microscopy examination. A user may quickly switch or toggle a predictive heat map overlay on and off while browsing through the investigation tool 111. A user may take screenshots of tissue viewable within the investigation tool and quickly share the screenshots. Different annotation tools, such as measurement tools and / or area highlighting, may also be available with the investigation tool. Additionally, a user may increase and decrease the magnification within the investigation tool independently of the main slide magnification level.

[0105] 12 illustrates an example output 1200 of the investigation tool 111 of the viewing application tool 101, according to an example embodiment. As illustrated in FIG. 12, the investigation window 1201 may display AI output in a magnified view to verify and assist in diagnosis. The investigation window 1201 may include a heatmap overlay that displays characteristics of the tissue sample predicted by the AI ​​system. For example, the heatmap overlay may identify areas in which the AI ​​system predicts that abnormalities may be present in the tissue sample.

[0106] 13 illustrates an exemplary workflow of the viewing application tool 101, according to an exemplary embodiment. The workflow may include a worklist, a case view, and / or a slide view as modes within a user's (e.g., a pathologist's) workflow. As illustrated in FIG. 13, a diagnostic report may be output from the case view, which may include information necessary for digital signature.

[0107] 14A, 14B, and 14C illustrate exemplary outputs for a slide view of the viewing application tool 101, according to an exemplary embodiment. For example, as illustrated at the bottom of FIG. 14A, a slide view toolbar 1401 may be located at the bottom of the display with a vertical orientation of the slides. Slides may be grouped by part and display related case information within that context. As illustrated in FIG. 14B, toolbar 1401 may be located on the left side of the display, within which the toolbar may include menu items such as tools, zoom functions, and thumbnail slides. As illustrated in FIG. 14C, two vertical toolbars 1401 and 1402 may be located on the left and right sides of the screen, which may allow easy access to various functions for touchscreen devices (e.g., tools and / or views).

[0108] FIG. 15 illustrates an exemplary workflow for a new user (e.g., patient, physician, etc.) to create an account for requesting and receiving consultations, according to an exemplary embodiment. A request to create a new account may be submitted through the hospital website and / or through the viewing application tool 101. Once an account is created, a pathologist may submit notes along with annotated images for quick consultation on any areas of interest on a slide. The patient may also receive consultation information from the pathologist and / or billing information related to their case. If the user is a physician, billing options may differ based on whether the party being billed is a referring physician, the patient, and / or an insurance company. If the user is a patient, the patient may be billed directly or through an insurance company.

[0109] 16A and 16B illustrate exemplary outputs for a pathology consultation dashboard, according to an exemplary embodiment. As illustrated in FIG. 16A, new requests and registered consultations may include patient name, referring physician, external ID, internal ID, facility, requested date, status, updated date, and / or a link to additional details. The consultation dashboard may include functionality for selecting slides to send, a send slides dialog box, consulting request notes, consulting attachments, and / or a consultation log. As illustrated in FIG. 16B, consultation requests may include patient name, referring physician, facility, requested date, status, updated date, and / or a link to further details about the request.

[0110] 17 illustrates an example output of a case view for the viewing application tool 101, according to an example embodiment. The case view may include all of the necessary information for a digital stamp workflow. In addition, as illustrated in FIG. 17, an annotation log may be integrated into the case view.

[0111] As shown in Figure 18, device 1800 may include a central processing unit (CPU) 1820. CPU 1820 may be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As will be understood by those skilled in the art, CPU 1820 may also be a single processor in a multi-core / multi-processor system, such a system operating alone or within a cluster of computing devices operating within a cluster or server farm. CPU 1820 may be connected to a data communications infrastructure 1810, for example, a bus, a message queue, a network, or a multi-core message passing scheme.

[0112] The device 1800 may also include a main memory 1840, e.g., random access memory (RAM), and may also include a secondary memory 1830. The secondary memory 1830, e.g., read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. The removable storage drive, in this embodiment, reads from and / or writes to a removable storage unit in a well-known manner. The removable storage unit may comprise a floppy disk, magnetic tape, optical disk, etc., which is read and written by the removable storage drive. As will be appreciated by those skilled in the art, such a removable storage unit generally includes a computer-usable storage medium having computer software and / or data stored therein.

[0113] In alternative implementations, secondary memory 1830 may include other similar means for allowing computer programs or other instructions to be loaded into device 1800. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 1800.

[0114] Device 1800 may also include a communications interface (“COM”) 1860. Communications interface 1860 allows software and data to be transferred between device 1800 and external devices. Communications interface 1860 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 1860 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1860. These signals may be provided to communications interface 1860 over a communications path of device 1800, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0115] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature and are assumed to be fully familiar to those skilled in the art. Device 1800 may also include input and output ports 1850 for connecting to input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of one computer hardware platform.

[0116] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. Components and modules can be implemented in software, hardware, or a combination of software and hardware.

[0117] The tools, modules, and functions described above may be implemented by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which may provide optional non-transitory storage for software programming.

[0118] The software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, the communication may allow the software to be loaded from one computer or processor into another. As used herein, unless limited to non-transitory, the term tangible "storage" medium, such as computer or machine "readable medium," refers to any medium that participates in providing instructions to a processor for execution.

[0119] The foregoing general description is exemplary and explanatory only and is not a limitation of the present disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only.

Claims

1. 1. A computer-implemented method for analyzing an electronic image corresponding to a specimen, the method comprising: receiving a first target electronic image corresponding to a target sample, the target sample comprising a tissue sample from a patient; applying a machine learning system to the first target electronic image to determine at least one property of the target sample and / or at least one property of the first target electronic image; outputting the first target electronic image identifying an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the first target electronic image; receiving a second target electronic image associated with the target sample; determining a first portion of the target specimen associated with the first target electronic image; determining a second portion of the target specimen associated with the second target electronic image; the machine learning system identifying whether the first portion and the second portion are the same or overlap; displaying a first representation of the first target electronic image and a second representation of the second target electronic image in a predetermined proximity to one another in response to identifying the first portion and the second portion as the same or overlapping; 11. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains a biomarker.

3. further comprising displaying a magnification window; The computer-implemented method of claim 2 , wherein the magnification window comprises a selectable icon for toggling the heat map overlay on the magnified image.

4. displaying a slide tray tool over the first target electronic image that identifies an outline of the target sample; applying the machine learning system to the first target electronic image to determine whether the portion of the target sample contains a biomarker; presenting an indicator of the biomarker in the slide tray tool in response to determining that the portion contains a biomarker. The computer-implemented method of claim 1 , further comprising:

5. The computer-implemented method of claim 1 , further comprising displaying an annotation log comprising an indicator identifying the area of ​​interest and a consultation request associated with the area of ​​interest.

6. Determining whether there is an area of ​​interest associated with the first target electronic image and / or an area of ​​interest associated with the second target electronic image by applying machine learning techniques; displaying an indicator associated with a first representation of the first target electronic image in response to determining that the area of ​​interest associated with the first target electronic image exists; displaying an indicator associated with a second representation of the second target electronic image in response to determining that the area of ​​interest associated with the second target electronic image exists; and The computer-implemented method of claim 1 , further comprising:

7. Displaying a magnification window over at least a portion of the first target electronic image; and presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the first target electronic image; further comprising The computer-implemented method of claim 1 , wherein the magnification window comprises one or more selectable icons that change the magnification level within the magnification window.

8. Receiving a plurality of target electronic images associated with the target sample; the machine learning system determining one or more portions of the target sample associated with the plurality of target electronic images; the machine learning system determining whether the one or more portions of the target specimen in the plurality of target electronic images are the same or overlap; responsive to determining that any of the one or more portions of the target specimen in the plurality of target electronic images are the same or overlapping, simultaneously displaying the target electronic images including the same or overlapping portions in a predetermined proximity to each other; The computer-implemented method of claim 1 , further comprising:

9. The machine learning system is capable of analyzing all of the first target electronic images and all of the second target electronic images to determine whether any area of ​​either image overlaps or is the same; upon determining that one or more areas overlap or are the same, displaying the first target electronic image and the second target electronic image in a predetermined proximity to each other; The computer-implemented method of claim 1 , further comprising:

10. The machine learning system further comprising determining a predicted likelihood that a location contains a biomarker; The computer-implemented method of claim 1 , wherein the biomarkers include indications of overexpression of proteins and / or gene products, amplification or mutation of specific genes.

11. 1. A system for analyzing an electronic image corresponding to a specimen, the system comprising: at least one memory for storing instructions; at least one processor; Equipped with the at least one processor executes the instructions; receiving a first target electronic image corresponding to a target sample, the target sample comprising a tissue sample from a patient; applying a machine learning system to the first target electronic image to determine at least one property of the target sample and / or at least one property of the first target electronic image; outputting the first target electronic image identifying an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the first target electronic image; receiving a second target electronic image associated with the target sample; determining a first portion of the target specimen associated with the first target electronic image; determining a second portion of the target specimen associated with the second target electronic image; the machine learning system identifying whether the first portion and the second portion are the same or overlap; displaying a first representation of the first target electronic image and a second representation of the second target electronic image in a predetermined proximity to one another in response to identifying the first portion and the second portion as the same or overlapping; A system that performs operations including:

12. 12. The system of claim 11, wherein identifying the area of ​​interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains a biomarker.

13. further comprising displaying a magnification window; The system of claim 12 , wherein the magnification window comprises a selectable icon for toggling the heat map overlay on the magnified image.

14. The operation is displaying a slide tray tool over the first target electronic image that identifies an outline of the target sample; applying the machine learning system to the first target electronic image to determine whether the portion of the target sample contains a biomarker; presenting an indicator of the biomarker in the slide tray tool in response to determining that the portion contains a biomarker. The system of claim 11 further comprising:

15. The system of claim 11, wherein the operations further include displaying an annotation log with an indicator identifying the area of ​​interest and a consultation request related to the area of ​​interest.

16. The operation is determining whether an area of ​​interest associated with the first target electronic image and / or an area of ​​interest associated with the second target electronic image exists by applying machine learning techniques; displaying an indicator associated with a first representation of the first target electronic image in response to determining the area of ​​interest associated with the first target electronic image; displaying an indicator associated with a second representation of the second target electronic image in response to determining the area of ​​interest associated with the second target electronic image; and The system of claim 11 further comprising:

17. The system of claim 11, wherein the operation further includes displaying a magnification window moving across the first target electronic image in response to user input.

18. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an electronic image corresponding to a sample, the method comprising: receiving a first target electronic image corresponding to a target sample, the target sample comprising a tissue sample from a patient; applying a machine learning system to the first target electronic image to determine at least one property of the target sample and / or at least one property of the first target electronic image; outputting the first target electronic image identifying an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the first target electronic image; receiving a second target electronic image associated with the target sample; determining a first portion of the target specimen associated with the first target electronic image; determining a second portion of the target specimen associated with the second target electronic image; the machine learning system identifying whether the first portion and the second portion are the same or overlap; displaying a first representation of the first target electronic image and a second representation of the second target electronic image in a predetermined proximity to one another in response to identifying the first portion and the second portion as the same or overlapping; 1. A non-transitory computer-readable medium comprising:

19. 20. The non-transitory computer-readable medium of claim 18, wherein identifying the area of ​​interest comprises displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on a predicted likelihood that a location contains a biomarker.

20. The non-transitory computer-readable medium of claim 18, wherein the method further includes displaying a magnification window moving across the first target electronic image in response to user input.

Images