System and method for processing slide images for digital pathology

Abstract

To provide a system and a method for processing an image of a slide for suitable digital pathology.SOLUTION: A system and a method include receiving a target electronic image corresponding to a target sample, the target sample including a tissue sample from a patient, and applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image. The machine learning system is generated by processing a plurality of training images and predicting at least one characteristic, the training images including an image of a human tissue and / or an algorithmically generated image, and for outputting, on the basis of at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the target electronic image identifying an area of interest.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 897,745, filed on September 9, 2019, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] (Field of the Disclosure) Various embodiments of the present disclosure generally relate to image-based sample analysis and related image processing methods. More specifically, certain embodiments of the present disclosure relate to systems and methods for identifying sample properties and providing an integrated pathology workflow based on processing images of tissue samples.

Background Art

[0003] (Background) For using digital pathology images within a hospital or research environment, it can be important to identify and categorize the tissue type of the sample, the nature of sample acquisition (e.g., prostate needle biopsy, breast biopsy, mastectomy, etc.), and other relevant properties of the sample or image.

[0004] A method for providing an integrated pathology workflow based on processing images of tissue samples is desired. The following disclosure is directed to systems and methods for providing a user interface and artificial intelligence (AI) tools that are integrated into the workflow and can facilitate and improve the pathologist's work solution.

[0005] The foregoing general description and the following detailed description are exemplary and explanatory only and are not limitations of the present disclosure. The background description provided herein is generally for the purpose of presenting the context of the present disclosure. Unless otherwise indicated herein, the matters described in this section are not prior art to the claims of the present application by virtue of their inclusion in this section, nor are they admitted, or suggested by, prior art.

Summary of the Invention

[0006] (summary) According to one aspect of this disclosure, a system and method for identifying sample properties and providing an integrated pathology workflow based on the processing of images of tissue samples are disclosed. [Means for solving the problem]

[0007] A computer implementation method for analyzing an electronic image corresponding to a sample includes receiving a target electronic image corresponding to a target sample, wherein the target sample comprises a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, wherein the machine learning system is generated by processing a plurality of training images and predicting at least one characteristic, wherein the training images comprise human images and / or algorithmically generated images; and outputting a target electronic image that identifies an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

[0008] A system for analyzing an electronic image corresponding to a sample includes a memory for storing instructions, a processor for executing instructions and receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample, applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images and predicting at least one characteristic, the training images comprising images of human tissue and / or algorithmically generated images, and outputting a target electronic image that identifies an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

[0009] A non-transient computer-readable medium, when executed by a processor, stores instructions causing the processor to perform a method for analyzing an image corresponding to a sample, the method comprising: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images and predicting at least one characteristic, the training images comprising human images and / or algorithmically generated images; and outputting a target electronic image that identifies an area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

[0010] Please understand that both the general description above and the detailed description below are illustrative and descriptive only, and do not limit the embodiments disclosed as claimed. This specification also provides, for example, the following items: (Item 1) A computer implementation method for analyzing an electronic image corresponding to a sample, wherein the method is The process involves receiving a target electronic image corresponding to a target sample, wherein the target sample comprises a patient's tissue sample. The method involves applying a machine learning system to the target electron image and determining at least one characteristic of the target sample and / or at least one characteristic of the target electron image, wherein the machine learning system is generated by processing multiple training images and predicting at least one characteristic, and the training images include images of human tissue and / or algorithmically generated images. Based on at least one characteristic of the target sample and / or at least one characteristic of the target electron image, the target electron image is output to identify the area of ​​interest. Computer implementation methods, including those mentioned above. (Item 2) The computer implementation method according to item 1, wherein identifying the area of ​​interest includes displaying a heatmap overlay on the target electronic image. (Item 3) Identifying the area of ​​interest comprises displaying a heatmap overlay on the target electronic image, wherein the heatmap overlay includes shading and / or coloring based on the predicted likelihood that a location contains an anomaly, according to the computer implementation method of item 1. (Item 4) Identifying the area of ​​interest includes displaying a heatmap overlay on the target electronic image, wherein the heatmap overlay includes shading and / or coloring based on the predicted likelihood that a location contains an anomaly. The computer implementation method described in item 1, wherein the heatmap overlay is transparent or semi-transparent. (Item 5) Displaying a magnification window over at least a portion of the target electronic image, Within the magnification window, an enlarged image of the target sample is presented at a magnification level different from that of the target electron image. The computer implementation method described in item 1, further including the method described in item 1. (Item 6) Displaying a magnification window over at least a portion of the target electronic image, Within the magnification window, an enlarged image of the target sample is presented at a magnification level different from that of the target electron image. It further includes, The computer implementation method according to item 1, wherein the magnification window comprises selectable icons for toggling a heatmap overlay on the magnified image. (Item 7) A slide tray tool that identifies the outline of the target sample is displayed on the target electronic image, The machine learning system is applied to the target electronic image to determine whether a part of the target sample contains abnormalities, In response to determining that the aforementioned portion contains an abnormality, the indicator of the abnormality is displayed within the slide tray tool. The computer implementation method described in item 1, further including the method described in item 1. (Item 8) The computer implementation method according to item 1, further comprising displaying an annotation log having an indicator for identifying the area of ​​interest, and a consultation request related to the area of ​​interest. (Item 9) Receiving a second target electron image corresponding to the aforementioned target sample, To determine the first portion of the target sample associated with the target electronic image, Determining a second portion of the target sample associated with the second target electron image, Identifying whether the first part and the second part are the same or overlap, In response to identifying that the first portion and the second portion are the same or overlap, the first representation of the target electron image and the second representation of the second target electron image are displayed relative to each other at a predetermined degree of proximity. To determine whether there is a focus area associated with the aforementioned target electron image and / or a focus area associated with the aforementioned second target electron image, In response to determining that the area of ​​interest is associated with the target electron image, an indicator associated with a first representation of the target electron image is displayed. In response to determining that the area of ​​interest is associated with the second target electron image, an indicator associated with the second representation of the second target electron image is displayed. The computer implementation method described in item 1, further including the method described in item 1. (Item 10) A system for analyzing an electronic image corresponding to a sample, wherein the system is At least one memory for storing instructions, At least one processor and Equipped with, The at least one processor executes the instructions and receives a target electronic image corresponding to a target sample, the target sample comprising a tissue sample of a patient, and applies a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images and predicting at least one characteristic, the training images comprising images of human tissue and / or algorithmically generated images, and outputs the target electronic image identifying the area of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image A system that performs a process including (Item 11) The system according to item 10, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image. (Item 12) The system according to item 10, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on the predicted likelihood that a location contains an abnormality. (Item 13) Identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and / or coloring based on the predicted likelihood that a location contains an abnormality, The system according to item 10, wherein the heat map overlay is transparent or translucent. (Item 14) displaying a magnification window over at least a portion of the target electronic image, and presenting an enlarged image of the target sample at a magnification level different from the magnification level of the target electronic image within the magnification window The system described in item 10, which further includes the system described in item 10. (Item 15) Displaying a magnification window over at least a portion of the target electronic image, Within the magnification window, an enlarged image of the target sample is presented at a magnification level different from that of the target electron image. It further includes, The system according to item 10, wherein the magnification window comprises selectable icons for toggling a heatmap overlay on the magnified image. (Item 16) A slide tray tool that identifies the outline of the target sample is displayed on the target electronic image, The machine learning system is applied to the target electronic image to determine whether a part of the target sample contains abnormalities, In response to determining that the aforementioned portion contains an abnormality, the indicator of the abnormality is displayed within the slide tray tool. The system described in item 10, which further includes the system described in item 10. (Item 17) The system according to item 10, further comprising displaying an annotation log having an indicator for identifying the area of ​​interest, and consultation requests related to the area of ​​interest. (Item 18) Receiving a second target electron image corresponding to the aforementioned target sample, To determine the first portion of the target sample associated with the target electronic image, Determining a second portion of the target sample associated with the second target electron image, Identifying whether the first part and the second part are the same or overlap, In response to identifying that the first portion and the second portion are the same or overlap, the first representation of the target electron image and the second representation of the second target electron image are displayed relative to each other at a predetermined degree of proximity. To determine whether there is a focus area associated with the aforementioned target electron image and / or a focus area associated with the aforementioned second target electron image, In response to determining that the area of ​​interest is associated with the target electron image, an indicator associated with a first representation of the target electron image is displayed. In response to determining that the area of ​​interest is associated with the second target electron image, an indicator associated with the second representation of the second target electron image is displayed. The system described in item 10, which further includes the system described in item 10. (Item 19) A non-transient computer-readable medium that, when executed by a processor, stores instructions causing the processor to perform a method for analyzing an electronic image corresponding to a sample, and the method is The process involves receiving a target electronic image corresponding to a target sample, wherein the target sample comprises a patient's tissue sample. The method involves applying a machine learning system to the target electron image and determining at least one characteristic of the target sample and / or at least one characteristic of the target electron image, wherein the machine learning system is generated by processing multiple training images and predicting at least one characteristic, and the training images include images of human tissue and / or algorithmically generated images. Based on at least one characteristic of the target sample and / or at least one characteristic of the target electron image, the target electron image is output to identify the area of ​​interest. Non-transient computer-readable media, including [specific examples of such media]. (Item 20) Identifying the area of ​​interest comprises displaying a heatmap overlay on the target electronic image, the heatmap overlay comprising shading and / or coloring based on the predicted likelihood that a location contains an anomaly, in a non-transient computer-readable medium as described in item 19. [Brief explanation of the drawing]

[0011] The accompanying drawings, incorporated and constituting part thereof within this specification, illustrate various exemplary embodiments and, together with descriptions, serve to illustrate the principles of the disclosed embodiments.

[0012] [Figure 1A] Figure 1A illustrates an exemplary block diagram of a system and network for determining sample properties or image property information relating to digital pathology images, according to an exemplary embodiment of the present disclosure.

[0013] [Figure 1B] Figure 1B illustrates an exemplary block diagram of a disease detection platform 100 according to an exemplary embodiment of the present disclosure.

[0014] [Figure 1C] Figure 1C illustrates an exemplary block diagram of a browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0015] [Figure 1D] Figure 1D illustrates an exemplary block diagram of a browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0016] [Figure 2] Figure 2 is a flowchart illustrating an exemplary method for outputting a target image that identifies an area of ​​interest, according to one or more exemplary embodiments of the present disclosure.

[0017] [Figure 3] Figure 3 illustrates the exemplary output of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0018] [Figure 4] Figure 4 illustrates the exemplary output of the overlay tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0019] [Figure 5]Figure 5 illustrates the exemplary output of the work list tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0020] [Figure 6] Figure 6 illustrates the exemplary output of the slide tray tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0021] [Figure 7] Figure 7 illustrates an exemplary output of the slide sharing tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0022] [Figure 8] Figure 8 illustrates the exemplary output of the annotation tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0023] [Figure 9] Figure 9 illustrates the exemplary output of the annotation tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0024] [Figure 10] Figure 10 illustrates the exemplary output of the annotation tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0025] [Figure 11] Figure 11 illustrates the exemplary output of the research tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0026] [Figure 12] Figure 12 illustrates the exemplary output of the research tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0027] [Figure 13]Figure 13 illustrates an exemplary workflow of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0028] [Figure 14A] Figures 14A, 14B, and 14C illustrate exemplary output for a slide view of the browsing application tool 101 according to an exemplary embodiment of the present disclosure. [Figure 14B] Figures 14A, 14B, and 14C illustrate exemplary output for a slide view of the browsing application tool 101 according to an exemplary embodiment of the present disclosure. [Figure 14C] Figures 14A, 14B, and 14C illustrate exemplary output for a slide view of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0029] [Figure 15-1] Figure 15 illustrates an exemplary workflow for creating an account for a new user and requesting consultation, according to an exemplary embodiment of the present disclosure. [Figure 15-2] Figure 15 illustrates an exemplary workflow for creating an account for a new user and requesting consultation, according to an exemplary embodiment of the present disclosure.

[0030] [Figure 16A] Figures 16A and 16B illustrate exemplary outputs for a pathology consultation dashboard according to an exemplary embodiment of the present disclosure. [Figure 16B] Figures 16A and 16B illustrate exemplary outputs for a pathology consultation dashboard according to an exemplary embodiment of the present disclosure.

[0031] [Figure 17] Figure 17 illustrates an exemplary output of a case view for a browsing application tool 101 according to an exemplary embodiment of the present disclosure.

[0032] [Figure 18] Figure 18 illustrates an exemplary system capable of performing the techniques presented herein. [Modes for carrying out the invention]

[0033] (Description of the embodiment) Herein, exemplary embodiments of the present disclosure will be referenced in detail, and such embodiments will be illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0034] The systems, devices, and methods disclosed herein are described in detail, with reference to the drawings, as examples. The embodiments discussed herein are merely examples and are provided to aid in the description of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be taken as essential for any specific implementation of any of these devices, systems, or methods unless specifically designated as essential.

[0035] Furthermore, with respect to any method described, whether or not the method is described in conjunction with a flowchart, unless otherwise specified or required by the context, any explicit or implicit ordering of the steps performed during the execution of the method does not imply that those steps must be performed in the order presented, but rather that they may be performed in a different order or in parallel.

[0036] As used herein, the term “exemplary” is used in the sense of “example” rather than “ideal.” Furthermore, the terms “a” and “an” herein do not indicate a limit on quantity, but rather indicate the presence of one or more of the items mentioned.

[0037] Pathology refers to the study of diseases. More specifically, pathology refers to the conduct of tests and analyses used to diagnose diseases. For example, a tissue sample may be placed on a slide for viewing under a microscope by a pathologist (e.g., a physician, a specialist who analyzes tissue samples and determines whether any abnormalities are present). That is, a pathological specimen may be cut into multiple pieces, stained, and prepared as slides for examination and diagnosis by a pathologist. When the diagnostic findings on a slide are uncertain, the pathologist may order additional cutting levels, staining, or other tests to gather further information from the tissue. The technician may then create a new slide, which may contain additional information for the pathologist to use when making a diagnosis. This process of creating additional slides can be time-consuming, not only because it may involve collecting blocks of tissue, cutting them, preparing new slides, and then staining the slides, but also because it may be batched for multiple orders. This can significantly delay the final diagnosis given by the pathologist. Furthermore, even after a delay, there may still be no guarantee that the new slides will contain enough information to provide a diagnosis.

[0038] A computer can be used to analyze images of tissue samples, quickly identify whether additional information about a particular tissue sample may be needed, and / or highlight areas that may require further examination by a pathologist. Thus, the process of obtaining additionally stained slides and tests can be automated before scrutiny by a pathologist. When paired with automated slide compartmentalization and staining machines, this can provide a fully automated slide preparation pipeline. This automation has the advantages of at least (1) minimizing the amount of time spent by pathologists determining whether slides are insufficient for diagnosis, (2) minimizing the (average total) time from sample acquisition to diagnosis by avoiding the additional time between when additional tests are ordered and when they are generated, (3) reducing the amount of time and material wasted per recut by allowing recutting to be performed while the tissue block (e.g., pathology specimen) is in the cutting desk, (4) reducing the amount of tissue material required during slide preparation, (5) reducing the cost of slide preparation by partially or completely automating the procedure, (6) enabling automated customized cutting and staining of slides which will yield more representative / useful slides from the sample, (7) enabling the generation of more slides per tissue block which will contribute to more informative / precise diagnoses by reducing the overhead of requiring additional tests on the pathologist, and / or (8) identifying or matching the correct properties (e.g., with respect to sample type) of digital pathology images, etc.

[0039] The process of using computers to assist pathologists is known as computational pathology. Computational methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and artificial intelligence (AI). AI may include, but are not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. By using computational pathology, lives may be saved by helping pathologists improve the accuracy, reliability, efficiency, and accessibility of their diagnoses. For example, computational pathology may be used to assist in detecting slides suspected of being cancerous, thereby allowing pathologists to check and confirm their initial assessment before giving a final diagnosis.

[0040] Histopathology refers to the study of specimens placed on slides. For example, digital pathology images can consist of digitized images of microscope slides containing specimens (e.g., smears). One method that pathologists may use is to analyze the images on the slides to identify nuclei and classify whether they are normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying nuclei, histological staining may be used to visualize cells. Many dye-based staining systems have been developed, including the periodate Schiff reaction, Masson's trichrome, Nissl and methylene blue, as well as hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method, where hematoxylin stains cell nuclei blue, eosin stains the cytoplasm and extracellular matrix pink, and other tissue areas exhibit variations of these colors. However, in many cases, histological preparations stained with H&E do not provide sufficient information for pathologists to visually identify biomarkers that may aid in diagnosis or guide treatment. In this situation, techniques such as immunohistochemical properties (IHC), immunofluorescence, in-situ hybridization (ISH), or fluorescent in-situ hybridization (FISH) may be used. IHC and immunofluorescence, for example, involve the use of antibodies that enable the visual detection of cells that bind to a specific antigen in the tissue and express a specific protein of interest, which may reveal biomarkers that are not reliably identifiable to a pathologist trained to analyze H&E-stained slides. ISH and FISH may be employed to assess the number of gene copies or the abundance of a specific RNA molecule, depending on the type of probe employed (e.g., DNA probes for gene copy number and RNA probes for assessing RNA expression). If these methods also fail to provide sufficient information to detect certain biomarkers, genetic testing of the tissue may be used to determine whether biomarkers (e.g., overexpression of a specific protein or gene product within a tumor, amplification of a given gene within a cancer) are present.

[0041] Digitized images may be prepared to show stained microscope slides, which may allow pathologists to manually view images on slides and estimate the number of stained abnormal cells in the images. However, this process can be time-consuming and prone to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computer processes and devices may be used to assist pathologists in detecting abnormalities that may otherwise be difficult to detect. For example, AI may be used to predict biomarkers (such as overexpression of proteins and / or gene products, amplification or mutation of specific genes) from prominent areas in digital images of tissue stained using H&E and other dye-based methods. Images of tissue may be whole slide images (WSI), images of tissue cores in a microarray, or selected areas of interest in a tissue section. Using staining methods such as H&E, these biomarkers may be difficult for humans to visually detect or quantify without the aid of additional testing. Inferring these biomarkers from digital images of tissue using AI has the potential to improve patient care, while also being faster and less expensive.

[0042] Only detected biomarkers or images can then be used to recommend specific cancer drugs or drug combination therapies to treat the patient, and AI can identify drugs or drug combinations with a high probability of success by correlating detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendations of immunotherapy drugs to target the patient's specific cancer. Furthermore, this can be used to enable personalized cancer treatment with respect to specific subsets of patients and / or rarer cancer types.

[0043] In today's field of pathology, providing systematic quality control ("QC") for pathology specimen preparation and quality assurance ("QA") for diagnostic quality throughout the entire histopathology workflow can be challenging. Systematic quality assurance is difficult because it is resource and time-intensive, potentially requiring the redundant work of two pathologists. Some methods for quality assurance include (1) a second review of the initial diagnosed cancer case, (2) periodic review of inconsistent or altered diagnoses by a quality assurance committee, and (3) random review of subsets of cases. These are non-comprehensive, primarily retrospective, and manual. By employing automated and systematic QC and QA mechanisms, quality can be ensured case by case and throughout the entire workflow. Laboratory quality control and digital pathology quality control can be crucial for the success of patient specimen collection, processing, diagnosis, and archiving. Manual and sampling approaches to QC and QA offer substantial advantages. Systematic QC and QA have the potential to provide efficiency and improve diagnostic quality.

[0044] As described above, the computer pathology processes and devices of this disclosure may provide an integrated platform, while integrating with a laboratory information system (LIS), enabling a fully automated process, including data acquisition, processing, and viewing of digital pathology images, via a web browser or other user interface. Furthermore, clinical information may be aggregated using cloud-based data analysis of patient data. The data may originate from hospitals, clinics, field researchers, etc., and may be analyzed by machine learning, computer vision, natural language processing, and / or statistical algorithms to enable real-time monitoring and prediction of health patterns at multiple geographical specificity levels.

[0045] This disclosure presents an integrated workflow for facilitating disease detection and / or cancer diagnosis by providing a workflow that integrates, for example, slide evaluation, tasks, image analysis and cancer detection AI, annotation, consultation, and recommendations on a single workstation. This disclosure describes various exemplary user interfaces available within the workflow, as well as AI tools that can be integrated into the workflow to facilitate and assist the work of pathologists.

[0046] Figure 1A illustrates a block diagram of a system and network for providing a workflow for determining and outputting sample or image property information relating to digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure.

[0047] Specifically, Figure 1A illustrates an electronic network 120 that may be connected to servers in a hospital, laboratory, and / or physician's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, etc., may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present application, the electronic network 120 may also be connected to a server system 127, which may include a processing device configured to implement a disease detection platform 100 according to an exemplary embodiment of the present disclosure, which includes a browsing application tool 101 for determining and outputting sample or image property information relating to digital pathology images using machine learning.

[0048] The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may create or otherwise acquire images of cytological samples, histopathological samples, slides of cytological samples, digitized images of slides of histopathological samples, or any combination thereof, from one or more patients. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may also acquire any combination of patient-specific information, such as age, medical history, history of cancer treatment, family history, and past biopsy or cytological information. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may transmit the digitized slide images and / or patient-specific information to the server system 127 via the electronic network 120. The server system 127 may include one or more storage devices 126 for storing images and data received from at least one of the physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The server system 127 may also include processing devices for processing images and data stored in the storage devices 126. The server system 127 may further include one or more machine learning tools or capabilities. For example, the processing device may include a machine learning tool for a disease detection platform 100, according to one embodiment. Alternatively, or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0049] The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to examine slide images. In a hospital setting, tissue type information may be stored in the LIS 125. However, correct tissue classification information is not always paired with image content. In addition, even when the LIS is used to access sample types for digital pathology images, the labeling may be incorrect due to the fact that many components of the LIS are manually entered, which can leave a large margin of error. According to exemplary embodiments of this disclosure, sample types and / or other sample information can be identified without requiring access to the LIS 125, or potentially identified against the correct LIS 125. For example, a third party may be given anonymized access to image content without the corresponding sample type labeling stored in the LIS. In addition, access to LIS content may be limited due to the content requiring careful handling.

[0050] Figure 1B illustrates an exemplary block diagram of a disease detection platform 100 that uses machine learning to determine and output sample or image property information related to digital pathology images.

[0051] Specifically, Figure 1B depicts components of a disease detection platform 100 according to one embodiment. For example, the disease detection platform 100 may include a browsing application tool 101, a data acquisition tool 102, a slide acquisition tool 103, a slide scanner 104, a slide management device 105, and / or a storage device 106.

[0052] The browsing application tool 101 described below may refer to a process and system for providing a user (e.g., a pathologist) with sample properties and / or image properties information relating to digital pathology images, according to exemplary embodiments. The information may be provided through various output interfaces (e.g., screens, monitors, storage devices, and / or web browsers).

[0053] The data acquisition tool 102 refers to a process and system for facilitating the transfer of digital pathology images to various tools, modules, components, and devices used for classifying and processing digital pathology images, according to exemplary embodiments.

[0054] The slide acquisition tool 103 refers to a process and system for scanning pathological images and converting them into a digital format, according to an exemplary embodiment. The slides may be scanned using a slide scanner 104, and the slide management device 105 may process the images on the slides into digitized pathological images and store the digitized images in a storage device 106.

[0055] Each of the browsing application tool 101 and its components may transmit and / or receive digitized slide images and / or patient information via the network 120 to and from the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. Furthermore, the server system 127 may include a storage device for storing images and data received from at least one of the browsing application tool 101, data acquisition tool 102, slide capture tool 103, slide scanner 104, and / or slide management device 105. The server system 127 may also include a processing device for processing the images and data stored in the storage device. The server system 127 may further include, for example, one or more machine learning tools or capabilities due to the processing device. Alternatively, or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0056] Any of the above devices, tools, and modules may be located on a device that can be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0057] Figure 1C illustrates an exemplary block diagram of a browsing application tool 101 according to an exemplary embodiment of the present disclosure. The browsing application tool 101 may include a work list tool 107, a slide tray tool 108, a slide sharing tool 109, and annotation tools 110, a survey tool 111, and / or an overlay tool 112. The work list tool 107 may provide an overview of an end-to-end workflow for slide browsing and / or case management. The slide tray tool 108 may organize case slides into parts and provide high-level case information, including case number, demographic information, etc. The slide sharing tool 109 may provide the user with the ability to share various slides, including and / or write brief annotations about the nature of the sharing. The annotation tools 110 may include a brush tool, an auto brush tool, a trajectory tool, a pinning tool, an arrow tool, a text field tool, a detail area tool, an ROI tool, a prediction tool, a measurement tool, a multi-measurement tool, an annotation tool, and / or a screenshot tool. The investigation tool 111 may provide an investigation window featuring a magnified view of the area of ​​interest on the target image. The overlay tool may provide a heatmap overlay on the magnified view of the area of ​​interest on the target image, and identify the area of ​​interest on the tissue sample in the magnified view of the target image.

[0058] Figure 1D illustrates an exemplary block diagram of a browsing application tool 101 according to an exemplary embodiment of the present disclosure. The browsing application tool 101 may include a training image platform 131 and / or a target image platform 135.

[0059] According to one embodiment, the training image platform 131 may include a training image acquisition module 132 and / or an image analysis module 133.

[0060] The training image platform 131 may, according to one embodiment, create or receive training images used to train a machine learning system and effectively analyze and classify digital pathology images. For example, training images may be received from any one or any combination of the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The images used for training may originate from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as H&E, hematoxylin only, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from a 3D imaging device such as a micro-CT.

[0061] The training image acquisition module 132 may create or receive a dataset comprising one or more training images corresponding to either or both images of human tissue and / or graphically / synthetically rendered images. For example, training images may be received from any one or any combination of the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. This dataset may be stored on a digital storage device. The image analysis module 133 may analyze the images and identify sample properties and / or image property information (e.g., sample type, overall quality of the sample cut, overall quality of the glass pathology slide itself, and / or tissue morphological and structural characteristics).

[0062] According to one embodiment, the target image platform 135 may include a target image acquisition module 136, a sample detection module 137, and an output interface 138. The target image platform 135 may receive a target image, apply a machine learning system to the received target image, and determine the characteristics of the target sample. For example, the target image may be received from any one or any combination of the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The target image acquisition module 136 may receive a target image corresponding to the target sample. The sample detection module 137 may apply a machine learning system to the target image to determine the characteristics of the target sample and / or the characteristics of the target image. For example, the sample detection module 137 may detect the sample type of the target sample. Furthermore, the sample detection module 137 may apply a machine learning system to determine whether the area of ​​the sample contains one or more anomalies.

[0063] The output interface 138 may be used to output information about the target image and target sample (for example, to a screen, monitor, storage device, web browser, etc.).

[0064] Figure 2 is a flowchart illustrating an exemplary method according to an exemplary embodiment of the present disclosure, which provides a browsing application tool 101 for identifying sample information and a user interface for browsing sample information. For example, exemplary method 200 (e.g., steps 202-206) may be performed by the browsing application tool 101 automatically or in response to a request from a user (e.g., a physician, pathologist, technician, etc.).

[0065] According to one embodiment, an exemplary method 200 for identifying sample information and providing a user interface for viewing sample information may include one or more of the following steps. Step 202 may include receiving a target image corresponding to a target sample, the target sample comprising a patient tissue sample. For example, the target image may be received from any one or any combination of the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125.

[0066] Step 204 may include applying a machine learning system to a target image to determine at least one characteristic of the target sample and / or at least one characteristic of the target image. Determining the characteristics of the target sample may include determining sample information of the target sample. For example, determining the characteristics may include determining whether an anomaly is present in the target sample.

[0067] A machine learning system may be generated by processing multiple training images and predicting at least one characteristic, and the training images may include images of human tissue and / or algorithmically / synthetically generated images. The machine learning system may be implemented using machine learning methods for classification and regression. The training input may include real or synthetic images. The training input may be augmented (e.g., by adding noise or creating variants of the input by inversion / distortion) or not. The exemplary machine learning system may include, but is not limited to, any one or any combination of neural networks, convolutional neural networks, random forests, logistic regression, and nearest neighbors. Convolutional neural networks can directly learn the image feature representations necessary to discriminate properties, which can work very well when there is a large amount of data to train on for each sample. Other methods can be used in conjunction with either conventional computer vision features, such as Speed-Up Robust Feature Transfer (SURF) or Scale-Invariant Feature Transfer (SIFT), or learned embeddings (e.g., descriptors) generated by a trained convolutional neural network, which can be advantageous when only a small amount of data to train is available. Training images may be received from any one or any combination of the server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. This dataset may be stored on a digital storage device. Images used for training may originate from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as H&E, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices such as micro-CT.

[0068] Step 206 may include outputting a target image that identifies the area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the target image.

[0069] Different methods for implementing machine learning algorithms and / or architectures may include, but are not limited to, ensemble methods (e.g., random forest), linear / nonlinear classifiers (e.g., SVM (support vector machine), MLP (multilayer perceptron), and / or dimensionality reduction techniques (e.g., PCA (principal component analysis), LDA (linear discriminant analysis), etc.), following (1) CNN (convolutional neural network), (2) MIL (multi-instance learning), (3) RNN (regressive neural network), (4) feature aggregation via CNN, and / or (5) feature extraction. Exemplary features may include vector embeddings from CNNs, single / multiclass outputs from CNNs, and / or multidimensional outputs from CNNs (e.g., mask overlays of the original images). CNNs may learn feature representations directly from pixels for classification tasks, which may lead to better diagnostic performance. When a large amount of labeled data is available, CNNs can directly MIL may be used to train a CNN or another neural network classifier when the labeling is at the overall slide level or only across a set of slides within a group (which in pathology may be called a "part"), and MIL is a diagnostic tool for classification tasks, leading to the ability to learn image regions without comprehensive annotation. RNNs may be used on features extracted from multiple image regions (e.g., tiles), which then process and make predictions. Other machine learning methods, e.g., random forests, SVMs, and many others, may be used in conjunction with any of the learned features by performing classification tasks using CNNs, CNNs with MIL, or manually created image features (e.g., SIFT or SURF), but they may perform poorly when trained directly from pixels. These methods may perform poorly compared to CNN-based systems when a large amount of annotated training data is available.Dimensionality reduction techniques can be used as a preprocessing step before using any of the described classifiers, which can be useful when there is little data available.

[0070] According to one or more embodiments, any of the above algorithms, architectures, methodologies, attributes, and / or features may be combined with any or all of the other algorithms, architectures, methodologies, attributes, and / or features. For example, any of the machine learning algorithms and / or architectures (e.g., neural network methods, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.) may be trained using any of the training methodologies (e.g., multi-instance learning, reinforcement learning, activation learning, etc.).

[0071] The following definitions of terms are illustrative and not intended to limit their meaning in any way.

[0072] A marker can refer to information about the input to a machine learning algorithm that the algorithm attempts to predict.

[0073] Given an image size of N×M, partitioning can be another image of size N×M, where each pixel in the original image is assigned a number that describes the class or type of that pixel. For example, in WSI, elements within a mask can categorize each pixel in the input image as belonging to a class such as background, tissue, and / or unknown.

[0074] Slide-level information generally refers to information about a slide, or, if not necessarily, the specific location of that information within the slide.

[0075] A heuristic can refer to a logical rule or function that deterministically generates an output based on given inputs. For example, it might output 1 if the slide is abnormal, if a prediction exists, and 0 otherwise.

[0076] Embedding can refer to a conceptual high-dimensional numerical representation of low-dimensional data. For example, if WSI is passed through CNN training to classify organization types, the numbers on the final layer of the network may provide a series of numbers (e.g., about several thousand) containing information about the slides (e.g., information about the organization type).

[0077] A slide-level prediction can refer to a concrete prediction about a slide as a whole. For example, a slide-level prediction might indicate that a slide contains anomalies. Furthermore, a slide-level prediction can refer to individual probability predictions across a defined set of classes.

[0078] A classifier can refer to a model and / or system that takes input data and is trained to associate it with categories.

[0079] According to one or more embodiments, machine learning models and / or systems may be trained in different ways. For example, training of a machine learning model may be carried out by any one or any combination of supervised training, semi-supervised training, unsupervised training, classifier training, mixed training, and / or uncertainty estimation. The type of training used may depend on the amount of data, the type of data, and / or the quality of the data. Table 1 below describes an unrestricted list of some types of training and their corresponding features. [Table 1]

[0080] Supervised training may be used in conjunction with a small amount of data to provide a seed for a machine learning model. In supervised training, the machine learning model may search for specific items (e.g., bubbles, tissue folds, etc.), flag them on slides, and quantify the amount of each specific item present on a slide.

[0081] According to one embodiment, exemplary fully supervised training may take WSI as input and may include partitioning labels. A pipeline for fully supervised training may include (1) 1, (2) 1, heuristic, (3) 1, 4, heuristic, (4) 1, 4, 5, heuristic, and / or (5) 1, 5, heuristic. Advantages of fully supervised training may be that (1) fewer slides may be required and / or (2) the area of ​​the images that contributed to the diagnosis can be grasped, so the output may be explainable. Disadvantages of using fully supervised training may be that it may require a large number of partitions, which may be difficult to obtain.

[0082] According to one embodiment, exemplary semi-supervised (e.g., weakly supervised) training may take WSI as input, or it may include slide-level information markers. A pipeline for semi-supervised training may include (1) 2, (2) 2, heuristic, (3) 2, 4, heuristic, (4) 2, 4, 5, heuristic, and / or (5) 2, 5, heuristic. The advantage of using semi-supervised training is that the output may be explainable because (1) the type of marker required may be present in many hospital records, and (2) the area of ​​the image that contributed most to the diagnosis can be identified. The disadvantage of using semi-supervised training is that it may be difficult to train. For example, the system may need to consider the fact that information about the location within the slide is limited, and that there is information that should lead to a decision, using training schemes such as multi-instance learning, activation learning, and / or distributed training.

[0083] According to one embodiment, exemplary unsupervised training may take WSI as input and may not require any markers. A pipeline for unsupervised training may include (1) 3, 4 and / or (2) 3, 4 heuristics. An advantage of unsupervised training may be that it does not require any markers. A disadvantage of using unsupervised training may be that (1) it may be difficult to train. For example, training schemes such as multi-instance learning, activation learning and / or distributed training may need to take into account the fact that there is limited information about the location within the slides, that there is information that should lead to a decision, (2) it may require additional slides, and / or (3) it may not be very explainable because it may output predictions and probabilities without explaining why the predictions were made.

[0084] According to one embodiment, exemplary mixed training comprises training one of the exemplary pipelines described above with respect to fully supervised training, semi-supervised training, and / or unsupervised training, and then using the resulting model as an initial point for any of the training methods. The advantages of mixed training may be that (1) it may require less data, (2) it may have improved performance, and / or (3) it may allow for a mixture of different levels of markers (e.g., partitioning, slide-level information, no information). The disadvantages of mixed training may be that (1) training may be more complex and / or expensive, and / or (2) it may require more code, which may increase the number and complexity of potential bugs.

[0085] According to one embodiment, exemplary uncertainty estimation may involve training one of the exemplary pipelines described above with respect to fully supervised training, semi-supervised training, and / or unsupervised training for any task related to slide data using uncertainty estimation at the end of the pipeline. Furthermore, a heuristic or classifier may be used to predict whether a slide is anomaly based on the amount of uncertainty in the prediction of the test. An advantage of uncertainty estimation may be that it is robust to out-of-distribution data. For example, when unfamiliar data is presented, it can still correctly predict that it is uncertain. Disadvantages of uncertainty estimation may be that (1) it may require more data, (2) it may have poor overall performance, and / or (3) the model may not be very explainable because it may not necessarily identify the degree to which a slide or slide embedding is anomaly.

[0086] According to one embodiment, ensemble training may simultaneously involve invoking models generated by any of the exemplary pipelines described above, and combining the outputs by heuristics or classifiers to produce robust and accurate results. The advantages of ensemble training may be that (1) it is robust to out-of-distribution data, and / or (2) it can combine the advantages and disadvantages of other models, resulting in minimizing the disadvantages (e.g., a supervised training model combined with an uncertainty estimation model, and heuristics using the supervised model when the incoming data is in-distribution and the uncertainty model when the data is out-of-distribution). The disadvantages of ensemble training may be that (1) it can be more complex, and / or (2) it can be expensive to train and invoke.

[0087] The training techniques discussed herein may also be progressive, with images with more annotations initially used for training, which may allow for more effective later training using slides with fewer annotations, less supervision, etc.

[0088] Training may begin with the most fully annotated slide compared to all possible training slide images. For example, training may begin using supervised learning. A first set of slide images is received, or determined, along with its associated annotations. Each slide may have marked and / or masked regions, and may contain information such as whether the slide is anomaly. The first set of slides may be provided to a training algorithm, for example, a CNN, which may determine the correlation between the first set of slides and its associated annotations.

[0089] After training with the first set of images is completed, a second set of slide images may be received or used for evaluation, with fewer annotations than the first set, e.g., with partial annotations. In one embodiment, the annotations may only indicate that the slide has a diagnostic or associated quality problem, and may not specify any disease or its location that may be found. The second set of slide images may be trained using a different training algorithm than the first, e.g., multiple instance learning. The first set of training data may be used to partially train the system, making the second training process more effective in generating an accurate algorithm.

[0090] Thus, training may be carried out in any number of stages based on the quality and type of the training slide images, using any number of algorithms. These techniques may be used in situations where multiple sets of training images are received, which may have varying quality, annotation levels, and / or annotation types.

[0091] Figure 3 illustrates an exemplary output 300 of the browsing application tool 101 according to an exemplary embodiment. As shown in Figure 3, the browsing application tool 101 may include a navigation menu 301 for navigating to a worklist view for the worklist tool 107, a slide view for the slide tray tool 108 and slide sharing tool 109, an annotation tool 110, a survey tool 111, and / or an overlay toggle for the overlay tool 112. The navigation menu 301 may also include a view mode input for adjusting the view (e.g., splitting the screen to view multiple slides and / or stains at once). A zoom menu 302 may be used to quickly adjust the zoom level of the target image 303. The target image may be displayed in any color. For example, a dark neutral color scheme may be used to display details of the target image 303. The browsing application tool 101 may include a positive / negative indicator regarding whether a possible disease is present at the (x,y) coordinates in the image. In addition, the user may verify and / or edit the positive / negative indicators.

[0092] Slides may be prioritized using prioritization features based on characteristics identified within the target image. Slides may be organized using a folder system, including a default system and a custom system. The browsing application tool 101 may include a case view, which may include results, notes, attachments, patient information, and status icons. Slides within a case may be organized by type and color-coded. The case sharing function may be designed to be secure and reliable. Cases may be archived to provide the necessary storage space. Functionality for switching between tumors within a single slide may exist. The browsing application tool 101 may provide notifications for new messages and / or new slides. Status icons may indicate whether a case and / or slide is new and whether it has been shared and / or ordered. Functionality for searching cases, patients, and / or projects may exist. The browsing application tool 101 may include patient queues and / or consulting queues.

[0093] The browsing application tool 101 may include browsing options, which may include browsing modes, windows, rotation, scale bars, and / or overlays. The exemplary output of the browsing application tool 101 may include patient information, slide information (e.g., image summary), and / or file information (e.g., slide map). The browsing modes may include a default view and browsing modes for different aspect ratios. The browsing application tool 101 may include fluorescence modes, magnifying view, scale bars, pan and zoom functions, image rotation functions, focus lesion, and / or other slide actions (e.g., ordering a new stain). The browsing application may analyze images and determine patient survival rates.

[0094] The viewing application tool 101 may include functionality for ordering slides and levels, and for recommending slides. For example, the viewing application tool 101 may include options for selecting a slide type, submitting a new slide order, receiving a slide order confirmation, a slide ordering indicator, viewing a new slide, and / or comparing a new slide with a previous slide. The viewing application tool 101 may also include functionality for recommending and selecting levels.

[0095] The browsing application tool 101 may include functionality for searching for similar cases using the browsing application tool 101, as in the exemplary embodiment. For example, the area of ​​focus may be identified, the area of ​​focus may be rotated and quantified, and cases with similar areas of focus may be identified and associated with patient cases.

[0096] The browsing application tool 101 may include functionality for providing mitotic counts. For example, the area of ​​interest may be identified, the area of ​​interest may be moved / rotated and quantified, and cases with similar areas of interest may be identified and associated with patient cases. The browsing application tool 101 may display the count results, including the number of mitosis and / or visual markers. The identified area may be magnified, and the count results may be reviewed and / or edited by the user. The results may be shared across the system.

[0097] Figure 4 illustrates an exemplary output 400 of the overlay tool 112 of the browsing application tool 101 according to an exemplary embodiment. The overlay tool 112 may provide a heatmap overlay 401 on the target image 303. The heatmap overlay may identify areas of interest on the tissue sample of the target image. For example, the heatmap overlay may identify areas in which the AI ​​system predicts that abnormalities may be present within the tissue sample. Predictive heatmap overlay visualization may be toggled on and off using the navigation menu 301. For example, the user may select the overlay icon on the navigation menu to toggle the heatmap overlay on or off. The predictive heatmap interface may show the user (e.g., a pathologist) one or more areas on the tissue that the user should investigate. The heatmap overlay may be transparent and / or semi-transparent so that the user can see the underlying tissue while browsing the heatmap overlay. The heatmap overlay may include different colors and / or shading to indicate the severity of the detected disease. Other types of overlays may also be available in context with respect to the type of tissue being viewed. For example, Figure 4 illustrates a prostate biopsy. However, other diseases may require different visualizations or associated AI systems.

[0098] Figure 5 illustrates an exemplary output 500 of the work list tool 107 of the browsing application tool 101 according to an exemplary embodiment. As shown in Figure 5, the browsing application tool 101 may include a work list 501 which can be opened by selecting the work list icon in the navigation menu 301. The main navigation menu 301 may provide an overview of the end-to-end workflow for slide viewing and / or case management. For example, the overview may include a work list 501 which can display cases for all users (e.g., pathologists). This may be accessed from the browsing user interface (UI) by opening the work list panel. Within the work list panel, the pathologist may view the patient's medical record number (MRN), patient ID number, patient name, suspected disease type, case status, surgical date, and select from various case actions such as viewing case details, sharing cases, and / or searching.

[0099] Figure 6 illustrates an exemplary output 600 of the slide tray tool 108 of the browsing application tool 101 according to an exemplary embodiment. As shown in Figure 6, the slide icon in the navigation menu 301 may open the slide tray 601 as a drawer / dropdown from the menu on the left side of the screen. The menu may present high-level case information, including case number, demographic information, etc. The slides of a case may be organized into parts and represented in the slide tray 601 as parent tabs in dark gray. Each part may be expanded to display slides within it. Thus, slides that are determined to be associated with the same part, which may be a predetermined area of ​​the sample or a predetermined sample area of ​​the patient, may be displayed within a predetermined proximity to each other. A red dot 602, which may be an indicator of a different color or a different shape, may indicate that the AI ​​system has found a possible disease (e.g., cancer) at a certain location within a part on that part. A red dot 603 on a slide may indicate that the AI ​​system has found a possible disease at a certain location on that slide. The parts and slides in which the AI ​​system identifies possible diseases may be brought to the top of the list for immediate viewing by the user (e.g., a pathologist). This workflow can help pathologists identify diseases more quickly.

[0100] Figure 7 illustrates an exemplary output 700 of the slide sharing tool 109 of the viewing application tool 101 according to an exemplary embodiment. From the slide panel, the user may select the “Share” button and select one, multiple, and / or all slides to share. The slide sharing panel 701 may be located to the right of the slide tray 601 and may allow the user to type in the recipients to share with, select the various slides to include, and / or write a brief note about the nature of the sharing. Once the send button is selected, the slides and brief note may be sent to the recipients, who may receive a notification and view these slides and / or note within their viewing application. User interaction may be captured in a log as described below.

[0101] Figure 8 illustrates an exemplary output 800 of the annotation tool 110 of the browsing application tool 101 according to an exemplary embodiment. For each annotation made, the annotation log 801 may capture the appearance of the actual annotation made, the type of annotation, any measurements, the annotation itself, the user who made the annotation, and / or the time of the annotation. The annotation log 801 can serve as a centralized view for consultation and information sharing to be examined within the context of the annotation or area of ​​interest. A user requesting consultation (e.g., a pathologist) and the pathologist providing the consultation may read the annotations, make annotations, and / or have a continuation dialogue specific to each annotation, each accompanied by a timestamp. The user may also select a thumbnail in the annotation log 801 and view its area of ​​interest at a large scale in the main viewer window.

[0102] Figure 9 illustrates exemplary output of the annotation tool 110 of the browsing application tool 101 according to an exemplary embodiment. For example, the annotation log illustrated in Figure 9 includes dialogue between pathologists for discussing a slide. Pathologists may send notes along with the annotated image to quickly consult on any area of ​​interest on the slide.

[0103] Figure 10 illustrates an exemplary output 1000 of the annotation tool 110 of the browsing application tool 101 according to an exemplary embodiment. The browsing application may include various annotation tools, illustrated in the annotation menu 1001. For example, the annotation tools may include a brush tool, an auto brush tool, a trajectory tool, a pinning tool, an arrow tool, a text field tool, a detail area tool, an ROI tool, a prediction tool, a measurement tool, a multi-measurement tool, a lesion area drawing tool, a region of interest tool, a tagging tool, an annotation tool, and / or a screenshot tool. Annotations may be used for research purposes and / or in a clinical setting. For example, a pathologist may make annotations, write annotations, and / or share them with colleagues to obtain a second opinion. Annotations may also be used in the final diagnostic report as supporting evidence for the diagnosis (e.g., the tumor has a mitotic count of y, with a length of x, etc.).

[0104] Figure 11 illustrates an exemplary output 1100 of the investigation tool 111 of the browsing application tool 101 according to an exemplary embodiment. The investigation tool 111 may include an investigation window 1101 featuring a magnified view of the area of ​​interest for a target image. Based on user input, the investigation window may be dragged across the image to quickly examine the image. Thus, a pathologist may be able to quickly move across the slide in a manner similar to the speed at which the slide is currently being moved within the microscope. While browsing through the investigation tool 111, the user may quickly switch or toggle a predictive heatmap overlay on and off. The user may take a screenshot of the tissue viewable within the investigation tool and quickly share the screenshot. Different annotation tools, such as measurement tools and / or area highlighting, may also be available with the investigation tool. In addition, the user may increase and decrease the magnification within the investigation tool independently of the magnification level of the main slide.

[0105] Figure 12 illustrates an exemplary output 1200 of the investigation tool 111 of the browsing application tool 101 according to an exemplary embodiment. As shown in Figure 12, the investigation window 1201 may display the AI ​​output in an enlarged view for verification and assistance of the diagnosis. The investigation window 1201 may include a heatmap overlay that displays the characteristics of the tissue sample predicted by the AI ​​system. For example, the heatmap overlay may identify areas in which the AI ​​system predicts that abnormalities may be present in the tissue sample.

[0106] Figure 13 illustrates an exemplary workflow of the browsing application tool 101 according to an exemplary embodiment. The workflow may include a work list, a case view, and / or a slide view as modes within the user's (e.g., a pathologist's) workflow. As illustrated in Figure 13, a diagnostic report may be output from the case view, which may include information necessary for digital signature.

[0107] Figures 14A, 14B, and 14C illustrate exemplary outputs for slide viewing of a browsing application tool 101 according to an exemplary embodiment. For example, as shown at the bottom of Figure 14A, the slide view toolbar 1401 may be positioned at the bottom of the display with vertical orientation of the slides. The slides may be grouped by part and may display relevant case information within that context. As shown in Figure 14B, the toolbar 1401 may be positioned on the left side of the display, and within the toolbar, the toolbar may include menu items such as tools, zoom functions, and thumbnail slides. As shown in Figure 14C, two vertical toolbars 1401 and 1402 may be positioned at the top left and right sides of the screen, which may allow easy access to various functions for touchscreen devices (e.g., tools and / or views).

[0108] Figure 15 illustrates an exemplary workflow for a new user (e.g., a patient, a physician, etc.) to create an account for requesting and receiving consultations, according to an exemplary embodiment. Requests to create a new account may be submitted through the hospital website and / or through the browsing application tool 101. Once an account is created, a pathologist may send notes, along with annotated images, for quick consultation on any area of ​​interest on a slide. The patient may also receive consultation information from the pathologist and / or billing information related to that case. If the user is a physician, billing options may differ based on whether the billing party is the commissioning physician, the patient, and / or the insurance company. If the user is a patient, the patient may bill directly or through the insurance company.

[0109] Figures 16A and 16B illustrate exemplary output for a pathology consultation dashboard according to an exemplary embodiment. As illustrated in Figure 16A, new requests and registered consultations may include patient name, referring physician, external ID, internal ID, facility, request date, status, update date, and / or links to additional details. The consultation dashboard may include functionality for selecting slides to submit, a submit slide dialogue box, a consultation request memo, a consultation attachment, and / or a consultation log. As illustrated in Figure 16B, a consultation request may include patient name, referring physician, facility, request date, status, update date, and / or links to further details about the request.

[0110] Figure 17 illustrates an exemplary output of a case view for a browsing application tool 101 according to an exemplary embodiment. The case view may include all the information necessary for the digital seal workflow. In addition, an annotation log may be integrated into the case view, as shown in Figure 17.

[0111] As shown in Figure 18, device 1800 may include a central processing unit (CPU) 1820. The CPU 1820 may be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As will be understood by those skilled in the art, the CPU 1820 may also be a single processor in a multicore / multiprocessor system, such a system operating alone or in a cluster of computing devices operating in a cluster or server farm. The CPU 1820 may be connected to a data communication infrastructure 1810, such as a bus, message queue, network, or multicore message transit scheme.

[0112] Device 1800 may also include main memory 1840, for example, random access memory (RAM), and may also include secondary memory 1830. The secondary memory 1830, for example, read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, flash memory, or the equivalent. In this embodiment, the removable storage drive is read from and / or written to in a well-known manner. The removable storage unit may comprise a floppy disk, magnetic tape, optical disk, etc., which is read from and written to by the removable storage drive. As will be understood by those skilled in the art, such a removable storage unit generally comprises a computer-usable storage medium having computer software and / or data stored therein.

[0113] In alternative implementations, the secondary memory 1830 may include other similar means for enabling computer programs or other instructions to be loaded into device 1800. Embodiments of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that enable software and data to be transferred from the removable storage units to device 1800.

[0114] Device 1800 may also include a communication interface ("COM") 1860. The communication interface 1860 enables the transfer of software and data between Device 1800 and external devices. The communication interface 1860 may include a modem, a network interface (such as an Ethernet® card), a communication port, a PCMCIA slot and card, or equivalent. The software and data transferred via the communication interface 1860 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 1860. These signals may be provided to the communication interface 1860 via a communication path of Device 1800, which may be implemented using, for example, wires or cables, optical fibers, telephone lines, cell phone links, RF links, or other communication channels.

[0115] The hardware elements, operating systems, and programming languages ​​of such devices are conventional in nature and are assumed to be well-known to those skilled in the art. Device 1800 also includes input and output ports 1850 and may be connected to input and output devices such as keyboards, mice, touchscreens, monitors, and displays. Naturally, various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming on a single computer hardware platform.

[0116] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or a related set of functions. Similar reference numbers are generally intended to refer to identical or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0117] The tools, modules, and functions described above may be implemented by one or more processors. The “storage” type medium may include any or all of the tangible memory of a computer, processor, or equivalent, or its associated modules such as various semiconductor memories, tape drives, disk drives, and equivalents, which may provide an ad-hoc, non-transient storage device for software programming.

[0118] Software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, communication may allow software to be loaded from one computer or processor into another. Unless limited to non-transient as used herein, the term computer or machine “readable medium” and other tangible “storage” medium refers to any medium involved in providing instructions to a processor for execution.

[0119] The general description set forth herein is illustrative and descriptive only and does not limit the disclosure. Other embodiments of the invention will be obvious to those skilled in the art from consideration of the specification and the practice of the invention disclosed herein. The specification and examples are intended to be considered illustrative only.

Claims

1. A computer implementation method for analyzing an electronic image corresponding to a sample, wherein the method is The process involves receiving a first target electronic image corresponding to a target sample, wherein the target sample comprises a patient's tissue sample. The machine learning system is applied to the first target electron image to determine at least one characteristic of the target sample and / or at least one characteristic of the first target electron image, Outputting a first target electron image that identifies the area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the first target electron image, Receiving a second target electronic image associated with the aforementioned target sample, Determining a first portion of the target sample associated with the first target electron image, Determining a second portion of the target sample associated with the second target electron image, The machine learning system identifies whether the first part and the second part are the same or overlap, In response to identifying that the first portion and the second portion are the same or overlap, the first representation of the first target electron image and the second representation of the second target electron image are displayed relative to each other at a predetermined degree of proximity. Computer implementation methods, including those mentioned above.

2. Identifying the area of ​​interest comprises displaying a heatmap overlay on the target electronic image, wherein the heatmap overlay includes shading and / or coloring based on the predicted likelihood that a location contains a biomarker, according to claim 1.

3. This further includes displaying a zoom window, The computer implementation method according to claim 2, wherein the magnification window comprises selectable icons for toggling the heatmap overlay on the magnified image.

4. A slide tray tool that identifies the outline of the target sample is displayed on the first target electronic image, The machine learning system is applied to the first target electronic image to determine whether a portion of the target sample contains a biomarker. In response to determining that the aforementioned portion contains a biomarker, the biomarker indicator is presented within the slide tray tool. The computer implementation method according to claim 1, further comprising:

5. The computer implementation method according to claim 1, further comprising displaying an annotation log having an indicator for identifying the area of ​​interest, and a consultation request related to the area of ​​interest.

6. Determining whether there is a focus area associated with the first target electron image and / or a focus area associated with the second target electron image by applying machine learning techniques, In response to determining that the area of ​​interest associated with the first target electron image exists, an indicator associated with the first representation of the first target electron image is displayed. In response to determining that the area of ​​interest associated with the second target electron image exists, an indicator associated with the second representation of the second target electron image is displayed. The computer implementation method according to claim 1, further comprising:

7. Displaying a magnification window over at least a portion of the first target electron image that is enlarged, Within the magnification window, an enlarged image of the target sample is presented at a magnification level different from that of the first target electron image. It further includes, The computer implementation method according to claim 1, wherein the magnification window comprises one or more selectable icons for changing the magnification level within the magnification window.

8. Receiving a plurality of target electronic images associated with the target sample, The machine learning system determines one or more parts of the target sample that are associated with the plurality of target electron images, The machine learning system determines whether one or more parts of the target sample in the plurality of target electron images are the same or overlap. In response to determining that one or more portions of the target sample in the plurality of target electron images are the same or overlapping, the target electron images containing the same or overlapping portions are displayed simultaneously at a predetermined proximity to each other. The computer implementation method according to claim 1, further comprising:

9. The machine learning system is capable of analyzing all of the first target electron images and all of the second target electron images and determining whether any area of ​​either image overlaps or is the same. When it is determined that one or more areas overlap or are the same, the first target electron image and the second target electron image are displayed at a predetermined proximity to each other. The computer implementation method according to claim 1, further comprising:

10. The machine learning system further includes determining the predicted likelihood that a location contains a biomarker, The computer implementation method according to claim 1, wherein the biomarker includes overexpression of a protein and / or gene product, amplification of a specific gene, or indication of mutation.

11. A system for analyzing an electronic image corresponding to a sample, wherein the system is At least one memory for storing instructions, At least one processor and Equipped with, The at least one processor executes the instruction, The process involves receiving a first target electronic image corresponding to a target sample, wherein the target sample comprises a patient's tissue sample. The machine learning system is applied to the first target electron image to determine at least one characteristic of the target sample and / or at least one characteristic of the first target electron image, Outputting a first target electron image that identifies the area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the first target electron image, Receiving a second target electronic image associated with the aforementioned target sample, Determining a first portion of the target sample associated with the first target electron image, Determining a second portion of the target sample associated with the second target electron image, The machine learning system identifies whether the first part and the second part are the same or overlap, In response to identifying that the first portion and the second portion are the same or overlap, the first representation of the first target electron image and the second representation of the second target electron image are displayed relative to each other at a predetermined degree of proximity. A system that performs operations including those mentioned above.

12. Identifying the area of ​​interest comprises displaying a heatmap overlay on the target electronic image, wherein the heatmap overlay includes shading and / or coloring based on the predicted likelihood that a location contains a biomarker, according to claim 11.

13. This further includes displaying a zoom window, The system according to claim 12, wherein the magnification window comprises selectable icons for toggling the heatmap overlay on the magnified image.

14. The operation described above is: A slide tray tool that identifies the outline of the target sample is displayed on the first target electronic image, The machine learning system is applied to the first target electronic image to determine whether a portion of the target sample contains a biomarker. In response to determining that the aforementioned portion contains a biomarker, the biomarker indicator is presented within the slide tray tool. The system according to claim 11, further comprising:

15. The system according to claim 11, wherein the operation further comprises displaying an annotation log having an indicator for identifying the area of ​​interest and a consultation request related to the area of ​​interest.

16. The operation described above is: By applying machine learning techniques, it is determined whether there is a focus area associated with the first target electron image and / or a focus area associated with the second target electron image. In response to determining the area of ​​interest associated with the first target electron image, an indicator associated with a first representation of the first target electron image is displayed. In response to determining the area of ​​interest associated with the second target electron image, an indicator associated with the second representation of the second target electron image is displayed. The system according to claim 11, further comprising:

17. The system according to claim 11, wherein the operation further comprises displaying a magnification window that moves across the first target electronic image in response to user input.

18. A non-transient computer-readable medium that, when executed by a processor, stores instructions causing the processor to perform a method for analyzing an electronic image corresponding to a sample, and the method is The process involves receiving a first target electronic image corresponding to a target sample, wherein the target sample comprises a patient's tissue sample. The machine learning system is applied to the first target electron image to determine at least one characteristic of the target sample and / or at least one characteristic of the first target electron image, Outputting a first target electron image that identifies the area of ​​interest based on at least one characteristic of the target sample and / or at least one characteristic of the first target electron image, Receiving a second target electronic image associated with the aforementioned target sample, Determining a first portion of the target sample associated with the first target electron image, Determining a second portion of the target sample associated with the second target electron image, The machine learning system identifies whether the first part and the second part are the same or overlap, In response to identifying that the first portion and the second portion are the same or overlap, the first representation of the first target electron image and the second representation of the second target electron image are displayed relative to each other at a predetermined degree of proximity. Non-transient computer-readable media, including [specific examples of such media].

19. Identifying the area of ​​interest comprises displaying a heatmap overlay on the target electronic image, wherein the heatmap overlay comprises shading and / or coloring based on the predicted likelihood that a location contains a biomarker, according to claim 18, a non-transient computer-readable medium.

20. The non-transient computer-readable medium according to claim 18, further comprising displaying a magnification window that moves across the first target electronic image in response to user input.

Images