Systems and methods for processing slide images for digital pathology

Abstract

A system and method are disclosed for receiving a target electronic image corresponding to a target sample, the target sample including a tissue sample from a patient, applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images to predict at least one characteristic, the training images including images of human tissue and / or images generated by the algorithm, and outputting a target electronic image identifying a region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

Description

[0001] (one or more) related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 897,745, filed September 9, 2019, the entire disclosure of which is hereby incorporated herein by reference. Technical Field

[0003] Various embodiments of this disclosure generally relate to image-based sample analysis and related image processing methods. More specifically, particular embodiments of this disclosure relate to systems and methods for identifying sample attributes and providing a comprehensive pathological workflow based on processed images of tissue samples. Background Technology

[0004] For digital pathology images to be used in hospital or research settings, it may be important to identify and classify the tissue type of the sample, the nature of the sample acquisition (e.g., prostate biopsy, breast biopsy, mastectomy, etc.), and other relevant attributes of the sample or image.

[0005] There is a desire for a way to provide a comprehensive pathology workflow based on the processing of tissue sample images. The following disclosure relates to systems and methods for providing user interfaces and artificial intelligence (AI) tools that can be integrated into workflows to accelerate and improve the work solutions for pathologists.

[0006] The foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. The background description provided herein is for the purpose of presenting the overall context of this disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art as claimed in this application, and are not admitted as prior art or suggestions of prior art by virtue of their inclusion in this section. Summary of the Invention

[0007] According to certain aspects of this disclosure, systems and methods for identifying sample attributes and providing a comprehensive pathological workflow based on processed images of tissue samples are disclosed.

[0008] A computer-implemented method for analyzing an electronic image corresponding to a sample includes: receiving a target electronic image corresponding to a target sample, the target sample including a tissue sample from a patient; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images to predict at least one characteristic, the training images including human images and / or algorithmically generated images; and outputting a target electronic image identifying a region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

[0009] A system for analyzing an electronic image corresponding to a sample includes: a memory storing instructions; and a processor that executes the instructions to perform a process comprising: receiving a target electronic image corresponding to a target sample, the target sample including a tissue sample from a patient; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images to predict at least one characteristic, the training images including images of human tissue and / or images generated by an algorithm; and outputting a target electronic image identifying a region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

[0010] A non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform a method for analyzing an image corresponding to a sample, the method comprising receiving a target electronic image corresponding to a target sample, the target sample including a tissue sample of a patient; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image, the machine learning system being generated by processing a plurality of training images to predict at least one characteristic, the training images including images of humans and / or images generated by an algorithm; and outputting a target electronic image identifying a region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and illustrative only, and are not intended to limit the scope of the disclosed embodiments as claimed. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate various exemplary embodiments and, together with the specification, serve to explain the principles of the disclosed embodiments.

[0013] Figure 1A An exemplary block diagram of a system and network for determining sample attribute or image attribute information associated with one or more digital pathology images according to an exemplary embodiment of the present disclosure is illustrated.

[0014] Figure 1B An exemplary block diagram of a disease detection platform 100 according to an exemplary embodiment of the present disclosure is shown.

[0015] Figure 1C An exemplary block diagram of a viewing application tool 101 according to an exemplary embodiment of the present disclosure is shown.

[0016] Figure 1D An exemplary block diagram of a viewing application tool 101 according to an exemplary embodiment of the present disclosure is shown.

[0017] Figure 2 This is a flowchart illustrating an exemplary method for outputting a target image identifying a region of interest, according to one or more exemplary embodiments of this disclosure.

[0018] Figure 3 The illustration shows an exemplary output of a viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0019] Figure 4 An exemplary output of the overlay tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure is illustrated.

[0020] Figure 5 The illustration shows an exemplary output of a work list tool of viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0021] Figure 6 An exemplary output of the slide tray tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure is illustrated.

[0022] Figure 7 The illustration shows an exemplary output of the slide sharing tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0023] Figure 8 The illustration shows exemplary output of the annotation tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0024] Figure 9 The illustration shows exemplary output of the annotation tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0025] Figure 10 The illustration shows exemplary output of the annotation tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0026] Figure 11 The illustration shows an exemplary output of the inspection tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0027] Figure 12 The illustration shows an exemplary output of the inspection tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

[0028] Figure 13 An exemplary workflow of a viewing application tool 101 according to an exemplary embodiment of the present disclosure is illustrated.

[0029] Figure 14A , 14BFigures 14C illustrate exemplary output of a view of a slide from an application tool 101 according to an exemplary embodiment of the present disclosure.

[0030] Figure 15 The illustration shows an exemplary workflow for creating an account and requesting advice for a new user according to an exemplary embodiment of the present disclosure.

[0031] Figure 16A and 16B An exemplary output of a pathology consultation dashboard according to an exemplary embodiment of the present disclosure is illustrated.

[0032] Figure 17 An exemplary output of a case view of a viewing application tool 101 according to an exemplary embodiment of the present disclosure is illustrated.

[0033] Figure 18 An example system that can perform the techniques presented in this paper is described.

[0034] Description of the Implementation Examples

[0035] Exemplary embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. Throughout the drawings, the same reference numerals will be used as much as possible to refer to the same or similar parts.

[0036] The systems, apparatuses, and methods disclosed herein are described in detail by way of example and with reference to the figures. The examples discussed herein are merely illustrative and are provided to assist in explaining the apparatuses, apparatuses, systems, and methods described herein. Unless specifically designated as mandatory, the features or components shown in the figures or discussed below should not be considered mandatory for any particular implementation of any of these apparatuses, systems, or methods.

[0037] Furthermore, for any of the methods described, whether or not the method is described in conjunction with a flowchart, it should be understood that, unless the context otherwise specifies or requires, any explicit or implicit order of the steps executed in the method execution does not imply that those steps must be executed in the presented order, but may instead be executed in a different order or in parallel.

[0038] As used herein, the term “exemplary” is used in the sense of “example” rather than in the sense of “ideal.” Furthermore, the terms “a” or “an” in this document do not indicate a limitation of quantity, but rather indicate the presence of one or more of the referenced items.

[0039] Pathology refers to the study of disease. More specifically, pathology refers to the performance of tests and analyses used to diagnose disease. For example, a tissue sample may be placed on a slide and examined under a microscope by a pathologist (e.g., a physician who is an expert in analyzing tissue samples to determine the presence of any abnormalities). That is, a pathological sample may be cut into multiple sections, stained, and prepared into slides for the pathologist to examine and make a diagnosis. When the diagnosis found on the slide is uncertain, the pathologist may order additional cutting levels, staining, or other tests to gather more information from the tissue. Then, one or more technicians may create one or more new slides that may contain additional information used by the pathologist in making a diagnosis. This process of creating additional slides can be time-consuming, not only because it may involve retrieving the tissue block, cutting it to make a new slide, and then staining the slide, but also because it may be done in batches for multiple orders. This can significantly delay the final diagnosis made by the pathologist. Furthermore, even after the delay, there may still be no guarantee that one or more new slides will have sufficient information to make a diagnosis.

[0040] Computers can be used to analyze images of tissue samples to quickly identify whether additional information about a particular tissue sample may be needed, and / or to highlight areas to the pathologist that he or she may need to examine more closely. Therefore, the process of obtaining additional stained slides and conducting tests can be automated before the pathologist's review. When paired with an automated slide splitting and staining machine, this can provide a fully automated slide preparation pipeline. This automation has at least the following benefits: (1) minimizing the amount of time a pathologist spends determining that a slide is insufficient to make a diagnosis; (2) minimizing the (average total) time from sample acquisition to diagnosis by avoiding the additional time between ordering additional tests and generating additional tests; (3) reducing the amount of time and material wasted per recutting by allowing recutting while the tissue block (e.g., pathological sample) is in the cutting table; (4) reducing the amount of tissue material required during slide preparation; (5) reducing the cost of slide preparation by partially or fully automating the process; (6) allowing automated custom cutting and staining of slides, which will result in more representative / informative slides from the sample; (7) allowing a higher number of slides to be generated per tissue block, contributing to a more informed / accurate diagnosis by reducing the overhead of requesting additional tests from the pathologist; and / or (8) identifying or verifying the correct attributes of digital pathology images (e.g., regarding sample type), etc.

[0041] The process of using computers to assist pathologists is called computational pathology. The computational methods used in computational pathology can include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI can include, but is not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. By using computational pathology, lives can be saved by helping pathologists improve the accuracy, reliability, efficiency, and accessibility of their diagnoses. For example, computational pathology can be used to assist in examining slides suspected of containing cancer, allowing pathologists to review and confirm their initial assessments before making a final diagnosis.

[0042] Histopathology refers to the study of samples placed on a glass slide. For example, a digital pathology image can consist of a digital image of a microscope slide containing a sample (e.g., a smear). One method pathologists can use to analyze images on a slide is to identify cell nuclei and classify them as normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying cell nuclei, histological staining can be used to make cells visible. Many dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hemolysin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method in which hematoxylin stains cell nuclei blue, eosin stains the cytoplasm and extracellular matrix pink, and other tissue areas exhibit variations of these colors. However, in many cases, H&E-stained histological preparations do not provide pathologists with sufficient information to visually identify biomarkers that can aid in diagnosis or guide treatment. In such cases, methods such as immunohistochemistry (IHC), immunofluorescence, etc., can be used. In situ Hybridization (ISH) or fluorescence In situ Techniques such as hybridization (FISH) are employed. IHC and immunofluorescence involve, for example, the use of antibodies that bind to specific antigens in tissues, enabling visual detection of cells expressing specific proteins of interest. This can reveal biomarkers that trained pathologists cannot reliably identify based on analysis of H&E-stained slides. Depending on the type of probe used (e.g., DNA probes for gene copy number and RNA probes for assessing RNA expression), ISH and FISH can be used to assess gene copy number or the abundance of specific RNA molecules. If these methods also fail to provide sufficient information to detect some biomarkers, tissue genetic testing can be used to confirm the presence of biomarkers (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a given gene in cancer).

[0043] Digital images can be prepared to show stained microscope slides, allowing pathologists to manually examine the images on the slides and estimate the number of abnormally stained cells in the images. However, this process can be time-consuming and prone to errors in identifying abnormalities, as some are difficult to detect. Computational processes and devices can be used to assist pathologists in detecting abnormalities that might otherwise be difficult to detect. For example, AI can be used to predict biomarkers (such as overexpression of protein and / or gene products, amplification or mutation of specific genes) from salient regions within digital images of tissues stained using H&E and other dye-based methods. The tissue image can be a whole slide image (WSI), an image of the tissue core within a microarray, or an image of a selected region of interest within a tissue section. Using staining methods like H&E, these biomarkers can be difficult to detect or quantify visually by humans without additional testing. Using AI to infer these biomarkers from digital images of tissues has the potential to improve patient care while also being faster and cheaper.

[0044] The detected biomarkers or images can then be used individually to recommend specific cancer drugs or drug combinations for treating patients, and AI can identify which drugs or drug combinations are likely to be successful by correlating the detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendations of immunotherapies for patients with specific cancers. Furthermore, this can be used to enable personalized cancer treatment for specific subsets of patients and / or rare cancer types.

[0045] In contemporary pathology, it can be challenging to provide systematic quality control (“QC”) relative to pathological sample preparation and quality assurance (“QA”) relative to diagnostic quality throughout the histopathology workflow. Systematic quality assurance is difficult because it is resource- and time-intensive, as it may require repetitive effort from two pathologists. Some approaches to quality assurance include (1) a second review of first-diagnosed cancer cases; (2) periodic reviews by a quality assurance committee for inconsistent or altered diagnoses; and (3) random review of case sample sets. These are all incomplete, largely retrospective, and manual. With automated and systematic QC and QA mechanisms, quality can be ensured throughout the workflow for each case. Laboratory quality control and digital pathology quality control can be critical for the successful inclusion, processing, diagnosis, and archiving of patient samples. Manual and sampling approaches to QC and QA offer substantial benefits. Systematic QC and QA have the potential to provide efficiency and improve diagnostic quality.

[0046] As described above, the computational pathology process and apparatus of this disclosure can provide an integrated platform that allows for a fully automated process, including the ingestion, processing, and viewing of digital pathology images via a web browser or other user interface, while integrating with a laboratory information system (LIS). Furthermore, cloud-based data analytics of patient data can be used to aggregate clinical information. The data may originate from hospitals, clinics, field researchers, etc., and can be analyzed using machine learning, computer vision, natural language processing, and / or statistical algorithms to enable real-time monitoring and prediction of health patterns at multiple geographic-specific levels.

[0047] This disclosure presents an integrated workflow that facilitates disease detection and / or cancer diagnosis by providing a workflow that integrates, for example, slide evaluation, tasks, image analysis, and cancer detection AI, annotation, consultation, and recommendation into a single workstation. This disclosure describes various exemplary user interfaces available in the workflow, as well as AI tools that can be integrated into the workflow to accelerate and assist pathologists' work.

[0048] Figure 1A The illustration shows a block diagram of a system and network according to an exemplary embodiment of the present disclosure, the system and network being configured to provide a workflow for determining and outputting sample attribute or image attribute information relating to one or more digital pathology images using machine learning.

[0049] Specifically, Figure 1A The illustration depicts an electronic network 120 that can connect to servers in locations such as hospitals, laboratories, and / or doctors' offices. Examples include a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, each of which can be connected to the electronic network 120, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of this application, the electronic network 120 can also be connected to a server system 127, which may include processing equipment configured to implement a disease detection platform 100, which includes a viewing application tool 101 for determining and outputting sample attribute or image attribute information related to one or more digital pathology images, and using machine learning according to an exemplary embodiment of this disclosure.

[0050] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may create or otherwise obtain images of one or more cytological samples, one or more histopathological samples, one or more slides of cytological samples, one or more slides of histopathological samples, or any combination thereof, from one or more patients. Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytological information, etc. Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may transmit digitized slide images and / or patient-specific information to server system 127 via electronic network 120. One or more server systems 127 may include one or more storage devices 126 for storing images and data received from at least one of a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125. Server system 127 may also include processing equipment for processing the images and data stored in the storage devices 126. Server system 127 may further include one or more machine learning tools or capabilities. For example, according to one embodiment, the processing equipment may include machine learning tools for a disease detection platform 100. Alternatively or additionally, this disclosure (or part of the systems and methods of this disclosure) may be executed on a local processing device (e.g., a laptop computer).

[0051] Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review slide images. In a hospital setting, tissue type information may be stored in LIS 125. However, correct tissue classification information is not always paired with image content. Additionally, even when using LIS to access the sample type of a digital pathology image, the label may be incorrect due to the fact that many components of LIS may be manually entered, leaving a large margin for error. According to exemplary embodiments of this disclosure, sample type and / or other sample information may be identified without requiring access to LIS 125, or may be identified to potentially correct LIS 125. For example, a third party may anonymously access image content where no corresponding sample type label is stored in LIS. Additionally, access to LIS content may be restricted due to its sensitive nature.

[0052] Figure 1B An exemplary block diagram of a disease detection platform 100 is shown, which uses machine learning to determine and output sample attribute or image attribute information related to one or more digital pathology images.

[0053] Specifically, Figure 1B Components of a disease detection platform 100 according to one embodiment are depicted. For example, the disease detection platform 100 may include a viewing application tool 101, a data ingestion tool 102, a slide ingestion tool 103, a slide scanner 104, a slide manager 105, and / or a storage device 106.

[0054] According to an exemplary embodiment, as described below, viewing application tool 101 can refer to a process and system for providing a user (e.g., a pathologist) with sample attribute and / or image attribute information related to one or more digital pathology images. The information can be provided through various output interfaces (e.g., screen, monitor, storage device, and / or web browser, etc.).

[0055] According to an exemplary embodiment, data ingestion tool 102 refers to a process and system that facilitates the transmission of digital pathology images to various tools, modules, components, and devices for classifying and processing digital pathology images.

[0056] According to an exemplary embodiment, the slide incorporation tool 103 refers to a process and system for scanning pathological images and converting them into digital form. A slide can be scanned using a slide scanner 104, and a slide manager 105 can process the images on the slides into digital pathological images and store the digital images in a storage device 106.

[0057] Viewing application tool 101 and each of its components can transmit and / or receive digitized slide images and / or patient information to server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 via network 120. Furthermore, server system 127 may include storage devices for storing images and data received from at least one of viewing application tool 101, data ingestion tool 102, slide ingestion tool 103, slide scanner 104, and / or slide manager 105. Server system 127 may also include processing devices for processing the images and data stored in the storage devices. Server system 127 may further include one or more machine learning tools or capabilities, for example, due to the processing devices. Alternatively or additionally, this disclosure (or part of the systems and methods of this disclosure) may be executed on a local processing device (e.g., a laptop computer).

[0058] Any of the above devices, tools, and modules may be located on a device that can be connected to an electronic network 120 (such as the Internet or a cloud service provider) via one or more computers, servers, and / or handheld mobile devices.

[0059] Figure 1C An exemplary block diagram of a viewing application tool 101 according to an exemplary embodiment of the present disclosure is illustrated. The viewing application tool 101 may include a worklist tool 107, a slide tray tool 108, a slide sharing tool 109, and annotation tools 110, inspection tools 111, and / or overlay tools 112. The worklist tool 107 can provide an overview of an end-to-end workflow for slide viewing and / or case management. The slide tray tool 108 can organize slides of a case by section and provides advanced case information including case number, demographic information, etc. The slide sharing tool 109 can provide users with the ability to share various slides and includes and / or write brief comments about the nature of the sharing. The annotation tool 110 may include a brush tool, an auto-brush tool, a trajectory tool, a lock tool, an arrow tool, a text field tool, a region of detail tool, a ROI tool, a prediction tool, a measurement tool, a multiple measurement tool, a comment tool, and / or a screenshot tool. The inspection tool 111 can provide an inspection window characterized by a magnified view of the region of interest of a target image. The overlay tool can provide a heatmap overlay on a magnified view of the region of interest in a target image, thereby identifying the region of interest on a tissue sample in a magnified view of the target image.

[0060] Figure 1D An exemplary block diagram of a viewing application tool 101 according to an exemplary embodiment of the present disclosure is illustrated. The viewing application tool 101 may include a training image platform 131 and / or a target image platform 135.

[0061] According to one embodiment, the training image platform 131 may include a training image inclusion module 132 and / or an image analysis module 133.

[0062] According to one embodiment, the training image platform 131 can create or receive training images for training a machine learning system to effectively analyze and classify digital pathology images. For example, training images can be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. Images used for training can be from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images can include (a) digitized slides stained with multiple staining agents, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, etc.; and / or (b) digitized tissue samples from a 3D imaging device such as microCT.

[0063] The training image inclusion module 132 can create or receive a dataset comprising one or more training images, which correspond to any one or both of human tissue images and graphic / synthetic rendered images. For example, training images can be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. This dataset can be stored on a digital storage device. The image analysis module 133 can analyze the images to identify sample attributes and / or image attribute information (e.g., sample type, overall quality of sample cuts, overall quality of the glass pathology slide itself, and / or tissue morphology characteristics).

[0064] According to one embodiment, the target image platform 135 may include a target image acquisition module 136, a sample detection module 137, and an output interface 138. The target image platform 135 can receive target images and apply a machine learning system to the received target images to determine the characteristics of the target samples. For example, target images can be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The target image acquisition module 136 can receive target images corresponding to target samples. The sample detection module 137 can apply a machine learning system to the target images to determine the characteristics of the target samples and / or the characteristics of the target images. For example, the sample detection module 137 can detect the sample type of the target samples. Furthermore, the sample detection module 137 can apply a machine learning system to determine whether a region of the sample contains one or more anomalies.

[0065] Output interface 138 can be used to output information about the target image and target sample (e.g., output to a screen, monitor, storage device, web browser, etc.).

[0066] Figure 2 This is a flowchart illustrating an exemplary method for providing a viewing application tool 101 for identifying sample information and a user interface for viewing sample information, according to an exemplary embodiment of the present disclosure. For example, exemplary method 200 (e.g., steps 202 to 206) may be executed automatically by the viewing application tool 101 or in response to a request from a user (e.g., a physician, pathologist, technician, etc.).

[0067] According to one embodiment, an exemplary method 200 for identifying sample information and providing a user interface for viewing the sample information may include one or more steps. In step 202, the method may include receiving a target image corresponding to a target sample, which may include a tissue sample from a patient. For example, the target image may be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125.

[0068] In step 204, the method may include applying a machine learning system to the target image to determine at least one characteristic of the target sample and / or at least one characteristic of the target image. Determining the characteristic of the target sample may include determining sample information of the target sample. For example, determining the characteristic may include determining whether an anomaly exists in the target sample.

[0069] Machine learning systems can be generated by processing multiple training images to predict at least one feature, and these training images may include images of human tissue and / or algorithmically / synthetically generated images. Machine learning systems can be implemented using machine learning methods for classification and regression. Training inputs may include real or synthetic images. Training inputs may or may not be augmented (e.g., by adding noise or creating variants of the input through flipping / distorting). Exemplary machine learning systems may include, but are not limited to, any or any combination of neural networks, convolutional neural networks, random forests, logistic regression, and nearest neighbors. Convolutional neural networks can directly learn the image feature representations necessary to distinguish between features, which works very well when there is a large amount of data to train on for each sample. Other methods can be used with traditional computer vision features (e.g., Speed-Up Robust Features (SURF) or Scale-Invariant Feature Transform (SIFT)) or with learned embeddings (e.g., descriptors) generated by trained convolutional neural networks—which can be advantageous when there is only a small amount of data to train on. Training images can be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The dataset can be stored on a digital storage device. Images used for training can be from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various staining agents, such as (but not limited to) H&E, IHC, molecular pathology, etc.; and / or (b) digitized tissue samples from 3D imaging devices such as microCT.

[0070] In step 206, the method may include outputting a target image that identifies the region of interest based on at least one feature of the target sample and / or at least one feature of the target image.

[0071] Different approaches to implementing machine learning algorithms and / or architectures may include, but are not limited to, (1) CNN (Convolutional Neural Network); (2) MIL (Multi-Instance Learning); (3) RNN (Recurrent Neural Network); (4) Feature aggregation via CNN; and / or (5) Feature extraction followed by ensemble methods (e.g., Random Forest), linear / nonlinear classifiers (e.g., SVM (Support Vector Machine), MLP (Multilayer Perceptron), and / or dimensionality reduction techniques (e.g., PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), etc.). Example features may include vector embeddings from CNNs, single / multi-class outputs from CNNs, and / or multidimensional outputs from CNNs (e.g., mask overlays of the original image). CNNs can learn feature representations for classification tasks directly from pixels, which can lead to better diagnostic performance. CNNs can be trained directly if there is a large amount of labeled data available when detailed annotations or pixel-by-pixel labels for regions are available. However, when labels are only at the level of the entire slide or on a set of slides in a group (which may be called "partial" in pathology), MIL may be used. MIL can be used to train CNNs or other neural network classifiers, where MIL learns image regions that are diagnostic for the classification task, resulting in the ability to learn even without extensive annotations. Features extracted from multiple image regions (e.g., image patches) can be used with RNNs and then processed to make predictions. Other machine learning methods, such as Random Forests, SVMs, and many others, can be used with features learned by CNNs, CNNs with MIL, or handcrafted image features (e.g., SIFT or SURF) to perform classification tasks, but they may perform poorly when trained directly from pixels. These methods may also underperform compared to CNN-based systems when a large amount of annotated training data is available. Dimensionality reduction techniques can be used as a preprocessing step before using any of the classifiers mentioned, which can be useful if there is very little available data.

[0072] According to one or more embodiments, any of the above algorithms, architectures, methods, attributes, and / or features can be combined with any or all of other algorithms, architectures, methods, attributes, and / or features. For example, any of the machine learning algorithms and / or architectures (e.g., neural network methods, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.) can be trained using any of the training methods (e.g., multi-instance learning, reinforcement learning, active learning, etc.).

[0073] The following descriptions of terms are merely illustrative and are not intended to limit the terminology in any way.

[0074] Labels can refer to information about the inputs to a machine learning algorithm that it is trying to predict.

[0075] For a given image of size N x M, a segmentation can be another image of size N x M, which assigns a number describing the class or type of each pixel in the original image. For example, in WSI, elements in a mask can classify each pixel in the input image into a class such as background, organization, and / or unknown.

[0076] Slide grade information can refer to general information about a slide, but it does not necessarily refer to the specific location of that information on the slide.

[0077] Heuristics can refer to logical rules or functions that deterministically produce outputs given input. For example, if the slide is predicted to have an anomaly, the output is 1; otherwise, the output is 0.

[0078] Embedding can refer to a conceptual high-dimensional numerical representation of low-dimensional data. For example, if a WSI is trained using a CNN to classify tissue types, the numbers on the last layer of the network can provide an array of numbers (e.g., approximately several thousand) containing information about the slide (e.g., information about the tissue type).

[0079] Slide-level prediction can refer to a specific prediction about the slide as a whole. For example, slide-level prediction could mean that the slide contains anomalies. Furthermore, slide-level prediction can refer to individual probability predictions over a defined set of classes.

[0080] A classifier can refer to a model and / or system that has been trained to take input data and associate it with categories.

[0081] According to one or more embodiments, machine learning models and / or systems can be trained in different ways. For example, training a machine learning model can be performed through any or any combination of supervised training, semi-supervised training, unsupervised classifier training, hybrid training, and / or uncertainty estimation. The type of training used can depend on the amount of data, the data type, and / or the data quality. Table 1 below describes a non-restrictive list of some training types and corresponding features.

[0082] Table 1

[0083]

[0084] Supervised training can be used with a small amount of data to provide seeds for machine learning models. In supervised training, the machine learning model can look for specific items (e.g., bubbles, tissue wrinkles, etc.), label slides, and quantify how many specific items are present on the slide.

[0085] According to one embodiment, example fully supervised training can take WSI as input and may include segmentation labels. A pipeline for fully supervised training may include (1) 1; (2) 1, heuristic; (3) 1, 4, heuristic; (4) 1, 4, 5, heuristic; and / or (5) 1, 5, heuristic. Advantages of fully supervised training may include: (1) it may require fewer slides and / or (2) the output is interpretable because it is known which regions of the image contribute to the diagnosis. Disadvantages of using fully supervised training may include that it may require a large number of segments, which may be difficult to obtain.

[0086] According to one embodiment, example semi-supervised (e.g., weakly supervised) training can take WSI as input and may include labels of slide-level information. A pipeline for semi-supervised training may include (1) 2; (2) 2, heuristic; (3) 2, 4, heuristic; (4) 2, 4, 5, heuristic; and / or (5) 2, 5, heuristic. Advantages of using semi-supervised training may include: (1) the desired label types may exist in many hospital records; and (2) the output is interpretable because it is known which regions of the image contribute most to the diagnosis. Disadvantages of using semi-supervised training include the potential difficulty in training. For example, the system may need to use training schemes such as multi-instance learning, activation learning, and / or distributed training to account for the fact that limited information about where information exists in the slide will lead to decision-making.

[0087] According to one embodiment, example unsupervised training can take WSI as input and may not require labels. Pipelines for unsupervised training may include (1) 3, 4; and / or (2) 3, 4, heuristics. An advantage of unsupervised training may be that it does not require any labels. A disadvantage of using unsupervised training may be that (1) it may be difficult to train. For example, it may require training schemes such as multi-instance learning, activation learning, and / or distributed training to take into account the fact that there is limited information about where information is present in the slide, which will lead to the decision; (2) additional slides may be required; and / or (3) it may be less interpretable because it may output predictions and probabilities without explaining why that prediction was made.

[0088] According to one embodiment, example hybrid training may include training any of the example pipelines described above for fully supervised training, semi-supervised training, and / or unsupervised training, and then using the resulting model as an initial point for any training method. The advantages of hybrid training may be: (1) it may require less data; (2) it may have improved performance; and / or (3) it may allow for the mixing of different levels of labels (e.g., segmentation, slide-level information, no information). The disadvantages of hybrid training may be: (1) training may be more complex and / or more expensive; and / or (2) it may require more code, which may increase the number and complexity of potential vulnerabilities.

[0089] According to one embodiment, example uncertainty estimation may include using uncertainty estimation at the end of a pipeline to train any of the example pipelines described above for fully supervised training, semi-supervised training, and / or unsupervised training for any task related to slide data. Furthermore, based on the amount of uncertainty in the test predictions, heuristics or classifiers may be used to predict whether the slides are anomalous. An advantage of uncertainty estimation may be its robustness to out-of-distribution data. For example, it may still correctly predict that it is uncertain when unfamiliar data is presented. Disadvantages of uncertainty estimation may include (1) potentially requiring more data; (2) potentially having poor overall performance; and / or (3) potentially being less interpretable because the model may not necessarily identify how the slides or slide embeddings are anomalous.

[0090] According to one embodiment, ensemble training may include simultaneously running models generated by any of the above example pipelines and combining the outputs via heuristics or classifiers to produce robust and accurate results. Advantages of ensemble training may include: (1) its robustness to out-of-distribution data; and / or (2) its ability to combine the strengths and weaknesses of other models, resulting in the minimization of weaknesses (e.g., a combination of a supervised training model with an uncertainty estimation model, and a heuristic using a supervised model when the incoming data is in the distribution and an uncertainty model when the data is out of distribution, etc.). Disadvantages of ensemble training may include: (1) it may be more complex; and / or (2) training and running may be expensive.

[0091] The training technique described in this paper can also be carried out in stages, with images that have more annotations initially used for training. This allows for more effective subsequent training using slides with fewer annotations and less supervision.

[0092] Training can begin with the slides most thoroughly annotated, relative to all possible training slide images. For example, training can begin using supervised learning. A first set of slide images with associated annotations can be received or determined. Each slide may have labeled and / or masked regions and may include information such as whether the slide has anomalies. This first set of slides can be fed to a training algorithm, such as a CNN, which can determine the correlation between the first set of slides and their associated annotations.

[0093] After training with a first set of images is complete, a second set of slide images can be received or determined, which has fewer annotations than the first set, such as partial annotations. In one embodiment, the annotations may only indicate that the slide has a diagnosis or quality problem associated with it, but may not specify what might be found or where the disease is found, etc. The second set of slide images can be trained using a different training algorithm than the first set (e.g., multi-instance learning). The first set of training data can be used to partially train the system, and can make the second round of training more efficient in producing an accurate algorithm.

[0094] Thus, based on the quality and type of the training slide images, any number of algorithms can be used for training at any number of stages. These techniques can be used when multiple training sets of images are received, and these multiple training sets of images can have different qualities, annotation levels, and / or annotation types.

[0095] Figure 3 The illustration shows an exemplary output 300 of a viewing application tool 101 according to an exemplary embodiment. Figure 3 As illustrated in the diagram, the viewing application tool 101 may include a navigation menu 301 for navigating to the worklist view of the worklist tool 107, the slide view for the slide tray tool 108 and the slide sharing tool 109, the annotation tool 110, the inspection tool 111, and / or the overlay toggle for the overlay tool 112. The navigation menu 301 may also include view mode inputs for adjusting the view (e.g., splitting the screen to view multiple slides and / or stains at once). A zoom menu 302 can be used to quickly adjust the zoom level of the target image 303. The target image can be displayed in any color. For example, a dark, neutral color scheme can be used to display details of the target image 303. The viewing application tool 101 may include a positive / negative indicator for the presence of a possible disease at (x, y) coordinates in the image. Additionally, the user can perform confirmation and / or editing of the positive / negative indicator.

[0096] Slides can be prioritized using prioritization features based on characteristics identified in the target image. Slides can be organized using a folder system, including a default system and a customized system. The application tool 101 may include a case view, which can include results, descriptions, attachments, patient information, and status icons. Slides within a case can be organized by type and can be color-coded. Case sharing functionality can be designed to be secure and trusted. Cases can be archived to provide necessary storage space. The ability to switch between qualities within a single slide may be available. The application tool 101 can provide notifications of new messages and / or new slides. Status icons can indicate whether a case and / or slide is new, and whether it has been shared and / or ordered. The ability to search for cases, patients, and / or items may be available. The application tool 101 may include patient queues and / or consultation queues.

[0097] Viewing application tool 101 may include viewing options, such as viewing modes, windows, rotation, scale bars, and / or overlay. Exemplary outputs of viewing application tool 101 may include patient information, slide information (e.g., image overview), and / or file information (e.g., slide map). Viewing modes may include a default view and viewing modes with different scales. Viewing application tool 101 may include a fluorescence mode, a magnifying glass view, a scale bar, pan and zoom functions, image rotation functions, a focus of interest, and / or other slide actions (e.g., selecting a new stain). The viewing application can analyze images and determine patient survival rates.

[0098] The viewing application tool 101 may include functions for selecting and recommending slides and grades. For example, the viewing application tool 101 may include options for selecting slide types, sending new slide orders, receiving slide order confirmations, slide order indicators, viewing new slides, and / or comparing new slides with previous slides. The viewing application tool 101 may also include functions for recommending and selecting grades.

[0099] According to an exemplary embodiment, the viewing application tool 101 may include the ability to search for similar cases using the viewing application tool 101. For example, it may identify regions of interest, rotate and quantize regions of interest, and identify cases with similar regions of interest and attach them to the patient case.

[0100] Viewing application tool 101 may include functionality for providing mitotic counting. For example, regions of interest (ROIs) can be identified, moved / rotated, and quantified, and cases with similar ROIs can be identified and attached to patient cases. Viewing application tool 101 can display counting results, including the number of mitotic divisions and / or visual markers. Identified regions can be zoomed in and out, and users can confirm and / or edit the counting results. Results can be shared across systems.

[0101] Figure 4 An exemplary output 400 of an overlay tool 112 of a viewing application tool 101 according to an exemplary embodiment is illustrated. The overlay tool 112 can provide a heatmap overlay 401 on a target image 303. The heatmap overlay can identify regions of interest on a tissue sample of the target image. For example, a heatmap overlay can identify areas in a tissue sample that an AI system predicts may contain abnormalities. The predictive heatmap overlay visualization can be toggled on and off using a navigation menu 301. For example, a user can select an overlay icon on the navigation menu to toggle the heatmap overlay on or off. The predictive heatmap interface can indicate to a user (e.g., a pathologist) one or more areas on the tissue that the user should examine. The heatmap overlay can be transparent and / or semi-transparent, so that the user can see the underlying tissue when viewing the heatmap overlay. The heatmap overlay can include different colors and / or shading to indicate the severity of the detected disease. Other types of overlays may be available in the context of the type of tissue being viewed. For example, Figure 4 The illustration depicts a prostate biopsy. However, other diseases may require different visualization or AI systems associated with them.

[0102] Figure 5 The illustration shows an exemplary output 500 of a worklist tool 107 for viewing application tool 101 according to an exemplary embodiment. For example... Figure 5 As illustrated in the diagram, the viewing application tool 101 may include a worklist 501, which can be opened by selecting the worklist icon in the navigation menu 301. The main navigation menu 301 provides an overview of the end-to-end workflow for slide viewing and / or case management. For example, the overview may include worklist 501, which can display cases for all users (e.g., pathologists). This can be accessed from the viewer user interface (UI) by opening the worklist panel. Within the worklist panel, pathologists can view the patient's Medical Record Number (MRN), Patient ID, Patient Name, Suspected Disease Type, Case Status, and Surgery Date, and select from various case actions, such as viewing case details, case sharing, and / or searching.

[0103] Figure 6The illustration shows an exemplary output 600 of the slide tray tool 108 of the viewing application tool 101 according to an exemplary embodiment. Figure 6 As illustrated in the diagram, the slide icon in navigation menu 301 opens slide tray 601, functioning as a drawer / drop-down menu from the menu on the left side of the screen. This menu can present advanced case information, including case number, demographic information, etc. Case slides can be organized into sections, represented in slide tray 601 by dark gray parent tabs. Each section can be expanded to display the slides within it. Thus, slides identified as associated with the same section can be displayed within a predetermined proximity to each other; this section could be a predetermined area of ​​a sample or a predetermined sample area of ​​a patient. Red dots 602 on sections can be indicators of different colors or shapes, indicating that the AI ​​system has detected a possible disease (e.g., cancer) somewhere within that section. Red dots 603 on slides indicate that the AI ​​system has detected a possible disease somewhere on that slide. These sections and slides where the AI ​​system has detected possible diseases can be moved to the top of the list for immediate viewing by users (e.g., pathologists). This workflow assists pathologists in identifying diseases more quickly.

[0104] Figure 7 An exemplary output 700 of the slide sharing tool 109 of a viewing application tool 101 according to an exemplary embodiment is illustrated. From the slide panel, a user can select a “Share” button and choose to share one, multiple, and / or all slides. The slide sharing panel 701 may be positioned to the right of the slide tray 601 and may allow the user to enter the recipients of the share, select various slides to include, and / or write a short comment about the nature of the share. Once the send button is selected, the slides and brief comment can be sent to recipients who can receive notifications to view these slides and / or comments in their viewing application. The conversation between users can be captured in the log described below.

[0105] Figure 8 An exemplary output 800 of the annotation tool 110 of a viewing application tool 101 according to an exemplary embodiment is illustrated. For each annotation made, the annotation log 801 can capture a visual image of the actual annotation made, the type of annotation, any measurements, comments, the user who made the annotation, and / or the time of the annotation. The annotation log 801 can be used as a centralized view for consultations and informal sharing, viewed in the context of annotations or areas of interest. Users requesting consultation (e.g., pathologists) and consulting pathologists can read, comment on, and / or have ongoing conversations specific to each annotation, each with a timestamp. Users can also select thumbnails within the annotation log 801 to view areas of interest at a large scale within the main viewer window.

[0106] Figure 9 The illustration shows exemplary output of the annotation tool 110 of the viewing application tool 101 according to an exemplary embodiment. For example, Figure 9 The annotation log shown in the image includes conversations between pathologists discussing slides. Pathologists can send descriptions along with annotated images for quick consultation on any area of ​​interest on the slide.

[0107] Figure 10 The illustration shows an exemplary output 1000 of the annotation tool 110 of a viewing application tool 101 according to an exemplary embodiment. The viewing application may include a variety of annotation tools illustrated in the annotation menu 1001. For example, annotation tools may include a brush tool, an auto-brush tool, a trajectory tool, a lock tool, an arrow tool, a text field tool, a region of detail tool, a region of interest (ROI) tool, a prediction tool, a measurement tool, a multiple measurement tool, a region-of-focus drawing tool, a region of interest tool, a labeling tool, a commenting tool, and / or a screenshot tool. Annotations may be used for research purposes and / or in a clinical setting. For example, a pathologist may make annotations, write comments, and / or share them with colleagues to obtain a second opinion. Annotations may also be used as supporting evidence for a diagnosis in the final diagnostic report (e.g., a tumor is...). x Length, having y The number of mitotic counts, etc.

[0108] Figure 11 The illustration shows an exemplary output 1100 of an inspection tool 111 of a viewing application tool 101 according to an exemplary embodiment. The inspection tool 111 may include an inspection window 1101 characterized by a magnified view of the region of interest of a target image. Based on user input, the inspection window can be dragged across the image to quickly inspect it. Therefore, pathologists can be able to move quickly across a slide in a manner similar to how slides are currently moved in a microscope. When viewing through the inspection tool 111, the user can quickly toggle or switch the predictive heatmap overlay on and off. The user can capture screenshots of viewable tissue within the inspection tool and quickly share them. Different annotation tools may also be available along with inspection tools such as measurement tools and / or region highlighting. Additionally, the user can increase and decrease the magnification within the inspection tool independently of the magnification level of the main slide.

[0109] Figure 12 The illustration shows an exemplary output 1200 of the inspection tool 111 of the viewing application tool 101 according to an exemplary embodiment. For example... Figure 12As illustrated in the diagram, inspection window 1201 can display the AI ​​output in a magnified view for verification and diagnostic assistance. Inspection window 1201 may include a heatmap overlay displaying the characteristics of the tissue sample predicted by the AI ​​system. For example, the heatmap overlay can identify areas in the tissue sample that the AI ​​system predicts may contain abnormalities.

[0110] Figure 13 The illustration depicts an exemplary workflow of a viewing application tool 101 according to an exemplary embodiment. The workflow may include a work list, a case view, and / or a slide view as patterns within a user's (e.g., a pathologist's) workflow. Figure 13 As shown in the diagram, a diagnostic report can be output from the case view, which may include the information necessary for digital check-out.

[0111] Figure 14A , Figure 14B and Figure 14C The illustration shows an exemplary output of a slide view from an application tool 101 according to an exemplary embodiment. For example, as... Figure 14A As shown in the bottom illustration, the slide view toolbar 1401 can be positioned at the bottom of the display with the slide vertically oriented. Slides can be grouped by section, and relevant case information within that context can be displayed. For example... Figure 14B As shown in the diagram, toolbar 1401 can be positioned on the left side of the monitor. The toolbar can include menu items such as tools, zoom functions, and thumbnail slides. Figure 14C As shown in the diagram, the two vertical toolbars 1401 and 1402 can be positioned on the left and right sides of the screen, which allows easy access to a variety of functions of the touchscreen device (e.g., tools and / or views).

[0112] Figure 15 The illustration depicts an exemplary workflow for requesting and receiving consultations by creating an account for a new user (e.g., a patient, physician, etc.) according to an exemplary embodiment. A request to create a new account can be submitted via the hospital website and / or by viewing application tool 101. Once an account is created, pathologists can send instructions along with annotated images for quick consultation on any area of ​​interest on a slide. Patients can also receive consultation information from pathologists and / or billing information related to their case. If the user is a physician, billing options may differ depending on whether the billing party is the referring physician, patient, and / or insurance company. If the user is a patient, billing can be made directly to the patient or through an insurance company.

[0113] Figure 16A and Figure 16B The illustration shows an exemplary output of a pathology consultation dashboard according to an exemplary embodiment. Figure 16AAs illustrated in the diagram, new requests and accesses to consultations may include the patient's name, referring physician, external ID, internal ID, institution, request date, status, update date, and / or links to additional details. The consultation dashboard may include functionality for selecting slides to send, a slide sending dialog box, consultation request instructions, consultation attachments, and / or a consultation log. Figure 16B As illustrated in the diagram, a consultation request may include the patient's name, referring physician, institution, request date, status, update date, and / or a link to additional details about the request.

[0114] Figure 17 The illustration shows an exemplary output of a case view of a viewing application tool 101 according to an exemplary embodiment. The case view may include all the information necessary for a digital stamp workflow. Additionally, as Figure 17 As shown in the diagram, the annotation log can be integrated into the case view.

[0115] like Figure 18 As shown, device 1800 may include a central processing unit (CPU) 1820. CPU 1820 can be any type of processor device, including, for example, any type of dedicated or general-purpose microprocessor device. As those skilled in the art will appreciate, CPU 1820 can also be a single processor in a multi-core / multi-processor system, such a system operating independently, or in a cluster of computing devices operating in a cluster or server group. CPU 1820 can be connected to data communication infrastructure 1810, such as a bus, message queue, network, or multi-core messaging scheme.

[0116] Device 1800 may also include main memory 1840, such as random access memory (RAM), and may also include auxiliary memory 1830. Auxiliary memory 1830, such as read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may include, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, flash memory, etc. The removable storage drive in this example reads from and / or writes to a removable storage unit in a known manner. The removable storage unit may include floppy disks, magnetic tapes, optical disks, etc., read from and written to by the removable storage drive. As those skilled in the art will appreciate, such a removable storage unit typically includes a computer-usable storage medium in which computer software and / or data are stored.

[0117] In an alternative implementation, auxiliary memory 1830 may include other similar means for allowing computer programs or other instructions to be loaded into device 1800. Examples of such means may include program cassette tapes and cassette tape interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, as well as other removable memory units and interfaces that allow software and data to be transferred from the removable storage unit to device 1800.

[0118] Device 1800 may also include a communication interface (“COM”) 1860. Communication interface 1860 allows software and data to be transferred between device 1800 and external devices. Communication interface 1860 may include a modem, network interface (such as an Ethernet card), communication port, PCMCIA slot, and cards, etc. Software and data transmitted via communication interface 1860 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by communication interface 1860. These signals may be provided to communication interface 1860 via communication paths of device 1800, which may be implemented using, for example, wires or cables, fiber optic cables, telephone lines, cellular telephone links, RF links, or other communication channels.

[0119] The hardware components, operating system, and programming language of such a device are conventional in nature and are assumed to be sufficiently familiar to those skilled in the art. Device 1800 may also include input and output ports 1850 for connection to input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. Of course, various server functions can be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the server can be implemented through appropriate programming of a single computer hardware platform.

[0120] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or a set of related functions. The same reference numerals are generally intended to refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0121] The aforementioned tools, modules, and functions can be executed by one or more processors. "Storage" type media can include any or all tangible memory of a computer, processor, etc., or associated modules such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time.

[0122] Software can communicate via the Internet, cloud service providers, or other telecommunications networks. For example, communication can enable the loading of software from one computer or processor into another. As used herein, unless limited to non-transitory tangible "storage" media, terms such as "computer or machine-readable medium" refer to any medium involved in providing instructions to the processor for execution.

[0123] The foregoing general description is exemplary and explanatory only and is not intended to limit this disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered exemplary only.

Claims

1. A computer-implemented method for analyzing electronic images corresponding to samples, the method comprising: Receive a target electronic image corresponding to a target sample, the target sample including a tissue sample from the patient; The machine learning system is applied to a target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image. The machine learning system is generated by processing multiple training images to predict at least one characteristic. The training images include images of human tissue and / or images generated by the algorithm. The processing of the multiple training images is carried out in multiple stages by using a first set of images with first-level annotations in a first stage and using images with fewer annotations than the first set of images in at least a second stage. Output a target electronic image that identifies the region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image; Display the target electronic image on the monitor; Receive a second target electronic image corresponding to the target sample; Identify the first portion of the target sample associated with the target electronic image; Identify the second portion of the target sample associated with the second target electronic image; Indicate whether the first and second parts are identical or overlapping; as well as In response to identifying that the first portion and the second portion are identical or overlapping, a first representation of the target electronic image and a second representation of the second target electronic image are displayed at a predetermined proximity to each other.

2. The computer-implemented method according to claim 1, wherein, Identifying regions of interest includes displaying a heatmap overlay on the target electronic image.

3. The computer-implemented method according to claim 1, wherein, Identifying regions of interest includes displaying a thermal overlay on a target electronic image, the thermal overlay including shading and / or coloring based on the predicted probability of anomalies at the location.

4. The computer-implemented method according to claim 1, wherein, Identifying regions of interest includes displaying a thermal overlay on the target electronic image, the thermal overlay comprising shading and / or coloring based on the predicted probability of anomalies at the location. The heatmap overlay is transparent or semi-transparent.

5. The computer-implemented method of claim 1, further comprising displaying a magnified window over at least a portion of the target electronic image, and A magnified image of the target sample is presented in the magnification window at a magnification level different from that of the target electronic image.

6. The computer-implemented method of claim 1, further comprising displaying a magnified window over at least a portion of the target electronic image; and A magnified image of the target sample is presented in the magnification window at a different magnification level than the target electronic image. The zoom-in window includes selectable icons for toggling the heatmap overlay onto the zoomed-in image.

7. The computer-implemented method according to claim 1, further comprising: A slide tray tool that displays an overview of the target sample on the target electronic image; Machine learning systems are applied to target electronic images to determine whether a portion of a target sample contains anomalies. and In response to determining that the portion contains an anomaly, an anomaly indicator is displayed in the slide tray tool.

8. The computer-implemented method according to claim 1, further comprising: Display the annotation log, which includes indicators that identify regions of interest and consultation requests related to those regions.

9. The computer-implemented method according to claim 1, further comprising: Determine whether there is a region of interest associated with the target electronic image and / or a region of interest associated with the second target electronic image; In response to determining a region of interest associated with a target electronic image, an indicator associated with a first representation of the target electronic image is displayed; and In response to determining a region of interest associated with a second target electronic image, an indicator associated with a second representation of the second target electronic image is displayed.

10. A system for analyzing an electronic image corresponding to a sample, the system comprising: At least one memory for storing instructions; and At least one processor executes instructions to perform a process that includes the following: Receive a target electronic image corresponding to a target sample, the target sample including a tissue sample from the patient; The machine learning system is applied to a target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image. The machine learning system is generated by processing multiple training images to predict at least one characteristic. The training images include images of human tissue and / or images generated by the algorithm. The processing of the multiple training images is carried out in multiple stages by using a first set of images with first-level annotations in a first stage and using images with fewer annotations than the first set of images in at least a second stage. Output a target electronic image that identifies the region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image; Display the target electronic image on the monitor; Receive a second target electronic image corresponding to the target sample; Identify the first portion of the target sample associated with the target electronic image; Identify the second portion of the target sample associated with the second target electronic image; Indicate whether the first and second parts are identical or overlapping; as well as In response to identifying that the first portion and the second portion are identical or overlapping, a first representation of the target electronic image and a second representation of the second target electronic image are displayed at a predetermined proximity to each other.

11. The system according to claim 10, wherein, Identifying regions of interest includes displaying a heatmap overlay on the target electronic image.

12. The system according to claim 10, wherein, Identifying regions of interest includes displaying a thermal overlay on a target electronic image, the thermal overlay including shading and / or coloring based on the predicted probability of anomalies at the location.

13. The system according to claim 10, wherein, Identifying regions of interest includes displaying a thermal overlay on the target electronic image, the thermal overlay comprising shading and / or coloring based on the predicted probability of anomalies at the location. The heatmap overlay is transparent or semi-transparent.

14. The system of claim 10, further comprising displaying a magnified window over at least a portion of the target electronic image, and A magnified image of the target sample is presented in the magnification window at a magnification level different from that of the target electronic image.

15. The system of claim 10, further comprising displaying a magnified window over at least a portion of the target electronic image; and A magnified image of the target sample is presented in the magnification window at a different magnification level than the target electronic image. The zoom-in window includes selectable icons for toggling the heatmap overlay onto the zoomed-in image.

16. The system of claim 10, further comprising: A slide tray tool that displays an overview of the target sample on the target electronic image; Machine learning systems are applied to target electronic images to determine whether a portion of a target sample contains anomalies. and In response to determining that the portion contains an anomaly, an anomaly indicator is displayed in the slide tray tool.

17. The system of claim 10, further comprising: Display the annotation log, which includes indicators that identify regions of interest and consultation requests related to those regions.

18. The system of claim 10, further comprising: Determine whether there is a region of interest associated with the target electronic image and / or a region of interest associated with the second target electronic image; In response to determining a region of interest associated with a target electronic image, an indicator associated with a first representation of the target electronic image is displayed; and In response to determining a region of interest associated with a second target electronic image, an indicator associated with a second representation of the second target electronic image is displayed.

19. A non-transitory computer-readable medium storing instructions, said instructions, when executed by a processor, causing the processor to perform a method for analyzing an electronic image corresponding to a sample, said method comprising: Receive a target electronic image corresponding to a target sample, the target sample including a tissue sample from the patient; The machine learning system is applied to a target electronic image to determine at least one characteristic of the target sample and / or at least one characteristic of the target electronic image. The machine learning system is generated by processing multiple training images to predict at least one characteristic. The training images include images of human tissue and / or images generated by the algorithm. The processing of the multiple training images is carried out in multiple stages by using a first set of images with first-level annotations in a first stage and using images with fewer annotations than the first set of images in at least a second stage. Output a target electronic image that identifies the region of interest based on at least one characteristic of the target sample and / or at least one characteristic of the target electronic image; Display the target electronic image on the monitor; Receive a second target electronic image corresponding to the target sample; Identify the first portion of the target sample associated with the target electronic image; Identify the second portion of the target sample associated with the second target electronic image; Indicate whether the first and second parts are identical or overlapping; as well as In response to identifying that the first portion and the second portion are identical or overlapping, a first representation of the target electronic image and a second representation of the second target electronic image are displayed at a predetermined proximity to each other.

20. The non-transitory computer-readable medium according to claim 19, wherein, Identifying regions of interest includes displaying a thermal overlay on a target electronic image, the thermal overlay including shading and / or coloring based on the predicted probability of anomalies at the location.

Images