Systems and methods for processing electronic images to generate tissue map visualizations - Patents.com

Abstract

A system and method for analyzing images of a slide corresponding to a specimen is disclosed, the method including receiving at least one digital image of the pathology specimen, using the digital image with an artificial intelligence (AI) system to determine at least one salient feature, the salient feature including a biomarker, cancer, cancer grade, parasite, toxicity, inflammation, and / or cancer subtype, determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel of the overlay, and suppressing one or more non-salient regions of the digital image based on the value of each pixel.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 041,778, filed June 19, 2020, the entire disclosure of which is incorporated herein by reference.

[0002] Various embodiments of the present disclosure relate generally to image-based tissue visualization and related image processing methods. More specifically, certain embodiments of the present disclosure relate to systems and methods for tissue visualization based on image processing of tissue specimens. [Background technology]

[0003] Pathology is a highly visual field requiring the identification and expert interpretation of morphological and histological patterns. Whole-slide images of pathology specimens consist of hundreds of thousands of pixels that pathologists must review. To assist them, artificial intelligence (AI) systems can be created to present pathologists with heat map overlays that highlight salient image regions, such as tumors. However, heat map overlays can obscure tissue and hinder the pathologist's ability to examine those regions.

[0004] The background discussion provided herein is intended to generally present the context of the disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and inclusion in this section is not admitted to be prior art or to suggest prior art. Summary of the Invention [Means for solving the problem]

[0005] According to one aspect of the present disclosure, a system and method for analyzing an image of a slide corresponding to a specimen is disclosed.

[0006] 1. A method for analyzing images of a slide corresponding to a specimen, the method comprising: receiving at least one digital image of a pathology specimen; using the digital image with an AI system to determine at least one salient feature, the salient feature including a biomarker, cancer, a cancer grade, a parasite, toxicity, inflammation, and / or a cancer subtype; determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel; and suppressing one or more non-salient regions of the digital image based on the value of each pixel.

[0007] A system for analyzing an image of a slide corresponding to a specimen includes a memory storing instructions and at least one processor, the at least one processor executing the instructions to perform a process including receiving at least one digital image of a pathology specimen, using the digital image with an AI system to determine at least one salient feature, the salient feature including a biomarker, cancer, a cancer grade, a parasite, toxicity, inflammation, and / or a cancer subtype, determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel of the overlay, and suppressing one or more unsalient regions of the digital image based on the value of each pixel.

[0008] 1. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing images of a slide corresponding to a specimen, the method including: receiving at least one digital image of a pathology specimen; using the digital image with an AI system to determine at least one salient feature, the salient feature including a biomarker, cancer, a cancer grade, a parasite, toxicity, inflammation, and / or a cancer subtype; determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel; and suppressing one or more non-salient regions of the digital image based on the value of each pixel.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed.

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. The present invention provides, for example, the following. (Item 1) 1. A computer-implemented method for analyzing an image of a slide corresponding to a specimen, comprising: receiving at least one digital image of a pathology specimen; using the digital image with an artificial intelligence (AI) system to determine at least one distinctive feature, the at least one distinctive feature comprising a biomarker, cancer, cancer grade, parasites, toxicity, inflammation, and / or cancer subtype; determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel; suppressing one or more non-salient regions of the digital image based on the value of each pixel; The computer-implemented method includes: (Item 2) Item 10. The computer-implemented method of item 1, further comprising converting the overlay of salient regions into a tissue map. (Item 3) Item 10. The computer-implemented method of item 1, further comprising normalizing the overlay of salient regions to obtain a parameter. (Item 4) The overlay of the salient regions comprises: a heatmap showing the score or probability of each pixel of a salient feature being present or absent; a set of superpixels associated with a score or probability of the salient feature being present or absent; a binary segmentation of the digital image indicating whether each pixel has the salient feature present; and a semantic segmentation of said digital image indicating a score or probability for each pixel; Item 4. The computer-implemented method of item 3, represented by one or more of: (Item 5) Item 10. The computer-implemented method of item 1, wherein detecting salient features uses image processing techniques, AI, and / or machine learning on the digital image to generate a tissue visualization. (Item 6) Item 10. The computer-implemented method of item 1, wherein the digital image includes associated case information, patient information, and information from clinical systems. (Item 7) Item 10. The computer-implemented method of item 1, further comprising alerting a user when the salient region overlay is available. (Item 8) Item 10. The computer-implemented method of item 1, wherein the salient region overlay is resized to the same size as the digital image. (Item 9) 2. The computer-implemented method of claim 1, further comprising developing a pipeline for archiving the plurality of processed images and / or prospective patient data. (Item 10) 1. A system for analyzing an image of a slide corresponding to a specimen, comprising: at least one memory for storing instructions; at least one processor configured to execute the instructions to perform operations, the operations comprising: receiving at least one digital image of a pathology specimen; using the digital image with an artificial intelligence (AI) system to determine at least one distinctive feature, the at least one distinctive feature comprising a biomarker, cancer, cancer grade, parasites, toxicity, inflammation, and / or cancer subtype; determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel; suppressing one or more non-salient regions of the digital image based on the value of each pixel; Including, The system. (Item 11) 11. The system of claim 10, further comprising converting the overlay of salient regions into a tissue map. (Item 12) Item 11. The system of item 10, further comprising normalizing the overlay of salient regions to obtain a parameter. (Item 13) The overlay of the salient regions comprises: a heatmap showing the score or probability of each pixel of a salient feature being present or absent; a set of superpixels associated with a score or probability of the salient feature being present or absent; a binary segmentation of the digital image indicating whether each pixel has the salient feature present; and a semantic segmentation of said digital image indicating a score or probability for each pixel; Item 13. The system of item 12, represented by one or more of: (Item 14) Item 11. The system of item 10, wherein detecting salient features uses image processing techniques, AI, and / or machine learning on the digital image to generate a tissue visualization. (Item 15) Item 11. The system of item 10, wherein the digital image includes associated case information, patient information, and information from clinical systems. (Item 16) Item 11. The system of item 10, further comprising alerting a user when the salient region overlay is available. (Item 17) Item 11. The system of item 10, wherein the salient region overlay is resized to the same size as the digital image. (Item 18) Item 11. The system of item 10, further comprising developing a pipeline for archiving a plurality of processed images and / or prospective patient data. (Item 19) 1. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image of a slide corresponding to a specimen, the method comprising: receiving at least one digital image of a pathology specimen; using the digital image with an artificial intelligence (AI) system to determine at least one distinctive feature, the at least one distinctive feature comprising a biomarker, cancer, cancer grade, parasites, toxicity, inflammation, and / or cancer subtype; determining with the AI ​​system an overlay of salient regions of the digital image, the AI ​​system indicating a value for each pixel; suppressing one or more non-salient regions of the digital image based on the value of each pixel; The non-transitory computer-readable medium comprising: (Item 20) 20. The non-transitory computer-readable medium of claim 19, further comprising converting the overlay of salient regions into a tissue map. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is an example of a heatmap showing the presence of disease in a biopsy. [Figure 2A] 1 shows an example block diagram of a system and network for tissue visualization of images, according to an example embodiment of the present disclosure. [Figure 2B] 2 shows an exemplary block diagram of a disease detection platform 200 according to an exemplary embodiment of the present disclosure. [Figure 3] 1 shows an exemplary visualization of specific regions that an AI detects as having diagnostic value, and suppresses the visualization of regions without diagnostic value, according to an exemplary embodiment of the present disclosure. [Figure 4] 10 illustrates an example of visualization of a specific region based on features, according to an exemplary embodiment of the present disclosure. [Figure 5] 1 is a flowchart of an exemplary method for providing tissue visualization of a digitized pathology image, according to an exemplary embodiment of the present disclosure. [Figure 6A] An example of a heatmap showing the presence of disease, where the tissue under consideration is obscured. [Figure 6B]10 shows an example of visualization of diseased areas without obscuring the tissue under consideration, according to an exemplary embodiment of the present disclosure. [Figure 7] 1 illustrates an example of a system that may implement the techniques presented herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0013] The systems, devices, and methods disclosed herein are described in detail, by way of example, with reference to the drawings. The examples described herein are merely examples and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. No feature or component shown in the drawings or described below should be considered essential to any particular implementation of these devices, systems, or methods, unless specifically designated as essential.

[0014] Also, with respect to a described method, whether or not the method is described in conjunction with a flow diagram, it should be understood that unless otherwise specified or required by context, the express or implied order of steps performed in the implementation of the method does not imply that these steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

[0015] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Furthermore, the words "a" and "an" as used herein do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced item.

[0016] As AI becomes increasingly integrated into pathologists' diagnostic and research workflows, the interpretability of AI technology results and the user experience when using AI are critical to pathologists' ability to effectively utilize them. Some techniques may use AI as a post-processing mechanism to overlay visualizations of predictions onto images; however, this can hinder pathologists' access to and understanding of results.

[0017] Once a pathologist recognizes an area of ​​interest (e.g., a tumor, an interesting morphological finding, and / or something that needs to be examined or reviewed), they may draw a border around the area, for example, with a marker directly on the slide. AI-based image analysis may be used to overlay a heat map on the digitized pathology image. In one approach, unimportant areas are given no overlay, while important areas are given an overlay. The overlay may be a gradient representing a probability map generated by an algorithm.

[0018] However, using the above method, there may be an unsightly overlay in the relevant area: because the heatmap may be displayed on top of the digital image, the pathologist may have to repeatedly toggle the overlay on and off to compare what the AI ​​has noted with the actual tissue underneath the heatmap.

[0019] The AI ​​system can generate a score and / or probability for each analyzed pixel relative to the question of interest (e.g., "Is cancer present?", "Is this a high-grade cancer?", etc.), and these scores can be used to create a heat map (e.g., Figure 1). While this can be a useful visualization for some use cases, for pathologists, it can be detrimental to the process of rendering a final interpretation. In fact, it can confuse pathologists when they try to determine the meaning of the heat map colors in relation to tissue morphology. For example, an AI might predict that a first region of a prostate needle biopsy is more likely to be Gleason Grade 3 than a second region (e.g., the heat map may indicate this). However, since the pathologist makes the final decision, knowing that one region is predicted to be more likely to be Gleason Grade 3 than the other may not be helpful to the pathologist.

[0020] FIG. 1 is an example of a heatmap indicating the presence of cancer or other disease in a biopsy (e.g., a prostate needle core biopsy). As shown in FIG. 1, a region of interest 1 has tissue 10 with a heatmap 11. The heatmap 11 may obscure the tissue 10 and suggest potentially misleading diagnostic associations with different colors. Additionally, a binary bitmap may be used to mark pixels predicted to have a particular score (e.g., probability) above a pre-specified threshold, indicating a significant region.

[0021] This disclosure allows pathologists to focus their attention on multiple salient image regions without obtrusive overlays on top of these regions of interest. This type of visualization optimizes the pathologist's digital pathology workflow by making AI results easier to identify, interpret, and engage with.

[0022] The present disclosure provides tissue visualization that does not obscure areas on a slide identified as important for review by a pathologist, resulting in more efficient interpretation and workflow (e.g., no need to turn output on and off), allowing for faster and more accurate diagnoses. Additionally, the output is easier for a pathologist to interpret.

[0023] The present disclosure can use AI techniques to detect features of interest (e.g., biomarkers, cancer, cancer grade, parasites, toxicity, inflammation, cancer subtype, etc.) that may be necessary for pathology evaluation and treatment decisions. The AI ​​can generate an overlay of salient regions and convert the overlay of salient regions into a visualization that suppresses irrelevant image regions or image regions determined to have no diagnostic value above a predetermined threshold.

[0024] Important regions of the entire slide image may be displayed and highlighted to a user (e.g., a pathologist) to help the user complete a specific task in the diagnostic process (e.g., cancer detection, grading, etc.), which requires minimal visual and usability overhead. One or more embodiments may include providing the tissue visualization to a user within a clinical workflow at a hospital, laboratory, medical center, etc., as at least one of a web application (e.g., cloud-based and / or on-premise), a mobile application, an interactive report, a static report, and / or a dashboard.

[0025] To improve usability and / or efficiency, the identified region(s) may be organized into a report with summary information. Additionally, interactive review / editing may be provided to the user during review of the digital image. Multiple features may be visualized across a single slide image.

[0026] A technical workflow according to one or more embodiments may be as follows: a digitized whole slide image may be created; metadata may be generated and / or determined; the metadata may be available from hospital and / or hardware databases; the image and corresponding data may be provided to an AI-based system, and an output may be generated; a portion of the output may be fed to one or more systems, which generate and / or display a visualization (e.g., one or more points or regions) to a user (e.g., a pathologist); the analysis and / or display may be generated based on a query of interest (e.g., cancer, nuclear features, cell count, etc.).

[0027] Additionally, one or more embodiments of the present disclosure may be used for pre-screening (i.e., before a pathologist reviews the images) and / or after a diagnosis has been made (e.g., quality assurance).

[0028] FIG. 2A shows an example block diagram of a system and network for tissue visualization in images, according to an example embodiment of the present disclosure.

[0029] 2A illustrates an electronic network 220 that can be connected to servers at a hospital, laboratory, doctor's office, etc. For example, physician server 221, hospital server 222, clinical trial server 223, laboratory server 224, and / or laboratory information system 225, etc., may each be connected to and communicate via electronic network 220, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present application, electronic network 220 may also be connected to server system 210, which may include a processing device configured to implement disease detection platform 200, which includes tissue visualization tool 201 for generating tissue visualization of digital pathology image(s) using machine learning, according to an exemplary embodiment of the present disclosure. Exemplary machine learning models may include, but are not limited to, any one or any combination of neural networks, convolutional neural networks, random forests, logistic regression, and / or nearest neighbors.

[0030] The physician server 221, hospital server 222, clinical trial server 223, laboratory server 224, and / or laboratory information system 225 may create or otherwise acquire one or more patient's cytology specimen(s), oncology specimen(s), slide(s) of cytology / oncology specimen(s), digital images of slide(s) of cytology / oncology specimen(s), or any combination thereof. The physician server 221, hospital server 222, clinical trial server 223, laboratory server 224, and / or laboratory information system 225 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsy or cytology information, etc. The physician server 221, hospital server 222, clinical trial server 223, laboratory server 224, and / or laboratory information system 225 may transmit the digitized slide images and / or patient-specific information to the server system 210 via the electronic network 220. The server system 210 may include one or more storage devices 209 for storing images and / or data received from at least one of the physician server 221, the hospital server 222, the clinical trial server 223, the laboratory server 224, and / or the laboratory information system 225. The server system 210 may also include a processing device for processing the images and / or data stored in the storage device 209. The server system 210 may further include one or more machine learning tools or functions. For example, according to one embodiment, the processing device may include machine learning tools for the disease detection platform 200. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

[0031] Physician server 221, hospital server 222, clinical trial server 223, laboratory server 224, and / or laboratory information system 225 refer to systems that a pathologist can use to review images of slides. In a hospital environment, tissue type information may be stored in laboratory information system 225.

[0032] FIG. 2B shows an exemplary block diagram of a disease detection platform 200 for generating tissue visualization of digital pathology image(s) using machine learning.

[0033] 2B illustrates components of a disease detection platform 200 according to one embodiment. For example, the disease detection platform 200 may include a tissue visualization tool 201, a data capture tool 202, a slide capture tool 203, a slide scanner 204, a slide manager 205, storage 206, and / or a viewing application tool 208.

[0034] As described below, tissue visualization tool 201 refers to a process and system for generating tissue visualization for digital pathology image(s) using machine learning, according to an exemplary embodiment.

[0035] Data ingestion tools 202 refer to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and / or devices used to classify and / or process the digital pathology images, according to an example embodiment.

[0036] According to an exemplary embodiment, slide capture tool 203 refers to a process and system for scanning pathology images and converting them into digital format. Slides may be scanned using slide scanner 204, and slide manager 205 may process the images on the slides into digitized pathology images and store the digital images in storage, such as storage 206 and / or storage device 209.

[0037] The viewing application tool 208 refers to processes and systems for providing specimen or image characteristic information related to digital pathology image(s) to a user (e.g., a pathologist), according to an exemplary embodiment. The information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0038] The tissue visualization tool 201, and each of its components, may transmit and / or receive digitized slide images and / or patient information to and / or from a server system 210, a physician server 221, a hospital server 222, a clinical trial server 223, a laboratory server 224, and / or a laboratory information system 225 via an electronic network 220. Additionally, the server system 210 may include a storage device for storing images and / or data from at least one of the tissue visualization tool 201, the data capture tool 202, the slide capture tool 203, the slide scanner 204, the slide manager 205, and / or the viewing application tool 208. The server system 210 may also include a processing device for processing the images and / or data stored in the storage device. The server system 210 may further include one or more machine learning tool(s) or functionality, e.g., via the processing device. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

[0039] The above devices, tools, and / or modules may be located on devices that may be connected to an electronic network 220, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0040] 3 shows an example of visualization of specific regions detected by AI as having diagnostic value, such as cancer, with visualization of non-disease regions or other regions lacking diagnostic value suppressed in accordance with the techniques described herein. In region of interest 1, tissue 10 has non-disease regions 12 suppressed compared to specific regions 13 of diagnostic value. Display icons 14 may also be included in the visualization.

[0041] 4 shows an example of visualization of specific regions based on features (e.g., cancer grade), with visualization of non-disease regions, or other regions lacking diagnostic value associated with the feature, suppressed according to the techniques described herein. In region of interest 1, tissue 10 has non-disease regions 12 suppressed compared to specific regions 13 of diagnostic value. Display icons 14 may also be included in the visualization.

[0042] 5 is a flowchart illustrating an exemplary method for providing tissue visualization of a digitized pathology image, according to an exemplary embodiment of the present disclosure. For example, exemplary method 500 (e.g., steps 502-508) may be performed by tissue visualization tool 201 automatically or in response to a request from a user (e.g., a pathologist, a patient, an oncologist, etc.).

[0043] An exemplary method 500 for developing a tissue visualization tool may include one or more of the following steps: In step 502, the method may include receiving at least one digital image of a pathology specimen (e.g., histology), which may also include associated case and patient information (e.g., specimen type, case and patient ID, portion within the case, overall description, etc.) and / or information from a clinical system (e.g., assigned pathologist, specimen available for testing, etc.). The method may include developing a pipeline for archiving processed images and / or prospective patient data. Additionally, the data may be stored on a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).

[0044] In step 504, the method may include using the digital image with an AI system to determine at least one distinctive feature, the at least one distinctive feature including a biomarker, cancer, a cancer grade, a parasite, toxicity, inflammation, and / or a cancer subtype.

[0045] In step 506, the method may include determining, in an AI system, an overlay of salient regions of the digital image, where the AI ​​system indicates a value for each pixel. The AI ​​system output may be an overlay M of salient regions of the digital input image indicating a value for each pixel. The AI ​​may represent M in multiple ways. For example, M may be represented by (1) a heat map indicating a score or probability for each pixel of the salient feature being present or absent; (2) a set of superpixels having associated scores or probabilities for the feature being present or absent; (3) a binary segmentation of the tissue indicating whether each pixel has the salient feature present; and / or (4) a semantic segmentation of the image indicating a score or probability for each pixel. Additionally, the salient regions overlay may be resized to the same size as the digital image.

[0046] In step 508, the method may include suppressing one or more non-salient regions of the digital image based on the value of each pixel. The salient region overlay may be processed (e.g., post-processed) and normalized to obtain S (e.g., normalized salient region overlay). For example, for a binary M, M may be left unmodified, i.e., S = M. As another example, using image processing techniques on a binary or continuous M, the method may include (1) setting S = M; (2) applying a smoothing operation, e.g., a Gaussian blur or a median blur, to S; (3) if S is not in the range of 0 to 1, normalizing S to this range, e.g., using linear contrast stretching; (4) if S has continuous values, thresholding S, setting all values ​​above a threshold to 1 and all values ​​below the threshold to 0; and / or (5) post-processing S using a morphological operator (e.g., closing, eroding, dilating) to improve visualization. As another example, using a segmentation of M using continuous scores or probabilities, a method may include (1) running a segmentation algorithm on M, such as a clustering method (e.g., k-means), region growing, graph-based methods, and / or other segmentation methods; (2) assigning a value to each segment based on the scores / probabilities of the pixels belonging to each segment, which can be done in many ways, for example, by taking the maximum score / probability, median, mean, and / or generalized mean of the underlying pixels; and / or (3) setting all values ​​in each segment with values ​​above a predetermined threshold to 1 and all values ​​below the threshold to 0 to obtain S from the segmentation and calculated segment values.

[0047] 6A is an example of a heatmap showing the presence of cancer, with obscured tissue. As mentioned above, heatmaps may obscure tissue relevant to a pathologist's diagnosis.

[0048] The techniques described with respect to Figure 5 and elsewhere herein may be used to generate the visualization of Figure 6B. Cancerous or other diagnosable problem areas are easily visible in the visualization without obscuring the tissue under consideration. Areas not relevant to the diagnosis, such as less salient areas, may be obscured or otherwise suppressed.

[0049] Once the visualization and / or report is generated, the user (e.g., pathologist, oncologist, patient, etc.) may be notified by a notification that the results are available. The user may be provided with the option to review the visualization and / or report. Alternatively, the visualization and report may be provided automatically. If the user chooses to view the visualization or the system is configured to display it automatically, the tissue map uses S to suppress pixels in S, i.e., pixel z, where the value of S(z) is 0. This indicates that the AI ​​will not interpret the pixel as diagnostic of the clinical feature of interest (it is not salient). This suppression may be done by at least one of "blacking out" these pixels, making them partially transparent, changing their brightness, and / or other visual differentiation methods.

[0050] Users can customize what the tissue visualization shows based on target outputs (which may include areas marked by the user or other users, morphological features, areas not seen by anyone, etc., based on tracking software). Tissue maps may be utilized to visualize multiple Ms (overlays of salient areas) on a single slide. If a slide has multiple Ms, the user may select which Ms (e.g., one or more) to display at a time. Users may add or remove areas within S based on expert assessment. For example, a user may use a mouse or input device to modify the visualization (e.g., to include more tissue), with corresponding details adjusted accordingly (e.g., as more tumor is identified, the display values ​​for tumor quantification are adjusted accordingly). Any changes may be reset to allow the user to return to the original projections and visualization.

[0051] The user's field of view may be moved to focus on each identified region of interest in order of priority, class, and / or other priority (e.g., a region of interest may be any area on the tissue map identified by the AI ​​or by the user as an area requiring further investigation). The output and / or visualized regions may be recorded as part of the medical history in a clinical reporting system.

[0052] Exemplary Cancer Detection Tool: According to one embodiment, a method includes identifying tissue regions that have cancer. The tissue map helps a user, e.g., a pathologist, more quickly identify tissue regions that have or are suspected of having cancer. A tissue visualization may be created using the steps described with respect to FIG. 5, using AI to generate an overlay M of salient regions that indicate which tissue regions have or are suspected of having cancer. The present disclosure provides systems and methods for modifying the tissue map, e.g., for target customization, editing capabilities, field of view, and / or recording / reporting purposes. A user may customize what the tissue map shows based on the definition of target outputs. For example, if the tissue visualization displays all regions detected as cancerous, a user may customize the output to consider or not consider certain features of the cancerous category (e.g., atypical ductal hyperplasia (ADH) may be considered cancerous at one hospital and not at another). Thus, a user at Hospital A may see all cancerous regions in breast biopsies that contain ADH, while a user at Hospital B may see all cancerous regions in breast biopsies that exclude ADH. The user may interact with and / or adjust the visualization to show more tissue (e.g., if they disagree with the results and believe more areas are cancerous) or less tissue (e.g., if they disagree with the results and believe the identified areas are not cancerous). The user's field of view may be moved to focus on each area of ​​interest in order of priority, class, or other priority. The output and / or visualized areas may be recorded as part of the medical history in a clinical reporting system.

[0053] Exemplary Cancer Grading Tool: According to one embodiment, a method includes characterizing tissue regions with cancer using a tissue map. The exemplary system and method may generate an overlay M of salient regions indicating which tissue regions have a particular grade of cancer, and the tissue visualization may be created using the steps described with respect to FIG. 5. A user can customize what the tissue visualization shows based on the definition of the target output. For example, cancer grading guidelines change over time. If a user wants to see how cancer was graded at different points in time, they can adjust it accordingly. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results and believe an area is cancerous and grade 3) or less tissue (e.g., if they disagree with the results and believe the identified area is neither cancerous nor grade 3). The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. The output and / or visualized areas may be recorded as part of the medical history in a clinical reporting system.

[0054] Exemplary Cancer Type or Precancerous Lesion Tool: According to one embodiment, the method includes a tissue visualization in which multiple forms of cancer (e.g., lobular carcinoma and / or ductal carcinoma) may occur. The tissue visualization may be created using the steps described with respect to FIG. 5 , with AI generating an overlay M of salient regions indicating which tissue regions are cancerous of a particular type. A user may customize what the tissue visualization shows based on the definition of the target output. For example, some users may prefer to see all potential precancerous and cancerous lesions on a slide, while other users may only want to report a few significant precancerous or atypical lesions. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results and believe a region is cancerous and ductal) or less tissue (e.g., if they disagree with the results and believe the identified region is neither cancerous nor ductal). The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. The output and / or visualized areas may be recorded as part of the medical history in a clinical reporting system.

[0055] Exemplary Non-Cancerous Feature Tool: According to one embodiment, the method includes identifying other non-cancerous features (e.g., fungi in skin pathology samples, bacteria in colon samples, atypia in breast samples, inflammation in many tissue types, etc.). A tissue visualization may be created using the steps described with respect to FIG. 5 , with AI generating an overlay M of salient regions indicating which tissue regions contain distinct biological features. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results and believe a region is fungal) or less tissue (e.g., if they disagree with the results and believe the identified region is not fungal). The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. The output and / or visualized regions may be recorded as part of the medical history in a clinical reporting system.

[0056] Exemplary Invasion Tool: According to one embodiment, a method includes determining the presence of invasion (e.g., microinvasion in breast cancer, muscularis propria invasion in bladder cancer, perineural invasion in prostate cancer, etc.). A tissue visualization may be created using the steps described with respect to FIG. 5, with AI generating an overlay M of salient regions indicating which tissue regions contain invasive cancer. A user may customize what the tissue visualization shows based on the definition of the target output. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results and believe a region is invasive) or less tissue (e.g., if they disagree with the results and believe the identified region is not invasive). The user's field of view may be moved to focus on each region of interest in order of priority, class, and / or other priority. Additionally, the output and / or visualized regions may be recorded as part of the medical history in a clinical reporting system.

[0057] Exemplary Differential Diagnosis Tool: According to one embodiment, a method includes distinguishing differential diagnoses (e.g., dermatofibroma and leiomyoma in a skin pathology specimen). A tissue visualization may be created using the steps described with respect to FIG. 5, with AI generating an overlay M of salient regions indicating which tissue regions contain invasive cancer. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results and believe a region is invasive) or less tissue (e.g., if they disagree with the results and believe the identified region is not invasive). The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. The output and / or visualized regions may be recorded as part of the medical history in a clinical reporting system. Additionally, a user may tag tissue regions with their multiple differential diagnoses.

[0058] Exemplary Preclinical Toxicity Detection Tool: According to one embodiment, a method includes visualization of salient tissues used in preclinical drug development, in which animals are administered a drug and their organs are evaluated by a pathologist to determine whether toxicity is present. A tissue visualization may be created using the steps described with respect to FIG. 5 , with AI generating an overlay M of salient regions indicating which tissue regions contain signs of toxicity. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results and believe a region indicates toxicity) or less tissue (e.g., if they disagree with the results and believe an identified region does not indicate toxicity). The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. A user may tag tissue regions with the user's multiple differential diagnoses. Output and / or visualized regions may be recorded and / or stored (e.g., to disk, cloud, laboratory information system, etc.).

[0059] Exemplary Parasitic Infestation Prediction Tool: According to one embodiment, a method includes visualization of AI detection of parasites across an entire slide image. A pathologist may be asked to diagnose a parasitic infection, such as a disease caused by a protozoan or helminth. For example, a diagnosis of Naegleria fowleri (commonly referred to as the "brain-eating amoeba") can be confirmed by pathological examination of brain tissue. A tissue visualization may be created using the steps described with respect to FIG. 5 , with the AI ​​generating an overlay M of salient regions indicating which tissue regions contain signs of parasitic infection. A user may interact with and / or edit the visualization. A user may adjust the visualization to show more tissue (e.g., if they disagree with the results) or less tissue (e.g., if they disagree with the results and believe the identified regions are not parasitic). The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. A user may tag tissue regions with their multiple differential diagnoses. The output and / or visualized regions may be recorded and / or stored (e.g., on disk, in the cloud, in a laboratory information system, etc.).

[0060] Exemplary Biomarker Tool: According to one embodiment, a method includes using AI to characterize and / or identify different biomarkers. In cancer pathology, one of the pathologist's tasks is to characterize and / or identify different biomarkers through additional testing (e.g., immunohistochemistry, sequencing, etc.). These may be applicable to all tissue types (e.g., HER2 in lung, breast, colon). A tissue visualization may be created using the steps described with respect to FIG. 5, with the AI ​​generating an overlay M of salient regions indicating which tissue regions contain invasive cancer. A user may customize what the tissue visualization shows based on the definition of the target output. For example, a user may want to see all current biomarkers, while another user may want to see only current biomarkers for which clinically actionable steps (e.g., drugs, treatment pathways, etc.) are available. The user's field of view may be moved to focus on each region of interest in order of priority, class, or other priority. The output and / or visualized regions may be recorded as part of the medical history in a clinical reporting system. A user may tag a tissue region with their multiple differential diagnoses.

[0061] 7 illustrates an example of a system 700 that may perform the techniques presented herein. Alternatively, multiple devices 700 may perform the techniques presented herein. The device 700 may include a central processing unit (CPU) 720. The CPU 720 may be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As those skilled in the art will appreciate, the CPU 720 may also be a single processor in a multi-core / multi-processor system, such as a system operating alone or in a cluster of computing devices operating in a cluster or server farm. The CPU 720 may be connected to a data communications infrastructure 710, for example, a bus, a message queue, a network, or a multi-core message passing scheme.

[0062] The device 700 may also include a main memory 740, e.g., random access memory (RAM), and may also include a secondary memory 730. The secondary memory 730, e.g., read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such removable storage drives may include floppy disk drives, magnetic tape drives, optical disk drives, flash memory, etc. The removable storage drive in this example reads from and / or writes to a removable storage unit in a well-known manner. The removable storage unit may include a floppy disk, magnetic tape, optical disk, etc., which are read from and written to by the removable storage drive. As those skilled in the art will appreciate, such removable storage units typically include computer-usable storage media having computer software and / or data stored thereon.

[0063] In alternative embodiments, secondary memory 730 may include other similar means for allowing computer programs or other instructions to be loaded into device 700. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 700.

[0064] Device 700 may also include a communications interface (“COM”) 760. Communications interface 760 allows software and data to be transferred between device 700 and external devices. Communications interface 760 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 760 may be in the form of signals, which may be electrical, electromagnetic, optical, or other signals capable of being received by communications interface 760. These signals may be provided to communications interface 760 via communications paths in device 700, which may be implemented using, for example, wire or cable, fiber optics, a telephone line, a cellular phone link, an RF link, or other communications channels.

[0065] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature and would be fully familiar to those skilled in the art. Device 700 may also include input / output ports 750 for connecting input / output devices such as a keyboard, mouse, touchscreen, monitor, display, etc. Naturally, various server functions may be implemented in a distributed manner across multiple similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of a single computer hardware platform.

[0066] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0067] The above tools, modules, and functions may be executed by one or more processors. A "storage" type medium may include tangible memory of a computer, processor, etc., or any or all of their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which may provide non-transitory storage at any time for software programming.

[0068] The software may be transmitted via the Internet, a cloud service provider, or other telecommunications network. For example, the software may be loaded from one computer or processor to another via communication. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0069] The foregoing general description is exemplary and explanatory only and is not intended to limit the present disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only.

Claims

1. 1. A computer-implemented method for analyzing an image of a slide corresponding to a specimen, comprising: receiving at least one digital image of a pathology specimen; using the digital image in an artificial intelligence (AI) system to determine at least one distinctive feature, wherein the at least one distinctive feature comprises a biomarker, cancer, cancer grade, parasites, toxicity, inflammation, and / or cancer subtype; determining an overlay of salient regions of the digital image with the AI ​​system, the AI ​​system indicating a diagnostic value for each pixel; suppressing one or more non-salient regions of the digital image based on the diagnostic value of each pixel; normalizing the salient regions overlay to obtain a parameter, the salient regions overlay being represented by a binary segmentation of the digital image, where each pixel indicates whether or not it has the salient feature present; Including, A computer-implemented method, wherein suppressing the one or more non-salient regions comprises making the pixels of the one or more non-salient regions partially transparent.

2. The computer-implemented method of claim 1 , further comprising converting the overlay of salient regions into a tissue map.

3. The computer-implemented method of claim 1 , wherein the overlay of salient regions is represented by a set of superpixels associated with a score or probability that the salient feature is present or absent.

4. 10. The computer-implemented method of claim 1, wherein detecting salient features uses image processing techniques, AI, and / or machine learning on the digital image to generate a tissue visualization.

5. The computer-implemented method of claim 1 , wherein the digital image includes associated case information, patient information, and information from clinical systems.

6. The computer-implemented method of claim 1 , further comprising alerting a user when the salient region overlay is available.

7. The computer-implemented method of claim 1 , wherein the salient region overlay is resized to the same size as the digital image.

8. The computer-implemented method of claim 1 , further comprising developing a pipeline that archives a plurality of processed images.

9. 1. A system for analyzing an image of a slide corresponding to a specimen, comprising: at least one memory for storing instructions; at least one processor configured to execute the instructions to perform operations, the operations comprising: receiving at least one digital image of a pathology specimen; using the digital image in an artificial intelligence (AI) system to determine at least one distinctive feature, wherein the at least one distinctive feature comprises a biomarker, cancer, cancer grade, parasites, toxicity, inflammation, and / or cancer subtype; determining an overlay of salient regions of the digital image with the AI ​​system, the AI ​​system indicating a diagnostic value for each pixel; suppressing one or more non-salient regions of the digital image based on the diagnostic value of each pixel; normalizing the salient regions overlay to obtain a parameter, the salient regions overlay being represented by a binary segmentation of the digital image, where each pixel indicates whether or not it has the salient feature present; Including, The system, wherein suppressing the one or more non-salient regions includes making the pixels of the one or more non-salient regions partially transparent.

10. The system of claim 9 , further comprising converting the overlay of salient regions into a tissue map.

11. The system of claim 9 , wherein the overlay of salient regions is represented by a set of superpixels associated with a score or probability that the salient feature is present or absent.

12. 10. The system of claim 9, wherein detecting salient features uses image processing techniques, AI, and / or machine learning on the digital image to generate a tissue visualization.

13. The system of claim 9 , wherein the digital image includes associated case information, patient information, and information from clinical systems.

14. The system of claim 9 , further comprising developing a pipeline for archiving prospective patient data.

15. 1. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image of a slide corresponding to a specimen, the method comprising: receiving at least one digital image of a pathology specimen; using the digital image in an artificial intelligence (AI) system to determine at least one distinctive feature, wherein the at least one distinctive feature comprises a biomarker, cancer, cancer grade, parasites, toxicity, inflammation, and / or cancer subtype; determining an overlay of salient regions of the digital image with the AI ​​system, the AI ​​system indicating a diagnostic value for each pixel; suppressing one or more non-salient regions of the digital image based on the diagnostic value of each pixel; normalizing the salient regions overlay to obtain a parameter, the salient regions overlay being represented by a binary segmentation of the digital image, where each pixel indicates whether or not it has the salient feature present; Including, The non-transitory computer-readable medium, wherein suppressing the one or more insalient regions includes making the pixels of the one or more insalient regions partially transparent.

16. The non-transitory computer-readable medium of claim 15 , further comprising converting the overlay of salient regions into a tissue map.

17. The computer-implemented method of claim 1 , wherein the overlay of salient regions is represented by a semantic segmentation of the digital image indicating a score or probability for each pixel.

Images