Contextually adaptive digital pathology interface

Abstract

To provide an image visualization system, method, and computer program product that include a contextually adaptive digital pathology interface.SOLUTION: A method includes the following steps. At least one image of a biological sample stained for the presence of one or more biomarkers is obtained (300). The image is displayed on a display screen at a first zoom level (310), in which a first subset of user selectable elements is contemporaneously displayed (320). As a result of user input, the image being is displayed at a second zoom level (330), in which a second subset of user selectable elements are contemporaneously displayed with the image (340). The one or more elements within the second subset of user selectable elements are disabled or hidden at the first zoom level, or one or more elements within the first subset of user selectable elements are disabled or hidden at the second zoom level.SELECTED DRAWING: Figure 3

Description

[Background technology]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 62 / 726,081, filed August 31, 2018, which is incorporated herein by reference in its entirety for all purposes. Digital pathology refers to the management and interpretation of pathology information in a digital environment. A scanning device is used to image slides of biological samples, which may be stained, resulting in a digital slide, e.g., the entire slide image. Using digital pathology software, the digital slide can be stored in a computer's memory device, displayed on a computer monitor, and the pathology information analyzed. Summary of the Invention

[0002] The present disclosure provides a graphical user interface adapted to provide a visualization of images of biological samples stained for the presence of one or more biomarkers and to enable the display of contextually relevant elements (e.g., analysis tools, viewer panels) to a user based on whether pre-established criteria are met. In some embodiments, contextually relevant elements are provided to a user based on a selected zoom level or the selection of an image having a particular stain. By providing only contextually relevant elements, a user is believed to be able to avoid being overwhelmed by visualized elements (e.g., elements that are inappropriate or ineffective in a particular context) and instead select the correct elements based on the condition (e.g., selected zoom level, selected image type, selected tissue type, selection of whole slide images versus slides obtained from tissue microarrays, selected slides to which a particular stain has been applied (e.g., H&E), selected slides stained for the presence of a particular biomarker, selected slides stained using immunohistochemistry as opposed to in situ hybridization, selected slides to which a particular image analysis algorithm has been applied). Furthermore, it is believed that providing only contextually relevant elements allows for faster and more accurate review of the presented image data. Furthermore, due to the limited size and resolution of display screens, it is often difficult to provide multiple visualizations at once, and if multiple visualizations are provided, they may overlap or obstruct the user's view of important elements. By displaying only contextually relevant elements given certain conditions (e.g., a particular zoom level selected by the user), the most appropriate elements can be prioritized and visualized given the limited display size and resolution.

[0003] In some embodiments, a computing device includes a display screen, wherein the computing device accesses at least one image of a biological sample stained for the presence of one or more biomarkers (e.g., HER2, PD-L1, etc.) from one or more memories communicatively coupled to the computing device; displays a first visualization of the at least one image on the display screen at a first zoom level and concurrently displays a first subset of user-selectable tools on the display screen while displaying the first visualization; and subsequently, in response to receiving a user input, displays a second visualization of the at least one image on the display screen at a second zoom level greater than the first zoom level and concurrently displays the second subset of user-selectable elements on the display screen while displaying the second visualization, wherein (i) one or more elements in the second subset of user-selectable elements are disabled or hidden at the first zoom level, or (ii) one or more elements in the first subset of user-selectable elements are disabled or hidden at the second zoom level. It is configured as follows.

[0004] In some embodiments, the second zoom level meets at least a predetermined zoom threshold. In some embodiments, the second subset of user-selectable elements are contextually relevant at the second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. By way of example, the tools and menus are context-dependent. When the user is fully "zoomed out," only relevant tools for use in that context are enabled (see FIG. 4A). When the user zooms in to a specified zoom level (see FIG. 4B), toolbar tools are enabled or context panels are displayed. These tools are available to the user when in a "zoomed in" context of the slide. Thus, the tools available in either the first set of tools or the second set of tools depend on the zoom level selected by the user, e.g., the second zoom level compared to the first zoom level, where the second zoom level meets or exceeds a predetermined zoom threshold level for enabling the second set of tools.

[0005] In some embodiments, the menu bar tools include an image annotation tool, a slide settings tool, a slide selection tool, a navigation tool, and a view tool. In some embodiments, the image annotation tool is enabled at a second zoom level. In some embodiments, the image annotation tool is hidden at a first zoom level. In some embodiments, the image annotation tools include a region of interest identification tool, a measurement tool, a marking tool, and a region exclusion tool.

[0006] In some embodiments, the computing device is further configured to display one or more viewer panels on the display screen at a second zoom level, and the one or more viewer panels for display at the second zoom level are disabled at the first zoom level. In some embodiments, the first zoom level is a 1x zoom level (i.e., the lowest zoom level, such as an image scanned or otherwise captured without magnification), and the first visualization representation includes at least one representation including at least one image of the biological sample. In some embodiments, the at least one representation further includes a portion including an identifying indicia.

[0007] In some embodiments, a computing device includes a display screen, and the computing device is configured to: access at least one image of a biological sample stained for the presence of one or more biomarkers from one or more memories communicatively coupled to the computing device; display a first representation of the at least one image on the display screen at a first zoom level and simultaneously display at least a first viewer panel on the display screen while displaying the first representation; and subsequently display a second representation of the at least one image on the display screen at a second zoom level greater than the first zoom level and simultaneously display at least a second viewer panel on the display screen in addition to the first viewer panel while displaying the second representation, wherein the at least second viewer panel is hidden at the first zoom level.

[0008] In some embodiments, a third viewer panel is displayed on the display screen simultaneously with the display of the second display panel, and the third viewer panel is hidden at the first zoom level. In some embodiments, a menu bar icon is enabled on the display screen simultaneously with the display of the second display panel. In some embodiments, the displayed menu bar icon is selected from the group consisting of a region of interest identification tool, a measurement tool, a marking tool, and a region exclusion tool. In some embodiments, at least The first representation includes (i) a first portion including at least one image of the biological sample at a first zoom level and (ii) a second portion including identifying indicia, in some embodiments, the identifying indicia including an identification of a biomarker.

[0009] In some embodiments, a method includes displaying, on a computing device having a display screen, a first visualization including at least one image of a stained biological sample, the first visualization being displayed in a first condition state; concurrently displaying a first viewer panel or at least one of a first set of user-selectable elements on the display screen while displaying the first visualization; and subsequently displaying a second visualization of the at least one image in a second condition state on the display screen, the second condition state being a result of a user selection; and concurrently displaying, on the display screen, at least one of a second set of user-selectable elements not enabled in the first condition state or a second viewer panel in addition to the first viewer panel, the second viewer panel being hidden at a first zoom level. In some embodiments, the first condition state is a default state. In some embodiments, the user selection is a zoom level, and the second condition is a zoom level greater than the default zoom level. In some embodiments, the user selection is an image selection, and the image selected for the second condition includes a different stain than the image for the first condition.

[0010] In some embodiments, a method includes displaying, on a computing device having a display screen, a first visualization including at least one image of a stained biological sample, the first visualization being displayed in a first condition state; concurrently displaying a first viewer panel or at least one of a first set of user-selectable elements on the display screen while displaying the first visualization; subsequently, displaying a second visualization of the at least one image on the display screen at a second zoom level, the second zoom level being greater than the first zoom level; and concurrently displaying the second visualization, at least one of a second set of user-selectable elements that are hidden or not enabled at the first zoom level or at least a second viewer panel in addition to the first viewer panel on the display screen while displaying the second visualization;

[0011] In some embodiments, the first zoom level is a default zoom level. In some embodiments, the second zoom level is one that sufficiently resolves cell clumps or sufficiently resolves cell nuclei. In some embodiments, the first zoom level is the lowest zoom level available and the second zoom level is at least 5x. In some embodiments, the first zoom level is the lowest zoom level available and the second zoom level is at least 10x.

[0012] In some embodiments, the second subset of user-selectable elements are contextually relevant at the second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. In some embodiments, the menu bar tools include an image annotation tool, a slide settings tool, a slide selection tool, a navigation tool, and a view tool. In some embodiments, the image annotation tools are enabled at the second zoom level. In some embodiments, the image annotation tools are hidden at the first zoom level. In some embodiments, the image annotation tools include a region of interest identification tool, a measurement tool, a marking tool, and a region exclusion tool.

[0013] In some embodiments, the first visualization comprises: (i) a biometric sub-image at a first zoom level; and (ii) a second portion including identifying indicia. In some embodiments, the identifying indicia includes an identification of a biomarker.

[0014] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors of a computing system, cause the computing system to display on a display screen a first visualization representation of at least one image at a first zoom level and, concurrently with displaying the first visualization representation, display a first subset of user-selectable tools on the display screen; and subsequently, display on the display screen a second visualization representation of the at least one image at a second zoom level greater than the first zoom level and, concurrently with displaying the second visualization representation, display the second subset of user-selectable tools on the display screen, wherein one or more tools in the second subset of user-selectable tools are not enabled at the first zoom level. [Brief explanation of the drawings]

[0015] For a general understanding of the features of the present disclosure, reference is made to the drawings, wherein like reference numerals are used throughout to identify identical elements.

[0016] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided to the Office upon request and payment of the necessary fee.

[0017] [Figure 1] 1 illustrates a system including a computer having one or more processors and a scanning device, the computer and scanning device being communicatively coupled, such as via a network, according to some embodiments.

[0018] [Figure 2A] 1 illustrates a system including a processing subsystem, a storage subsystem, an output device, and an input device, each of which is communicatively coupled via a bus, network, or other wired or wireless interconnection, according to some embodiments. The system may also include software that allows remote access, i.e., a client portal or interface.

[0019] [Figure 2B] 1 illustrates a block diagram of a system communicatively coupled with a client interface over a network, according to some embodiments.

[0020] [Figure 3] 1 shows a flowchart providing general steps for displaying a visualization, according to some embodiments.

[0021] [Figure 4A] 1 illustrates a viewer window containing a visualization including multiple representations at a first zoom level, according to some embodiments.

[0022] [Figure 4B] 4B illustrates a viewer window including a visualization representation including a representation at a second zoom level, the second zoom level being greater than the first zoom level of FIG. 4A, according to some embodiments.

[0023] [Figure 4C] 4A and 4B, respectively. A viewer window is shown containing a visualization representation including multiple representations at a third zoom level, according to some embodiments, the third zoom level being between the first and second zoom levels of FIGS. 4A and 4B.

[0024] [Figure 5A] 4B illustrates a drop-down menu for selecting a particular viewer panel, such as those available at a first zoom level (eg, the zoom level of FIG. 4A), according to some embodiments.

[0025] [Figure 5B] 4B shows a drop-down menu for selecting a particular viewer panel, such as one available at a second zoom level (e.g., the zoom level of FIG. 4B), according to some embodiments.

[0026] [Figure 6A] 1 illustrates a viewer window containing a first visualization representation including three representations at a first zoom level, according to some embodiments.

[0027] [Figure 6B] 6A and 6B show a viewer window including a first visualization representation including three representations at a second zoom level according to some embodiments, which is an intermediate zoom level compared to the zoom levels shown in FIGS. 6A and 6C.

[0028] [Figure 6C] 6B shows a viewer window including a first visualization representation including three representations at a third zoom level, according to some embodiments, the third zoom level being greater than the zoom levels of FIGS. 6A and 6B. DETAILED DESCRIPTION OF THE INVENTION

[0029] Detailed Description It should also be understood that, unless expressly stated to the contrary, in methods claimed herein that include multiple steps or acts, the order of the method steps or acts is not necessarily limited to the order in which the method steps or acts are described.

[0030] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly dictates otherwise. The term "comprising" is defined inclusively, such that "including A or B" means including A, B, or A and B.

[0031] As used in this specification and the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as inclusive, that is, including a plurality, but including at least one of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as "only one" or "exactly one," or, when used in the claims, "consisting of," shall refer to the inclusion of exactly one element of a number or list of elements. In general, the term "or" as used herein shall only be interpreted as indicating exclusive alternatives (i.e., "one or the other, but not both") when preceded by terms of exclusivity, such as "either," "either," "only," or "exactly either." "Consisting essentially of" when used in the claims shall have its ordinary meaning as used in the field of patent law.

[0032] Terms such as "comprises," "includes," and "having" are used interchangeably and have the same meaning. Similarly, terms such as "comprises," "includes," and "having" are used interchangeably and have the same meaning. Specifically, each term is defined consistent with the general U.S. patent law definition of "comprises," and is therefore to be interpreted as open terms meaning "at least the following" and not excluding additional features, limitations, aspects, etc. Thus, for example, "an apparatus having components a, b, and c" means that the apparatus includes at least components a, b, and c. Similarly, the phrase "a method comprising steps a, b, and c" means that the method comprises at least steps a, b, and c. Furthermore, although steps and processes may be outlined in a particular order herein, one of ordinary skill in the art will recognize that the ordering of steps and processes may vary.

[0033] As used in this specification and claims, the phrase "at least one" in connection with a list of one or more elements should be understood to mean at least one element selected from any one or more elements of the list of elements, but does not necessarily include at least one of every element specifically listed in the list of elements, and does not exclude combinations of elements in the list of elements. This definition also allows for elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related to the specifically identified elements or not, may optionally be present. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B," or, equivalently, "at least one of A and / or B") can, in some embodiments, refer to at least one, optionally including two or more As, and no B (and optionally including elements other than B); in some embodiments, it can refer to at least one, optionally including two or more Bs, and no A (and optionally including elements other than A); in yet other embodiments, it can refer to at least one, optionally including two or more As, and at least one, optionally including two or more Bs (and optionally including other elements); and so forth.

[0034] As used herein, the term "image data" encompasses raw image data acquired from a biological tissue sample, such as by an optical sensor or sensor array, or pre-processed image data. In particular, image data can include a matrix of pixels.

[0035] As used herein, the terms "image," "image scan," or "scanned image" encompass raw or pre-processed image data obtained from a biological tissue sample, such as by an optical sensor or sensor array. In particular, the image data may include a matrix of pixels.

[0036] As used herein, the terms "biological sample," "tissue sample," "specimen," and the like refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans, veterinary animals such as cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies of tissue), cell samples (such as cytological smears, such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biomolecules derived from an initial biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0037] As used herein, the term "slide" refers to any substrate (e.g., glass, stone, etc.) of any suitable dimensions on which a biological specimen is placed for analysis. The term "slide" refers to a substrate (such as a substrate made of glass, plastic, silicon, etc.), more specifically a "microscope slide" such as a standard 3" x 1" microscope slide or a standard 75mm x 25mm microscope slide. Examples of biological specimens that can be placed on a slide include, but are not limited to, cytological smears, thin tissue sections (such as from a biopsy), and arrays of biological specimens, e.g., tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in some embodiments, tissue sections, DNA samples, RNA samples, and / or proteins are placed at specific locations on the slide. In some embodiments, the term "slide" can refer to SELDI and MALDI chips as well as silicon wafers.

[0038] As used herein, the terms "staining," "staining," and the like generally refer to any treatment of a biological specimen to detect and / or differentiate the presence, location, and / or amount (e.g., concentration) of a specific molecule (e.g., lipid, protein, or nucleic acid) or a specific structure (e.g., normal or malignant cells, cytosol, nucleus, Golgi apparatus, or cytoskeleton) in the biological specimen. For example, staining can provide contrast between a specific molecule or specific cellular structure and the surrounding area of ​​the biological specimen, and the intensity of the staining can provide a measure of the amount of a specific molecule in the specimen. Staining can be used to aid in the observation of molecules, cellular structures, and organisms using not only brightfield microscopes but also other observation tools such as phase contrast microscopes, electron microscopes, and fluorescence microscopes. Some staining performed by the system can be used to visualize cell contours. Other staining performed by the system can rely on the specific cellular component (e.g., molecule or structure) to be stained, with no or relatively little staining of other cellular components. Examples of types of staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on reactions between molecules (including non-covalent interactions), such as hybridization reactions between nucleic acid molecules. Specific staining methods include, but are not limited to, primary staining methods (e.g., H&E staining, Pap staining, etc.), enzyme-linked immunohistochemical methods, and in situ RNA and DNA hybridization methods, such as fluorescent in situ hybridization (FISH).

[0039] As used herein, the term "user interface" refers to an interface that allows a user, e.g., an end user such as a histologist and / or pathologist, to input commands and data and receive results, such as a graphical user interface (GUI). The terms "user interface" and "graphical user interface" are used interchangeably herein.

[0040] As described in further detail herein, the present disclosure relates to a graphical user interface that allows a user to view and / or analyze one or more images of a biological sample stained for the presence of one or more biomarkers, whereby visualization of certain elements and / or availability of certain analysis and / or processing tools are provided to the user on a context-based basis. In some embodiments, the graphical user interface is adapted to provide contextually relevant visualizations and elements for selection based on whether a pre-established condition is met. In some embodiments, the pre-established condition is a selection made by the user. For example, the selection may be a zoom level or magnification level selected by the user, where selection of a different zoom level or magnification level causes the graphical user interface to adaptively generate a particular visualization or enable a particular user-configurable item, e.g., annotation tools, image processing tools, etc. In some embodiments, the graphical user interface adjusts the visualizations, analysis tools, and / or viewer panels provided to the user (e.g., for display on a display) according to the selected zoom level, e.g., a first subset of contextually relevant tools are , may be presented to the user at the lowest zoom level (e.g., no optical magnification compared to 10x, 20x, or 40x optical magnification), while a more comprehensive second subset of tools is presented to the user at a larger zoom level of 10x, with additional tools included in the second subset again contextually relevant to the 10x zoom level. As another example, the selection may be a particular type of tissue (i.e., an image of the tissue) selected by the user, or an image of a tissue sample stained for the presence of a particular biomarker.

[0041] It is believed that a user may be able to interact with the software more efficiently by presenting only visualization representations, analysis tools, viewer panels, etc. that are contextually relevant to pre-established criteria, such as a selected zoom level, a selected tissue type, selected slides to which a particular stain has been applied, selected slides stained for the presence of a particular biomarker, a selection of whole slide images as opposed to a tissue microarray, etc. Stated differently, an operator may be able to interact with the graphical user interface more efficiently because the user is not overwhelmed by the availability of numerous tools and / or viewer panels that are unrelated to a particular zoom level, ultimately resulting in a more rapid review and analysis of presented images of tissue samples.

[0042] In some embodiments, the disclosed system is configured to facilitate the interpretation and reporting of image data obtained from a subject (e.g., a human patient). In some embodiments, image data is acquired from a scanning device (such as a VENTANA DP 200 scanner available from Ventana Medical Systems, Inc., Tucson, AZ), and the image data can be stored in a database, such as a networked database, for later visualization and analysis. For example, image data can be acquired using a scanning device, and the scanned image data can be stored in a file located on the storage subsystem 104 or a networked server, whereby the file can be later retrieved for visualization and analysis (see FIG. 1). In some embodiments, software such as an image visualization and analysis application runs directly on the system, and the image data is retrieved from the networked server for interpretation and reporting by a user interacting with the software (see FIG. 2A). In other embodiments, software such as an image visualization and analysis application runs on a remote system, and a client interface or client portal is used to access the system, whereby the image data can be retrieved from the storage subsystem for visualization and analysis (see FIG. 2B).

[0043] The systems and methods provided herein can be applied to the visualization and analysis of any type of image of tissue stained for the presence of one or more biomarkers. For example, a biological sample can be stained by application of one or more stains, and the resulting image or image data includes signals corresponding to each of the one or more stains. In some embodiments, the input image is a simple image having only a single stain (e.g., stained with 3,3'-diaminobenzidine (DAB)). In some embodiments, the biological sample can be stained with multiplex assays for two or more stains (thus providing a multiplexed image). In some embodiments, the biological sample is stained for at least two biomarkers. In some embodiments, the biological sample is stained for the presence of at least two biomarkers and also stained with a primary stain (e.g., hematoxylin). In some embodiments, the biological sample is stained for the presence of at least one protein biomarker and at least two nucleic acid biomarkers (e.g., DNA, RNA, microRNA, etc.).

[0044] In some embodiments, the biological sample contains one or more protein biomarkers. For example, biological samples can be stained for the presence of human epidermal growth factor receptor 2 protein (HER2 protein). Currently, in the United States, there are two Food and Drug Administration (FDA)-approved methods for HER2 assessment: HerceptTest™ (DAKO, Glostrup, Denmark) and HER2 / neu (4B5) rabbit monoclonal primary antibody (Ventana, Tucson, Arizona).

[0045] In some embodiments, the biological sample is stained for the presence of estrogen receptor (ER), progesterone receptor (PR), or Ki-67. In yet other embodiments, the biological sample is stained for the presence of EGFR or HER3. Other examples of protein biomarkers are described by Zamay et al., "Current and Prospective Biomarkers of Long Cancer," Cancers (Basel), November 2018;9(11), the disclosure of which is incorporated herein by reference in its entirety. Examples of protein biomarkers described by Zamay include CEACAM, CYFRA21-1, PKLK, VEGF, BRAF, and SCC.

[0046] In some embodiments, biological samples are stained for the presence of one or more nucleic acids, including mRNA, in an in situ hybridization (ISH) assay. U.S. Patent No. 7,087,379 (the disclosure of which is incorporated herein by reference in its entirety) describes a method for staining samples with ISH probes so that individual spots (or dots) representing single gene copies can be observed and detected. In some embodiments, several target genes are analyzed simultaneously by exposing cell or tissue samples to multiple nucleic acid probes labeled with multiple different nucleic acid tags.

[0047] FIG. 1 illustrates a system 100 (computer or computing device) that includes a scanning device 110 communicatively coupled to a processing subsystem 102. The scanning device 110 can be coupled to the processing subsystem 102 directly (e.g., via one or more communication cables) or via one or more wired and / or wireless networks 130. In some embodiments, the processing subsystem 102 can be included in or integrated with the scanning device 110. In some embodiments, the system 100 can include software that instructs the scanning device 110 to perform certain operations using certain user-configurable parameters and transmit the resulting acquired image data to the processing subsystem 102 or a storage subsystem (e.g., a local storage subsystem or a network storage device). In some embodiments, either the processing subsystem 102 or the scanning device 110 can be coupled to a network 130. In some embodiments, a storage device is coupled to the network 130 for storing or retrieving image data, subject information, and / or other tissue data. The processing subsystem 102 can include a display 108 and one or more input devices (not shown) for receiving commands from a user or operator (e.g., a technician, histologist, or pathologist).

[0048] In some embodiments, a user interface is rendered by the processing subsystem 102 and provided to the display 108 to (i) facilitate analysis, interpretation, and / or reporting of image data and / or subject data, (ii) acquire data from the scanning device, and (iii) acquire image data, subject information, or other clinical information, such as databases available over the network. In some embodiments, the network 130 provides remote access to the processing subsystem 102 and / or the scanning device 110, such as through a client interface or client portal (not shown). In this manner, remote users can access the processing subsystem 102 such that image visualization and analysis software can be executed remotely on the processing subsystem 102. In some embodiments, the client interface or client portal may also enable retrieval of stored reports following analysis of the image data.

[0049] 2A is a block diagram of a system 100 according to one embodiment of the present disclosure. System 100 can be implemented using any type of user-operable computing device, including a desktop computer, a laptop computer, a tablet computer, a handheld device (e.g., a smartphone, a media player), etc. System 100 can include several interconnected components, such as a processing subsystem 102, a storage subsystem 104, a user input device 106, a display 108, and a network interface 112 that communicate via a bus 114, as discussed in more detail below. In some embodiments, system 100 shown in FIG. 2A may be accessed remotely; for example, one or more remote users may access system 100, such as via a network, so that image data stored in storage subsystem 104 can be acquired, interpreted, analyzed, and / or reported.

[0050] The processing subsystem 102 can include a single processor, which can have one or more cores, or multiple processors, each having one or more cores. In some embodiments, the processing subsystem 102 can include one or more general-purpose processors (e.g., CPUs), special-purpose processors such as graphics processors (GPUs), digital signal processors, or any combination of these and other types of processors. In some embodiments, some or all of the processors in the processing subsystem can be implemented using customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuitry itself. In some embodiments, the processing subsystem 102 can retrieve and execute instructions stored on the storage subsystem 104, and the instructions can be executed by the processing subsystem 102 regardless of whether a user accesses the system locally or remotely, such as through the client portal 116. As an example, the processing subsystem 102 can receive and process image data stored in a local or networked storage system and execute instructions to display the image data (e.g., displaying an entire slide scan image or a magnified portion of an entire slide scan image).

[0051] The storage subsystem 104 may include various memory units, such as system memory, read-only memory (ROM), and persistent storage. ROM may store static data and instructions needed by the processing subsystem 102 and other processing modules of the system 100. The persistent storage may be a read-and-write memory device. This permanent storage may be a non-volatile memory unit that stores instructions and data even when the system 100 is powered off. In some embodiments, mass storage devices (such as magnetic or optical disks or flash memory) may be used as permanent storage devices. Other embodiments may use removable storage devices (e.g., flash drives) as permanent storage devices. The system memory may be a read-and-write memory device or a volatile read-and-write memory, such as dynamic random access memory. The system memory may store some or all of the instructions and data needed by the processor during execution.

[0052] The storage subsystem 104 may include any combination of non-transitory computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and the like. Magnetic and / or optical disks may also be used. In some embodiments, the storage subsystem 104 may include removable storage media, which may be readable and / or writable. Examples of such media include compact discs (CDs), read-only digital versatile discs (DVD-ROMs, dual-layer DVD-ROMs, etc.), read-only and recordable Blu-ray discs, ultra-high density optical discs, flash memory cards (SD cards, mini SD cards, micro SD cards, etc.), and the like. In some embodiments, the image data and / or subject data may be stored in one or more remote locations, e.g., cloud storage, and synchronized with other components of the system 100. When the terms “memory” or “memory” are used herein, they may refer to one or more memories, such as multiple memories.

[0053] In some embodiments, the storage subsystem 104 can store one or more software programs executed by the processing subsystem 102, such as the image visualization and analysis application 120. "Software" generally refers to a set of instructions that, when executed by the processing subsystem 102, causes the system 100 to perform various operations and thus defines one or more specific machine implementations that perform and execute the operations of the software programs. Thus, "software" may also include firmware or embedded applications, or any other type of instructions readable and executable by the processing subsystem 102. Software may be implemented as a single program or as a collection of individual programs or program modules that interact as necessary. In some embodiments, programs and / or data may be stored in non-volatile storage and copied in whole or in part to volatile working memory during program execution. From the storage subsystem 104, the processing subsystem 102 can retrieve program instructions to execute and data to process to perform various operations, including those described below.

[0054] In some embodiments, software may run locally on system 100 but may be accessed and / or controlled remotely, such as via client portal 116. For example, an instance of image visualization and analysis application 120 may run locally on system 100, but a remote operator may access image visualization and analysis application 120 by way of networked client portal 116, such that the remote user may control the instance of image visualization and analysis application 120 to facilitate review, interpretation, and analysis of image data (e.g., scanned images of biological samples retrieved from storage subsystem 104 and presented to the remote user for analysis).

[0055] The user interface can be provided on the display 108 and / or one or more other user output devices (not shown). The user interface may include, for example, visualizations and other representations, including images obtained from scans of biological samples (e.g., samples stained for the presence of one or more biomarkers or stained with hematoxylin and eosin), menu bars, drop-down menus, and / or panels. The user interface provided on the display can be adapted to provide only contextually relevant tools and / or viewer panels to the user based on user selections, including, for example, but not limited to, a user-selected zoom or magnification level. User input device 106 can include any device by which a user can provide signals to system 100, and system 100 can receive and transmit signals. The codes may be interpreted as indicating a particular user request or information. In some embodiments, user input device 106 may include any or all of a keyboard touchpad, a touchscreen (e.g., a touch-sensitive overlay on the display surface of display 108), a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, etc.

[0056] Display 108 can display visualizations generated by system 100 (e.g., image data, viewer panels to convey information to the user, or context menus providing user-selectable configuration options) and can include a variety of image generation technologies, such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), light emitting diodes (LEDs) including organic light emitting diodes (OLEDs), projection systems, etc., and supporting electronics (digital-to-analog or analog-to-digital converters, signal processors, etc.). Some embodiments can include devices such as touchscreens that function as both input and output devices. In some embodiments, other user output devices can be provided in addition to or instead of display 108.

[0057] In some embodiments, the user interface may provide a graphical user interface in which visible image elements within a particular region of the display 108 are defined as active, interactive, or control elements that the user selects using the user input device 106. For example, the user may manipulate the user input device 106 to position an on-screen cursor or pointer over a control element and "click" a button to indicate a selection, which transmits a signal to perform a specified action or routine. For example, the user may manipulate the user input device 106 to select an icon within the user interface (e.g., an icon within a viewer panel, menu bar, or drop-down menu), which initiates an action or selection of a tool, such as annotation of one or more displayed representations of an image of a tissue sample. As another example, the user may click a menu bar icon to initiate tool selection, such that the user may select a region of interest based on received input. In some embodiments, the user may manipulate the user input device 106 to interact with a drop-down menu to select one or more panels, including interactive panels. In some embodiments, these selections may be performed by the user only if the tools and / or viewer panels are enabled based on whether pre-established conditions are met, e.g., whether a particular zoom level is selected by the user, whether a particular tissue type is selected, whether a particular slide with particular biomarkers is selected.

[0058] Alternatively, a user can touch (e.g., with a finger or stylus) a control element on a touchscreen device. In some embodiments, the user can speak one or more words associated with the control element (the words can be, for example, a label on the element or a function associated with the element). In some embodiments, user gestures on the touch-sensitive device can be recognized and interpreted as input commands. These gestures can be, but need not be, associated with particular areas on the display 108. Other user interfaces can also be implemented.

[0059] The network interface 112 can provide data communication capabilities for the system 100. In some embodiments, the network interface 112 can include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., cellular technologies, advanced data network technologies such as 3G, 4G, or EDGE, 5G, WiFi (IEEE 802.11 fiber), etc.). (Using a wireless LAN, a wireless LAN cable ...

[0060] The bus 114 may include various system, peripheral, and chipset buses that communicatively couple the various components of the system 100. For example, the bus 114 may communicatively couple the processing subsystem 102 to the storage subsystem 104. The bus 114 may also connect to the input device 106 and the display 108. The bus 114 may also couple the processing subsystem 102 to a network via the network interface 112. In this manner, the system 100 may be connected to a network of multiple computer systems (e.g., a network of networks such as a local area network (LAN), a wide area network (WAN), an intranet, or the Internet). Those skilled in the art will appreciate that additional components, such as a scanning device, a tissue processing system, etc., may be connected to the bus 114.

[0061] Some embodiments include electronic components such as a microprocessor, storage devices, and memories that store computer program instructions on a computer-readable storage medium. Many of the functions described herein may be implemented as a process specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing units to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code, such as produced by a compiler, and files containing high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0062] Through appropriate programming, the processing subsystem 102 can provide various functions to the system 100. For example, the processing subsystem 102 can execute an image visualization and analysis application 120 having a user interface that facilitates review and interpretation of scanned images of biological samples. The image visualization and analysis application 120 can provide various functions, such as the ability to select user-configurable options or user-selectable panels, or the ability to control navigation and annotate images. In some embodiments, the analysis application 120 includes logic such that only relevant user items (informational items, user-selectable items, interactive items) are presented to the user based on whether pre-established conditions are met, e.g., whether the user selects a zoom level that meets or exceeds a predetermined threshold, or whether a particular type of image is selected for review that stains for the presence of a particular biomarker. In some embodiments, additional components are provided as described in U.S. Patent Application Publication No. and may be incorporated into the systems and software of the present disclosure, including components identified in 2012 / 0320094, the disclosure of which is incorporated herein by reference in its entirety.

[0063] In some embodiments, the image visualization and analysis application 120 incorporates various interoperable modules (e.g., blocks of code) that, when executed by one or more processors in the processing subsystem 102, implement aspects of interface operations. For example, the image visualization and analysis application 120 may include a content fetcher 122, a content renderer 124, a GUI renderer 126, and a UI interpreter 128.

[0064] In some embodiments, the content fetcher 122 can include instructions for interacting with (e.g., accessing) a local database (e.g., storage subsystem 104) or network interface 112 to fetch or otherwise obtain content items such as image data and / or subject data. In some embodiments, the content fetcher 122 is configured to access multiple scan images, each of which is derived from a subject sample, and each of which may be stained for the presence of one or more biomarkers or for hematoxylin and eosin. In some embodiments, the content fetcher 122 is configured to obtain subject information, image metadata, case history information, etc. In some embodiments, the content fetcher 122 can include instructions for interacting with the scanning device 110 such that image data can be obtained from one or more slides having tissue samples stained for the presence of one or more biomarkers.

[0065] In some embodiments, the content renderer 124 may include instructions for interpreting content items fetched from one or more sources and then delivering the rendered content to or into image placeholders or other representations generated by the GUI renderer 126. For example, the content renderer 124 may populate one or more rendered representations with image data obtained from the content fetcher 122 (see representation 401 in FIG. 4A ). In some embodiments, the content renderer 124 may deliver subject information to other GUI elements, such as one or more viewer panels, or place the obtained subject information into portions of the GUI representation. In some embodiments, the content renderer 124 may deliver metadata, such as tissue type, applied stains, scan parameters, z-stack layers, focus layers, etc., to other GUI elements. In some embodiments, the content renderer 124 may also process the obtained image data, e.g., apply any pre-processing to the obtained images.

[0066] In some embodiments, the GUI renderer 126 creates graphical user interface (GUI) elements that are presented to the user along with content items rendered by the content renderer 124 or other system modules. The GUI renderer 126 may include code that defines the position and appearance of GUI elements, such as menu bar items, viewer panels, and configuration panels, each of which may be or include interactive elements. In some embodiments, the GUI renderer 126, along with signals received from the UI interpreter 128, may determine whether to enable or make available to the user a particular menu bar item or viewer panel depending on whether pre-established conditions are met, e.g., whether a zoom threshold level or magnification level is selected. For example, a menu bar item may be activated by the user, thereby allowing the user to select a configuration option or panel view from a drop-down menu (e.g., see FIGS. 5A and 5B). In some embodiments, the GUI renderer 126 can incorporate the retrieved image data provided from the content fetcher 122 or the content renderer 124 into some or all of the GUI elements (e.g., an actual image of the scanned biological sample can be displayed in a representation within the user interface).

[0067] As an example, GUI renderer 126 can generate a series of representations 401 that can be populated with image data retrieved with information retrieved by content fetcher 122. Example representations are shown in Figures 4A, 4B, and 4C. These representations can be interactive representations. For example, if a user clicks on any particular representation (e.g., representation 401 in Figure 4A) (e.g., as interpreted by UI interpreter 128), GUI renderer 126 can update the corresponding display of a viewer panel, such as a slide navigator viewer panel.

[0068] Similarly, the GUI renderer 126 may generate a series of viewer panels. In some embodiments, the generated viewer panels are interactive panels that allow a user to select specific configurable options. For example, a zoom panel may include a slider bar that allows a user to select a specific preset zoom level, such as 1x, 2x, 10x, 40x, or to enter a specific zoom level value. In some embodiments, the viewer panels are configured to convey relevant information to the user; for example, a case log panel may provide a history of user-configurable selections made by the user during analysis of the image data. Additionally, the GUI renderer 126 may render visualizations that show items that are hidden or unavailable or not enabled for the user to interact with.

[0069] The UI interpreter 128 can receive user input, for example, via the user input device 106, and interpret the input to determine actions to be performed by the analysis application 120. For example, the UI interpreter 128 can determine which GUI element the user selected (e.g., an icon or a selectable item such as a menu, context menu, drop-down list, button, or representation) and initiate the corresponding action (e.g., adding an annotation, displaying additional content information, zooming to a selected zoom level, or generating a report for export). For example, the UI interpreter 128 can detect whether the user selected an annotation tool (see annotation tool 405 in FIGS. 4A and 4B ) and send a signal to the GUI renderer 126 to display additional user selections. In some embodiments, the annotation tools include a manual region-of-interest (ROI) generation tool, an automatic ROI generation tool, a tool that allows drawing shapes (e.g., arrows), a measurement tool, or a text input generation tool. Each of these tools can be individually disabled or hidden based on the context of the user interaction. In some embodiments, menu items that may be selected include those that perform specific image processing algorithms, such as a membrane detection algorithm, a cell detection and counting algorithm, a nucleus detection algorithm, a scoring algorithm, a heatmap generation algorithm, a tissue masking algorithm, a tissue type identification algorithm, etc. (See, e.g., PCT Publication Nos. WO2016 / 120442 and WO2015 / 113895, and U.S. Patent Application Publication Nos. 2017 / 0154420, 2017 / 0372117, 2017 / 0103521, 2017 / 0140246, 2015 / 0347702, 2017 / 0082627, 2014 / 0377753, 2017 / 0337695, 2017 / 0323148, and 2017 / 0243051, the disclosures of which are incorporated herein by reference in their entireties). The input received from the UI interpreter 128 may be used to determine whether a pre-established condition is met.

[0070] It will be understood that the system 100 is exemplary and that variations and modifications are possible. Additionally, while system 100 has been described with reference to particular blocks, it should be understood that these blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of components. Furthermore, the blocks need not correspond to physically distinct components. The blocks may be configured to perform various operations, such as programming a processor or providing appropriate control circuitry, and the various blocks may or may not be reconfigured depending on how the initial configuration is accessed. Embodiments of the present disclosure may be realized in a variety of devices, including electronic devices implemented using any combination of circuitry and software. The image visualization and analysis application 120 is also exemplary, and particular implementations may include more or fewer modules than those described herein. Furthermore, while particular modules may be described as performing particular functions, such description is not intended to imply particular functions performed by modules or particular sets of instructions contained within such modules.

[0071] FIG. 2B illustrates a client interface 140 in communication with network 130 and system 100 (such as the systems shown in FIGS. 1 and 2). Client interface 140 can be a standalone application (e.g., standalone image visualization and analysis software) or a web browser or other interface software that allows remote access to image visualization and analysis application 120. For example, client interface 140 allows a remote operator to log into system 100 (such as the systems shown in FIGS. 1 and 2) and access stored image data (such as data stored in storage subsystem 104 or other network-attached storage) or image data uploaded to system 100 for processing. In some embodiments, client interface 140 can include any of the software modules described herein. In this manner, a remote user can remotely interact with elements (e.g., configurable elements) of the system (e.g., a histologist or pathologist can select user-configurable parameters, such as menu bar tools and / or viewer panels) so that the image data can be analyzed and / or interpreted.

[0072] In some embodiments, the graphical user interface is adapted to display only certain features depending on the selected zoom level. For example, depending on the selected zoom level, certain tools in the menu bar may not be enabled (see FIG. 4A). Similarly, certain panels displayed near the accessed visualization of the image may not be available. In general, the system 100 may be configured to "limit" access to certain tools and panels that are not contextually relevant during image analysis, given a particular zoom level. For example, at a particular zoom level, it may not be possible to annotate certain portions of an image if there is not enough resolution between certain features in the image. Thus, if the software determines that a tool is not relevant at the selected zoom level, the tool is not enabled; as discussed above, this facilitates quick review of the data displayed in the visualization and provides an improved and potentially less confusing user experience.

[0073] 3 shows a flowchart illustrating a method for visualizing associated image data derived from a biological sample stained for the presence of one or more biomarkers. In step 300, at least one image of the biological sample is accessed. In some embodiments, the biological sample is stained for the presence of one or more biomarkers. In some embodiments, multiple images are acquired, such as multiple images from the same biological sample, each image containing stains indicative of the presence or absence of a particular biomarker.

[0074] Subsequently, a first visualization representation is rendered within the graphical user interface (step 310), where the first visualization representation includes a rendering of at least one image at at least a first zoom level. In some embodiments, the first visualization representation 400 includes a rendering of a plurality of accessed images (see, e.g., FIG. 4A ). In some embodiments, the first visualization representation includes a series of representations 401 (e.g., image placeholders), where each representation 401 may include one of the accessed images 410A or 410B. In some embodiments, each representation 401 is the same size and / or shape. In some embodiments, each representation 401 includes a first portion 402 that includes one of the accessed images 410A and a second portion 403 that includes identifying indicia. In some embodiments, the identifying indicia include an identification of a stain or stained biomarker appearing in the image. As an example, second portion 403 of representation 401 indicates that a particular image 410A in first portion 402 was stained with hematoxylin and eosin.

[0075] In some embodiments, a first set of tools (e.g., annotation tools 405) in menu bar 404 are displayed simultaneously with first visualization representation 400 (step 320). As described above and as shown in FIG. 4A, some tools are unavailable for selection, i.e., they are “grayed out” (e.g., see setup calibration tool 408), while other tools are available for selection and are displayed as white icons (e.g., see rotation tool 407). As a further example, each of the five annotation tools 405 is grayed out and therefore disabled. In some embodiments, certain tools may be completely hidden from the user at certain zoom levels, as described further herein. Thus, the first set of tools (i.e., selectable tools and tools represented by white icons) represents a subset of all tools available to the user. Given the four image zoom levels shown in representation 401 of FIG. 4A, these unavailable, i.e., “grayed out,” tools, as shown in FIG. 4A, are considered irrelevant at a given zoom level of 1x. In other words, tools that are not enabled or hidden from user selection are deemed invalid at the selected zoom level. For example, it may not be worthwhile for a histologist or pathologist to perform a measurement or draw an arrow on a cell because at a particular zoom level (e.g., 1x), not enough cellular features can be resolved to accurately perform the measurement or correctly place the arrow pointing to a particular structure of interest.

[0076] In some embodiments, the first set of panels may also be displayed simultaneously with the first visualization and the first set of tools. For example, the zoom panel 406 and the slide navigator panel 409 may be displayed simultaneously with the first visualization and the first set of tools. As with the first set of tools, only panels that are contextually relevant at a given zoom level are displayed. For example, FIG. 5A illustrates panels that a user can select in a context menu or drop-down menu. Unavailable items are not selectable and are "grayed out." Similarly, selectable panels are displayed in white and can be selected by the user. Additionally, panels that have already been selected may be marked with an indicia, such as a check mark.

[0077] Following the simultaneous presentation of the first visualization representation (e.g., the three representations 401 in FIG. 4A ) and the first set of tools (those not “grayed out” in the menu bar 404) and / or the first set of panels, the user may then interact with the visualization representation, such as by changing the zoom level, e.g., increasing the zoom of one or more of the accessed images presented in the first visualization representation, allowing the user to view at least a portion of the accessed image in greater detail, thereby providing at least a second visualization representation at a second zoom level 430. “Increasing the zoom” or “zooming into an image” means that a portion of the image is enlarged, thereby increasing the visual resolution of that portion of the image. For example, it should be appreciated that accessed image 410B is presented in FIG. 4A at a 1x zoom level, and that at this zoom level certain tissue structures, e.g., 420A, are difficult to decipher. However, when the zoom level of image 410B is increased as shown in FIG. 4B (e.g., increasing the zoom level from 1x to about 10x), the magnification and / or resolution of tissue structure 420B is revealed in greater detail, e.g., at a level where individual cells can be resolved.

[0078] Concurrent with the display of the second visualization (step 330) at a second zoom level 430, a second set of tools is displayed in the menu bar 404 (step 340). Like the first set of tools, the second set of tools represents a subset of all tools available to the user. As an example, compared to the first set of available tools (see FIG. 4A), the second set of tools includes each of the five annotation tools 405 (see FIG. 4B).

[0079] In the particular embodiment shown in FIG. 4B , the second set of tools includes the first series of tools, i.e., the second set of tools includes all of the tools available in the first series of tools. In some embodiments, the second set of tools does not include all of the tools provided in the first series of tools. In some embodiments, the second series of tools includes at least one different tool than those provided in the first series of tools.

[0080] In some embodiments, the second set of panels is displayed simultaneously with the second visualization and the second set of tools at a second zoom level 430. As with the second set of tools, only panels that are contextually relevant at a given zoom level are displayed. In the embodiment shown in FIG. 4B, compared to the embodiment of FIG. 4A, the sliding panel 425 is automatically displayed at a zoom level of approximately 10x. See FIG. 5B, unavailable items are not selectable and are "grayed out." Similarly, selectable viewer panels are displayed in white and can be selected by the user. Additionally, panels that have already been selected are marked with an indicia, such as a check mark.

[0081] FIG. 4C illustrates the selection of a zoom level (e.g., an intermediate zoom level) between the zoom levels shown in FIG. 4A and FIG. 4B. While four representations 401 were shown in FIG. 4A at a 1x zoom level, only three representations are visualized at the zoom level in FIG. 4C, and only one of the three representations is fully visualized. Notably, the same tools appear to be available in FIG. 4C as in FIG. 4A. Similarly, the same panels appear in FIG. 4C as in FIG. 4A. Thus, FIG. 4C indicates that the zoom threshold has not been reached by the user, such that the image visualization and analysis application 420 would make other tools available in the menu bar 404, or, for that matter, other viewer panels. In this regard, FIG. 4C illustrates that the items displayed and made available to the user are contextually dependent, here on the level of zoom selected by the user.

[0082] In some embodiments, each of the menu items and / or viewer panels has a pre-programmed zoom threshold level that must be achieved before enabling the respective menu item and / or viewer panel to be available. For example, referring to FIG. 4A, the annotation tool 405 may only be available if the user selects a zoom level above a predetermined zoom threshold value, e.g., 5x. In some embodiments, each individual tool within the menu bar 404 may have a different predetermined threshold. For example, a first annotation tool may have a predetermined threshold of 2x, a second annotation tool may have a predetermined threshold of 6x, and a slide calibration tool may have a predetermined threshold of 10x. In some implementations, In morphology, the zoom threshold may depend on the tissue type or stain being observed.

[0083] In some embodiments, the threshold for whether a menu bar item or viewer panel is available need not be tied to a predetermined zoom threshold, i.e., 1x, 2x, 4x, 8x, 16x, 32x, etc. Rather, the threshold may be tied to whether a zoom level is selected at which individual cells or individual nuclei can be resolved by the user. Alternatively, the threshold may be associated with whether a zoom level is selected at which a certain number of cells are present in a predefined area (pixel x pixel), e.g., 100 cells in a 500 pixel x 500 pixel area. In some embodiments, different elements may be visualized depending on the available display resolution. For example, the threshold may be predefined as "p" for an "m x n" display resolution, but may be predefined as "p * q" for an (m * q x n * q) display resolution, where q is a scaling factor to account for differences in display resolution. In still other embodiments, whether a particular viewer panel is displayed or hidden may be tied to the available display resolution. For example, five viewer panels may become available or hidden (i.e., become contextually relevant and available) as a zoom level threshold is reached, but if available screen "real estate" is unavailable due to low or limited display resolution, the system may continue to hide certain panels; the system determines which of the available viewer panels are most relevant, taking into account the display resolution limitations, and prioritizes those panels for display. Further, by way of example, viewer panels may be cycled as needed to accommodate limited display resolutions.

[0084] As described herein, in some embodiments, entire menu items may be hidden until a particular zoom level is selected by the user. For example, suppose a menu bar includes items A, B, C, D, E, F, G, and H. Further, assume a zoom level of 1x, as shown in FIG. 6A, where only menu bar items A, B, E, and H are shown in the graphical user interface. According to the present disclosure, these menu bar items A, B, E, and H are the only tools relevant and useful to the user at the 1x zoom level. When a user zooms in on one of the images (see FIG. 6B), additional menu bar items may be displayed, e.g., menu bar items C and G, if the menu bar items (i.e., C and G) are associated with the selected level of zoom (e.g., 3x). When a user zooms in further on one of the images (see FIG. 6C), here 601B, even additional menu bar items, i.e., D and F, may be displayed if the additional menu bar items are associated with the selected level of zoom (e.g., 8x). FIG. 6C further shows that a second viewer panel, i.e., “Viewer Panel 2,” becomes visible through the graphical user interface, but only when a certain zoom level is reached (i.e., a zoom level at least greater than 3x, e.g., 8x).

[0085] In some embodiments, an input image is received by the visualization system and a visualization is provided at a default zoom level, e.g., a zoom level of 1x. In some embodiments, the system receives user input, e.g., a selection of an updated zoom level, a selection of a particular image. In some embodiments, a comparison is made between the received user input and a threshold condition, e.g., a comparison is made between received user inputs of zoom levels to determine whether a zoom threshold level has been reached, or a comparison is made between received user inputs of image selections to determine biomarkers identified in the image. In some embodiments, if the zoom threshold level is met, an additional visualization element (e.g., a tool, a panel) may be presented to the user. In some embodiments, if different biomarkers are selected in the second image compared to the first image, an additional visualization element may be presented to the user. In some embodiments, at least one GUI element is changed based on the user's selection.

[0086] Additional Embodiments

[0087] In some embodiments, each displayed visualization representation or element has a location within a coordinate system of a display provided within an interface application, such as a browser. For example, an icon for a tool, such as an image analysis tool, has a location within the display coordinate system. For example, if a display resolution is 4,000 x 3,000 and each pixel is considered a point in the coordinate system, an icon for an annotation tool may have a location defined by pixel bounds [150, 200] (top left corner), [160, 200] (top right corner), [150, 210] (bottom left corner), and [160, 210] (bottom right corner). In some embodiments, viewer panels, image data, and other representations may have locations within the display coordinate system. In some embodiments, a first viewer panel may have a first location bound, and a second viewer element may have a second location bound. In some embodiments, each viewer panel may have a variable location bound, depending on the type of information being displayed and the amount of information available. In some embodiments, the positions of various elements within the display coordinate system may be fixed or may vary depending on the context.

[0088] Embodiments of the subject matter and operations described herein can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus. Any of the modules described herein may include logic executed by a processor. As used herein, "logic" refers to any information in the form of instruction signals and / or data that can be applied to affect the operation of a processor. Software is an example of logic.

[0089] A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Furthermore, a computer storage medium is not a propagating signal, but a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described herein may be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0090] The term "programmed processor" encompasses all kinds of devices, apparatus, and machines for processing data, including, for example, a programmable microprocessor, a computer, a system on a chip, or two or more of the foregoing, or a combination thereof. An apparatus may include special purpose logic circuitry such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may be integrated into a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. This can be realized.

[0091] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative, or procedural languages, and can be deployed in any form. It can be included as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0092] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by manipulating input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), or may implement an apparatus.

[0093] Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data, transfer data, or both. However, such devices are not required for a computer. Furthermore, a computer can be incorporated into another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0094] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer that has a display device, e.g., an LCD (liquid crystal display), an LED (light emitting diode) display, or an OLED (organic light emitting diode) display, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. In some implementations, a touch screen can be used to display information and receive input from the user. Other types of devices can be used to A computer may also provide for interaction with a user. For example, feedback provided to a user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from a user may be received in any form, including acoustic, speech, or tactile input. A computer may also interact with a user by sending documents to or receiving documents from a device used by the user. For example, a computer may send a web page to a web browser on a user's client device in response to a request received from the web browser.

[0095] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component such as an application server, or includes a front-end component. The end component, e.g., a client computer with a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described in this specification, or one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), inter-networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example, network 20 of FIG. 1 can include one or more local area networks.

[0096] A computing system may include any number of clients and servers. Clients and servers are typically remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship. In some embodiments, a server sends data (e.g., HTML pages) to a client device (e.g., for the purposes of displaying the data and receiving user input from a user interacting with the client device). Data generated on the client device (e.g., the result of a user's operation) can be received from the client device at the server.

[0097] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned herein and / or listed in the Application Data Sheet are incorporated herein by reference in their entirety. Aspects of some embodiments can be modified, if necessary, to employ concepts from the various patents, applications, and publications to provide further embodiments.

[0098] While the present disclosure has been described with reference to certain exemplary embodiments, it should be understood that many other modifications and embodiments may be devised by those skilled in the art which would fall within the spirit and scope of the principles of the present disclosure. More particularly, reasonable variations and modifications are possible in the components and / or arrangements of the subject matter combined and arranged within the scope of the foregoing disclosure, the drawings, and the appended claims without departing from the spirit of the disclosure. In addition to variations and modifications of the components and / or arrangements, alternative uses will also be apparent to those skilled in the art.

Claims

1. 1. A computing device including a display screen, the computing device comprising: accessing at least one image of the biological sample stained for the presence of one or more biomarkers from one or more memories communicatively coupled to the computing device; displaying a first visualization of the at least one image on the display screen at a first zoom level, and displaying a first subset of user-selectable elements on the display screen concurrently with displaying the first visualization; responsive to receiving user input, display a second visualization representation of the at least one image on the display screen at a second zoom level different from the first zoom level, and while displaying the second visualization representation, display a second subset of user-selectable elements on the display screen, wherein (i) one or more elements in the second subset of user-selectable elements are disabled or hidden at the first zoom level, or (ii) one or more elements in the first subset of user-selectable elements are disabled or hidden at the second zoom level.

2. The computing device of claim 1 , wherein the second zoom level meets at least a predetermined zoom threshold.

3. 2. The computing device of claim 1, wherein the second subset of user-selectable elements that are disabled or hidden at the first zoom level are contextually relevant at the second zoom level but not contextually relevant at the first zoom level.

4. The computing device of claim 1 , wherein the second subset of user-selectable elements comprises one or more image annotation tools that are disabled or hidden at the first zoom level.

5. The image annotation tools include a region of interest identification tool, a measurement tool, an indicia drawing tool, and The computing device of claim 4 , further comprising a region exclusion tool.

6. The computing device of claim 1 , wherein the second subset of user-selectable elements comprises one or more image processing tools that are disabled or hidden at the first zoom level.

7. The computing device of claim 1 , wherein the second zoom level is greater than the first zoom level.

8. 10. The computing device of claim 1, wherein the computing device is further configured to display one or more viewer panels on the display screen at the second zoom level, the one or more viewer panels for display at the second zoom level being disabled or hidden at the first zoom level.

9. 10. The computing device of claim 1, wherein the first zoom level is a non-magnified zoom level, and the first visualization comprises at least one representation that includes the at least one image of the biological sample.

10. The computing device of claim 9 , wherein the at least one representation further includes a portion that includes identifying indicia.

11. 1. A computing device including a display screen, the computing device comprising: accessing at least one image of the biological sample stained for the presence of one or more biomarkers from one or more memories communicatively coupled to the computing device; displaying at least a first visualization on the display screen, the visualization including the at least one image at a first zoom level, and displaying at least a first viewer panel on the display screen while displaying the first visualization; responsive to receiving user input, display a second visualization of the at least one image on the display screen at a second zoom level different from the first zoom level, and concurrently displaying the second visualization, display at least a second viewer panel on the display screen in addition to the first viewer panel, wherein the at least the second viewer panel is disabled or hidden at the first zoom level.

12. 12. The computing device of claim 11, wherein a third viewer panel is displayed on the display screen simultaneously with the display of the second viewer panel, and wherein the third viewer panel is hidden at the first zoom level.

13. 12. The computing device of claim 11, wherein previously displayed or hidden user-selectable tools are enabled on the display screen simultaneously with the display of the second viewer panel.

14. The computing device of claim 13 , wherein the user-selectable tool is an image annotation tool.

15. 12. The computing device of claim 11, wherein the at least the first visualization representation comprises: (i) a first portion including the at least one image of the biological sample at the first zoom level; and (ii) a second portion including identifying indicia.

16. The computing device of claim 15 , wherein the identifying indicia includes an identification of a biomarker.

17. displaying, on a computing device having a display screen, a first visualization including at least one image of the stained biological sample, the first visualization being displayed in a first condition; displaying at least one of a first viewer panel or a first set of user selectable elements on the display screen while displaying the first visualization; displaying, in response to receiving a user input, a second visualization representation of the at least one image in a second condition state on the display screen, the second condition state resulting from a user selection; and and simultaneously displaying the second visualization representation, displaying on the display screen at least one of: (i) a second set of user-selectable elements that are not enabled in the first condition state; or (ii) a second viewer panel that is hidden in the first condition state in addition to the first viewer panel.

18. 18. The method of claim 17, wherein the first condition state is a default state.

19. 20. The method of claim 18, wherein the user selection is a zoom level selection, the first condition state is a first zoom level, and the second condition state is a second zoom level different from the first zoom level.

20. 18. The method of claim 17, wherein the user selection is a selection of an image, and the image selected for the second condition state includes a different stain than the image for the first condition state.

21. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to: displaying, on a display screen, a first visualization of at least one image at a first zoom level, and displaying a first subset of user-selectable tools on the display screen concurrently with displaying the first visualization; subsequently displaying a second visualization representation of the at least one image on the display screen at a second zoom level different from the first zoom level, and displaying a second subset of user-selectable tools on the display screen while displaying the second visualization representation, wherein one or more tools in the second subset of user-selectable tools are not enabled at the first zoom level.

Images