Context-Adaptive Digital Pathology Interface

Abstract

To provide an image visualization system, method, and computer program product that include a contextually adaptive digital pathology interface.SOLUTION: A method includes the following steps. At least one image of a biological sample stained for the presence of one or more biomarkers is obtained (300). The image is displayed on a display screen at a first zoom level (310), in which a first subset of user selectable elements is contemporaneously displayed (320). As a result of user input, the image being is displayed at a second zoom level (330), in which a second subset of user selectable elements are contemporaneously displayed with the image (340). The one or more elements within the second subset of user selectable elements are disabled or hidden at the first zoom level, or one or more elements within the first subset of user selectable elements are disabled or hidden at the second zoom level.SELECTED DRAWING: Figure 3

Description

Background Art

[0001] (Cross - reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 62 / 726,081, filed Aug. 31, 2018, which is hereby incorporated by reference in its entirety for all purposes. Digital pathology refers to the management and interpretation of pathological information in a digital environment. A scanning device is used to image slides of biological samples that can be stained, resulting in the generation of digital slides, e.g., the entire slide image. Using digital pathology software, digital slides can be stored in a computer's memory device, displayed on a computer monitor, and pathological information can be analyzed.

Summary of the Invention

[0002] The present disclosure provides a visual representation of an image of a biological sample stained for the presence of one or more biomarkers and is adapted to enable the display of contextually relevant elements (e.g., analysis tools, viewer panels) to a user based on whether a pre-established criterion is met. In some embodiments, contextually relevant elements are provided to the user based on a selected zoom level or a selection of an image having a particular stain. By providing only contextually relevant elements, the user is not overwhelmed by visualized elements (e.g., elements that are not appropriate or ineffective in a particular context) and can instead select the correct elements based on conditional states (e.g., the selected zoom level, the selected image type, the selected tissue type, the selection of a slide from a tissue microarray of the entire slide image, the selected slide to which a particular stain has been applied (e.g., H&E), the selected slide stained for the presence of a particular biomarker, the selected slide stained using immunohistochemistry as opposed to in situ hybridization, the selected slide to which a particular image analysis algorithm has been applied). Further, it is believed that providing only contextually relevant elements enables a more rapid and accurate review of the presented image data. Further, due to limited display screen size and resolution, it is often difficult to provide multiple visual representations at once, and if multiple visualizations are provided, they may overlap or interfere with the display of important elements to the user. By displaying only contextually relevant elements when given a particular condition (e.g., a particular zoom level selected by the user), the most appropriate elements can be prioritized and visualized taking into account the limited display size and resolution.

[0003] In some embodiments, the computing device includes a display screen, and the computing device accesses at least one image of a biological sample stained for the presence of one or more biomarkers (e.g., HER2, PD-L1, etc.) from one or more memories communicatively coupled to the computing device, displays a first visualization representation of the at least one image at a first zoom level on the display screen, and while displaying the first visualization representation, displays a first subset of user-selectable tools on the display screen. Subsequently, in response to receiving user input, a second visualization representation of the at least one image is displayed on the display screen at a second zoom level greater than the first zoom level, and while displaying the second visualization representation, a second subset of user-selectable elements is displayed on the display screen, where (i) one or more elements within the second subset of user-selectable elements are disabled or not displayed at the first zoom level, or (ii) one or more elements within the first subset of user-selectable elements are disabled or not displayed at the second zoom level, and does the following. It is configured to do so.

[0004] In some embodiments, the second zoom level meets at least a predetermined zoom threshold. In some embodiments, the second subset of user-selectable elements is contextually relevant at the second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. As an example, the tools and menus are context-dependent. When the user is fully "zoomed out", only the relevant tools used in that context are enabled (see Figure 4A). When the user zooms in to a specified zoom level (see Figure 4B), the tools on the toolbar are enabled or a context panel is displayed. When in the "zoomed in" context of the slide, the user can use these tools. Thus, the tools that can be used with either the first set of tools or the second set of tools depend on the zoom level selected by the user, e.g., the second zoom level compared to the first zoom level, and the second zoom level meets or exceeds a predetermined zoom threshold level for enabling the second set of tools.

[0005] In some embodiments, the menu bar tools include image annotation tools, slide setting tools, slide selection tools, navigation tools, and display tools. In some embodiments, the image annotation tools are enabled at the second zoom level. In some embodiments, the image annotation tools are hidden at the first zoom level. In some embodiments, the image annotation tools include region of interest identification tools, measurement tools, marker drawing tools, and region exclusion tools.

[0006] In some embodiments, the computing device is further configured to display one or more viewer panels on the display screen at a second zoom level, and the one or more viewer panels for display at the second zoom level are disabled at the first zoom level. In some embodiments, the first zoom level is a 1x zoom level (i.e., the lowest zoom level such as an image captured by scanning or other means without magnification), and the first visualization representation includes at least one representation including at least one image of the biological sample. In some embodiments, the at least one representation further includes a portion including an identification mark.

[0007] In some embodiments, the computing device includes a display screen, and the computing device accesses at least one image of a biological sample stained for the presence of one or more biomarkers from one or more memories communicatively coupled to the computing device, displays a first representation of the at least one image on the display screen at a first zoom level, and while displaying the first representation, displays at least a first viewer panel on the display screen, and subsequently displays a second representation of the at least one image on the display screen at a second zoom level greater than the first zoom level, and while displaying the second representation, displays at least a second viewer panel on the display screen in addition to the first viewer panel, wherein at least the second viewer panel is hidden at the first zoom level.

[0008] In some embodiments, the third viewer panel is displayed on the display screen simultaneously with the display of the second display panel, and the third viewer panel is made invisible at the first zoom level. In some embodiments, the menu bar icon is enabled on the display screen simultaneously with the display of the second display panel. In some embodiments, the displayed menu bar icon is selected from the group consisting of an area of interest identification tool, a measurement tool, a marker drawing tool, and an area exclusion tool. In some embodiments, at least The first representation also includes, in at least some embodiments, (i) a first portion including at least one image of the biological sample at the first zoom level, and (ii) a second portion including an identifier. In some embodiments, the identifier includes the identification of biomarkers.

[0009] In some embodiments, the method comprises displaying, on a computing device having a display screen, a first visualization representation including at least one image of a stained biological sample, the first visualization representation being displayed in a first condition state; displaying, simultaneously with the display of the first visualization representation, at least one of a first viewer panel or a first set of user-selectable elements on the display screen; subsequently, displaying, on the display screen, a second visualization representation of at least one image in a second condition state, the second condition state being obtained as a result of a user selection; and displaying, simultaneously with the display of the second visualization representation, at least one of a second set of user-selectable elements not enabled in the first condition state, or a second viewer panel in addition to the first viewer panel, on the display screen, the second viewer panel being made invisible at the first zoom level. In some embodiments, the first condition state is a default state. In some embodiments, the user selection is a zoom level, and the second condition is a zoom level greater than the default zoom level. In some embodiments, the user selection is an image selection, and the image selected for the second condition includes a different stain than the image in the first condition state.

[0010] In some embodiments, the method comprises displaying, on a computing device having a display screen, a first visualization representation including at least one image of a stained biological sample, the first visualization representation being displayed in a first condition state; displaying, simultaneously with the first visualization representation, at least one of a first viewer panel or a first series of user-selectable elements on the display screen; subsequently, displaying, on the display screen, a second visualization representation of at least one image at a second zoom level, the second zoom level being greater than the first zoom level; and displaying, simultaneously with the second visualization representation, at least one of a second series of user-selectable elements that were hidden or disabled at the first zoom level, or at least a second viewer panel in addition to the first viewer panel, the second viewer panel being hidden at the first zoom level.

[0011] In some embodiments, the first zoom level is a default zoom level. In some embodiments, the second zoom level is one that sufficiently resolves cell clusters or sufficiently resolves cell nuclei. In some embodiments, the first zoom level is the lowest available zoom level and the second zoom level is at least 5 times greater. In some embodiments, the first zoom level is the lowest available zoom level and the second zoom level is at least 10 times greater.

[0012] In some embodiments, the second subset of user-selectable elements is contextually relevant at the second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. In some embodiments, the menu bar tools include image annotation tools, slide setting tools, slide selection tools, navigation tools, and display tools. In some embodiments, the image annotation tools are enabled at the second zoom level. In some embodiments, the image annotation tools are hidden at the first zoom level. In some embodiments, the image annotation tools include region of interest identification tools, measurement tools, marker drawing tools, and region exclusion tools.

[0013] In some embodiments, the first visualization representation includes (i) a first portion including at least one image of a biological sample at the first zoom level, and (ii) a second portion including an identifier. In some embodiments, the identifier includes the identification of biomarkers. In some embodiments, the non-transitory computer-readable medium stores instructions that, when executed by one or more processors of a computing system, cause the computing system to display, on a display screen, a first visualization representation of at least one image at the first zoom level, and simultaneously with displaying the first visualization representation, display, on the display screen, a first subset of user-selectable tools, and subsequently display, on the display screen, a second visualization representation of at least one image at a second zoom level greater than the first zoom level, and simultaneously with displaying the second visualization representation, display, on the display screen, a second subset of user-selectable tools, wherein one or more tools within the second subset of user-selectable tools are not enabled at the first zoom level.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] ​For a general understanding of the features of the present disclosure, reference is made to the drawings. In the drawings, like reference numbers are used throughout to identify like elements.

[0016] The patent or application file contains at least one drawing created in color. A copy of this patent or patent application publication with color drawings will be provided to the office upon request and payment of the required fee.

[0017]

Figure 1

[0018]

Figure 2A

[0019]

Figure 2B

[0020]

Figure 3

[0021]

Figure 4A

[0022]

Figure 4B

[0023]

Figure 4C

[0024]

Figure 5A

[0025]

Figure 5B

[0026]

Figure 6A

[0027]

Figure 6B

[0028]

Figure 6C

Best Mode for Carrying Out the Invention

[0029] Detailed Description It should also be understood that, unless explicitly stated to the contrary, in a method claimed in this specification that includes a plurality of steps or acts, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts of the method are described.

[0030] As used in this specification, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly indicates otherwise. The term "comprising" is defined inclusively such that "comprising A or B" means including A, B, or both A and B.

[0031] When used in this specification and the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as inclusive. That is, it includes at least one of the number or list of elements, and optionally, additional unlisted items, but is interpreted as including more than one. Only terms such as "only one" or "exactly one", or when used in the claims, terms such as "consisting of", which are explicitly stated to the contrary, refer to exactly one element of a number or list of elements. Generally, the term "or" as used in this specification shall be interpreted only as indicating an exclusive alternative when preceded by an exclusive term such as "either", "any one", "only", "exactly either" (i.e., "one or the other but not both"). "Consisting essentially of", when used in the claims, shall have the ordinary meaning as used in the field of patent law.

[0032] Terms such as "comprising", "including", "having" are used interchangeably and have the same meaning. Similarly, "comprise", "include", "have" are used with the same meaning and have the same meaning. Specifically, each term is defined in accordance with the general US patent law definition of "including", and thus is interpreted as an open term meaning "at least the following" and is also interpreted not to exclude additional features, limitations, aspects, etc. Thus, for example, "an apparatus having components a, b, and c" means that the apparatus includes at least components a, b, and c. Similarly, the phrase "a method including steps a, b, and c" means that the method includes at least steps a, b, and c. Further, steps and processes may be outlined in a particular order herein, but one of ordinary skill in the art will recognize that the ordered steps and processes may vary.

[0033] As used in this specification and the claims, the phrase "at least one" in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements. However, it is not necessary to include at least one of all the elements specifically listed within the list of elements, nor does it exclude combinations of elements within the list of elements. This definition also allows for the optional presence of elements other than those specifically identified within the list of elements referred to by the phrase "at least one", whether or not such other elements are related to the specifically identified elements. Thus, by way of non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently "at least one of A and / or B") can, in some embodiments, refer to including at least one, and optionally two or more, of A and no B (and optionally including elements other than B), can, in some embodiments, refer to including at least one, and optionally two or more, of B and no A (and optionally including elements other than A), and can, in yet another embodiment, refer to including at least one, and optionally two or more, of A and at least one, and optionally two or more, of B (and optionally including other elements), and so on.

[0034] As used herein, the term "image data" encompasses raw image data obtained from a biological tissue sample by, for example, an optical sensor or sensor array, or preprocessed image data. In particular, the image data can include a pixel matrix.

[0035] As used herein, the terms "image", "image scan", or "scanned image" encompass raw image data obtained from a biological tissue sample by, for example, an optical sensor or sensor array, or preprocessed image data. In particular, the image data can include a pixel matrix.

[0036] As used herein, the terms "biological sample", "tissue sample", "specimen", etc. refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as veterinary animals like humans, cats, dogs, horses, cows, and pigs, as well as laboratory animals like mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or tissue needle biopsies), cell samples (such as cytological smear specimens like Pap smear specimens or blood smear specimens, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components by centrifugation or the like). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucosa, tears, sweat, pus, biopsy tissue (such as obtained by surgical biopsy or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, rinse fluid (such as cheek swabs), or any material containing biomolecules derived from an initial biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a part thereof obtained from a subject (such as a homogenized or liquefied sample).

[0037] As used herein, the term "slide" refers to any substrate of any suitable dimension on which a biological specimen is placed for analysis (e.g., glass, stone, in whole or in part) a substrate made of, for example, plastic, silicon, etc., and more specifically, refers to a "microscope slide" such as a standard 3-inch × 1-inch microscope slide or a standard 75 mm × 25 mm microscope slide. Examples of biological specimens that can be placed on the slide include, but are not limited to, cytological smear specimens, thin tissue sections (such as those from biopsies), and arrays of biological specimens, such as tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in some embodiments, tissue sections, DNA samples, RNA samples, and / or proteins are placed at specific locations on the slide. In some embodiments, the term slide can refer to SELDI and MALDI chips, as well as silicon wafers.

[0038] As used herein, terms such as "staining" and "stained" as used herein generally refer to any treatment of a biological specimen to detect and / or distinguish the presence, location, and / or amount (such as concentration) of a particular molecule (such as a lipid, protein, or nucleic acid) or a particular structure (such as a normal or malignant cell, cytoplasmic sol, nucleus, Golgi apparatus, or cytoskeleton) in the biological specimen. For example, staining can provide contrast between a particular molecule or a particular cell structure and the surrounding portion of the biological specimen, and the intensity of the staining can provide a measure of the amount of a particular molecule in the specimen. Staining can be used to assist in the observation of molecules, cell structures, and organisms using not only bright-field microscopes but also other observation tools such as phase-contrast microscopes, electron microscopes, and fluorescence microscopes. Some stains performed by the system can be used to visualize the cell outline. Other stains performed by the system can depend on the particular cell component (such as a molecule or structure) being stained without or with relatively less staining of other cell components. Examples of types of staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on intermolecular reactions (including non-covalent interaction) such as hybridization reactions between nucleic acid molecules. Particular staining methods include, but are not limited to, primary staining methods (such as H&E staining, Pap staining, etc.), enzyme-linked immunohistochemical methods, and in situ RNA and DNA hybridization methods such as fluorescence in situ hybridization (FISH).

[0039] As used herein, the term "user interface" refers to an interface that enables a user, such as an end user such as a histologist and / or pathologist, to input commands and data and receive results such as a graphical user interface (GUI). The terms "user interface" and "graphical user interface" are used interchangeably herein.

[0040] As will be described in more detail herein, the present disclosure relates to a graphical user interface that enables a user to display and / or analyze one or more images of a biological sample stained for the presence of one or more biomarkers, whereby visualization of specific elements and / or availability of specific analysis and / or processing tools are provided to the user in a context-based manner. In some embodiments, the graphical user interface is adapted to provide contextually relevant visualization representations and elements for selection based on whether a pre-established condition is met. In some embodiments, the pre-established condition is a selection made by the user. For example, the selection can be a zoom level or magnification level selected by the user, and selection of different zoom levels or magnification levels causes the graphical user interface to adaptively generate a specific visualization representation or enable specific user-configurable items, such as an annotation tool, an image processing tool, etc. In some embodiments, the graphical user interface adjusts the visualization representation, analysis tool, and / or viewer panel provided to the user (such as for display on a display) according to the selected zoom level. For example, a first subset of contextually relevant tools can be presented to the user at the lowest zoom level (e.g., no optical magnification as compared to an optical magnification of 10x, 20x, or 40x), while a more comprehensive second subset of tools is presented to the user at a greater zoom level of 10x, and additional tools included in the second subset are again contextually related to the 10x zoom level. As another example, the selection can be a specific type of tissue (i.e., an image of the tissue) selected by the user, or an image of a tissue sample stained for the presence of a specific biomarker. and can be presented to the user at the lowest zoom level (e.g., no optical magnification as compared to an optical magnification of 10x, 20x, or 40x), while a more comprehensive second subset of tools is presented to the user at a greater zoom level of 10x, and additional tools included in the second subset are again contextually related to the 10x zoom level.

[0041] For example, by presenting only visualization representations, analysis tools, viewer panels, etc. that are contextually relevant to pre-established conditions, such as a selected zoom level, a selected tissue type, a selected slide to which a specific stain has been applied, a selected slide stained for the presence of a specific biomarker, a selection of the entire slide image as opposed to a tissue microarray, it is believed that the user may be able to interact with the software more efficiently. Stated another way, since the user is not overwhelmed by the availability of a large number of tools and / or viewer panels that are not relevant to a specific zoom level, the operator may be able to interact with the graphical user interface more efficiently, ultimately resulting in a more rapid review and analysis of the presented image of the tissue sample.

[0042] In some embodiments, the system of the present disclosure is configured to facilitate the interpretation and reporting of image data obtained from a subject (e.g., a human patient). In some embodiments, the image data is obtained from a scanning device (such as the VENTANA DP 200 scanner available from Ventana Medical Systems, Inc., Tucson, AZ), and the image data can be stored in a database such as a network database for later visualization and analysis. For example, the image data can be obtained using a scanning device, and the scanned image data can be stored in a file located on the memory subsystem 104 or a networked server, whereby the file can be obtained later for visualization and analysis (see FIG. 1). In some embodiments, software such as an image visualization and analysis application is executed directly on the system, and the image data is obtained from a networked server for interpretation and reporting by a user interacting with the software (see FIG. 2A). In other embodiments, software such as an image visualization and analysis application is executed on a remote system, and a client interface or client portal is used for the client to access the system, whereby the image data can be obtained from the memory subsystem for visualization and analysis (see FIG. 2B).

[0043] The systems and methods provided herein can be applied to the visualization and analysis of any type of image of a tissue stained for the presence of one or more biomarkers. For example, a biological sample can be stained by the application of one or more stains, and the resulting image or image data includes signals corresponding to each of the one or more stains. In some embodiments, the input image is a simple image having only a single stain (e.g., stained by 3,3'-diaminobenzidine (DAB)). In some embodiments, a biological sample can be stained in a multiplex assay for two or more stains (thus providing a multiplex image). In some embodiments, a biological sample is stained for at least two biomarkers. In some embodiments, a biological sample is stained for the presence of at least two biomarkers and is also stained by a primary stain (e.g., hematoxylin). In some embodiments, a biological sample is stained for the presence of at least one protein biomarker and at least two nucleic acid biomarkers (e.g., DNA, RNA, microRNA, etc.).

[0044] In some embodiments, a biological sample is stained in an immunohistochemistry assay for the presence of one or more protein biomarkers For example, a biological sample can be stained for the presence of the human epidermal growth factor receptor 2 protein (HER2 protein). Currently in the United States, there are two Food and Drug Administration (FDA)-approved methods for HER2 evaluation: HerceptTest™ (DAKO, Glostrup, Denmark) and the HER2 / neu (4B5) rabbit monoclonal primary antibody (Ventana, Tucson, Arizona).

[0045] In some embodiments, the biological sample is stained for the presence of estrogen receptor (ER), progesterone receptor (PR), or Ki-67. In still other embodiments, the biological sample is stained for the presence of EGFR or HER3. Examples of other protein biomarkers are described by Zamay et al., "Current and Prospective Biomarkers of Lung Cancer", Cancers (Basel), November 2018; 9(11), the disclosure of which is incorporated herein by reference in its entirety. Examples of protein biomarkers described by Zamay include CEACAM, CYFRA21-1, PKLK, VEGF, BRAF, and SCC.

[0046] In some embodiments, the biological sample is stained in an in situ hybridization (ISH) assay for the presence of one or more nucleic acids, including mRNA. U.S. Patent No. 7,087,379 (the disclosure of which is incorporated herein by reference in its entirety) describes a method of staining a sample with an ISH probe such that individual spots (or dots) representing single gene copies can be observed and detected. In some embodiments, several target genes are analyzed simultaneously by exposing the cell or tissue sample to a plurality of nucleic acid probes labeled with a plurality of different nucleic acid tags.

[0047] FIG. 1 shows a system 100 (a computer or computing device) that includes a scanning device 110 communicatively coupled to a processing subsystem 102. The scanning device 110 can be coupled to the processing subsystem 102 directly (e.g., via one or more communication cables) or via one or more wired and / or wireless networks 130. In some embodiments, the processing subsystem 102 can be included in or integrated with the scanning device 110. In some embodiments, the system 100 includes software that uses specific user-configurable parameters to perform specific operations and commands the scanning device 110 to transmit the acquired resulting image data to the processing subsystem 102 or a storage subsystem (e.g., a local storage subsystem or a network storage device). In some embodiments, either the processing subsystem 102 or the scanning device 110 can be coupled to the network 130. In some embodiments, a storage device is coupled to the network 130 to store or retrieve image data, object information, and / or other organized data. The processing subsystem 102 can include a display 108 and one or more input devices (not shown) for receiving commands from a user or operator (e.g., a technician, histologist, or pathologist).

[0048] In some embodiments, the user interface is rendered by the processing subsystem 102 and provided to the display 108 to (i) facilitate the analysis, interpretation, and / or reporting of image data and / or object data, (ii) acquire data from the scanning device, and (iii) acquire image data, object information, or other clinical information such as a database available via the network. In some embodiments, the network 130 provides remote access to the processing subsystem 102 and / or the scanning device 110, such as via a client interface or client portal (not shown). Enable access. In this way, a remote user can access the processing subsystem 102 so that the image visualization and analysis software can be remotely executed on the processing subsystem 102. In some embodiments, the client interface or client portal may also enable the retrieval of stored reports after the analysis of the image data.

[0049] FIG. 2A is a block diagram of a system 100 according to one embodiment of the present disclosure. The system 100 can be implemented using any type of user-operable computing device, including a desktop computer, a laptop computer, a tablet computer, a handheld device (e.g., a smartphone, a media player), and the like. The system 100 can include several interconnected components such as a processing subsystem 102, a memory subsystem 104, a user input device 106, a display 108, and a network interface 112 that communicates via a bus 114, as discussed in more detail below. In some embodiments, the system 100 shown in FIG. 2A may be remotely accessed. For example, one or more remote users can access the system 100 via a network so that the image data stored in the memory subsystem 104 can be retrieved, interpreted, analyzed, and / or reported.

[0050] The processing subsystem 102 can include a single processor that can have one or more cores, or multiple processors each having one or more cores. In some embodiments, the processing subsystem 102 can include one or more general-purpose processors (e.g., CPUs), dedicated processors such as graphics processors (GPUs), digital signal processors, or any combination of these and other types of processors. In some embodiments, some or all of the processors within the processing subsystem can be implemented using customized circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored within the circuit itself. In some embodiments, the processing subsystem 102 can fetch and execute instructions stored in the storage subsystem 104, and the instructions can be executed by the processing subsystem 102 regardless of whether the user accesses the system locally or remotely, such as via the client portal 116. By way of example, the processing subsystem 102 can receive and process image data stored within a local or networked storage system and execute instructions to display the image data (e.g., display an entire slide scan image, or an enlarged portion of an entire slide scan image).

[0051] The memory subsystem 104 can include various memory units such as system memory, read-only memory (ROM), and persistent storage devices. The ROM can store static data and instructions required by the processing subsystem 102 of the system 100 and other modules. The persistent storage device can be a read and write memory device. This persistent storage device can be a non-volatile memory unit that stores instructions and data even when the power to the system 100 is turned off. In some embodiments, a mass storage device (such as a magnetic or optical disk or flash memory) can be used as the persistent storage device. Other embodiments can use a removable storage device (e.g., a flash drive) as the persistent storage device. The system memory can be a read and write memory device or a volatile read and write memory such as dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor at runtime.

[0052] The memory subsystem 104 can include any combination of non-transitory computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), etc. Magnetic disks and / or optical disks can also be used. In some embodiments, the memory subsystem 104 can include removable storage media that can be readable and / or writable. Examples of such media include compact discs (CDs), read-only digital versatile discs (DVD-ROM, dual-layer DVD-ROM, etc.), read-only and recordable Blu-ray (registered trademark) discs, ultra-high density optical discs, flash memory cards (SD cards, mini SD cards, micro SD cards, etc.). In some embodiments, the image data and / or target data can be stored at one or more remote locations, such as cloud storage, and can be synchronized with other components of the system 100. When the terms "memory" or "memories" are used herein, they may refer to one or more memories, such as multiple memories.

[0053] In some embodiments, the memory subsystem 104 can store one or more software programs that are executed by the processing subsystem 102, such as the image visualization and analysis application 120. "Software" generally refers to a series of instructions that, when executed by the processing subsystem 102, cause the system 100 to perform various operations, and thus define one or more specific machine implementations that execute and perform the operations of the software program. Thus, "software" can also include firmware or embedded applications, or any other type of instructions that are readable and executable by the processing subsystem 102. The software can be implemented as a single program or, if desired, as a collection of individual programs or program modules that interact with each other. In some embodiments, the program and / or data are stored in non-volatile memory and can be copied in whole or in part to volatile working memory during program execution. From the memory subsystem 104, the processing subsystem 102 can obtain the program instructions to execute and the data to process in order to perform various operations, including the operations described below.

[0054] In some embodiments, the software can be executed locally on the system 100 but can be accessed and / or controlled remotely, such as via the client portal 116. For example, an instance of the image visualization and analysis application 120 may be executed locally on the system 100, but a remote operator may access the image visualization and analysis application 120 via the network-connected client portal 116, and as a result, a remote user can control an instance of the image visualization and analysis application 120 to facilitate review, interpretation, and analysis of image data (e.g., a scanned image of a biological sample obtained from the memory subsystem 104 and presented to the remote user for analysis).

[0055] The user interface can be provided to the display 108 and / or one or more other user output devices (not shown). The user interface may include, for example, visualizations and other representations, and the representations may include scans of biological samples (e.g., samples stained for the presence of one or more biomarkers, or samples stained with hematoxylin and eosin), menu bars, drop-down menus, and / or images obtained from panels. The user interface provided to the display can be adapted such that only contextually relevant tools and / or viewer panels are provided to the user based on user selections, including but not limited to, for example, the zoom or magnification level selected by the user. The user input device 106 can include any device by which a user can provide a signal to the system 100, and the system 100 can interpret the signal as indicative of a specific user request or information. In some embodiments, the user input device 106 can include any or all of a keyboard, touchpad, touch screen (e.g., a touch-sensitive overlay on the display surface of the display 108), mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, etc. The signal can be interpreted as indicative of a specific user request or information.

[0056] The display 108 can display visualization representations (such as image data, a viewer panel for communicating information to the user, or a context menu providing user-selectable configuration options, etc.) generated by the system 100, and can include various image generation technologies, such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), light-emitting diodes (LEDs) including organic light-emitting diodes (OLEDs), projection systems, etc., and supporting electronic devices (from digital to analog, or analog-digital converters, signal processors, etc.). Some embodiments can include devices such as touchscreens that function as both input and output devices. In some embodiments, in addition to or instead of the display 108, other user output devices can be provided.

[0057] In some embodiments, the user interface can provide a graphical user interface in which visible image elements within a specific region of the display 108 are defined as active, interactive, or control elements that the user selects using the user input device 106. For example, the user can operate the user input device 106 to control a cursor or pointer on the screen to be placed on a control element and "click" a button to indicate a selection, and the selection sends a signal to execute a specified action or routine. For example, the user can operate the user input device 106 to select an icon within the user interface (e.g., an icon within a viewer panel, a menu bar, or a drop-down menu), which results in the start of an operation or the selection of a tool, such as the start of annotation of one or more displayed representations of an image of a tissue sample. As another example, the user can click a menu bar icon to initiate a tool selection so that the user can select a region of interest based on received input. In some embodiments, the user can operate the user input device 106 to interact with a drop-down menu and select one or more panels including an interactive panel. In some embodiments, these selections can be made by the user only if the pre-established conditions are met, e.g., whether a specific zoom level has been selected by the user, whether a specific tissue type has been selected, whether a specific slide with a specific biomarker has been selected, and the tool and / or viewer panel is enabled.

[0058] Alternatively, the user can touch (e.g., with a finger or a stylus) a control element on the touch screen device. In some embodiments, the user can speak one or more words associated with the control element (the words can be, for example, labels on the element or functions associated with the element). In some embodiments, user gestures on the touch-sensitive device can be recognized and interpreted as input commands. These gestures can be associated with specific regions on the display 108, but need not be. Other user interfaces can also be implemented.

[0059] The network interface 112 can provide data communication capabilities to the system 100. In some embodiments, the network interface 112 can include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., mobile phone technology, advanced data network technologies such as 3G, 4G or EDGE, 5G, WiFi (IEEE 802.11 fa (e.g., millimeter standard), or other mobile communication technologies, or any combination thereof, using a GPS receiver component and / or other components). In some embodiments, network interface 112 can provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface. Network interface 112 can be implemented using a combination of hardware (e.g., antennas, modulators / demodulators, encoders / decoders, and other analog and / or digital signal processing circuits) and software components. Network interface 112 can facilitate remote access to system 100, such as via client portal 116 (e.g., a remote user can access system 100 via a remote computer, and the remote computer can interact with system 100 via network interface 112). In some embodiments, client portal 116 is a stand-alone application executed by a remote user on a remote computer or other computing device. In some embodiments, client portal 116 is a web browser executed on a remote computer or other computing device that accesses system 100 via the network.

[0060] Bus 114 can include various systems, peripheral devices, and chipset buses that communicatively connect a number of components of system 100. For example, bus 114 can communicatively couple processing subsystem 102 to memory subsystem 104. Bus 114 can also be connected to input device 106 and display 108. Bus 114 can also couple processing subsystem 102 to a network via network interface 112. In this way, system 100 can be connected to a network of multiple computer systems (e.g., a network of networks such as a local area network (LAN), a wide area network (WAN), an intranet, or the Internet). One of ordinary skill in the art will understand that additional components such as a scanning device, an organization processing system, etc. can be connected to bus 114.

[0061] Some embodiments include electronic components such as a microprocessor, a storage device, and a memory that store computer program instructions on a computer-readable storage medium. Many of the functions described herein can be implemented as a process specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing unit to perform various operations indicated by the program instructions. Examples of program instructions or computer code include machine code such as that generated by a compiler, and files containing high-level code that is executed by a microprocessor using a computer, an electronic component, or an interpreter.

[0062] Through appropriate programming, the processing subsystem 102 can provide various functions to the system 100. For example, the processing subsystem 102 can execute an image visualization and analysis application 120 that has a user interface to facilitate the review and interpretation of scanned images of biological samples. The image visualization and analysis application 120 can provide various functions such as the function of selecting user-configurable options or user-selectable panels, or the function of controlling navigation to annotate images. In some embodiments, the analysis application 120 is based on whether pre-established conditions are met, for example, whether the user selects a zoom level that meets or exceeds a predetermined threshold, or whether a particular type of image staining a specific biomarker is selected for review, and includes logic such that only relevant user items (information items, user-selectable items, interactive items) are presented to the user. In some embodiments, additional components include the components specified in U.S. Patent Application Publication No. 2012 / 0320094, which can be incorporated into the systems and software of the present disclosure, the disclosure of which is hereby incorporated by reference in its entirety.

[0063] In some embodiments, when the image visualization and analysis application 120 is executed by one or more processors within the processing subsystem 102, it incorporates various interoperability modules (e.g., blocks of code) that implement aspects of the interface operation. For example, the image visualization and analysis application 120 can include a content fetcher 122, a content renderer 124, a GUI renderer 126, and a UI interpreter 128.

[0064] In some embodiments, content fetcher 122 can fetch or otherwise obtain content items such as image data and / or subject data by including instructions for interacting with (e.g., accessing) a local database (e.g., memory subsystem 104) or network interface 112. In some embodiments, content fetcher 122 is configured to access a plurality of scanned images, each of which is derived from a subject sample and each of which can be stained for the presence of one or more biomarkers or for hematoxylin and eosin. In some embodiments, content fetcher 122 is configured to obtain subject information, image metadata, case history information, and the like. In some embodiments, content fetcher 122 can include instructions for interacting with scanner device 110 such that the image data can be obtained from one or more slides having tissue samples stained for the presence of one or more biomarkers.

[0065] In some embodiments, the content renderer 124 can interpret content items fetched from one or more sources and then include instructions for delivering the rendered content to or for placement in an image placeholder or other representation generated by the GUI renderer 126. For example, the content renderer 124 can transfer the image data obtained from the content fetcher 122 into one or more rendered representations (see representation 401 in FIG. 4A). In some embodiments, the content renderer 124 can deliver the target information to other GUI elements such as one or more viewer panels or place the obtained target information as part of a GUI representation. In some embodiments, the content renderer 124 can deliver metadata such as the type of tissue, applied stain, scan parameters, z-stack layer, focus layer, etc. to other GUI elements. In some embodiments, the content renderer 124 can also process the obtained image data, for example, apply any pre-processing to the obtained image.

[0066] In some embodiments, the GUI renderer 126 creates graphical user interface (GUI) elements that are presented to the user along with content items rendered by the content renderer 124 or other system modules. The GUI renderer 126 can include code that defines the position and appearance of GUI elements such as menu bar items, viewer panels, configuration panels, etc., each of which can be an interactive element or can include interactive elements. In some embodiments, the GUI renderer 126, along with signals received from the UI interpreter 128, can determine whether to enable or make available certain menu bar items or viewer panels depending on whether pre-established conditions are met, e.g., whether a zoom threshold level or magnification level has been selected. For example, a menu bar item can be activated by the user, thereby enabling the user to select configuration options or panel views from a drop-down menu (see, e.g., FIGS. 5A and 5B). In some embodiments, the GUI renderer 126 can incorporate the acquired image data provided by the content fetcher 122 or the content renderer 124 into some or all of the GUI elements (e.g., an actual image of a scanned biological sample can be displayed within a representation within the user interface).

[0067] As an example, the GUI renderer 126 can generate a series of representations 401 that can be populated with image data obtained with information acquired by the content fetcher 122. Examples of the representations are shown in FIGS. 4A, 4B, and 4C. These representations can be interactive representations. For example, if the user clicks on any particular representation (e.g., representation 401 in FIG. 4A) (e.g., as interpreted by the UI interpreter 128), the GUI renderer 126 can update the corresponding display of a viewer panel such as a slide navigator viewer panel.

[0068] Similarly, the GUI renderer 126 can generate a series of viewer panels. In some embodiments, the generated viewer panels are interactive panels where the user can select specific configurable options. For example, a zoom panel may include a slider bar where the user can select specific pre-set zoom levels such as 1x, 2x, 10x, 40x, etc., or where the user can enter a specific zoom level value. In some embodiments, the viewer panels are configured to convey relevant information to the user. For example, a case log panel may provide a history of user-configurable selections made by the user during the analysis of image data. Further, the GUI renderer 126 may render a visual representation indicating items that are hidden or not available or not enabled for the user to interact with.

[0069] The UI interpreter 128 can receive user input via, for example, the user input device 106, interpret the input, and determine actions to be performed by the analysis application 120. For example, the UI interpreter 28 can determine GUI elements selected by the user (e.g., icons, or selectable items such as menus, context menus, drop-down lists, buttons, representations, etc.) and initiate corresponding actions (e.g., adding annotations, displaying additional content information, zooming to a selected zoom level, generating a report for export). For example, the UI interpreter 128 can detect whether the user has selected an annotation tool (see the annotation tool 405 in FIGS. 4A and 4B) and can send a signal to the GUI renderer 126 to display additional user-selectable items. In some embodiments, the annotation tool includes a manual region of interest (ROI) generation tool, an automatic ROI generation tool, a tool that enables drawing of shapes (e.g., arrows), a measurement tool, or a text input generation tool. Each of these tools can be individually disabled or hidden based on the context of the user interaction. In some embodiments, selectable menu items include those that execute specific image processing algorithms, such as, for example, a film detection algorithm, a cell detection and counting algorithm, a nucleus detection algorithm, a scoring algorithm, a heat map generation algorithm, a tissue masking algorithm, a tissue type identification algorithm, etc. (see, for example, PCT Publications WO2016 / 120442 and WO2015 / 113895, and U.S. Patent Application Publications 2017 / 0154420, 2017 / 0372117, 2017 / 0103521, 2017 / 0140246, 2015 / 0347702, 2017 / 0082627, 2014 / 0377753, 2017 / 0337695, 2017 / 0323148, and 2017 / 0243051, the disclosures of which are incorporated herein by reference in their entireties). The input received from the UI interpreter 128 can be used to determine whether pre-established conditions are met.

[0070] System 100 is illustrative, and it is to be understood that variations and modifications are possible. Furthermore, although System 100 is described with reference to specific blocks, it is to be understood that these blocks are defined for convenience of explanation and are not intended to imply a particular physical arrangement of components. Further, the blocks need not correspond to physically distinct components. The blocks can be configured to perform various operations, such as, for example, programming a processor or providing appropriate control circuitry, and various blocks can or may not be reconfigured depending on how the initial configuration is accessed. Embodiments of the present disclosure can be implemented in a variety of devices including electronic devices implemented using any combination of circuitry and software. Image visualization and analysis application 120 is also illustrative, and a particular implementation can include more or fewer modules than those described herein. Further, although a particular module can be described as performing a particular function, such description is not intended to mean a particular function performed by the module or a particular set of instructions included within such module.

[0071] Figure 2B shows a client interface 140 that communicates with network 130 and system 100 (such as the systems shown in FIGS. 1 and 2). The client interface 140 can be a stand-alone application (e.g., stand-alone image visualization and analysis software), or a web browser or other interface software that enables remote access to the image visualization and analysis application 120. For example, the client interface 140 enables a remote operator to log in to system 100 (such as the systems shown in FIGS. 1 and 2) and access stored image data (such as data stored in memory subsystem 104 or other network-connected storage devices), or image data uploaded to system 100 for processing. In some embodiments, the client interface 140 can include any of the software modules described herein. In this way, a remote user can remotely interact with elements of the system (e.g., configurable elements) (e.g., an organizational scientist or pathologist can select user-configurable parameters such as menu bar tools and / or viewer panels) so that the remote user can analyze and / or interpret the image data.

[0072] In some embodiments, the graphical user interface is adapted to display only certain features depending on the selected zoom level. For example, depending on the selected zoom level, certain tools within the menu bar may not be enabled (see FIG. 4A). Similarly, certain panels that are displayed near the visualization of the accessed image may not be available. In general, system 100 may be configured to "restrict" access to certain tools and panels that are not contextually relevant during image analysis when a particular zoom level is given. For example, at a particular zoom level, if there is not sufficient resolution between certain features within the image, it may not be possible to annotate a particular portion of the image. Thus, if the software determines that a tool is not relevant at the selected zoom level, that tool is not enabled, and as described above, this facilitates a quick review of the data displayed within the visualization and also provides an improved user experience and an experience free of potential confusion.

[0073] FIG. 3 shows a flowchart depicting a method for visualizing associated image data derived from a biological sample stained for the presence of one or more biomarkers. At step 300, at least one image of the biological sample is accessed. In some embodiments, the biological sample is stained for the presence of one or more biomarkers. In some embodiments, multiple images are obtained, such as multiple images from the same biological sample, each image including a stained one indicating the presence or absence of a particular biomarker.

[0074] Subsequently, the first visualization representation is rendered within the graphical user interface (step 310), and the first visualization representation includes rendering of at least one image at least at a first zoom level. In some embodiments, the first visualization representation 400 includes rendering of a plurality of accessed images (see, e.g., FIG. 4A). In some embodiments, the first visualization representation includes a series of representations 401 (e.g., image placeholders), and each representation 401 may include one of the accessed images 410A or 410B. In some embodiments, each representation 401 is of the same size and / or shape. In some embodiments, each representation 401 includes a first portion 402 that includes one of the accessed images 410 and a second portion 403 that includes an identifier. In some embodiments, the identifier includes identification of a stain or stained biomarker that appears in the image. By way of example, the second portion 403 of the representation 401 indicates that a particular image 410A within the first portion 402 is stained with hematoxylin and eosin.

[0075] In some embodiments, a first series of tools (e.g., annotation tool 405) within the menu bar 404 are displayed simultaneously with the first visualization representation 400 (step 320). As described above and as shown in FIG. 4A, some tools are not available for selection, i.e., they are “grayed out” (see, e.g., setup calibration tool 408), and other tools are available for selection and are displayed as white icons (see, e.g., rotation tool 407). As a further example, each of the five annotation tools 405 are grayed out and thus disabled. In some embodiments, certain tools may be completely hidden from the user at a particular zoom level, as further described herein. Thus, the first series of tools (i.e., tools that can be selected or are represented by white icons) represent a subset of all the tools that the user may use. Four zoom levels of the images shown in the representation 401 of FIG. 4A are provided, and as shown in FIG. 4A, these tools that are not available, i.e., the “grayed out” tools, are considered irrelevant at a given zoom level of 1×. Said another way, tools that are not enabled or are hidden from the user's selection are determined to be invalid at the selected zoom level. For example, it may be considered of no value for a histologist or pathologist to perform measurements or draw arrows on cells, as at a particular zoom level (e.g., 1×) sufficient cell features are not obtained and the cell features cannot be resolved well enough to accurately perform the measurements or correctly place an arrow indicating a particular targeted structure.

[0076] In some embodiments, the first set of panels may also be displayed simultaneously with the first visualization representation and the first set of tools. For example, the zoom panel 406 and the slide navigator panel 409 may be displayed simultaneously with the first visualization representation and the first set of tools. Similar to the first set of tools, only the panels relevant to the context at a given zoom level are displayed. For example, FIG. 5A shows panels that can be selected by a user in a context menu or a drop-down menu. Unavailable items cannot be selected and are "grayed out". Similarly, selectable panels are displayed in white and can be selected by the user. Further, a selected panel may be marked, for example, with a checkmark.

[0077] Following the simultaneous presentation of the first visualization representation (e.g., the three representations 401 of FIG. 4A) and the first set of tools (those not "grayed out" in the menu bar 404), and / or the first set of panels, the user may then interact with the visualization representation, such as by changing the zoom level. For example, increasing the zoom of one or more of the accessed images presented in the first visualization representation, enabling the user to view at least a portion of the accessed image in more detail, thereby providing at least a second visualization representation at a second zoom level 430. "Zoom in" or "zoom into an image" means that a portion of the image is enlarged, thereby increasing the visual resolution of that portion of the image. For example, it should be understood that the accessed image 410B is presented in FIG. 4A at a 1x zoom level, and at this zoom level, a particular tissue structure, e.g., 420A, is difficult to interpret. However, when the zoom level of the image 410B is increased as shown in FIG. 4B (e.g., increasing the zoom level from 1x to about 10x), the magnification and / or resolution of the tissue structure 420B is shown in more detail, for example, at a level where individual cells can be resolved.

[0078] Simultaneously with the display of the second visualization representation (step 330) at the second zoom level 430, a second set of tools within the menu bar 404 is displayed (step 340). Similar to the first set of tools, the second set of tools also represents a subset of all the tools available to the user. As an example, compared to the first set of available tools (see FIG. 4A), the second set of tools includes each of five annotation tools 405 (see FIG. 4B).

[0079] In the specific embodiment shown in FIG. 4B, the second set of tools includes the first set of tools, that is, the second set of tools includes all the tools available in the first set of tools. In some embodiments, the second set of tools does not include all of the tools provided in the first set of tools. In some embodiments, the second set of tools includes at least one different tool from those provided in the first set of tools.

[0080] In some embodiments, the second set of panels is displayed simultaneously with the second visualization representation and the second set of tools at the second zoom level 430. Similar to the second set of tools, only the panels relevant to the context at a given zoom level are displayed. In the embodiment shown in FIG. 4B, compared to the embodiment of FIG. 4A, the slide panel 425 is automatically displayed at a zoom level of approximately 10 times. Referring to FIG. 5B, items that are not available cannot be selected and are "grayed out". Similarly, selectable viewer panels are displayed in white and can be selected by the user. Further, panels that have already been selected are marked, for example, with a check mark.

[0081] Figure 4C shows the selection of a zoom level (e.g., an intermediate zoom level) that is between the zoom level shown in Figure 4A and the zoom level shown in Figure 4B. Four representations 401 were shown at a 1x zoom level in Figure 4A at the zoom level of Figure 4C, but only three representations are visualized, and only one of the three representations is fully visualized. In particular, the same tools appear to be available in Figure 4C as in Figure 4A. Similarly, the same panel appears in Figure 4C as in Figure 4A. Thus, Figure 4C indicates that the user has not reached the zoom threshold such that the image visualization and analysis application 420 enables other tools in the menu bar 404, or, even further, enables other viewer panels. In this regard, Figure 4C indicates that the items that are displayed and made available to the user are contextually dependent, here contextually dependent on the level of zoom selected by the user.

[0082] In some embodiments, each of the menu items and / or viewer panels may have a pre-programmed zoom threshold level that must be achieved before enabling each respective menu item and / or viewer panel that is available. For example, referring to Figure 4A, the annotation tool 405 may become available only when the user selects a zoom level that exceeds a predetermined zoom threshold level, e.g., 5x. In some embodiments, the individual tools within the menu bar 404 may each have different predetermined thresholds. For example, a first annotation tool may have a predetermined threshold of 2x, a second annotation tool may have a predetermined threshold of 6x, and a slide calibration tool may have a predetermined threshold of 10x. In some embodiments, the zoom threshold may depend on the tissue type or staining being observed.

[0083] In some embodiments, the threshold for enabling a menu bar item or viewer panel need not be tied to a predetermined zoom threshold, i.e., values such as 1x, 2x, 4x, 8x, 16x, 32x, etc. Rather, the threshold can be tied to whether the zoom level at which individual cells or individual nuclei can be resolved by the user is selected. Alternatively, the threshold can be associated with whether a particular number of cells are present in a pre-defined area (pixels x pixels) at a selected zoom level, e.g., 100 cells in a 500 pixel x 500 pixel area. In some embodiments, different elements can be visualized depending on the available display resolution. For example, the threshold can be pre-defined as "p" when the display resolution is "m x n", but may be pre-defined as "p * q" when the display resolution is (m * q x n * q), where q is a scaling factor to account for differences in the display resolution. In yet other embodiments, whether to display or not display a particular viewer panel can be tied to the available display resolution. For example, five viewer panels may become available or unavailable upon reaching a zoom level threshold (i.e., contextually relevant), but if the "physical property" of the available screen is such that it cannot be utilized due to low or restricted display resolution, the system may continue to keep a particular panel unavailable, and the system determines which of the available viewer panels are most relevant in view of the display resolution limitations and prioritizes those panels for display. Further, by way of example, viewer panels can be cycled as needed to accommodate limited display resolution.

[0084] As described herein, in some embodiments, the entire menu item may be hidden until a particular zoom level is selected by the user. For example, assume that the menu bar includes items A, B, C, D, E, F, G, and H. Further, assume a 1x zoom level where only menu bar items A, B, E, and H are shown within the graphical user interface as shown in FIG. 6A. According to the present disclosure, these menu bar items A, B, E, and H are the only tools relevant and useful to the user at the 1x zoom level. When the user zooms in on one of the images (see FIG. 6B), additional menu bar items, e.g., menu bar items (i.e., C and G), may be displayed if they are relevant to the selected zoom level (such as 3x). When the user zooms in further on one of the images (see FIG. 6C), here to 601B, additional menu bar items, i.e., D and F, may be displayed if they are relevant to the selected zoom level (e.g., 8x). FIG. 6C further shows that a second viewer panel, i.e., “Viewer Panel 2,” becomes displayed by the graphical user interface, but only when a particular zoom level (i.e., a zoom level greater than at least 3x, such as 8x) is reached.

[0085] In some embodiments, the input image is received by a visualization system and the visualization is provided at a default zoom level, e.g., a 1x zoom level. In some embodiments, the system receives user input, e.g., a selection of an updated zoom level, a selection of a particular image. In some embodiments, a comparison is made between the received user input and a threshold condition, e.g., comparing between received user inputs of zoom levels to determine if a zoom threshold level has been reached, or comparing between received user inputs of image selections to determine a biomarker identified within the image. In some embodiments, if the zoom threshold level is met, additional visualization elements (e.g., tools, panels) may be presented to the user. In some embodiments, if a different biomarker is selected in a second image compared to a first image, additional visualization elements may be presented to the user. In some embodiments, at least one GUI element is changed based on the user's selection.

[0086] Additional embodiments

[0087] In some embodiments, each visualization representation or element being displayed has a position within the coordinate system of a display provided within an interface application such as a browser. For example, an icon of a tool such as an image analysis tool has a position within the display coordinate system. For example, if the display has a resolution of 4,000 x 3,000 and each pixel is considered a point in the coordinate system, the icon of an annotation tool may have a position defined by the pixel regions [150,200] (top left corner), [160,200] (top right corner), [150,210] (bottom left corner), and [160,210] (bottom right corner). In some embodiments, a viewer panel, image data, and other representations may have positions within the coordinate system of the display. In some embodiments, a first viewer panel may have a first position region, and a second viewer element may have a second position region. In some embodiments, each viewer panel may have a variable position region depending on the type of information being displayed and the amount of information available. In some embodiments, the positions of various elements within the coordinate system of the display may be fixed or variable depending on the situation.

[0088] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, hardware, or in combinations of one or more of them that include the structures disclosed in this specification and their structural equivalents. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, a data processing apparatus. Any of the modules described in this specification can include logic executable by a processor. As used herein, "logic" refers to any information having the form of an instruction signal and / or data that can be applied to affect the operation of a processor. Software is an example of logic.

[0089] A computer storage medium can be, or can include, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Further, a computer storage medium is not a propagated signal, but can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be, or can include, one or more distinct physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described herein can be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0090] The term "programmed processor" encompasses any kind of device, apparatus, and machine for processing data, including, for example, programmable microprocessors, computers, system-on-chips, or combinations of the foregoing, or combinations thereof. The apparatus can include special purpose logic circuitry such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or code constituting one or more combinations thereof. The apparatus and execution environment can implement various different computing model infrastructures such as web services, distributed computing, grid computing infrastructures. can be realized.

[0091] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and can be deployed in any form. It can be included as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. The program may be part of a file that holds other programs or data (e.g., one or more scripts saved in a markup language document), a single file dedicated to the program in question, or multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0092] The processes and logical flows described herein can be executed by one or more programmable processors executing one or more computer programs to perform actions by operating input data to produce output. The processes and logical flows can also be executed by, or can implement, special-purpose logic circuits, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0093] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing the instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as, magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data, transfer data, or both. However, such devices are not necessarily required for a computer. Further, a computer may be incorporated in another device, by way of a few examples, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive). Devices suitable for storing the instructions and data of a computer program include, by way of example, all forms of non-volatile memory, media and memory devices including semiconductor memory devices such as, EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented or incorporated by special purpose logic circuits.

[0094] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as an LCD (Liquid Crystal Display), LED (Light Emitting Diode) display, or OLED (Organic Light Emitting Diode) display, a display for presenting information to the user, and a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. In some implementations, a touch screen can be used to present information and receive input from the user. Other types of devices can be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Also, the input from the user can be received in any form, including acoustic, voice, or tactile input. Further, the computer can interact with the user by sending and receiving documents between the devices the user uses. For example, in response to a request received from a web browser, a web page is sent to the web browser on the user's client device.

[0095] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, backend components as a data server, or includes middleware components such as an application server, or includes frontend components. An end component, for example, a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), networks between networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example, network 20 of FIG. 1 can include one or more local area networks.

[0096] A computing system can include any number of clients and servers. Clients and servers are typically remote from each other and typically interact via a communication network. The relationship between a client and a server arises from computer programs that are executed on each computer and have a client-server relationship. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., for the purpose of displaying the data and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., as a result of a user's operation) can be received at the server from the client device.

[0097] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned in this specification and / or listed in the application data sheet are hereby incorporated by reference in their entirety. Aspects of some embodiments can, if necessary, be modified to provide further embodiments using the concepts of various patents, applications, and publications.

[0098] Although the present disclosure has been described with reference to some exemplary embodiments, it is to be understood that many other modifications and embodiments can be devised by those skilled in the art that will fall within the spirit and scope of the principles of the present disclosure. More specifically, reasonable variations and modifications are possible in the components and / or arrangements of the combination configuration of the foregoing disclosure, drawings, and the subject matter of the appended claims without departing from the spirit of the disclosure. In addition to variations and modifications in components and / or arrangements, alternative uses will also be apparent to those skilled in the art.

Claims

1. Accessing a plurality of images of a biological sample stained for the presence of one or more biomarkers, wherein the biological sample comprises a plurality of cells; Displaying, on a display screen, a graphical user interface in a first state, the graphical user interface in the first state including the plurality of images at a first magnification level, a first set of menu items of a menu bar, and a first viewer panel; Receiving user input for changing a zoom level of an image among the plurality of images from the first magnification level to a second magnification level; In response to receiving the user input for changing the zoom level of the image among the plurality of images from the first magnification level to the second magnification level, displaying, on the display screen, a graphical user interface in a second state, the graphical user interface in the second state including the image among the plurality of images at the second magnification level, a second set of menu items of the menu bar, and the first viewer panel; A method comprising the above steps.

2. The method according to claim 1, wherein at least one menu item in the first set of menu items is included in the second set of menu items.

3. The method according to claim 1, wherein the number of menu items in the second set of menu items is greater than the number of menu items in the first set of menu items.

4. The method according to claim 1, wherein the second magnification level is greater than the first magnification level.

5. Receiving user input for changing the zoom level of the image among the plurality of images from the second magnification level to a third magnification level In response to receiving the user input for changing the zoom level of the image among the plurality of images from the second magnification level to the third magnification level, displaying, on the display screen, the graphical user interface in a third state, wherein the graphical user interface in the third state includes the image among the plurality of images at the third magnification level, a third set of menu items of the menu bar, and a second viewer panel, the step of The method according to claim 1, further comprising.

6. The method according to claim 5, wherein the number of menu items in the third set of menu items is greater than the number of menu items in the second set of menu items.

7. The method according to claim 5, wherein the third magnification level is greater than the second magnification level.

8. A system comprising a processor and a memory, wherein the memory stores instructions, when the instructions are executed by the processor, causing the processor to access a plurality of images of a biological sample stained for the presence of one or more biomarkers, the biological sample comprising a plurality of cells, the step of displaying, on a display screen, a graphical user interface in a first state, wherein the graphical user interface in the first state includes the plurality of images at a first magnification level, a first set of menu items of a menu bar, and a first viewer panel, the step of receiving a user input for changing the zoom level of an image among the plurality of images from the first magnification level to a second magnification level; In response to receiving the user input for changing the zoom level of the image among the plurality of images from the first magnification level to the second magnification level, displaying, on the display screen, the graphical user interface in a second state, wherein the graphical user interface in the second state includes the image among the plurality of images at the second magnification level, a second set of menu items of the menu bar, and the first viewer panel, the step of A system that causes an operation including.

9. The system according to claim 8, wherein at least one menu item in the first set of menu items is included in the menu items of the second set.

10. The system according to claim 8, wherein the number of menu items in the second set of menu items is greater than the number of menu items in the first set of menu items.

11. The system according to claim 8, wherein the second magnification level is greater than the first magnification level.

12. The operation is receiving a user input for changing the zoom level of the image among the plurality of images from the second magnification level to a third magnification level; displaying, on the display screen, the graphical user interface in a third state in response to receiving the user input for changing the zoom level of the image among the plurality of images from the second magnification level to the third magnification level, wherein the graphical user interface in the third state includes the image among the plurality of images at the third magnification level, a third set of menu items of the menu bar, and a second viewer panel; The system according to claim 8, further comprising.

13. The system according to claim 12, wherein the number of menu items in the third set of menu items is greater than the number of menu items in the second set of menu items.

14. The system according to claim 12, wherein the third magnification level is greater than the second magnification level.

15. A non-transitory computer-readable medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, the processor is caused to access a plurality of images of a biological sample stained for the presence of one or more biomarkers, the biological sample comprising a plurality of cells; displaying, on a display screen, a graphical user interface in a first state, the graphical user interface in the first state including the plurality of images at a first magnification level, a first set of menu items of a menu bar, and a first viewer panel; Receiving a user input for changing the zoom level of an image among the plurality of images from the first magnification level to the second magnification level; In response to receiving the user input for changing the zoom level of the image among the plurality of images from the first magnification level to the second magnification level, displaying, on the display screen, the graphical user interface in a second state, wherein the graphical user interface in the second state includes the image among the plurality of images at the second magnification level, a second set of menu items of the menu bar, and the first viewer panel; A non-transitory computer-readable medium for causing an operation including.

16. The non-transitory computer-readable medium according to claim 15, wherein at least one menu item in the first set of menu items is included in the second set of menu items.

17. The non-transitory computer-readable medium according to claim 15, wherein the number of menu items in the second set of menu items is greater than the number of menu items in the first set of menu items.

18. The non-transitory computer-readable medium according to claim 15, wherein the second magnification level is greater than the first magnification level.

19. The operation includes Receiving a user input for changing the zoom level of the image among the plurality of images from the second magnification level to a third magnification level; and In response to receiving the user input for changing the zoom level of the image among the plurality of images from the second magnification level to the third magnification level, displaying, on the display screen, the graphical user interface in a third state, wherein the graphical user interface in the third state includes the image among the plurality of images at the third magnification level, a third set of menu items of the menu bar, and a second viewer panel; The non-transitory computer-readable medium according to claim 15, further including.

20. The non-transitory computer-readable medium according to claim 19, wherein the number of menu items in the third set of menu items is greater than the number of menu items in the second set of menu items.

Images