Context Adaptive Digital Pathology Interface

Abstract

The present disclosure provides a method, system, and computer program product for an image visualization system (120) that includes a context-adaptive digital pathology interface. At least one image (300) of a biological sample stained for the presence of one or more biomarkers is obtained. The image is displayed on a display screen at a first zoom level (310), where a first subset of user-selectable elements is displayed simultaneously (320). As a result of user input, the image being displayed is displayed at a second zoom level (330), where a second subset of user-selectable elements is displayed simultaneously with the image (340). One or more elements within the second subset of user-selectable elements are disabled or hidden at the first zoom level, or one or more elements within the first subset of user-selectable elements are disabled or hidden at the second zoom level.

Description

[0001] Cross - reference to Related Patent Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 726,081, filed on August 31, 2018, the entire disclosure of which is incorporated herein by reference for all purposes. Background Art

[0003] Digital pathology refers to the management and interpretation of pathological information in a digital environment. Scanning devices are used to image slides of biological samples, which may be stained, to generate digital slides, such as whole slide images. Digital pathology software enables the storage of digital slides in a computer storage device, viewing them on a computer monitor, and analyzing them to obtain pathological information. Summary of the Invention

[0004] The present disclosure provides a graphical user interface that is adapted to provide visualization of images of biological samples stained for the presence of one or more biomarkers and is adapted to enable the display of context - relevant elements (e.g., analysis tools, viewer panels) to a user based on whether a preset criterion is met. In some embodiments, context - relevant elements are provided to the user based on a selected zoom level or selection of an image having a particular stain. It is believed that by providing only those context - relevant elements, the user is not presented with an overabundance of visualization elements (e.g., those that are inappropriate or those that are not valid in a given context), but rather can be based on a conditional state (e.g., a selected zoom level, a selected image type, a selected tissue type, a selection of a whole slide image relative to a slide from a tissue microarray, a selected slide having a particular stain applied (by way of example, H&E), a selected slide stained for the presence of a particular biomarker, a selected slide using immunohistochemistry rather than in situ hybridization staining, a selected slide to which certain image analysis algorithms have been applied). It is further believed that providing only context - relevant elements enables faster and more accurate examination of the presented image data. Additionally, given the limited size and resolution of a display screen, it is often difficult to provide multiple visualizations at once, or if multiple visualizations are provided, they may overlap or obstruct the user's view of important elements. By displaying only those context - relevant elements when a given specific condition (e.g., a particular zoom level selected by the user) is met, the most appropriate elements can be prioritized and visualized considering the limited display size and resolution.

[0005] In some embodiments, a computing device includes a display screen, and the computing device is configured to: access, from one or more memories communicatively coupled to the computing device, at least one image of a biological sample stained for the presence of one or more biomarkers (such as HER2, PD-L1, etc.); display a first visualization of the at least one image at a first zoom level on the display screen, and while displaying the first visualization, display a first subset of user-selectable elements on the display screen; and subsequently display a second visualization of the at least one image at a second zoom level on the display screen, where the second zoom level is greater than the first zoom level, and while displaying the second visualization, display a second subset of user-selectable elements on the display screen, (i) one or more elements within the second subset of user-selectable elements are disabled or hidden at the first zoom level, or (ii) one or more elements within the first subset of user-selectable elements are disabled or hidden at the second zoom level.

[0006] In some embodiments, the second zoom level at least meets a predetermined threshold zoom value. In some embodiments, the second subset of user-selectable elements is context-dependent at the second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. For example, the tools and menus are context-sensitive. When the user fully "zooms out", only the relevant tools to be used in that context are enabled (see Figure 4A ). When the user zooms in to a specified zoom level (see Figure 4B ), the tools in the toolbar are enabled and / or a context panel appears. The user can now use these tools in the "zoomed-in" context of the slide. Thus, the tools available in the first set of tools or in the second set of tools depend on the zoom level selected by the user, e.g., the second zoom level compared to the first zoom level, where the second zoom level reaches or exceeds a predetermined threshold zoom level for enabling the second set of tools.

[0007] In some embodiments, the menu bar tools include image annotation tools, slide setting tools, slide selection tools, navigation tools, and viewing tools. In some embodiments, the image annotation tools are enabled at the second zoom level. In some embodiments, the image annotation tools are hidden at the first zoom level. In some embodiments, the image annotation tools include region of interest recognition tools, measurement tools, marker drawing tools, and region exclusion tools.

[0008] In some embodiments, the computing device is further configured to display one or more viewer panels on the display screen at a second zoom level, wherein the one or more viewer panels for display at the second zoom level are disabled or hidden at the first zoom level. In some embodiments, the first zoom level is the 1x zoom level (i.e., the lowest zoom level, such as an image scanned or otherwise captured without magnification), and wherein the first visualization includes at least one representation, the at least one representation including at least one image of a biological sample. In some embodiments, the at least one representation further includes a portion that includes an identification marker.

[0009] In some embodiments, a computing device includes a display screen, and the computing device is configured to: access at least one image of a biological sample stained for the presence of one or more biomarkers from one or more memories communicatively coupled to the computing device; display at least a first representation including the at least one image on the display screen at a first zoom level, and while displaying the first representation, display at least a first viewer panel on the display screen; subsequently display a second representation of the at least one image on the display screen at a second zoom level, wherein the second zoom level is greater than the first zoom level, and while displaying the second representation, display at least one second viewer panel on the display screen other than the first viewer panel, wherein at least the second viewer panel is hidden and not displayed at the first zoom level.

[0010] In some embodiments, while displaying the second display panel, a third viewer panel is displayed on the display screen, wherein the third viewer panel is hidden and not displayed at the first zoom level. In some embodiments, while displaying the second display panel, a menu bar icon is enabled on the display screen. In some embodiments, the displayed menu bar icon is selected from a target area recognition tool, a measurement tool, a marker drawing tool, and an area exclusion tool. In some embodiments, the at least first representation includes: (i) a first portion that includes at least one image of a biological sample at the first zoom level, and (ii) a second portion that includes an identification marker. In some embodiments, the identification marker includes an identification of a biomarker.

[0011] In some embodiments, a method includes: displaying a first visualization including at least one image of a stained biological sample on a computing device having a display screen, wherein the first visualization is displayed in a first condition state, and while the first visualization is being displayed, displaying at least one of a first viewer panel or a first series of user-selectable elements on the display screen; and subsequently displaying a second visualization of the at least one image on the display screen in a second condition state, wherein the second condition state is derived as a result of a user selection, and while the second visualization is being displayed, displaying a second series of user-selectable elements that were not enabled in the first condition state or a second viewer panel other than the first viewer panel on the display screen, wherein the second viewer panel is hidden and not displayed at a first zoom level. In some embodiments, the first condition state is a default state. In some embodiments, the user selection is a zoom level, and wherein the second condition is a zoom level greater than the default zoom level. In some embodiments, the user selection is an image selection, wherein the image selected for the second condition includes a different stain than the image of the first condition state.

[0012] In some embodiments, a method includes: displaying a first visualization including at least one image of a stained biological sample on a computing device having a display screen, wherein the first visualization is displayed at a first zoom level, and while the first visualization is being displayed, displaying at least one of a first viewer panel or a first series of user-selectable elements on the display screen; and subsequently displaying a second visualization of the at least one image on the display screen at a second zoom level, wherein the second zoom level is greater than the first zoom level, and while the second visualization is being displayed, displaying at least one of a second series of user-selectable elements that were hidden or not enabled at the first zoom level or at least a second viewer panel other than the first viewer panel on the display screen, wherein the second viewer panel is hidden and not displayed at the first zoom level.

[0013] In some embodiments, the first zoom level is a default zoom level. In some embodiments, the second zoom level is a zoom level sufficient to resolve cell clusters or sufficient to resolve cell nuclei. In some embodiments, the first zoom level is the lowest available zoom level, and the second zoom level is at least 5x. In some embodiments, the first zoom level is the lowest available zoom level, and the second zoom level is at least 10x.

[0014] In some embodiments, a second subset of user-selectable elements is contextually relevant at a second zoom level. In some embodiments, user-selectable elements include menu bar tools and context menu items. In some embodiments, menu bar tools include an image annotation tool, a slide setting tool, a slide selection tool, a navigation tool, and a viewing tool. In some embodiments, the image annotation tool is enabled at the second zoom level. In some embodiments, the image annotation tool is hidden at the first zoom level. In some embodiments, the image annotation tool includes a region of interest recognition tool, a measurement tool, a marker drawing tool, and a region exclusion tool.

[0015] In some embodiments, a first visualization includes at least one representation, the at least one representation including: (i) a first portion that includes the at least one image of the biological sample at the first zoom level, and (ii) a second portion that includes identification markers. In some embodiments, the identification markers include the identification of biomarkers.

[0016] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors of a computing system, cause the computing system to display a first visualization of at least one image at a first zoom level on a display screen, and while displaying the first representation, display a first subset of user-selectable tools on the display screen; and subsequently display a second visualization of the at least one image at a second zoom level on the display screen, where the second zoom level is greater than the first zoom level, and while displaying the second representation, display a second subset of user-selectable elements on the display screen, where one or more tools within the second subset of user-selectable tools are not enabled at the first zoom level. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features of the present disclosure are understood generally with reference to the accompanying drawings. In the drawings, the same reference numerals are always used to identify the same elements.

[0018] The patent or application file contains at least one color drawing. Copies of the patent or patent application publication with color drawings will be provided to the Patent Office upon request and payment of the necessary fees.

[0019] Figure 1 FIG. shows a system according to some embodiments, the system including a computer having one or more processors and a scanning device, where the computer and the scanning device are communicatively coupled (e.g., via a network).

[0020] Figure 2AA system according to some embodiments is shown, the system including a processing subsystem, a storage subsystem, an output device, and an input device, each component communicatively coupled via a bus, network, or other wired or wireless interconnect. The system may also include software for enabling remote access, namely a client portal or client interface.

[0021] Figure 2B A block diagram of a system communicatively coupled to a client interface via a network according to some embodiments is presented.

[0022] Figure 3 A flowchart illustrating general steps for providing a display visualization according to some embodiments is presented.

[0023] Figure 4A A viewer window including a visualization according to some embodiments is shown, the visualization including multiple representations at a first zoom level.

[0024] Figure 4B A viewer window including a visualization according to some embodiments is shown, the visualization including a representation at a second zoom level, the second zoom level being greater than Figure 4A the first zoom level.

[0025] Figure 4C A viewer window including a visualization according to some embodiments is shown, the visualization including multiple representations at a third zoom level, the third zoom level being between Figure 4A the first zoom level and Figure 4B the second zoom level.

[0026] Figure 5A A drop-down menu for selecting certain viewer panels according to some embodiments is shown, such as those viewer panels available at a first zoom level (e.g., the zoom level in Figure 4A ).

[0027] Figure 5B A drop-down menu for selecting certain viewer panels according to some embodiments is shown, such as those viewer panels available at a second zoom level (e.g., the zoom level in Figure 4B ).

[0028] Figure 6A A viewer window including a first visualization according to some embodiments is shown, the first visualization including three representations at a first zoom level.

[0029] Figure 6B A viewer window including a first visualization according to some embodiments is shown, the first visualization including three representations at a second zoom level, the second zoom level being the same as Figure 6A andFigure 6C The zoom level depicted in

[0030] Figure 6C is a middle zoom level compared to Figure 6A and Figure 6B the zoom levels in DETAILED DESCRIPTION

[0031] It should also be understood that, unless there is an express contrary indication, in any method claimed herein that includes a plurality of steps or acts, the order of the steps or acts need not be limited to the order of the steps or acts recited in the method.

[0032] As used herein, unless otherwise specified, the singular terms "a / an" and "the" include plural referents. Similarly, unless the context clearly indicates otherwise, the word "or" is intended to include "and". The term "comprising" is defined as inclusive such that "comprising A or B" means including A, B, or A and B.

[0033] As used in the specification and claims herein, "or" shall be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as inclusive, that is, including at least one of a plurality of elements or a list of elements and optionally other unlisted items, but also including more than one. Only terms expressly indicating the contrary, such as "only one" or "exactly one", or when used in the claims, "consisting of" will refer to including only one of one or more elements. Generally, when preceded by exclusive terms such as "either", "one", "only one", "exactly one", the term "or" as used herein should only be interpreted as indicating an exclusive choice (e.g., "one or the other, but not both"). When used in the claims, "consisting essentially of" shall have the ordinary meaning used in the field of patent law.

[0034] The terms "comprising", "including", "having", etc. are used interchangeably and have the same meaning. Similarly, "comprising", "including", "having", etc. are used interchangeably and have the same meaning. Specifically, the definition of each term is consistent with the definition of "comprising" in ordinary United States patent law, and thus each term can be understood as an open-ended term meaning "at least the following", and can also be construed as not excluding additional features, limitations, aspects, etc. Thus, for example, "a device having components a, b, and c" means that the device includes at least components a, b, and c. Similarly, the phrase: "a method involving steps a, b, and c" means that the method includes at least steps a, b, and c. Additionally, although the steps and processes may be outlined herein in a particular order, those skilled in the art will recognize that the order of the steps and processes may vary.

[0035] As used herein in the specification and claims, when referring to a list of one or more elements, the phrase "at least one" shall be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for the optional presence of other elements, whether related or unrelated to those specifically identified elements, in addition to the elements specifically identified in the list of elements referred to by the phrase "at least one". Thus, as a non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently, "at least one of A and / or B") in one embodiment can mean at least one optionally including more than one A, but no B (and optionally including elements other than B); in some embodiments, it means at least one optionally including more than one B, but no A (and optionally including elements other than A); in some embodiments, it means at least one optionally including more than one A, and at least one optionally including more than one B (and optionally including other elements), etc.

[0036] As used herein, the term "image data" encompasses raw image data collected from a biological tissue sample, such as by an optical sensor or sensor array, or preprocessed image data. In particular, the image data may include a pixel matrix.

[0037] As used herein, the terms "image", "image scan", or "scanned image" encompass raw image data collected from a biological tissue sample, such as by an optical sensor or sensor array, or preprocessed image data. In particular, the image data may include a pixel matrix.

[0038] As used herein, the terms "biological sample", "tissue sample", "specimen", or similar terms refer to any sample obtained from any living organism, including viruses, that includes biomolecules (e.g., proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof). Examples of other living organisms include mammals (e.g., humans; veterinary animals such as cats, dogs, horses, cows, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (e.g., tissue sections and needle biopsies of tissue), cell samples (e.g., cytology smears such as cervical smears or blood smears or obtained by microdissection), or cell fractions, fragments, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or other means). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, cerumen, milk, vaginal secretions, saliva, swabs (e.g., oral swabs), or any material containing biomolecules and derived from a first biological sample. In some embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (e.g., a sample that has been homogenized or liquefied).

[0039] As used herein, the term "slide" refers to any substrate of any suitable size on which a biological specimen can be placed for analysis (e.g., a substrate made in whole or in part of glass, quartz, plastic, silicon, etc.), and more particularly refers to a "microscope slide" such as a standard 3 x 1 inch microscope slide or a standard 75 mm x 25 mm microscope slide. Examples of biological specimens that can be placed on a slide include, but are not limited to, cytology smears, thin tissue sections (e.g., from a biopsy), and biological specimen arrays such as tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in some embodiments, tissue sections, DNA samples, RNA samples, and / or proteins are placed at specific locations on a slide. In some embodiments, the term "slide" can refer to SELDI and MALDI chips, as well as silicon wafers.

[0040] As used herein, the terms "stain", "staining" or similar terms generally refer to any treatment of a biological specimen that detects and / or differentiates the presence, location, and / or amount (e.g., concentration) of a specific molecule (e.g., lipid, protein, or nucleic acid) or a specific structure (e.g., normal or malignant cells, cytoplasm, nucleus, Golgi apparatus, or cytoskeleton) in the biological specimen. For example, staining can compare a specific molecule or a specific cellular structure of a biological specimen with the surrounding parts, and the intensity of the staining can determine the amount of the specific molecule in the specimen. Staining can be used not only with bright-field microscopes but also with other observation tools such as phase-contrast microscopes, electron microscopes, and fluorescence microscopes to assist in observing molecules, cellular structures, and organisms. Some stains performed by the system can make the outline of the cells clearly visible. Other stains performed by the system can rely on specific cellular components (e.g., molecules or structures) that are stained and that do not stain or stain relatively few other cellular components. Examples of various staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on intermolecular reactions (including non-covalent binding interactions) such as hybridization reactions between nucleic acid molecules. Specific staining methods include, but are not limited to, primary staining methods (such as H&E staining, cervical staining, etc.), enzyme-linked immunohistochemical methods, and in situ RNA and DNA hybridization methods such as fluorescence in situ hybridization (FISH).

[0041] As used herein, the term "user interface" refers to an interface that allows a user (e.g., an end user such as a histologist and / or pathologist) to input commands and data and receive results, such as a graphical user interface (GUI). The terms "user interface" and "graphical user interface" may be used interchangeably herein.

[0042] As described in further detail herein, the present disclosure relates to a graphical user interface that allows a user to view and / or analyze one or more images of a biological sample stained for the presence of one or more biomarkers, thereby providing the user with visualization of certain elements and / or availability of certain analysis and / or processing tools in a context-based manner. In some embodiments, the graphical user interface is adapted to provide contextually relevant visualizations and elements for selection based on whether a preset condition is met. In some embodiments, the preset condition is a selection made by the user. For example, the selection can be a zoom level or magnification level selected by the user, where different selections of the zoom level or magnification level cause the graphical user interface to adaptively generate certain visualizations or enable certain user-configurable items, such as annotation tools, image processing tools, etc. In some embodiments, the graphical user interface adjusts the visualizations, analysis tools, and / or viewer panels provided to the user (e.g., for display on a monitor) based on the selected zoom level. For example, a first subset of contextually relevant tools can be presented to the user at a lowest zoom level (e.g., no optical magnification as compared to, e.g., 10x, 20x, or 40x optical magnification), while a more inclusive second subset of tools can be presented to the user at a greater zoom level of 10x, where additional tools included within the second subset are contextually relevant at the 10x zoom level. As another example, the selection can be a particular type of tissue (i.e., an image of the tissue) selected by the user or an image of a tissue sample stained for the presence of a particular biomarker.

[0043] It is believed that by presenting only those visualizations, analysis tools, viewer panels, etc. that are contextually relevant to a predetermined condition, such as a selected zoom level, a selected tissue type, a selected slide with a particular stain applied, a selected slide stained for the presence of a particular biomarker, a selection of a whole slide image rather than a tissue microarray, etc., the user may be able to interact with the software more effectively. In other words, since the user is not bogged down by the availability of numerous tools and / or viewer panels that are not relevant at a particular zoom level, the operator may be able to interact with the graphical user interface more effectively and ultimately may be able to examine and analyze the presented images of the tissue sample more quickly.

[0044] In some embodiments, the systems of the present disclosure are adapted to assist in the interpretation and reporting of image data obtained from a subject (e.g., a human patient). In some embodiments, the image data is obtained from a scanning device (e.g., a VENTANA DP 200 scanner, available from Ventana Medical Systems, Inc., Tucson, AZ), and the image data may be stored in a database, such as a networked database, for later visualization and analysis. For example, a scanning device may be used to obtain the image data, and the scanned image data is stored in a file located on storage subsystem 104 or a networked server so that the file can be retrieved later for visualization and analysis (see Figure 1 ). In some embodiments, software (e.g., an image visualization and analysis application) is run directly on the system, and a user interacting with the software retrieves the image data from a networked server for interpretation and reporting (see Figure 2A ). In some embodiments, software (e.g., an image visualization and analysis application) is run on a remote system, and the system is accessed using a client interface or a client portal so that the image data can be retrieved from the storage subsystem for visualization and analysis (see Figure 2B ).

[0045] The systems and methods provided herein can be applied to visualize and analyze any type of image of tissue stained for the presence of one or more biomarkers. For example, a biological sample can be stained by applying one or more stains, and the resulting image or image data includes signals corresponding to each of the one or more stains. In some embodiments, the input image is a simplex image having only a single stain (e.g., stained with 3,3'-diaminobenzidine (DAB)). In some embodiments, the biological sample can be stained in a multiplex assay with two or more stains (thereby providing a multiplex image). In some embodiments, the biological sample is stained for at least two biomarkers. In some embodiments, the biological sample is stained for the presence of at least two biomarkers and is also stained with a primary stain (e.g., hematoxylin). In some embodiments, the biological sample is stained for the presence of at least one protein biomarker and at least two nucleic acid biomarkers (e.g., DNA, RNA, microRNA, etc.).

[0046] In some embodiments, the biological sample is stained for the presence of one or more protein biomarkers in an immunohistochemistry assay. For example, the biological sample can be stained for the presence of the human epidermal growth factor receptor 2 protein (HER2 protein). Currently, there are two Food and Drug Administration (FDA)-approved methods for HER2 assessment in the United States. HerceptTestTM (DAKO, Glostrup, Demark) and HER2 / neu (4B5) rabbit monoclonal primary antibody (Ventana, Tucson, Arizona).

[0047] In some embodiments, the biological sample is stained for the presence of estrogen receptor (ER), progesterone receptor (PR), or Ki-67. In other embodiments, the biological sample is stained for the presence of EGFR or HER3. Zamay et al., “Current and Prospective Biomarkers of Long Cancer”, Cancers (Basel), November 2018; 9(11) describes examples of other protein biomarkers, the disclosure of which is incorporated herein by reference in its entirety. Examples of protein biomarkers described by Zamay include CEACAM, CYFRA21-1, PKLK, VEGF, BRAF, and SCC.

[0048] In some embodiments, the biological sample is stained in an in situ hybridization (ISH) assay for the presence of one or more nucleic acids, including mRNA. U.S. Patent No. 7,087,379 (the disclosure of which is incorporated herein by reference in its entirety) describes a method of staining a sample with an ISH probe to observe and detect individual spots (or dots) representative of individual gene copies. In some embodiments, multiple target genes are analyzed simultaneously by exposing the cell or tissue sample to a plurality of nucleic acid probes that have been labeled with a plurality of different nucleic acid tags.

[0049] Figure 1Describes a system 100 (computer or computing device) that includes a scanning device 110 communicatively coupled to a processing subsystem 102. The scanning device 110 can be coupled to the processing subsystem 102 directly (e.g., via one or more communication cables) or via one or more wired and / or wireless networks 130. In some embodiments, the processing subsystem 102 can be included in or integrated into the scanning device 110. In some embodiments, the system 100 can include software to command the scanning device 110 to perform certain operations using certain user-configurable parameters and send the resulting imaging data obtained to the processing subsystem 102 or a storage subsystem (e.g., a local storage subsystem or a networked storage device). In some embodiments, the processing subsystem 102 or the scanning device 110 can be coupled to the network 130. In some embodiments, a storage device is coupled to the network 130 for storing or retrieving image data, subject information, and / or other tissue data. The processing subsystem 102 can include a display 108 and one or more input devices (not shown) for receiving commands from a user or operator (e.g., a technician, a histologist, or a pathologist).

[0050] In some embodiments, a user interface is rendered by the processing subsystem 102 and provided on the display 108 to (i) facilitate the analysis, interpretation, and / or reporting of imaging data and / or subject data; (ii) retrieve data from the scanning device; or (iii) retrieve imaging data, subject information, or other clinical information from a database (e.g., one available via a network). In some embodiments, the network 130 enables remote access to the processing subsystem 102 and / or the scanning device 110 (e.g., via a client interface or a client portal, not shown). In this way, a remote user can access the processing subsystem 102 such that image visualization and analysis software can be run remotely on the processing subsystem 102. In some embodiments, the client interface or the client portal can also enable the retrieval of stored reports after analyzing the imaging data.

[0051] Figure 2A is a block diagram of a system 100 according to an embodiment of the present disclosure. The system 100 can be implemented using any type of user-operable computing device, including a desktop computer, a portable computer, a tablet computer, a handheld device (e.g., a smartphone, a media player), and the like. The system 100 can include a plurality of interconnected components, such as a processing subsystem 102, a storage subsystem 104, a user input device 106, a display 108, and a network interface 112 that communicate via a bus 114, as discussed in more detail below. In some embodiments, Figure 2AThe system 100 depicted herein is remotely accessible, for example, one or more remote users can access the system 100 such as via a network, enabling the inspection, interpretation, analysis, and / or reporting of image data stored within the storage subsystem 104.

[0052] The processing subsystem 102 can include a single processor (which can have one or more cores), or can include multiple processors (each having one or more cores). In some embodiments, the processing subsystem 102 can include one or more general-purpose processors (e.g., CPU), a dedicated processor (e.g., a graphics processing unit, GPU), a digital signal processor, or any combination of these and other types of processors. In some embodiments, some or all of the processors in the processing subsystem can be implemented using custom circuits such as, for example, application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself. In some embodiments, the processing subsystem 102 can retrieve and execute instructions stored in the storage subsystem 104, and the instructions can be executed by the processing subsystem 102 regardless of whether the user accesses the system locally or remotely, for example, via the client portal 116. By way of example, the processing subsystem 102 can execute instructions to receive and process image data stored in a local or network storage system and display the image data (e.g., display a whole-slide scan image or a magnified portion of any whole-slide scan image).

[0053] The storage subsystem 104 can include various memory units, such as system memory, read-only memory (ROM), and a permanent storage device. The ROM can store static data and instructions required by the processing subsystem 102 and other modules of the system 100. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage unit that stores instructions and data even when the system 100 is powered off. In some embodiments, a mass storage device (e.g., a magnetic disk or an optical disk or flash memory) can be used as the permanent storage device. Other embodiments can use a removable storage device (e.g., a flash drive) as the permanent storage device. The system memory can be a read-write storage device or a volatile read-write memory (e.g., dynamic random access memory). The system memory can store some or all of the instructions and data required by the processor during operation.

[0054] The storage subsystem 104 may include any combination of non-transitory computer-readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), etc. Disk and / or optical disks may also be used. In some embodiments, the storage subsystem 104 may include removable storage media that is readable and / or writable; examples of such media include optical discs (CDs), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), read-only and readable magnetic disks, high density optical discs, flash memory storage cards (e.g., SD cards, mini SD cards, micro SD cards, etc.), and the like. In some embodiments, the image data and / or subject data may be stored at one or more remote locations, such as in cloud storage, and synchronized with other components of the system 100. When the terms "memory" or "a memory" are used herein, they may refer to one or more memories, such as multiple memories.

[0055] In some embodiments, the storage subsystem 104 may store one or more software programs to be executed by the processing subsystem 102, such as the image visualization and analysis application 120. "Software" generally refers to a sequence of instructions that, when executed by the processing subsystem 102, causes the system 100 to perform various operations, thereby defining one or more specific machine-implemented manners of performing and implementing the operations of the software program. Thus, "software" may also include firmware or embedded applications or any other type of instructions readable and executable by the processing subsystem 102. The software may be implemented as a single program or as a collection of separate programs or program modules that interact as desired. In some embodiments, the program and / or data may be stored in non-volatile memory and copied, in whole or in part, to volatile working memory during program execution. The processing subsystem 102 may retrieve the program instructions to be executed and the data to be processed from the storage subsystem 104 in order to perform various operations including those described below.

[0056] In some embodiments, the software may run locally on the system 100 but be remotely accessed and / or controlled, for example, via the client portal 116. For example, an instance of the image visualization and analysis application 120 may run locally on the system 100, but a remote operator may access the image visualization and analysis application 120 via the network-connected client portal 116 such that the remote user can control the instance of the image visualization and analysis application 120 to assist in viewing, interpreting, and analyzing image data (e.g., a scanned image of a biological sample retrieved from the storage subsystem 104 and presented to the remote user for analysis).

[0057] A user interface can be provided to the display 108 and / or one or more other user output devices (not shown). The user interface can include, for example, visual effects and other representations that include images derived from the scanning of a stained biological sample (e.g., a sample stained for the presence of one or more biomarkers or stained with hematoxylin and eosin), a menu bar, drop-down menus, and / or panels. The user interface provided to the display can be adapted such that, based on, for example, user selections (including but not limited to the zoom or magnification level selected by the user), only contextually relevant tools and / or viewer panels are provided to the user. The user input device 106 can include any device through which a user can provide signals to the system 100; the system 100 can interpret the signals as indicating a specific user request or information. In some embodiments, the user input device 106 can include any one or all of a keyboard touchpad, a touchscreen (e.g., a touch-sensitive overlay on the display surface of the display 108), a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keyboard, a microphone, and the like.

[0058] The display 108 can display visualizations generated by the system 100 (such as representations of image data, viewer panels that convey information to the user, or context menus that provide user-selectable configuration options, etc.), and can include various image generation technologies, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a light-emitting diode (LED) (including an organic light-emitting diode (OLED)), a projection system, etc., as well as support electronics (such as a digital-to-analog converter or an analog-to-digital converter, a signal processor, etc.). Some embodiments can include devices that act as both input devices and output devices, such as a touchscreen. In some embodiments, other user output devices can be provided in addition to or instead of the display 108.

[0059] In some embodiments, the user interface may provide a graphical user interface, where visible image elements in certain regions of the display 108 are defined as active elements, interactive elements, or control elements for the user to select using the user input device 106. For example, the user may manipulate the user input device 106 to position the on-screen cursor or pointer over the control element and then "click" a button to indicate selection, where the selection sends a signal to perform a specified action or routine. For example, the user may manipulate the user input device 106 to select an icon within the user interface (such as an icon in a viewer panel, in a menu bar, or within a drop-down menu), which will initiate the activation of a tool operation or selection, such as initiating an annotation of one or more display representations of an image of an tissue sample. As another example, the user may click on a menu bar icon to initiate the selection of a tool, enabling the user to select a target region based on the received input. In some embodiments, the user may manipulate the user input device 106 to interact with a drop-down menu to select one or more panels, including interactive panels. In some embodiments, these selections can only be made by the user based on whether a preset condition is met, determining whether to enable a tool and / or a viewer panel, where the preset condition may be, for example, whether the user has selected a certain zoom level, whether a specific tissue type has been selected, whether a specific slide with a certain biomarker has been selected.

[0060] Alternatively, the user may touch a control element on a touchscreen device (e.g., with a finger or a stylus). In some embodiments, the user may speak one or more words associated with the control element (the word may be, for example, a label on the element or a function associated with the element). In some embodiments, user gestures on a touch-sensitive device may be recognized and interpreted as input commands; these gestures may or may not be associated with any specific region on the display 108. Other user interfaces may also be implemented.

[0061] Network interface 112 can provide data communication capabilities for system 100. In some embodiments, network interface 112 may include radio frequency (RF) transceiver components, GPS receiver components, and / or other components for accessing wireless voice and / or data networks (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G or EDGE, 5G, WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof). In some embodiments, in addition to or instead of a wireless interface, network interface 112 can provide wired network connectivity (e.g., Ethernet). Network interface 112 can be implemented using a combination of hardware (e.g., antennas, modulators / demodulators, encoders / decoders, and other analog and / or digital signal processing circuits) and software components. Network interface 112 can facilitate remote access to system 100, such as through client portal 116 (e.g., a remote user can access system 100 through a remote computer, and the remote computer interacts with system 100 through network interface 112). In some embodiments, client portal 116 is a stand-alone application run by a remote user on a remote computer or other computing device. In some embodiments, client portal 116 is a web browser running on a remote computer or other computing device that accesses system 100 through the network.

[0062] Bus 114 can include various system buses, peripheral buses, and chipset buses that communicatively connect a large number of components of system 100. For example, bus 114 can communicatively couple processing subsystem 102 to storage subsystem 104. Bus 114 can also be connected to user input device 106 and display 108. Bus 114 can also couple processing subsystem 102 to the network through network interface 112. In this way, system 100 can be connected to a network of multiple computer systems, such as a local area network (LAN), a wide area network (WAN), an intranet, or a network of networks (such as the Internet). Those skilled in the art will understand that other components (such as scanning devices, tissue processing systems) can be connected to bus 114.

[0063] Some embodiments include electronic components such as microprocessors, storage devices, and memories that store computer program instructions on a computer-readable storage medium. Many of the features described herein can be implemented as a process of a set of program instructions specified as being encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing units to perform the various operations indicated in the program instructions. Examples of program instructions or computer code include machine code (e.g., generated by a compiler) and files (including high-level code executed by a computer, an electronic component, or a microprocessor using an interpreter).

[0064] Through appropriate programming, the processing subsystem 102 can provide various functions for the system 100. For example, the processing subsystem 102 can execute an image visualization and analysis application 120 with a user interface that facilitates the inspection and interpretation of scanned images of biological samples. The image visualization and analysis application 120 can provide various functions, such as the ability to select user-configurable options or user-selectable panels, or the ability to control the navigation and annotation of images. In some embodiments, the analysis application 120 includes logic such that based on whether a preset condition is met or not, only relevant user items are presented to the user, such as only relevant user items (information items, user-selectable items, interaction items) are satisfied, where the preset condition is, for example, whether the user has selected a zoom level that reaches or exceeds a predefined threshold, or whether the user has selected an image stained for the presence of a particular biomarker for inspection. In some embodiments, additional components can be incorporated into the systems and software of the present disclosure, including those identified in U.S. Patent Application Publication No. 2012 / 0320094, the entire disclosure of which is incorporated herein by reference in its entirety.

[0065] In some embodiments, the image visualization and analysis application 120 incorporates various interoperable modules (e.g., code blocks) that, when executed by one or more processors within the processing subsystem 102, implement various aspects of the interface operations. For example, the image visualization and analysis application 120 can include a content fetcher 122, a content renderer 124, a GUI renderer 126, and a UI interpreter 128.

[0066] In some embodiments, the content fetcher 122 can include instructions for interacting (e.g., accessing) with a local database (e.g., the storage subsystem 104) or with the network interface 112 to obtain or otherwise retrieve content items, such as image data and / or subject data. In some embodiments, the content fetcher 122 is configured to access multiple scanned images, each of the scanned images originating from a subject sample, and wherein each of the scanned images can be stained for the presence of one or more biomarkers or with hematoxylin and eosin. In some embodiments, the content fetcher 122 is configured to retrieve subject information, image metadata, medical record information, etc. In some embodiments, the content fetcher 122 can include instructions for interacting with the scanning device 110 such that image data can be obtained from one or more slides having tissue samples stained for the presence of one or more biomarkers.

[0067] In some embodiments, the content renderer 124 may include instructions for interpreting content items obtained from one or more sources and then populating or delivering the rendered content to an image placeholder or other representation generated by the GUI renderer 126. For example, the content renderer 124 may populate one or more rendered representations with image data retrieved from the content fetcher 122 (see Figure 4A representation 401). In some embodiments, the content renderer 124 may pass subject information to other GUI elements, such as one or more viewer panels, or place the retrieved subject information within a portion of the GUI representation. In some embodiments, the content renderer 124 may pass metadata to other GUI elements, e.g., tissue type, stain applied, scan parameters, z-stack layer, focus layer, etc. In some embodiments, the content renderer 124 may also process the acquired image data, e.g., perform any preprocessing on the acquired images.

[0068] In some embodiments, the GUI renderer 126 creates graphical user interface (GUI) elements to be presented to the user together with content items rendered by the content renderer 124 or other system modules. The GUI renderer 126 may include code defining the location and appearance of GUI elements, such as menu bar items and viewer panels, each of which is itself an interactive element or may include interactive elements. In some embodiments, the GUI renderer 126, together with signals received from the UI interpreter 128, may determine whether certain menu bar items or viewer panels should be enabled or otherwise available to the user, e.g., depending on whether a preset condition is met, e.g., whether a threshold zoom level or magnification level has been selected. For example, a menu bar item may be activated by the user, which will then allow the user to select configuration options or panel views from a drop-down menu (see, e.g., Figure 5A and Figure 5B ). In some embodiments, the GUI renderer 126 may incorporate acquired image data provided by the content fetcher 122 or the content renderer 124 into some or all of the GUI elements (e.g., an actual image of a scanned biological sample may be displayed within a representation within the user interface).

[0069] As an example, the GUI renderer 126 may generate a series of representations 401, which may be populated with image data retrieved from the content fetcher 122. Examples of representations are shown in Figure 4A , Figure 4B and Figure 4C . These representations may be interactive representations. For example, if the user clicks on any particular representation (e.g., Figure 4AIf the representation 401) (e.g., interpreted by the UI interpreter 128), the GUI renderer 126 can update the corresponding display in the viewer panel (e.g., the slide navigator viewer panel).

[0070] Similarly, the GUI renderer 126 can generate a series of viewer panels. In some embodiments, the generated viewer panels are interactive panels where the user can select certain configurable options. For example, the zoom panel can include a slider where the user can select a specific preset zoom level, e.g., 1x, 2x, 10x, 40x, etc., or the user can enter a specific zoom level value therein. In some embodiments, the viewer panel is configured to convey relevant information to the user, e.g., the case log panel can provide a history of user-configurable selections made by the user during the analysis of the image data. Additionally, the GUI renderer 126 can render a visualization that shows items that are hidden or unavailable or non-interactable by the user.

[0071] The UI interpreter 128 can receive user input (e.g., via the user input device 106) and can interpret the input to determine the action to be performed by the analysis application 120. For example, the UI interpreter 28 can determine which GUI element (e.g., icon, or selectable item in a menu, context menu, drop-down list, button, representation, etc.) the user has selected and initiate the corresponding action (e.g., add a comment, display additional content information, zoom to a selected zoom level, generate a report for export). For example, the UI interpreter 128 can detect whether the user has selected an annotation tool (see Figure 4A and Figure 4Bannotation tool 405), and can send a signal to the GUI renderer 126 to display other user-selected items. In some embodiments, the annotation tool includes a manual region of interest (ROI) generation tool, an automatic ROI generation tool, a tool that allows drawing of shapes (such as arrows), a measurement tool, or a text input generation tool. Each of these tools can be disabled or hidden independently based on the context of the user interaction. In some embodiments, selectable menu items include running certain image processing algorithms, such as a membrane detection algorithm, a cell detection and counting algorithm, a nucleus detection algorithm, a scoring algorithm, a heat map generation algorithm, a tissue masking algorithm, a tissue type recognition algorithm, etc. (see, for example, PCT Publication Nos. WO2016 / 120442 and WO2015 / 113895 and U.S. Patent Application Publication Nos. 2017 / 0154420, 2017 / 0372117, 2017 / 0103521, 2017 / 0140246, 2015 / 0347702, 2017 / 0082627, 2014 / 0377753, 2017 / 0337695, 2017 / 0323148, and 2017 / 0243051, the disclosures of which are incorporated herein by reference in their entireties). Input received from the UI interpreter 128 can be used to determine whether a preset condition is met.

[0072] It should be understood that the system 100 is illustrative, and variations and modifications are possible. Additionally, although the system 100 is described with reference to specific blocks, it should be understood that these blocks are defined for ease of description and are not intended to imply a particular physical arrangement of components. Moreover, these blocks need not correspond to physically distinct components. The blocks can be configured (e.g., by programming a processor or providing appropriate control circuitry) to perform various operations, and depending on how the initial configuration is accessed, various blocks may or may not be reconfigurable. Embodiments of the present disclosure can be implemented in a variety of devices of an electronic apparatus that includes any combination of circuitry and software. The image visualization and analysis application 120 is also illustrative, and a particular implementation may include more or fewer modules than those described herein. Moreover, although a particular module may be described as performing a particular function, such a description is not intended to imply a particular function performed by the module or a particular instruction set included within such module.

[0073] Figure 2B depicts a client interface 140 communicating with the network 130 and the system 100 (such as Figure 1 and the system shown in FIG. 2). The client interface 140 can be a stand-alone application (such as stand-alone image visualization and analysis software) or a web browser or other interface software that allows remote access to the image visualization and analysis application 120. For example, the client interface 140 allows a remote operator to log in to the system 100 (such asFigure 1 and the system shown in FIG. 2) and access stored image data (such as data stored in storage subsystem 104 or other network-connected storage devices) or image data uploaded to system 100 for processing. In some embodiments, client interface 140 may include any of the software modules described herein. In this way, a remote user can remotely interact with elements of the system (e.g., configurable elements) such that image data can be analyzed and / or interpreted (e.g., a histologist or pathologist can select user-configurable parameters such as menu bar tools and / or viewer panels).

[0074] In some embodiments, the graphical user interface is adapted to display only certain features according to a selected zoom level. For example, according to a selected zoom level, certain tools within the menu bar may not be enabled (see Figure 4A ). Similarly, certain panels presented near the visualization of the accessed image may not be available. Generally, system 100 may be configured to "restrict" access to certain tools and panels that would not be contextually relevant during image analysis given a certain zoom level. For example, at certain zoom levels, annotating a particular portion of an image may not be feasible if there is not sufficient resolution between certain features within the image. Thus, if the software determines that a particular tool is not relevant at a selected zoom level, that tool will not be enabled, and as described above, this helps to more quickly examine the data presented within the visualization and also provides an enhanced user experience and no potential confusion.

[0075] Figure 3 A flowchart is presented that illustrates a method for visualizing associated image data that is derived from a biological sample stained for the presence of one or more biomarkers. At step 300, at least one image of the biological sample is accessed. In some embodiments, the biological sample is stained for the presence of one or more biomarkers. In some embodiments, multiple images are retrieved, such as multiple images from the same biological sample, but where each image includes staining indicative of the presence or absence of a particular biomarker.

[0076] Subsequently, a first visualization is rendered within the graphical user interface (step 310), where the first visualization includes at least a rendering of the at least one image at a first zoom level. In some embodiments, first visualization 400 includes a rendering of multiple accessed images (e.g., see Figure 4A)。In some embodiments, the first visualization effect includes a series of representations 401 (e.g., image placeholders), where each representation 401 may include one of the accessed images 410A or 410B. In some embodiments, each representation 401 has the same size and / or shape. In some embodiments, each representation 401 includes a first portion 402 and a second portion 403, the first portion including one of the accessed images 410, and the second portion including an identification marker. In some embodiments, the identification marker includes the identification of a stained or stainable biomarker present in the image. By way of example, the second portion 403 of the representation 401 indicates that a particular image 410A within the first portion 402 is stained with hematoxylin and eosin.

[0077] In some embodiments, a first series of tools (e.g., annotation tool 405) within the menu bar 404 is displayed simultaneously with the first visualization effect 400 (step 320). As described above, and as Figure 4A shown, some tools are not available for selection, i.e., they are "grayed out" (see, e.g., the set calibration tool 408), while other tools are available for selection and are displayed as white icons (see, e.g., the rotate tool 407). As a further example, each of the five annotation tools 405 is grayed out and thus disabled. In some embodiments, certain tools may be completely hidden from the user at a particular zoom level as further described herein. Thus, the first series of tools (i.e., those that are available for selection and / or are represented as white icons) represents a subset of all those tools that may be available to the user. Given Figure 4A the zoom level of the four images depicted in the representation 401, it is believed that those tools that are not available, i.e., those that are "grayed out", would be irrelevant at the given 1x zoom level as Figure 4A shown. In other words, those tools that are not enabled or are hidden from user selection are determined to be ineffective at the selected zoom level. For example, it is believed that for a histologist or pathologist, it would be of no value to make measurements or draw arrows to cells because at the given zoom level (e.g., 1x), there would not be sufficient cellular features resolvable to make accurate measurements, or to correctly place an arrow to point to a particular desired structure.

[0078] In some embodiments, a first set of panels may also be displayed simultaneously with the first visualization effect and the first series of tools. For example, the zoom panel 406 and the slide navigator panel 409 may be displayed simultaneously with the first visualization effect and with the first series of tools. Similar to the first series of tools, only those panels that are contextually relevant at the given zoom level are displayed. For example, Figure 5ADisplays panels that can be selected by the user in a context menu or a drop-down menu. Those items that are not available cannot be selected and are "grayed out". Similarly, those panels that can be selected are displayed in white and can be selected by the user. Moreover, those panels that have been selected can be marked, such as with a checkmark.

[0079] After presenting a first visualization (e.g., Figure 4A one of the three representations 401 in Figure 4A ) and a first set of tools (those that are not "grayed out" in the menu bar 404) and / or a first set of panels, the user can then interact with the visualization, such as changing the zoom level to increase the zoom of one or more of the accessed images presented in the first visualization, enabling the user to view at least a portion of one of the accessed images in more detail, thereby providing at least a second visualization at a second zoom level 430. "Increasing the zoom" or "zooming in on an image" means that a portion of the image is enlarged, thereby increasing the visual resolution of that portion of the image. For example, Figure 4B shows the accessed image 410B presented at a 1x zoom level in

[0080] , and it should be understood that at this zoom level, some organizational structures such as 420A are difficult to distinguish. However, as Figure 4A shown, when the zoom level of the image 410B is increased (e.g., from 1x to approximately 10x), the magnification and / or resolution of the organizational structure 420B is shown in more detail at a level where individual cells can be distinguished. Figure 4B

[0081] While displaying the second visualization (step 330) at the second zoom level 430, display a second set of tools in the menu bar 404 (step 340). Like the first set of tools, the second set of tools again presents a subset of all those tools that are available to the user. By way of example, compared to the available first set of tools (see Figure 4B ), the second set of tools includes each of the five annotation tools 405 (see

[0082] ). Figure 4B In the specific embodiment shown in Figure 4AIn contrast to the embodiments, the slide panel 425 is automatically displayed at a magnification level of approximately 10x. Go to Figure 5B In , those items that are not available are again not selectable and are "grayed out". Similarly, those viewer panels that can be selected are displayed in white and can be selected by the user. Also, those panels that have been selected are marked with a marker, such as a checkmark.

[0083] Figure 4C Shows the selection of a magnification level (e.g., the middle magnification level), which falls Figure 4A between the magnification levels depicted in Figure 4B and the magnification levels depicted in Figure 4A Although four representations 401 are shown at a magnification level of 1x in Figure 4C at the magnification level in , only three representations are visualized, and only one of the three representations is fully visualized. It is worth noting that, as in Figure 4A the same tools are shown as available in Figure 4C Similarly, as in Figure 4A the same panels appear in Figure 4C In this way, Figure 4C shows that the user has not reached the threshold magnification value, such that the image visualization and analysis application 420 will make other tools in the menu bar 404 available, or make other available viewer panels available for this purpose. In this regard, Figure 4C shows that the items shown and available to the user are context-dependent, and in this context, it depends on the magnification level selected by the user.

[0084] In some embodiments, each of the menu items and / or viewer panels has a pre-programmed threshold magnification level that must be reached before the corresponding menu item and / or viewer panel is enabled. For example, referring to Figure 4A the annotation tool 405 becomes available only when the user selects a magnification level that exceeds a predetermined threshold magnification level value (e.g., 5x). In some embodiments, each individual tool within the menu bar 404 can have a different predetermined threshold. For example, the first annotation tool can have a predetermined threshold of 2x, the second annotation tool can have a predetermined threshold of 6x, and the slide calibration tool can have a predetermined threshold of 10x. In some embodiments, the threshold magnification value can depend on the type of tissue or staining being observed.

[0085] In some embodiments, the threshold for determining the availability of menu bar items or viewer panels need not be related to a predetermined threshold scaling value (i.e., values 1x, 2x, 4x, 8x, 16x, 32x, etc.). Instead, the threshold can be related to whether a zoom level has been selected at which the user can distinguish individual cells or individual nuclei. Alternatively, the threshold can be related to whether a zoom level has been selected at which a certain number of cells are presented in a predetermined area (e.g., pixel x pixel), such as 100 cells in an area of 500 pixels x 500 pixels. In some embodiments, depending on the available display resolution, different elements can be visualized. For example, if the display resolution is "m x n", the threshold can be predefined as "p", and if the display resolution is (m*q x n*q), the threshold can be predefined as "p*q", where q is a scaling factor to account for differences in display resolution. In other embodiments, whether certain viewer panels are shown or hidden can be related to the available display resolution. For example, although at a threshold zoom level, 5 viewer panels can be made available or not hidden (e.g., they become contextually relevant and available), if the available screen "real estate" is unavailable due to low or limited display resolution, the system may continue to hide certain panels, and the system can determine which of the available viewer panels are most relevant given the resolution limitations and prioritize these panels for display. Additionally, as an example, viewer panels can be cycled as needed to accommodate limited display resolution.

[0086] As described herein, in some embodiments, an entire menu bar item can be hidden until the user selects a particular zoom level. For example, assume the menu bar contains items A, B, C, D, E, F, G, and H. Further assume the zoom level is 1x, then only menu bar items A, B, E, and H are shown within the graphical user interface, as Figure 6A shown. According to the present disclosure, those menu bar items A, B, E, and H are the only tools relevant and useful to the user at a zoom level of 1x. Once the user zooms in on one of the images (see Figure 6B ), additional menu bar items, such as menu bar items C and G, can be shown provided the menu bar items (i.e., C and G) are relevant to the selected zoom level (e.g., 3x). Once the user further zooms in on one of the images (see Figure 6C ), here 601B, additional menu bar items, namely D and F, can be shown, again assuming those additional menu bar items are relevant at the selected zoom level (e.g., 8x). Figure 6C Further shown is that a second viewer panel (i.e., "viewer panel 2") becomes displayed by the graphical user interface, but only when a particular zoom level is reached (i.e., at least a zoom level greater than 3x, such as at 8x).

[0087] In some embodiments, an input image is received by a visualization system and visualization is provided at a default zoom level (e.g., a zoom level of 1x). In some embodiments, the system receives user input, such as a selection of an updated zoom level, a selection of a particular image. In some embodiments, a comparison is made between the received user input and a threshold condition, such as comparing the received user input of the zoom level to determine if a threshold zoom level is reached, or comparing the received user input of the image selection to identify biomarkers in the image. In some embodiments, if the threshold zoom level is met, additional visualization elements (e.g., tools, panels) can be presented to the user. In some embodiments, if a different biomarker is selected in a second image compared to a first image, additional visualization elements can be presented to the user. In some embodiments, at least one GUI element is changed based on the user selection.

[0088] Additional Embodiments

[0089] In some embodiments, each displayed visualization or element has a position within the coordinate system of a display provided within an interface application (such as a browser). For example, an icon for a tool (such as an image analysis tool) has a position within the display coordinate system. For example, if the resolution of the display is 4,000 x 3,000 and each pixel is considered a point in the coordinate system, the icon for an annotation tool can have a position defined by the pixel regions [150,200] (upper left), [160,200] (upper right), [150,210] (lower left), and [160,210] (lower right). In some embodiments, viewer panels, image data, and other representations can have positions within the coordinate system of the display. In some embodiments, a first viewer panel can have a first position area, while a second viewer element can have a second position area. In some embodiments, each viewer panel can have a variable position area depending on the type of information being displayed and the amount of information available. In some embodiments, the positions of various elements within the coordinate system of the display can be fixed or can be variable depending on the context.

[0090] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by, or to control the operation of, a data processing apparatus. Any module described herein can include logic executed by a processor. As used herein, "logic" refers to any information having the form of an instruction signal and / or data that can be applied to affect the operation of a processor. Software is an example of logic.

[0091] A computer storage medium can be, or can include in whole or in part, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, although a computer storage medium is not a propagated signal, a computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be one or more separate physical components or media (such as multiple CDs, disks, or other storage devices), or can be included in them. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0092] The term "programmable processor" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable microprocessor, a computer, a system-on-a-chip, or multiple or combinations of the foregoing. The apparatus can include dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the apparatus can also include code that creates an execution environment for the computer programs being discussed, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and the execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0093] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages), and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0094] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logical flows can also be performed by, and apparatus can also be implemented as, special-purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0095] By way of example, processors suitable for the execution of a computer program include general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing the instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, or transfer data to, or both receive and transfer data to, one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks). However, a computer need not have such devices. In addition, a computer can be embedded in another device, by way of example only, such as a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, by way of example, including semiconductor storage devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0096] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, a keyboard, and a pointing device (such as a mouse or a trackball), the display device being, for example, an LCD (liquid crystal display), an LED (light emitting diode) display, or an OLED (organic light emitting diode) display for displaying information to the user, and the user can provide input to the computer via the keyboard and the pointing device. In some implementations, a touch screen can be used to display information and receive input from the user. Other kinds of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form (including sound, voice, or tactile input). Additionally, a computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user client device in response to a request received from the web browser.

[0097] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a backend component (such as a data server), or includes a middleware component (such as an application server), or includes a frontend component (such as a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), the Internet (such as the Internet), and peer-to-peer networks (such as ad hoc peer-to-peer networks). For example, Figure 1 Network 20 can include one or more local area networks.

[0098] A computing system can include any number of clients and servers. Clients and servers are typically physically separate from each other and typically interact via a communication network. The relationship between a client and a server is created by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, a server sends data (such as an HTML page) to a client device (for example, for the purpose of displaying the data to a user interacting with the client device and receiving user input therefrom). Data generated at the client device (such as the result of a user interaction) can be received at the server from the client device.

[0099] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned in this specification and / or listed in the Application Data Sheet are hereby incorporated by reference in their entirety. Where necessary, various aspects of some embodiments may be modified so as to adopt the concepts of various patents, applications, and publications to provide further other embodiments.

[0100] Although the present disclosure has been described with reference to a number of illustrative embodiments, it should be understood that many other modifications and embodiments can be designed by those skilled in the art, which will fall within the spirit and scope of the principles of the present disclosure. More particularly, reasonable variations and modifications in the components and / or arrangements of the subject combination arrangements are possible within the scope of the foregoing disclosure, the drawings, and the appended claims without departing from the spirit of the present disclosure. In addition to variations and modifications in the components and / or arrangements, alternative uses will also be apparent to those skilled in the art.

Claims

1. A computing device including a display screen, the computing device configured to: access at least one image of a biological sample stained for the presence of one or more biomarkers from one or more memories communicatively coupled to the computing device, and the biological sample is a cell sample; display a first visualization of the at least one image at a first zoom level on the display screen, and while displaying the first visualization, display on the display screen at least one of a first viewer panel or a first series of user-selectable elements; receive user input to change the zoom level from the first zoom level to a second zoom level, where the second zoom level is greater than the first zoom level and is a zoom level at which a certain number of cells within the biological sample are presented within a predetermined area; and in response to receiving the user input, display a second visualization of the at least one image at the second zoom level on the display screen, and while displaying the second visualization, display on the display screen: (i) a second series of user-selectable elements disabled at the first zoom level, or (ii) a second viewer panel in addition to the first viewer panel, where the second viewer panel is hidden and not displayed at the first zoom level, and where the display of the first viewer panel, the first series of user-selectable elements, the second viewer panel, and the second series of user-selectable elements is determined based on a threshold related to: (i) whether a zoom level at which a user can distinguish individual cells or individual nuclei is selected; or (ii) whether a zoom level at which a certain number of cells are presented within a predetermined area is selected.

2. The computing device according to claim 1, wherein the second series of user-selectable elements disabled or hidden at the first zoom level are contextually relevant at the second zoom level, but not contextually relevant at the first zoom level.

3. The computing device according to claim 1, wherein the second series of user-selectable elements includes one or more image annotation tools disabled or hidden at the first zoom level.

4. The computing device according to claim 3, wherein the image annotation tools include a region of interest recognition tool, a measurement tool, a marker drawing tool, and a region exclusion tool.

5. The computing device according to claim 1, wherein the second series of user-selectable elements includes one or more image processing tools disabled or hidden at the first zoom level.

6. The computing device according to claim 1, wherein the first zoom level is an unzoomed zoom level, and wherein the first visualization includes at least one representation including at least one image of the biological sample.

7. The computing device according to claim 6, wherein the at least one representation further includes a portion including identification markers.

8. A computing device including a display screen, the computing device configured to: Access, from a communication device connected to the computing device, at least one image of a biological sample stained for the presence of one or more biomarkers, and the biological sample is a cell sample; Display, on the display screen, at least a first visualization effect including the at least one image at a first zoom level, and while displaying the first visualization effect, display on the display screen at least one of a first viewer panel or a first series of user-selectable elements; Receive user input to change the zoom level from the first zoom level to a second zoom level, where the second zoom level is greater than the first zoom level and is a zoom level at which a certain number of cells within the biological sample are presented within a predetermined area; And In response to receiving the user input, display, on the display screen, a second visualization effect of the at least one image at the second zoom level, and while displaying the second visualization effect, display on the display screen: (i) a second series of user-selectable elements disabled at the first zoom level, or (ii) a second viewer panel other than the first viewer panel, where the second viewer panel is hidden and not displayed at the first zoom level, where the display of the first viewer panel, the first series of user-selectable elements, the second viewer panel, and the second series of user-selectable elements is determined based on a threshold related to: (i) whether a zoom level at which a user can distinguish individual cells or individual nuclei is selected; or (ii) whether a zoom level at which a certain number of cells are presented within a predetermined area is selected, and where while the second viewer panel is being displayed, a third viewer panel is displayed on the display screen, where the third viewer panel is hidden and not displayed at the first zoom level.

9. The computing device according to claim 8, where while the second viewer panel is being displayed, a previously displayed or hidden user-selectable tool is enabled on the display screen.

10. The computing device according to claim 9, where the user-selectable tool is an image annotation tool.

11. The computing device according to claim 8, where the at least first visualization effect includes: (i) a first part that includes the at least one image of the biological sample at the first zoom level, and (ii) a second part that includes identification marks.

12. The computing device according to claim 11, where the identification marks include the identification of biomarkers.

13. A method for visualizing and analyzing an image of tissue stained for the presence of one or more biomarkers, which includes: Display, on a computing device having a display screen, a first visualization effect including at least one image of a stained biological sample, where in the first visualization effect, the at least one image is displayed at a first zoom level, and the biological sample is a cell sample; While displaying the first visualization effect, display at least one of a first viewer panel or a first series of user-selectable elements on the display screen; Receive user input to change the zoom level from the first zoom level to a second zoom level, where the second zoom level is a zoom level at which a certain number of cells within the biological sample are presented within a predetermined area, and where the second zoom level is greater than the first zoom level; In response to receiving the user input, display a second visualization effect of the at least one image on the display screen, where in the second visualization effect, the at least one image is displayed at the second zoom level; And While displaying the second visualization effect, display on the display screen: (i) a second series of user-selectable elements that are disabled at the first zoom level; or (ii) a second viewer panel other than the first viewer panel, where the second viewer panel is hidden and not displayed at the first zoom level, where the display of the first viewer panel, the first series of user-selectable elements, the second viewer panel, and the second series of user-selectable elements is determined based on a threshold related to: (i) whether a zoom level at which a user can distinguish individual cells or individual nuclei has been selected; or (ii) whether a zoom level at which a certain number of cells are presented within a predetermined area has been selected.

14. The method according to claim 13, where the first zoom level is the default state.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to: Display a first visualization effect of at least one image including a stained biological sample at a first zoom level on a display screen, and while displaying the first visualization effect, display at least one of a first viewer panel or a first series of user-selectable elements on the display screen, and the biological sample is a cell sample; Receive user input to change the zoom level from the first zoom level to a second zoom level, where the second zoom level is greater than the first zoom level and is a zoom level at which a certain number of cells within the biological sample are presented within a predetermined area; and Subsequently display a second visualization effect of the at least one image at the second zoom level on the display screen, and while displaying the second visualization effect, display on the display screen: (i) a second series of user-selectable elements that are disabled at the first zoom level, or (ii) a second viewer panel other than the first viewer panel, where the second viewer panel is hidden and not displayed at the first zoom level, and wherein the display of the first viewer panel, the first series of user-selectable elements, the second viewer panel, and the second series of user-selectable elements is determined based on a threshold related to: (i) whether a zoom level has been selected at which the user can resolve individual cells or individual nuclei; or (ii) whether a zoom level has been selected at which a certain number of cells are presented within a predefined area.

Images