Context-adaptive digital pathology interface

Abstract

To provide an image visualization system, method, and computer program product that include a contextually adaptive digital pathology interface.SOLUTION: A method includes the following steps. At least one image of a biological sample stained for the presence of one or more biomarkers is obtained (300). The image is displayed on a display screen at a first zoom level (310), in which a first subset of user selectable elements is contemporaneously displayed (320). As a result of user input, the image being is displayed at a second zoom level (330), in which a second subset of user selectable elements are contemporaneously displayed with the image (340). The one or more elements within the second subset of user selectable elements are disabled or hidden at the first zoom level, or one or more elements within the first subset of user selectable elements are disabled or hidden at the second zoom level.SELECTED DRAWING: Figure 3

Description

Background Art

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 62 / 726,081, filed Aug. 31, 2018, which is hereby incorporated by reference in its entirety for all purposes. Digital pathology refers to the management and interpretation of pathological information in a digital environment. A scanning device is used to image slides of biological samples that can be stained, resulting in the generation of digital slides, e.g., an entire slide image. Using digital pathology software, digital slides can be stored in a computer's memory device, displayed on a computer monitor, and pathological information can be analyzed.

Summary of the Invention

[0002] This disclosure provides a visualized representation of images of biological samples stained for the presence of one or more biomarkers, and provides a graphical user interface adapted to enable the user to view contextually relevant elements (e.g., analysis tools, viewer panels) based on whether pre-established criteria are met. In some embodiments, contextually relevant elements are provided to the user based on a selected zoom level or a selection of images with a particular stain. By providing only contextually relevant elements, it is believed that the user can avoid being overwhelmed by visualized elements (e.g., inappropriate elements or elements that are ineffective in a particular context) and instead select the correct elements based on the conditional state (e.g., selected zoom level, selected image type, selected tissue type, selection of slides obtained from tissue-versus-microarrays of the entire slide image, selected slides to which a particular stain has been applied (e.g., H&E), selected slides stained for the presence of a particular biomarker, selected slides stained using immunohistochemistry as opposed to insitu hybridization, selected slides to which a particular image analysis algorithm has been applied). Furthermore, it is believed that providing only contextually relevant elements enables a faster and more accurate review of the presented image data. Furthermore, due to the limited size and resolution of display screens, it is often difficult to provide multiple visualizations simultaneously, and when multiple visualizations are provided, they may overlap with or obstruct the display of important elements for the user. By displaying only context-relevant elements under specific conditions (e.g., a specific zoom level selected by the user), it is possible to prioritize and visualize the most appropriate elements, taking into account the limited display size and resolution.

[0003] In some embodiments, the computing device includes a display screen, and the computing device accesses at least one image of a stained biological sample for the presence of one or more biomarkers (e.g., HER2, PD-L1, etc.) from one or more memories communicably coupled to the computing device, displays a first visualization representation of at least one image on the display screen at a first zoom level, and simultaneously displays a first subset of user-selectable tools on the display screen in response to receiving user input, and subsequently displays a second visualization representation of at least one image on the display screen at a second zoom level greater than the first zoom level, and simultaneously displays a second subset of user-selectable elements on the display screen in response to displaying the second visualization representation, wherein (i) one or more elements within the second subset of user-selectable elements are disabled or hidden at the first zoom level, or (ii) one or more elements within the first subset of user-selectable elements are disabled or hidden at the second zoom level. It is structured in such a way.

[0004] In some embodiments, the second zoom level satisfies at least a predetermined zoom threshold. In some embodiments, a second subset of user-selectable elements are contextually relevant at the second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. For example, tools and menus are context-dependent. When the user is fully “zoomed out,” only the relevant tools for use in that context are enabled (see Figure 4A). When the user zooms in to a specified zoom level (see Figure 4B), toolbar tools are enabled or a context panel is displayed. The user can use these tools when in the “zoomed in” context of the slide. Thus, the tools available in either the first set of tools or the second set of tools depend on the zoom level selected by the user, for example, the second zoom level compared to the first zoom level, where the second zoom level satisfies or exceeds a predetermined zoom threshold level for enabling the second set of tools.

[0005] In some embodiments, the menu bar tools include an image annotation tool, a slide setting tool, a slide selection tool, a navigation tool, and a display tool. In some embodiments, the image annotation tool is enabled at a second zoom level. In some embodiments, the image annotation tool is hidden at a first zoom level. In some embodiments, the image annotation tool includes a region of interest identification tool, a measurement tool, a marking tool, and a region exclusion tool.

[0006] In some embodiments, the computing device is further configured to display one or more viewer panels on a display screen at a second zoom level, and one or more viewer panels for display at the second zoom level are disabled at the first zoom level. In some embodiments, the first zoom level is a 1x zoom level (i.e., the lowest zoom level, such as an image scanned or otherwise captured without magnification), and the first visualization representation includes at least one representation containing at least one image of a biological sample. In some embodiments, the at least one representation further includes a portion containing an identifying mark.

[0007] In some embodiments, the computing device includes a display screen and is configured to access at least one image of a stained biological sample for the presence of one or more biomarkers from one or more memories communicably coupled to the computing device; to display a first representation of at least one image on the display screen at a first zoom level, and simultaneously displaying at least a first viewer panel on the display screen; and subsequently display a second representation of at least one image on the display screen at a second zoom level greater than the first zoom level, and simultaneously displaying at least a second viewer panel on the display screen in addition to the first viewer panel, wherein at least the second viewer panel is hidden at the first zoom level.

[0008] In some embodiments, the third viewer panel is displayed on the display screen simultaneously with the display of the second display panel, and the third viewer panel is hidden at the first zoom level. In some embodiments, the menu bar icons are enabled on the display screen simultaneously with the display of the second display panel. In some embodiments, the displayed menu bar icons are selected from a group consisting of region of interest identification tools, measurement tools, marking tools, and region exclusion tools. In some embodiments, at least Each first representation includes (i) a first portion containing at least one image of a biological sample at a first zoom level, and (ii) a second portion containing an identification mark. In some embodiments, the identification mark includes the identification of a biomarker.

[0009] In some embodiments, the method includes displaying a first visualization representation on a computing device having a display screen, the first visualization representation being displayed in a first condition state, and simultaneously displaying the first visualization representation and at least one of a first viewer panel or a first set of user-selectable elements on the display screen, and subsequently displaying a second visualization representation on the display screen of at least one image in a second condition state, the second condition state being obtained as a result of user selection, and simultaneously displaying the second visualization representation and at least one of a second set of user-selectable elements not enabled in the first condition state, or a second viewer panel in addition to the first viewer panel, the second viewer panel being hidden at a first zoom level. In some embodiments, the first condition state is a default state. In some embodiments, the user selection is a zoom level, and the second condition is a zoom level greater than the default zoom level. In some embodiments, the user selection is image selection, and the image selected for the second condition includes a different staining than the image for the first condition state.

[0010] In some embodiments, the method includes displaying a first visualization representation on a computing device having a display screen, the first visualization representation being displayed under first conditional conditions; displaying the first visualization representation while simultaneously displaying at least one of a first viewer panel or a first set of user-selectable elements on the display screen; subsequently displaying a second visualization representation of at least one image at a second zoom level, the second zoom level being greater than the first zoom level; and displaying the second visualization representation while simultaneously displaying at least one of a second set of user-selectable elements that are hidden or disabled at the first zoom level, or at least a second viewer panel in addition to the first viewer panel, the second viewer panel being hidden at the first zoom level.

[0011] In some embodiments, the first zoom level is the default zoom level. In some embodiments, the second zoom level is sufficient to adequately resolve cell clumps or cell nuclei. In some embodiments, the first zoom level is the lowest available zoom level and the second zoom level is at least 5x. In some embodiments, the first zoom level is the lowest available zoom level and the second zoom level is at least 10x.

[0012] In some embodiments, a second subset of user-selectable elements are contextually relevant at a second zoom level. In some embodiments, the user-selectable elements include menu bar tools and context menu items. In some embodiments, the menu bar tools include image annotation tools, slide setting tools, slide selection tools, navigation tools, and display tools. In some embodiments, the image annotation tools are enabled at a second zoom level. In some embodiments, the image annotation tools are hidden at a first zoom level. In some embodiments, the image annotation tools include region of interest identification tools, measurement tools, marking tools, and region exclusion tools.

[0013] In some embodiments, the first visualization representation is (i) a biological sample at a first zoom level. The system comprises (ii) a first portion including at least one image of a sample, and (ii) a second portion including an identification mark. In some embodiments, the identification mark includes the identification of a biomarker.

[0014] In some embodiments, a non-temporary computer-readable medium stores instructions, and when an instruction is executed by one or more processors of a computing system, the computing system causes the computing system to display on a display screen a first visualization representation of at least one image at a first zoom level, and simultaneously display a first subset of user-selectable tools on the display screen; and subsequently display on the display screen a second visualization representation of at least one image at a second zoom level greater than the first zoom level, and simultaneously display a second subset of user-selectable tools on the display screen, wherein one or more tools within the second subset of user-selectable tools are not enabled at the first zoom level. [Brief explanation of the drawing]

[0015] For a general understanding of the features of this disclosure, refer to the drawings. In the drawings, the same reference numbers are used throughout to identify identical elements.

[0016] Each patent or application file must include at least one drawing created in color. A copy of the publication of this patent or patent application, including the color drawing, will be provided to the Office upon request and payment of the required fees.

[0017] [Figure 1] The system, according to several embodiments, includes a computer having one or more processors and a scanning device, wherein the computer and the scanning device are connected in a way that allows them to communicate via a network or the like.

[0018] [Figure 2A] The system includes, in several embodiments, a processing subsystem, a storage subsystem, an output device, and an input device, each component being communicably coupled via a bus, network, or other wired or wireless interconnection. The system may also include software enabling remote access, i.e., a client portal or client interface.

[0019] [Figure 2B] The following are block diagrams of systems, in several embodiments, that are connected to a client interface for communication via a network.

[0020] [Figure 3] A flowchart is shown that provides general steps for displaying a visualization representation according to several embodiments.

[0021] [Figure 4A] The following describes a viewer window containing a visualization representation that includes multiple representations at a first zoom level, according to several embodiments.

[0022] [Figure 4B] Shows a viewer window including a visualization representation including the representation at the second zoom level according to some embodiments, where the second zoom level is larger than the first zoom level in FIG. 4A.

[0023] [Figure 4C] Shows a viewer window including a visualization representation including a plurality of representations at the third zoom level according to some embodiments, where the third zoom level is between the first zoom level and the second zoom level in FIGS. 4A and 4B, respectively.

[0024] [Figure 5A] Shows a drop-down menu for selecting a specific viewer panel, such as those available at the first zoom level (e.g., the zoom level in FIG. 4A).

[0025] [Figure 5B] Shows a drop-down menu for selecting a specific viewer panel, such as those available at the second zoom level (e.g., the zoom level in FIG. 4B).

[0026] [Figure 6A] Shows a viewer window including a first visualization representation including three representations at the first zoom level according to some embodiments.

[0027] [Figure 6B] Shows a viewer window including a first visualization representation including three representations at the second zoom level according to some embodiments, which is an intermediate zoom level compared to the zoom levels shown in FIGS. 6A and 6C.

[0028] [Figure 6C] Shows a viewer window including a first visualization representation including three representations at the third zoom level according to some embodiments, where the third zoom level is larger than the zoom levels in FIGS. 6A and 6B. [Modes for carrying out the invention]

[0029] Detailed explanation Conversely, unless explicitly stated otherwise, in any method claimed herein that includes multiple steps or actions, the order of the steps or actions of the method is not necessarily limited to the order in which they are described.

[0030] As used herein, the singular forms “a,” “an,” and “the” include multiple referents unless the context explicitly indicates otherwise. Similarly, the word “or” is intended to include “and” unless the context explicitly indicates otherwise. The term “includes” is defined inclusively, so as “includes A or B” means to include A, B, or A and B.

[0031] Where used herein and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as inclusive; that is, it shall be interpreted as including, but not limited to, at least one of the number or list of elements, and optionally, additional unlisted items. Only terms explicitly indicated as “exactly one” or, where used in the claims, “consisting of,” refer to including exactly one element of the number or list of elements. In general, where used herein, the term “or” shall be interpreted only as indicating an exclusive alternative when preceded by terms of exclusivity such as “either,” “either,” “only,” or “exactly either” (i.e., “one or the other, but not both”). Where used in the claims, “essentially consisting of” shall have the usual meaning as used in the field of patent law.

[0032] Terms such as “equipped,” “included,” and “possessed” are interchangeable and have the same meaning. Similarly, “equipped,” “included,” and “possessed” are used interchangeably and have the same meaning. Specifically, each term is defined in accordance with the general U.S. patent law definition of “included,” and is therefore interpreted as an open term meaning “at least the following,” and is not interpreted as excluding additional functions, limitations, aspects, etc. Therefore, for example, “an apparatus having components a, b, and c” means that the apparatus has at least component a, This means including steps a, b, and c. Similarly, the phrase “a method comprising steps a, b, and c” means that the method comprises at least steps a, b, and c. Furthermore, while steps and processes may be outlined herein in a specific order, those skilled in the art will recognize that the ordered steps and processes may vary.

[0033] As used herein and in the claims, the phrase “at least one” relating to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements. However, it is not necessary to include at least one of all elements specifically listed in the list of elements, nor is it excluded from combinations of elements in the list of elements. This definition also allows for the existence of elements other than those specifically identified in the list of elements referred to by the phrase “at least one,” whether or not they relate to the specifically identified elements, at the discretion of the definition. Therefore, as a non-restrictive example, “at least one of A and B” (or equivalently, “at least one of A or B,” or equivalently, “at least one of A and / or B”) may, in some embodiments, include at least one A, optionally two or more, and B may not be present (and optionally include elements other than B); in some embodiments, include at least one B, optionally two or more, and A may not be present (and optionally include elements other than A); and in yet another embodiment, include at least one A, optionally two or more, and B, optionally two or more (and optionally include other elements), and so on.

[0034] As used herein, the term “image data” includes raw image data or preprocessed image data obtained from a biological tissue sample by an optical sensor or sensor array, etc. In particular, image data may include a pixel matrix.

[0035] As used herein, the terms “image,” “image scan,” or “scanned image” include raw or pre-processed image data obtained from a biological tissue sample by an optical sensor or sensor array, etc. In particular, the image data may include a pixel matrix.

[0036] As used herein, terms such as “biological sample,” “tissue sample,” and “specimen” refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (veterinary animals such as humans, cats, dogs, horses, cattle, and pigs, as well as laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies), cell samples (such as cytological smears like Pap smears or blood smears, or cell samples obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components by centrifugation, etc.). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biomolecules derived from the initial biological sample. In certain embodiments, the term “biological sample” as used herein refers to a sample (such as a homogenized or liquefied sample) prepared from a tumor or a portion thereof obtained from the subject.

[0037] As used herein, the term “slide” refers to any substrate of any appropriate dimensions (e.g., glass, stone, whole or in part) on which a biological specimen is placed for analysis. The term "microscope slide" refers to a substrate made from materials such as plastic or silicon, more specifically, a standard 3-inch x 1-inch microscope slide or a standard 75mm x 25mm microscope slide. Examples of biological specimens that can be placed on a slide are, but are not limited to, cytological smears, thin tissue sections (such as those from biopsies), and sequences of biological specimens, e.g., tissue sequences, cell sequences, DNA sequences, RNA sequences, protein sequences, or any combination thereof. Thus, in some embodiments, tissue sections, DNA samples, RNA samples, and / or proteins are placed in specific positions on the slide. In some embodiments, the term slide may refer to SELDI and MALDI chips, as well as silicon wafers.

[0038] As used herein, terms such as “staining” and “stained” generally refer to any treatment of a biological specimen that detects and / or distinguishes the presence, location, and / or amount (such as concentration) of specific molecules (such as lipids, proteins, or nucleic acids) or specific structures (such as normal or malignant cells, cytosol, nucleus, Golgi apparatus, or cytoskeleton) within the biological specimen. For example, staining can provide contrast between a specific molecule or specific cellular structure and the surrounding part of the biological specimen, and the intensity of the stain can provide a measure of the amount of a particular molecule in the specimen. Staining can be used to assist in the observation of molecules, cellular structures, and organisms using bright-field microscopy as well as other observation tools such as phase-contrast microscopy, electron microscopy, and fluorescence microscopy. Some stains performed by the system can be used to visualize the contours of cells. Other stains performed by the system may depend on the specific cellular component (such as a molecule or structure) being stained, either without staining other cellular components or with relatively little staining. Examples of the types of staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on intermolecular reactions (including non-covalent interactions), such as hybridization reactions between nucleic acid molecules. Specific staining methods include, but are not limited to, primary staining methods (e.g., H&E staining, Pap staining), enzyme-linked immunohistochemical methods, and in-situ RNA and DNA hybridization methods such as fluorescence in-situ hybridization (FISH).

[0039] As used herein, the term “user interface” refers to an interface that enables a user, such as an end user (e.g., histologist and / or pathologist), to input commands and data and receive results, such as a graphical user interface (GUI). The terms “user interface” and “graphical user interface” are used interchangeably herein.

[0040] As will be described in further detail herein, this disclosure relates to a graphical user interface that enables a user to view and / or analyze one or more images of a biological sample stained for the presence of one or more biomarkers, thereby providing the user with contextually relevant visualizations of specific elements and / or availability of specific analysis and / or processing tools. In some embodiments, the graphical user interface is adapted to provide contextually relevant visualizations and elements for selection based on whether pre-established conditions are met. In some embodiments, the pre-established conditions are selections made by the user. For example, the selection may be a zoom level or magnification level selected by the user, and the selection of different zoom levels or magnification levels causes the graphical user interface to adaptively generate specific visualizations or enable specific user-configurable items, such as annotation tools, image processing tools, etc. In some embodiments, the graphical user interface adjusts the visualizations, analysis tools, and / or viewer panels provided to the user (for display on a display, for example) according to the selected zoom level, for example, a first subset of contextually relevant tools Some tools may be presented to the user at the lowest zoom level (e.g., no optical magnification compared to 10x, 20x, or 40x optical magnification), while a more comprehensive second subset of tools may be presented to the user at a larger zoom level of 10x, and any additional tools included in the second subset may again be contextually related to the 10x zoom level. As another example, the selection may be a particular type of tissue selected by the user (i.e., an image of tissue), or an image of a tissue sample stained for the presence of a particular biomarker.

[0041] For example, it is believed that users can interact with the software more efficiently by presenting only visualizations, analysis tools, and viewer panels that are contextually relevant to pre-established conditions, such as a selected zoom level, selected tissue type, selected slides with specific staining applied, selected slides stained for the presence of specific biomarkers, and the selection of entire slide images in contrast to tissue microarrays. In other words, because users are not overwhelmed by the availability of numerous tools and / or viewer panels unrelated to a specific zoom level, operators may be able to interact with the graphical user interface more efficiently, ultimately leading to faster review and analysis of images of the presented tissue samples.

[0042] In some embodiments, the systems of the present disclosure are configured to facilitate the interpretation and reporting of image data obtained from subjects (e.g., human patients). In some embodiments, image data is acquired from a scanning device (such as a VENTANA DP 200 scanner, available from Ventana Medical Systems, Inc., Tucson, AZ), and the image data may be stored in a database, such as a network database, for later visualization and analysis. For example, image data may be acquired using a scanning device, and the scanned image data is stored in a file located on the storage subsystem 104 or a networked server, thereby the file may be retrieved later for visualization and analysis (see Figure 1). In some embodiments, software, such as an image visualization and analysis application, runs directly on the system, and image data is retrieved from a networked server for interpretation and reporting by a user interacting with the software (see Figure 2A). In other embodiments, software, such as an image visualization and analysis application, runs on a remote system, and a client interface or client portal is used to access the system, thereby the image data may be retrieved from the storage subsystem for visualization and analysis (see Figure 2B).

[0043] The systems and methods provided herein can be applied to the visualization and analysis of any type of image of tissue stained for the presence of one or more biomarkers. For example, a biological sample can be stained by the application of one or more stains, and the resulting image or image data contains signals corresponding to each of the one or more stains. In some embodiments, the input image is a simple image having only a single stain (e.g., stained with 3,3'-diaminobenzidine (DAB)). In some embodiments, the biological sample can be stained with a multiplex assay for two or more stains (thus providing a multiplex image). In some embodiments, the biological sample is stained for at least two biomarkers. In some embodiments, the biological sample is stained for the presence of at least two biomarkers and also stained with a primary stain (e.g., hematoxylin). In some embodiments, the biological sample is stained for the presence of at least one protein biomarker and at least two nucleic acid biomarkers (e.g., DNA, RNA, microRNA, etc.).

[0044] In some embodiments, the biological sample contains one or more protein biomarkers. The presence of HER2 is detected by staining in immunohistochemical assays. For example, a biological sample can be stained for the presence of human epidermal growth factor receptor 2 protein (HER2 protein). Currently in the United States, there are two FDA-approved methods for HER2 evaluation: HerceptTest™ (DAKO, Grostrup, Denmark) and HER2 / neu(4B5) rabbit monoclonal primary antibody (Ventana, Tucson, Arizona).

[0045] In some embodiments, the biological sample is stained for the presence of estrogen receptor (ER), progesterone receptor (PR), or Ki-67. In yet other embodiments, the biological sample is stained for the presence of EGFR or HER3. Examples of other protein biomarkers are described by Zamay et al., “Current and Prospective Biomarkers of Long Cancer,” Cancers (Basel), November 2018; 9(11), the disclosure of which is incorporated herein by reference in its entirety. Examples of protein biomarkers described by Zamay include CEACAM, CYFRA21-1, PKLK, VEGF, BRAF, and SCC.

[0046] In some embodiments, a biological sample is stained with an in-situ hybridization (ISH) assay for the presence of one or more nucleic acids, including mRNA. U.S. Patent No. 7,087,379 (whose disclosure is incorporated herein by reference in its entirety) describes a method for staining a sample with an ISH probe so that individual spots (or dots) representing a single gene copy can be observed and detected. In some embodiments, several target genes are analyzed simultaneously by exposing a cell or tissue sample to multiple nucleic acid probes labeled with multiple different nucleic acid tags.

[0047] Figure 1 shows a system 100 (computer or computing device) including a scanning device 110 communicatively coupled to a processing subsystem 102. The scanning device 110 can be coupled to the processing subsystem 102 directly (e.g., via one or more communication cables) or via one or more wired and / or wireless networks 130. In some embodiments, the processing subsystem 102 may be included in or integrated with the scanning device 110. In some embodiments, the system 100 may include software that instructs the scanning device 110 to perform a specific operation using specific user-configurable parameters and to transmit the acquired resulting image data to the processing subsystem 102 or a storage subsystem (e.g., a local storage subsystem or a network storage device). In some embodiments, either the processing subsystem 102 or the scanning device 110 can be coupled to the network 130. In some embodiments, a storage device is coupled to the network 130 to store or retrieve image data, subject information, and / or other tissue data. The processing subsystem 102 may include a display 108 and one or more input devices (not shown) for receiving commands from a user or operator (e.g., a technician, histologist, or pathologist).

[0048] In some embodiments, the user interface is rendered by the processing subsystem 102 and provided to the display 108 to (i) facilitate the analysis, interpretation, and / or reporting of image data and / or target data, (ii) acquire data from the scanning device, and (iii) acquire image data, target information, or other clinical information such as databases available via the network. In some embodiments, the network 130 allows remote access to the processing subsystem 102 and / or scanning device 110, for example, via a client interface or client portal (not shown). This enables access. In this way, remote users can access the processing subsystem 102 so that image visualization and analysis software can be run remotely on the processing subsystem 102. In some embodiments, a client interface or client portal may also enable the retrieval of stored reports after the analysis of image data.

[0049] Figure 2A is a block diagram of System 100 according to one embodiment of the present disclosure. System 100 can be implemented using any type of user-operable computing device, including desktop computers, laptop computers, tablet computers, and handheld devices (e.g., smartphones, media players). System 100 may include several interconnected components, such as a processing subsystem 102, a storage subsystem 104, a user input device 106, a display 108, and a network interface 112 that communicates via a bus 114, as will be discussed in more detail below. In some embodiments, System 100 shown in Figure 2A may be remotely accessible, for example, one or more remote users may access System 100 via a network, so that image data stored in the storage subsystem 104 can be retrieved, interpreted, analyzed, and / or reported.

[0050] The processing subsystem 102 may include a single processor having one or more cores, or multiple processors, each having one or more cores. In some embodiments, the processing subsystem 102 may include one or more general-purpose processors (e.g., CPUs), dedicated processors such as graphics processors (GPUs), digital signal processors, or any combination of these and other types of processors. In some embodiments, some or all of the processors in the processing subsystem may be implemented using customized circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored in the circuit itself. In some embodiments, the processing subsystem 102 may retrieve and execute instructions stored in the storage subsystem 104, and the instructions may be executed by the processing subsystem 102 regardless of whether the user accesses the system locally or remotely, such as through the client portal 116. As an example, the processing subsystem 102 may receive and process image data stored in a local or networked storage system and execute instructions to display the image data (e.g., display the entire slide scan image, or an enlarged portion of the entire slide scan image).

[0051] The memory subsystem 104 may include various memory units such as system memory, read-only memory (ROM), and persistent memory. ROM can store static data and instructions required by processing the processing subsystem 102 and other modules of system 100. Persistent memory can be a read-and-write memory device. This persistent memory may be a non-volatile memory unit that stores instructions and data even when system 100 is powered off. In some embodiments, mass storage devices (such as magnetic or optical disks or flash memory) can be used as persistent memory. In other embodiments, removable storage devices (e.g., flash drives) can be used as persistent memory. System memory can be a read-and-write memory device or volatile read-and-write memory such as dynamic random-access memory. System memory can store some or all of the instructions and data required by the processor at runtime.

[0052] The storage subsystem 104 may include any combination of non-temporary computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory, etc.). Magnetic disks and / or optical disks may also be used. In some embodiments, the storage subsystem 104 may include removable storage media that may be readable and / or writable. Examples of such media include compact discs (CDs), read-only digital multipurpose discs (DVD-ROMs, dual-layer DVD-ROMs, etc.), read-only and recordable Blu-ray® discs, ultra-high-density optical discs, flash memory cards (SD cards, miniSD cards, microSD cards, etc.), etc. In some embodiments, image data and / or target data may be stored in one or more remote locations, e.g., cloud storage, and may be synchronized with other components of system 100. Where the terms “memory” or “storage” are used herein, they may refer to one or more memories, such as multiple memories.

[0053] In some embodiments, the storage subsystem 104 can store one or more software programs executed by the processing subsystem 102, such as an image visualization and analysis application 120. “Software” generally refers to a set of instructions that, when executed by the processing subsystem 102, cause the system 100 to perform various operations, and thus define one or more specific machine implementations that perform and execute the operations of the software program. Therefore, “software” may also include firmware or embedded applications, or any other type of instructions that are readable and executable by the processing subsystem 102. The software may be implemented as a single program, or as a collection of individual programs or program modules that interact as needed. In some embodiments, the program and / or data may be stored in non-volatile storage and copied in whole or in part to volatile working memory during program execution. From the storage subsystem 104, the processing subsystem 102 can retrieve program instructions to execute and data to process in order to perform various operations, including those described below.

[0054] In some embodiments, the software may run locally on system 100 but may be remotely accessible and / or controlled, such as through a client portal 116. For example, an instance of the image visualization and analysis application 120 may run locally on system 100, but a remote operator may access the image visualization and analysis application 120 via a networked client portal 116, and as a result, the remote user may control an instance of the image visualization and analysis application 120 to facilitate the review, interpretation, and analysis of image data (e.g., scanned images of a biological sample acquired from the storage subsystem 104 and presented to the remote user for analysis).

[0055] The user interface may be provided on the display 108 and / or one or more other user output devices (not shown). The user interface may include, for example, visualizations and other representations, the representations of which may include scans of biological samples (e.g., samples stained for the presence of one or more biomarkers, or stained with hematoxylin and eosin), menu bars, drop-down menus, and / or images obtained from panels. The user interface provided on the display may be adapted so that only contextually relevant tools and / or viewer panels are provided to the user based on user selections, including, but not limited to, a zoom or magnification level selected by the user. The user input device 106 may include any device on which the user can provide signals to the system 100, and the system 100 may provide signals The numbers can be interpreted as indicating a specific user request or information. In some embodiments, the user input device 106 may include any or all of the following: a keyboard touchpad, a touchscreen (e.g., a touch-sensitive overlay on the display surface of the display 108), a mouse or other pointing device, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, a microphone, and so on.

[0056] The display 108 can display a visualization generated by the system 100 (e.g., image data, a viewer panel for conveying information to the user, or a context menu providing user-selectable configuration options), and may include various image generation technologies, such as cathodes, photoreceptor tubes (CRTs), liquid crystal displays (LCDs), light-emitting diodes (LEDs) including organic light-emitting diodes (OLEDs), projection systems, and supporting electronic equipment (digital-to-analog or analog-to-digital converters, signal processors, etc.). Some embodiments may include devices such as touchscreens that function as both input and output devices. In some embodiments, other user output devices may be provided in addition to or instead of the display 108.

[0057] In some embodiments, the user interface can provide a graphical user interface in which visible image elements within a specific area of ​​the display 108 are defined as active elements, interactive elements, or control elements that the user selects using a user input device 106. For example, the user can operate the user input device 106 to position a cursor or pointer on the screen over a control element, and to indicate a selection by "clicking" a button, which sends a signal to perform a specified action or routine. For example, the user can operate the user input device 106 to select an icon within the user interface (e.g., an icon in a viewer panel, menu bar, or drop-down menu), which results in the initiation of an action or selection of a tool, such as the initiation of annotation of one or more displayed representations of an image of a tissue sample. As another example, the user may click a menu bar icon to initiate a tool selection so that the user can select an area of ​​interest based on the input received. In some embodiments, the user can operate the user input device 106 to interact with a drop-down menu and select one or more panels, including an interactive panel. In some embodiments, these selections can only be made by the user if the tool and / or viewer panel is enabled, based on whether pre-established conditions are met, for example, whether a particular zoom level has been selected by the user, whether a particular tissue type has been selected, or whether a particular slide having a particular biomarker has been selected.

[0058] Alternatively, the user can touch control elements on the touchscreen device (e.g., with a finger or stylus). In some embodiments, the user can speak one or more words related to the control elements (the words may be, for example, labels on the elements or functions related to the elements). In some embodiments, user gestures on the touch-sensitive device can be recognized and interpreted as input commands. These gestures can, but are not required to be, associated with specific areas on the display 108. Other user interfaces can also be implemented.

[0059] The network interface 112 can provide data communication functionality to the system 100. In some embodiments, the network interface 112 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., cellular technology, advanced data network technologies such as 3G, 4G or EDGE, 5G, WiFi (IEEE 802.11)). (Using millimeter-standard, or other mobile communication technologies, or any combination thereof, GPS receiver components, and / or other components). In some embodiments, the network interface 112 may provide a wired network connection (e.g., Ethernet) in addition to, or instead of, the wireless interface. The network interface 112 may be implemented using a combination of hardware (e.g., antennas, modulators / demodulators, encoders / decoders, and other analog and / or digital signal processing circuits) and software components. The network interface 112 may facilitate remote access to the system 100, such as through a client portal 116 (e.g., a remote user may access the system 100 via a remote computer, and the remote computer may interact with the system 100 via the network interface 112). In some embodiments, the client portal 116 is a standalone application run by a remote user on a remote computer or other computing device. In some embodiments, the client portal 116 is a web browser run on a remote computer or other computing device that accesses the system 100 over a network.

[0060] Bus 114 may include various systems, peripherals, and chipset buses that communicate with numerous components of system 100. For example, bus 114 can communicate with the processing subsystem 102 and the storage subsystem 104. Bus 114 can also connect to the input device 106 and the display 108. Bus 114 can also connect the processing subsystem 102 to a network via the network interface 112. In this way, system 100 can be connected to a network of multiple computer systems (e.g., a network of networks such as a local area network (LAN), wide area network (WAN), intranet, or the internet). Those skilled in the art will understand that additional components such as scanning devices and tissue processing systems can be connected to bus 114.

[0061] Some embodiments include electronic components such as microprocessors, storage devices, and memory that store computer program instructions in a computer-readable storage medium. Many of the functions described herein can be implemented as processes designated as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing units to perform the various operations indicated by the program instructions. Examples of program instructions or computer code include machine code, such as that generated by a compiler, and files containing high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0062] Through appropriate programming, the processing subsystem 102 can provide various functions to the system 100. For example, the processing subsystem 102 can run an image visualization and analysis application 120 having a user interface that facilitates the review and interpretation of scanned images of biological samples. The image visualization and analysis application 120 can provide various functions such as the ability to select user-configurable options or user-selectable panels, or the ability to control navigation and annotate images. In some embodiments, the analysis application 120 includes logic such that only relevant user items (information items, user-selectable items, interactive items) are presented to the user based on whether pre-established conditions are met, for example, whether the user selects a zoom level that meets or exceeds a predetermined threshold, or whether a particular type of image staining the presence of a specific biomarker is selected for review. In some embodiments, additional components are described in U.S. Patent Application Publication No. This includes components identified in No. 2012 / 0320094, which may be incorporated into the Systems and Software of this Disclosure, and which is incorporated herein in its entirety by reference.

[0063] In some embodiments, the image visualization and analysis application 120, when executed by one or more processors in the processing subsystem 102, incorporates various interoperability modules (e.g., blocks of code) that implement a mode of interface operation. For example, the image visualization and analysis application 120 may include a content fetcher 122, a content renderer 124, a GUI renderer 126, and a UI interpreter 128.

[0064] In some embodiments, the content fetcher 122 may include instructions for interacting with (e.g., accessing) a local database (e.g., a storage subsystem 104) or a network interface 112 to fetch or otherwise acquire content items such as image data and / or subject data. In some embodiments, the content fetcher 122 is configured to access multiple scan images, each of which originates from a subject sample, and each of which may be stained for the presence of one or more biomarkers, or for hematoxylin and eosin. In some embodiments, the content fetcher 122 is configured to acquire subject information, image metadata, case history information, etc. In some embodiments, the content fetcher 122 may include instructions for interacting with the scanning device 110 so that image data may be acquired from one or more slides having tissue samples stained for the presence of one or more biomarkers.

[0065] In some embodiments, the content renderer 124 may interpret content items fetched from one or more sources and then include instructions for distributing the rendered content to or to image placeholders or other representations generated by the GUI renderer 126. For example, the content renderer 124 may populate one or more rendered representations with image data acquired from the content fetcher 122 (see representation 401 in Figure 4A). In some embodiments, the content renderer 124 may distribute the subject information to other GUI elements such as one or more viewer panels, or place the acquired subject information as part of a GUI representation. In some embodiments, the content renderer 124 may distribute metadata such as tissue type, applied staining, scan parameters, z-stack layer, and focus layer to other GUI elements. In some embodiments, the content renderer 124 may also process the acquired image data, for example, by applying any preprocessing to the acquired image.

[0066] In some embodiments, the GUI renderer 126 creates graphical user interface (GUI) elements that are presented to the user along with content items rendered by the content renderer 124 or other system modules. The GUI renderer 126 may include code that defines the position and appearance of GUI elements, such as menu bar items, viewer panels, and configuration panels, each of which may be or contain interactive elements. In some embodiments, the GUI renderer 126 may, along with signals received from the UI interpreter 128, determine whether to enable or make available to the user a particular menu bar item or viewer panel depending on whether pre-established conditions are met, such as whether a zoom threshold level or magnification level is selected. For example, a menu bar item may be activated by the user, thereby allowing the user to select configuration options or panel views from a dropdown menu (see, for example, Figures 5A and 5B). (This means that). In some embodiments, the GUI renderer 126 may incorporate acquired image data provided by the content fetcher 122 or content renderer 124 into some or all of the GUI elements (for example, an actual image of a scanned biological sample may be displayed within a representation in the user interface).

[0067] For example, the GUI renderer 126 can generate a series of representations 401 that can be populated with image data acquired with information obtained by the content fetcher 122. Examples of representations are shown in Figures 4A, 4B, and 4C. These representations may be interactive. For example, if a user clicks on any particular representation (e.g., representation 401 in Figure 4A) (e.g., interpreted by the UI interpreter 128), the GUI renderer 126 may update the corresponding display in a viewer panel, such as a slide navigator viewer panel.

[0068] Similarly, the GUI renderer 126 can generate a set of viewer panels. In some embodiments, the generated viewer panels are interactive panels from which the user can select certain configurable options. For example, a zoom panel may include a slider bar from which the user can select a specific pre-set zoom level, such as 1x, 2x, 10x, or 40x, or into which the user can input a specific zoom level value. In some embodiments, the viewer panels are configured to convey relevant information to the user; for example, a case log panel may provide a history of user-configurable selections made by the user during the analysis of image data. Furthermore, the GUI renderer 126 may render visualizations indicating items that are hidden or unavailable or not enabled for user interaction.

[0069] The UI interpreter 128 can receive user input, for example, via the user input device 106, interpret the input, and determine the actions to be performed by the analysis application 120. For example, the UI interpreter 28 can determine the GUI element selected by the user (e.g., an icon, or selectable items such as menus, context menus, dropdown lists, buttons, or representations) and initiate the corresponding action (e.g., adding annotations, displaying additional content information, zooming to a selected zoom level, or generating a report for export). For example, the UI interpreter 128 can detect whether the user has selected an annotation tool (see annotation tool 405 in Figures 4A and 4B) and send a signal to the GUI renderer 126 to display additional user selection items. In some embodiments, annotation tools include manual region of interest (ROI) generation tools, automatic ROI generation tools, tools that enable the drawing of shapes (e.g., arrows), measurement tools, or text input generation tools. Each of these tools may be individually disabled or hidden based on the context of the user interaction. In some embodiments, selectable menu items include those that perform specific image processing algorithms, such as membrane detection algorithms, cell detection and counting algorithms, nucleus detection algorithms, scoring algorithms, heatmap generation algorithms, tissue masking algorithms, tissue type identification algorithms, etc. (see, for example, PCT Publications WO2016 / 120442 and WO2015 / 113895, and U.S. Patent Application Publications 2017 / 0154420, 2017 / 0372117, 2017 / 0103521, 2017 / 0140246, 2015 / 0347702, 2017 / 0082627, 2014 / 0377753, 2017 / 0337695, 2017 / 0323148, and 2017 / 0243051, the disclosures of which are incorporated herein by reference in their entirety). The input received from the UI interpreter 128 can be used to determine whether pre-established conditions are met.

[0070] System 100 is illustrative and should be understood to be deformable and modifiable. Furthermore, although System 100 is described with reference to certain blocks, it should be understood that these blocks are defined for explanatory purposes only and are not intended to imply a specific physical arrangement of components. Moreover, the blocks do not need to correspond to physically different components. Blocks can be configured to perform a variety of operations, such as programming a processor or providing appropriate control circuits, and the various blocks may or may not be reconfigurable depending on how the initial configuration is accessed. Embodiments of this disclosure can be realized in a variety of devices, including electronic devices implemented using any combination of circuitry and software. The image visualization and analysis application 120 is also illustrative, and a particular implementation may include more or fewer modules than those described herein. Furthermore, although certain modules may be described as performing certain functions, such descriptions are not intended to imply a specific function performed by the module or a specific set of instructions contained within such a module.

[0071] Figure 2B shows a client interface 140 that communicates with network 130 and system 100 (such as the system shown in Figures 1 and 2). The client interface 140 may be a standalone application (e.g., standalone image visualization and analysis software) or a web browser or other interface software that enables remote access to the image visualization and analysis application 120. For example, the client interface 140 allows a remote operator to log in to system 100 (such as the system shown in Figures 1 and 2) and access stored image data (such as data stored in the storage subsystem 104 or other network-attached storage device) or image data uploaded to system 100 for processing. In some embodiments, the client interface 140 may include any of the software modules described herein. In this way, a remote user can remotely interact with elements of the system (e.g., configurable elements) so that they can analyze and / or interpret the image data (e.g., a histologist or pathologist may select user-configurable parameters such as menu bar tools and / or viewer panels).

[0072] In some embodiments, the graphical user interface is adapted to display only certain features depending on the selected zoom level. For example, depending on the selected zoom level, certain tools in the menu bar may not be enabled (see Figure 4A). Similarly, certain panels that appear near the visualized representation of the accessed image may not be available. In general, the system 100 may be configured to "restrict" access to certain tools and panels that are not contextually relevant during image analysis, given a certain zoom level. For example, at a certain zoom level, it may not be possible to annotate certain parts of the image if there is insufficient resolution between certain features in the image. Therefore, if the software determines that a tool is irrelevant at the selected zoom level, that tool will not be enabled, and as mentioned above, this facilitates a quick review of the data displayed in the visualized representation and provides an improved user experience and an experience free from potential confusion.

[0073] Figure 3 shows a flowchart illustrating a method for visualizing relevant image data derived from a biological sample stained for the presence of one or more biomarkers. In step 300, at least one image of the biological sample is accessed. In some embodiments, the biological sample is stained for the presence of one or more biomarkers. In some embodiments, multiple images are acquired, such as multiple images from the same biological sample, but each image includes staining indicating the presence or absence of a particular biomarker.

[0074] Next, the first visualization representation is rendered within a graphical user interface (step 310), and the first visualization representation includes rendering at least one image at at least a first zoom level. In some embodiments, the first visualization representation 400 includes renderings of multiple accessed images (see, for example, Figure 4A). In some embodiments, the first visualization representation includes a series of representations 401 (e.g., image placeholders), each representation 401 may include one of the accessed images 410A or 410B. In some embodiments, each representation 401 is the same size and / or shape. In some embodiments, each representation 401 includes a first portion 402 containing one of the accessed images 410 and a second portion 403 containing an identification mark. In some embodiments, the identification mark includes identification of stains or stained biomarkers appearing in the image. For example, the second part 403 of representation 401 shows that a particular image 410A in the first part 402 was stained with hematoxylin and eosin.

[0075] In some embodiments, a first set of tools in the menu bar 404 (e.g., annotation tools 405) is displayed simultaneously with the first visualization representation 400 (step 320). As described above and as shown in Figure 4A, some tools are unavailable for selection, i.e., they are “displayed in gray” (see, e.g., setup calibration tool 408), while others are available for selection and are displayed as white icons (see, e.g., rotation tool 407). As a further example, each of the five annotation tools 405 is displayed in gray and therefore disabled. In some embodiments, certain tools may be completely hidden from the user at certain zoom levels, as will be further described herein. Thus, the first set of tools (i.e., the selectable tools and the tools represented by white icons) represents a subset of all tools that the user may use. Given the zoom levels of the four images shown in representation 401 of Figure 4A, these tools that are unavailable, i.e., “displayed in gray” as shown in Figure 4A, are considered irrelevant at a given zoom level of 1x. In other words, tools that are not enabled or hidden from the user's selection are considered ineffective at the selected zoom level. For example, a histologist or pathologist might not consider it worthwhile to perform measurements or draw arrows on cells because, at a certain zoom level (e.g., 1x), sufficient cellular features cannot be obtained, and the cellular features are not resolved enough to perform accurate measurements or correctly position arrows pointing to specific structures of interest.

[0076] In some embodiments, the first set of panels may also be displayed simultaneously with the first visualization representation and the first set of tools. For example, the zoom panel 406 and the slide navigator panel 409 may be displayed simultaneously with the first visualization representation and the first set of tools. As with the first set of tools, only panels relevant to the context at a given zoom level are displayed. For example, Figure 5A shows panels that the user can select from a context menu or drop-down menu. Unavailable items cannot be selected and are "grayed out". Similarly, selectable panels are displayed in white and are available for the user to select. Furthermore, panels that have already been selected may have a mark, such as a checkmark.

[0077] Following the simultaneous presentation of a first visualization representation (e.g., the three representations 401 in Figure 4A) and a first set of tools (those not “grayed out” in menu bar 404), and / or a first set of panels, the user may then interact with the visualization representation, such as by changing the zoom level, for example by increasing the zoom of one or more accessed images presented in the first visualization representation, allowing the user to see at least a portion of the accessed image in more detail, thereby providing at least a second visualization representation at a second zoom level 430. To “increase the zoom” or “zoom in on the image” means that a portion of the image is enlarged, thereby increasing the visual resolution of that portion of the image. For example, consider the accessed image 410B presented in Figure 4A at a 1x zoom level. At this zoom level, it should be understood that certain tissue structures, such as 420A, are difficult to decipher. However, when the zoom level of image 410B is increased as shown in Figure 4B (for example, increasing the zoom level from 1x to approximately 10x), the magnification and / or resolution of the tissue structure 420B are shown in more detail, for example, to a level where individual cells can be resolved.

[0078] Simultaneously with the display of the second visualization at the second zoom level 430 (step 330), a second set of tools appears in the menu bar 404 (step 340). Similar to the first set of tools, the second set of tools also represents a subset of all tools available to the user. For example, compared to the first set of available tools (see Figure 4A), the second set of tools includes each of the five annotation tools 405 (see Figure 4B).

[0079] In the particular embodiment shown in Figure 4B, the second set of tools includes the first set of tools, i.e., the second set of tools includes all the tools available in the first set of tools. In some embodiments, the second set of tools does not include all the tools provided in the first set of tools. In some embodiments, the second set of tools includes at least one tool that is different from those provided in the first set of tools.

[0080] In some embodiments, the second set of panels is displayed simultaneously with the second visualization representation and the second set of tools at a second zoom level 430. Similar to the second set of tools, only panels relevant to the context are displayed at a given zoom level. In the embodiment shown in Figure 4B, compared to the embodiment in Figure 4A, the slide panel 425 is automatically displayed at approximately 10x zoom level. Referring to Figure 5B, unavailable items cannot be selected and are "grayed out." Similarly, selectable viewer panels are displayed in white and can be selected by the user. Furthermore, panels that have already been selected are marked, for example, with a checkmark.

[0081] Figure 4C shows a selection of zoom levels (e.g., intermediate zoom levels) between the zoom levels shown in Figure 4A and Figure 4B. While four representations 401 were shown at a 1x zoom level in Figure 4A, only three representations are visualized, and only one of these three is fully visualized. In particular, the same tools appear to be available in Figure 4C as they are in Figure 4A. Similarly, the same panel appears in Figure 4C as it does in Figure 4A. Thus, Figure 4C indicates that the user has not reached the zoom threshold so that the image visualization and analysis application 420 makes other tools available in the menu bar 404, or even more so, other viewer panels. In this regard, Figure 4C shows that the items displayed and made available to the user are contextually dependent, and in this case, contextually dependent on the zoom level selected by the user.

[0082] In some embodiments, each menu item and / or viewer panel has a pre-programmed zoom threshold level that must be achieved before each available menu item and / or viewer panel can be enabled. For example, referring to Figure 4A, the annotation tool 405 may only become available when the user selects a zoom level exceeding a predetermined zoom threshold level value, e.g., 5x. In some embodiments, each individual tool in the menu bar 404 may have a different predetermined threshold. For example, the first annotation tool may have a predetermined threshold of 2x, the second annotation tool may have a predetermined threshold of 6x, and the slide calibration tool may have a predetermined threshold of 10x. Morphologically, the zoom threshold may depend on the type of tissue being observed or the staining.

[0083] In some embodiments, the threshold for making a menu bar item or viewer panel available does not need to be tied to a predetermined zoom threshold, i.e., a value such as 1x, 2x, 4x, 8x, 16x, 32x, etc. Rather, the threshold may be tied to whether a zoom level is selected in which individual cells or individual nuclei can be resolved by the user. Alternatively, the threshold may be associated with whether a zoom level is selected in which a certain number of cells reside in a predefined area (pixels x pixels), for example, 100 cells in a 500 pixels x 500 pixel area. In some embodiments, different elements may be visualized depending on the available display resolution. For example, the threshold may be predefined as "p" when the display resolution is "mxn", but as "p*q" when the display resolution is (m*qxn*q), where q is a scaling factor to account for the difference in display resolution. In yet another embodiment, whether a particular viewer panel is shown or hidden may be tied to the available display resolution. For example, five viewer panels may become available or hidden when a zoom level threshold is reached (i.e., they become available in contextual context), but if the available screen "physical property" is unavailable due to a low or limited display resolution, the system may continue to hide certain panels. The system will take display resolution limitations into consideration to determine which of the available viewer panels is most relevant and prioritize those panels for display. Furthermore, as an example, viewer panels may be rotated as needed to accommodate limited display resolutions.

[0084] As described herein, in some embodiments, entire menu items may be hidden until a specific zoom level is selected by the user. For example, suppose the menu bar contains items A, B, C, D, E, F, G, and H. Furthermore, assume a 1x zoom level where only menu bar items A, B, E, and H are displayed within the graphical user interface, as shown in Figure 6A. According to this disclosure, these menu bar items A, B, E, and H are the only tools relevant and useful to the user at the 1x zoom level. When the user zooms in on one of the images (see Figure 6B), additional menu bar items, for example, menu bar items C and G, may be displayed if they are relevant to the selected zoom level (e.g., 3x). When the user zooms further in on one of the images (see Figure 6C), here 601B, additional menu bar items, namely D and F, may be displayed if they are relevant to the selected zoom level (e.g., 8x). Figure 6C further shows that a second viewer panel, or "Viewer Panel 2," is displayed via a graphical user interface, but only when a certain zoom level is reached (i.e., a zoom level greater than 3x, e.g., 8x).

[0085] In some embodiments, an input image is received by a visualization system, and the visualization is provided at a default zoom level, e.g., 1x zoom level. In some embodiments, the system receives user input, e.g., an updated zoom level selection, a specific image selection. In some embodiments, a comparison is made between the received user input and a threshold condition, e.g., a comparison is made between received user inputs for zoom levels to determine if the zoom threshold level has been reached, or a comparison is made between received user inputs for image selection to determine biomarkers identified in the image. In some embodiments, if the zoom threshold level is met, additional visualization elements (e.g., tools, panels) may be presented to the user. In some embodiments, if different biomarkers are selected in a second image compared to a first image, additional visualization elements may be presented to the user. In some embodiments, at least one GUI element is modified based on the user's selection.

[0086] Additional Embodiments

[0087] In some embodiments, each displayed visualization or element has a position within the coordinate system of the display provided within an interface application such as a browser. For example, the icon of a tool such as an image analysis tool has a position within the display coordinate system. For example, if the display resolution is 4,000 x 3,000 and each pixel is considered a point in the coordinate system, the icon of an annotation tool may have a position defined by the pixel regions [150,200] (upper left corner), [160,200] (upper right corner), [150,210] (lower left corner), and [160,210] (lower right corner). In some embodiments, viewer panels, image data, and other representations may have positions within the coordinate system of the display. In some embodiments, a first viewer panel may have a first position region, and a second viewer element may have a second position region. In some embodiments, each viewer panel may have a variable position region depending on the type of information being displayed and the amount of information available. In some embodiments, the positions of various elements within the coordinate system of the display may be fixed or variable depending on the context.

[0088] The subjects and embodiments of operation described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subjects described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Any of the modules described herein may include logic executed by a processor. As used herein, “logic” means any information having the form of instruction signals and / or data that can be applied to affect the operation of a processor. Software is an example of logic.

[0089] Computer storage media may be, or may be, computer-readable storage devices, computer-readable storage boards, random or serial access memory arrays or devices, or one or more combinations thereof. Furthermore, although computer storage media are not propagating signals, computer storage media may be the source or destination of computer program instructions encoded into artificially generated propagating signals. Computer storage media may also be, or may be, one or more distinct physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described herein may be performed as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0090] The term "programmed processor" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable microprocessors, computers, systems on a chip, or a combination of the aforementioned. A device may include special-purpose logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device may also include code that creates the execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or code that constitutes one or more of these. Devices and execution environments can be used in a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. It can be achieved.

[0091] Computer programs (also called programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and can be deployed in any form. They can be included as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not have to, correspond to files in a file system. A program can be part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or a set of coordinated files (e.g., a file containing one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on one computer, or on multiple computers located in one site, or distributed across multiple sites and interconnected by a communication network.

[0092] The processes and logic flows described herein are executed by one or more programmable processors running one or more computer programs, and can perform actions by manipulating input data to produce outputs. The processes and logic flows can also be executed by specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and can be implemented in devices.

[0093] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operationally coupled to receive data, transfer data, or both. However, a computer does not necessarily require such devices. Furthermore, a computer can be incorporated into other devices, to name just a few, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be complemented or incorporated by special-purpose logic circuits.

[0094] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as an LCD (liquid crystal display), an LED (light-emitting diode) display, or an OLED (organic light-emitting diode) display, a display for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, on which the user can provide input to the computer. In some implementations, a touchscreen can be used to display information and receive input from the user. Other types of devices can be used, It can also provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or haptic feedback. Furthermore, user input can be received in any form, including acoustic, voice, or haptic input. In addition, the computer can interact with the user by sending and receiving documents to and from the device the user is using. For example, it can send a web page to a web browser on the user's client device in response to a request received from a web browser.

[0095] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component such as an application server, or a frontend component. The end component may be, for example, a client computer with a graphical user interface or web browser on which a user can interact with the implementation of the subject matter described in this specification, or one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), inter-network networks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks). For example, network 20 in Figure 1 may include one or more local area networks.

[0096] A computing system can include any number of clients and servers. Clients and servers are typically remote to each other and usually interact via a communication network. The client-server relationship is established by computer programs running on each computer that have a client-server relationship. In some embodiments, the server sends data (e.g., an HTML page) to client devices (e.g., for the purpose of displaying data and receiving user input from a user interacting with the client device). Data generated by the client devices (e.g., the results of user operations) can be received from the server's client devices.

[0097] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications referred to herein and / or listed in the application datasheet are incorporated herein by reference in their entirety. Some embodiments may be modified to provide further embodiments using the concepts of various patents, applications, and publications as needed.

[0098] While this disclosure has been described with reference to several exemplary embodiments, it should be understood that many other modifications and embodiments that would fall within the spirit and scope of the principles of this disclosure can be devised by those skilled in the art. More specifically, reasonable variations and modifications are possible in the components and / or arrangements of the subject matter in combination arrangements within the foregoing disclosure, drawings, and appended claims, without departing from the spirit of the disclosure. In addition to variations and modifications of components and / or arrangements, alternative uses will also be apparent to those skilled in the art.

Claims

1. The steps include providing a graphical user interface that includes an area for displaying one or more images and a first set of menu items in a menu bar, A step of receiving a selection to change the zoom level of the image of a biological sample, Steps of providing an updated version of the graphical user interface in response to receiving the selection, wherein the updated version of the graphical user interface includes a second set of menu items in the area and the menu bar, and the number of menu items in the second set of menu items is greater than the number of menu items in the first set of menu items, A method that includes this.

2. The method according to claim 1, wherein at least one menu item in the first set of menu items is included in the second set of menu items.

3. The method according to claim 1, wherein the region for presenting one or more images is on which the images of the biological sample are arranged.

4. The method according to claim 1, wherein the graphical user interface further includes a zoom panel, and the selection for changing the zoom level is received in response to user interaction with the zoom panel.

5. The method according to claim 1, wherein the step of providing the updated version of the graphical user interface includes changing the zoom level of the image contained in the region from a first zoom level to a second zoom level.

6. The method according to claim 1, wherein the graphical user interface further comprises a first set of panels, and the updated version of the graphical user interface further comprises a second set of panels, wherein at least one panel in the second set of panels is different from at least one panel in the first set of panels.

7. The method according to claim 1, further comprising the step of determining that the zoom level of the image of the biological sample is equal to or greater than a zoom threshold level, before the step of providing the updated version of the graphical user interface.

8. A system comprising a processor and memory, The memory stores instructions, When the aforementioned instruction is executed by the processor, the processor will: The steps include providing a graphical user interface that includes an area for displaying one or more images and a first set of menu items in a menu bar, A step of receiving a selection to change the zoom level of the image of a biological sample, Steps of providing an updated version of the graphical user interface in response to receiving the selection, wherein the updated version of the graphical user interface includes a second set of menu items in the area and the menu bar, and the number of menu items in the second set of menu items is greater than the number of menu items in the first set of menu items, A system that performs actions including those mentioned above.

9. The system according to claim 8, wherein at least one menu item in the first set of menu items is included in the second set of menu items.

10. The system according to claim 8, wherein the region for presenting one or more images is configured to contain the images of the biological sample.

11. The system according to claim 8, wherein the graphical user interface further includes a zoom panel, and the selection for changing the zoom level is received in response to user interaction with the zoom panel.

12. The system according to claim 8, wherein the step of providing the updated version of the graphical user interface includes changing the zoom level of the image contained in the region from a first zoom level to a second zoom level.

13. The system according to claim 8, wherein the graphical user interface further comprises a first set of panels, and the updated version of the graphical user interface further comprises a second set of panels, wherein at least one panel in the second set of panels is different from at least one panel in the first set of panels.

14. The aforementioned operation, The system according to claim 8, further comprising the step of determining that the zoom level of the image of the biological sample is equal to or greater than a zoom threshold level, prior to the step of providing the updated version of the graphical user interface.

15. A non-temporary computer-readable medium for storing computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, the processor receives The steps include providing a graphical user interface that includes an area for displaying one or more images and a first set of menu items in a menu bar, A step of receiving a selection to change the zoom level of the image of a biological sample, Steps of providing an updated version of the graphical user interface in response to receiving the selection, wherein the updated version of the graphical user interface includes a second set of menu items in the area and the menu bar, and the number of menu items in the second set of menu items is greater than the number of menu items in the first set of menu items, A non-temporary, computer-readable medium that causes an action to be performed, including the execution of such an action.

16. The non-temporary computer-readable medium according to claim 15, wherein at least one menu item in the first set of menu items is included in the second set of menu items.

17. The non-temporary computer-readable medium according to claim 15, wherein the graphical user interface further includes a zoom panel, and the selection for changing the zoom level is received in response to user interaction with the zoom panel.

18. The non-temporary computer-readable medium according to claim 15, wherein the step of providing the updated version of the graphical user interface includes changing the zoom level of the image contained in the region from a first zoom level to a second zoom level.

19. The non-temporary computer-readable medium according to claim 15, wherein the graphical user interface further comprises a first set of panels, and the updated version of the graphical user interface further comprises a second set of panels, wherein at least one panel in the second set of panels is different from at least one panel in the first set of panels.

20. The aforementioned operation, The non-temporary computer-readable medium according to claim 15, further comprising the step of determining that the zoom level of the image of the biological sample is equal to or greater than a zoom threshold level, prior to the step of providing the updated version of the graphical user interface.

Images