Image processing system and method for displaying multiple images of biological specimens

Abstract

A system and method for displaying multiple simultaneous views of the same region of a biological tissue sample. A processor executes logic instructions to perform operations such as receiving multiple images of the biological tissue sample, transforming the multiple images to a common reference frame based on metadata associated with each image, and arranging the multiple images into a display pattern for simultaneously viewing different aspects of the imaged biological tissue sample on a display screen. The multiple images are generated by preprocessing the images of the biological tissue sample. Each image shows a view pattern of the same region of the biological tissue sample, and each image includes metadata describing the spatial orientation of the image (such as translation, rotation, and magnification) to bring the multiple images into a common view.

Description

[0001] This application is a divisional application of the application with the application date of August 25, 2016, application number 201680050177.X and the invention name of "Image processing system and method for displaying multiple images of biological specimens". Technical Field

[0002] The subject disclosure relates to imaging for medical diagnosis. More particularly, the subject disclosure relates to consistent display and transformation of field of view (FOV) images. Background Art

[0003] In the analysis of biological specimens such as tissue sections, blood, cell cultures, etc., one or more combinations of staining agents are utilized to dye the biological specimens, and the test substance produced by the results is checked or the test substance is imaged for further analysis. Observing this test substance has realized various processes, including the diagnosis of disease, the assessment of therapeutic response, and the development of new drugs that are used to resist disease. The test substance comprises one or more staining agents that are combined with antibodies, and this antibody is combined with the protein, protein fragment, or other interested object (hereinafter referred to as target or target object) in the specimen. Antibodies or other compounds that make the target in the specimen be combined with staining agents are referred to as biomarkers in this subject disclosure. Some biomarkers have a fixed relationship with staining agents (for example, the counterstain hematoxylin that is often used), yet for other biomarkers, the selection of staining agents can be used to develop and produce new test substances. After staining, the test substance can be imaged for the further analysis of the content of the tissue sample. The image of the entire slide is usually referred to as a full slide image, or is simply referred to as a full slide.

[0004] Typically, in immune score calculations, scientists use multiple assays involving staining a single piece of tissue or a single assay involving staining adjacent serial tissue sections to detect or quantify, for example, multiple proteins or nucleic acids in the same tissue block. When stained slides are available, immunological data (such as the type, density, and location of immune cells) can be estimated from tumor tissue samples. This data has been reported to predict colorectal cancer patient survival and has shown significant prognostic value.

[0005] In the traditional workflow for immunoscore calculation, an expert reader, such as a pathologist or biologist, manually selects a representative field of view (FOV) or region of interest (ROI) as an initial step by examining the slide under a microscope or reading an image of a scanned / digitized slide on a monitor. When the tissue slide is scanned, an independent reader reviews the scanned image and manually labels the FOV based on the reader's personal preference. After selecting the FOV, a computer generates a count of immune cells in each FOV via an automated algorithm, or the pathologist / reader manually counts the immune cells within the selected FOV. Manual FOV selection and counting is highly subjective and subject to reader bias, as different readers may select different FOVs for counting. Consequently, immunoscore studies are no longer reproducible. By automating FOV selection, a unified method is applied that reduces individual reader subjectivity. The use of low-resolution images to perform FOV selection further improves computational efficiency, allowing analysts to rapidly analyze tissue regions.

[0006] It is often the case that any single view of a tissue sample may lead to several possible disease state diagnoses. The tedious review of several different views must rely on the memory of an expert reader to limit the focus to any particular diagnosis.

[0007] Prior art includes, for example, US 2003 / 0210262 to Graham et al., which generally teaches displaying at least two views of the same area adjacent to each other on a microscope slide, where each view provides different lighting conditions and the viewing device provides similar linear translation.

[0008] Finally, US 2012 / 0320094 to Ruddle et al. generally teaches displaying at least two microscope slide images of the same area adjacent to each other on a viewing screen at different magnifications.

[0009] Automatic identification of FOV is disclosed in US 62 / 005,222 and is incorporated herein in its entirety by reference to PCT / EP2015 / 062015. Summary of the Invention

[0010] The invention provides an image processing method for displaying a plurality of images of a biological tissue region and a corresponding image processing system as claimed in independent claims 1 and 9. Embodiments of the invention and further aspects of the invention are provided in the other independent and dependent claims.

[0011] As understood herein, a 'tissue sample' is any biological sample obtained from a tissue region, such as a surgical biopsy specimen obtained from a human or animal body for anatomical pathology. The tissue sample may be a prostate tissue sample, a breast tissue sample, a colon tissue sample, or a tissue sample obtained from another organ or body region.

[0012] As understood herein, a 'multi-channel image' includes a digital image obtained from a biological tissue sample in which different biological structures, such as nuclei and tissue structures, are simultaneously stained with specific fluorescent dyes, each of which fluoresces in a different spectral band, thereby constituting one of the channels of the multi-channel image. The biological tissue sample can be stained with multiple stains and / or with a stain and a counterstain, the latter also being referred to as a 'single marker image'.

[0013] As understood herein, a 'demixed image' comprises a gray value or scalar image obtained for one channel of a multi-channel image. By demixing the multi-channel image, one demixed image is obtained for each channel.

[0014] A 'color channel' as understood herein is one channel of an image sensor. For example, the image sensor may have three color channels, such as red (R), green (G), and blue (B).

[0015] A 'heat map' as understood herein is a graphical representation of data that represents individual values ​​contained in a matrix as colours.

[0016] 'Thresholding' as understood herein comprises the application of a predefined threshold or classification of local maxima to provide a classified list and selecting a predetermined number of local maxima from the top of the classified list.

[0017] As understood herein, 'spatial low-pass filtering' includes spatial filtering using a spatial filter that performs a low-pass filtering operation (particularly a linear or nonlinear operation) on a neighborhood of image pixels. In particular, spatial low-pass filtering can be performed by applying a convolution filter. Spatial filtering is known in the art (see Digital Image Processing, 3rd Edition, Rafael C. Gonzalez, Richard E. Woods, page 145, chapter 3.4.1).

[0018] As understood herein, 'local maximum filtering' includes a filtering operation in which a pixel is considered a local maximum if it is equal to the maximum value in a sub-image region. Local maximum filtering can be implemented by applying a so-called maximum filter (see Digital Image Processing, 3rd Edition, Rafael C. Gonzalez, Richard E. Woods, page 326, Chapter 5).

[0019] A 'field of view (FOV)' as understood herein comprises an image portion having a predetermined size and shape, such as a rectangular or circular shape.

[0020] According to an embodiment of the present invention, a tissue region of a cancer biopsy tissue sample is sliced ​​into adjacent tissue slices. The tissue slices may be labeled with a single or multiple stains for identification of corresponding biometric features. A digital image is acquired from each of the labeled tissue slices using an image sensor having multiple color channels, such as an RGB image sensor.

[0021] An image registration algorithm is performed on the acquired multiple digital images. Various suitable image registration algorithms known in the art can be used to perform image registration (see https: / / en.wikipedia.org / wiki / Image_registration and http: / / tango.andrew.cmu.edu / ~gustavor / 42431-intro-bioimaging / readings / ch8.pdf). In particular, affine transformations can be utilized to perform image registration.

[0022] The image registration algorithm generates a geometric transformation that aligns corresponding points of the images.The geometric transformations may be provided in the form of mappings, where each mapping maps a point in one of the images to a corresponding point in the other of the images.

[0023] The images are aligned according to image registration. In other words, a geometric transformation generated by an image registration algorithm is applied to the images to align the images so that the aligned images are displayed in a two-dimensional plan on a display. Thus, the display shows a plurality of images after registration and alignment such that each of the images displayed in the two-dimensional plan shows a matching tissue region.

[0024] An image transformation command with respect to one of the displayed images may be input via a graphical user interface, such as by performing a mouse click on the image, rotating a mouse wheel, or performing a gesture input via a touch-sensitive display screen. For example, the image transformation command is a command to zoom in or out, rotate, or perform another image transformation, such as by selecting a field of view.

[0025] In response to inputting an image transformation command to transform one of the displayed images, the other images are simultaneously transformed in the same manner. This is accomplished using a geometric transformation (such as a mapping) generated by an image registration algorithm. Thus, in response to an image transformation command in all images, image transformation is uniformly performed.

[0026] Embodiments of the present invention are particularly advantageous because a user, such as a pathologist, can easily view and manipulate images obtained from tissue sections of a tissue region in an intuitive manner that facilitates performing diagnostic tasks.

[0027] According to an embodiment of the present invention, to acquire a multi-channel image, at least one of the tissue sections is labeled with multiple stains. The multi-channel images are demixed to provide a set of demixed images. The demixed images do not need to be registered with respect to each other or with respect to the multi-channel images, as they are all based on the same dataset acquired from one of the tissue sections by an optical sensor. The multi-channel image is selected as a reference image for performing an image registration algorithm on multiple images other than the set of demixed images. This provides a mapping of each of the multiple images other than the demixed image to the reference image.

[0028] Using a multi-channel image as a reference image is advantageous for image registration because it reduces the computational cost of performing image registration and alignment of images, since no image registration and alignment is required for the demixed image.

[0029] According to an embodiment of the present invention, an image transformation command is a zoom-in or zoom-out command received via a graphical user interface using gesture recognition. For example, the user gesture input by which the image transformation command is to be zoomed in or out is a pinch gesture performed by placing two fingers on one of the displayed images. Thus, an image transformation command is received with respect to one of the displayed images on which the user placed their fingers, and the image transformation command is executed with respect to that image and also synchronously with respect to the other displayed images.

[0030] According to another embodiment of the present invention, the plurality of acquired images are stored on a server computer. The images are then transmitted from the server computer to a mobile battery-powered telecommunication device (such as a smartphone or mobile computer) via a telecommunication network for display on the telecommunication device. This provides maximum flexibility in accessing and viewing the images.

[0031] According to an embodiment of the present invention, at least the execution of the image registration algorithm is performed by a server computer, and the resulting geometric transformation (such as a mapping) is transmitted from the server computer to the telecommunications device along with the image. This can be advantageous because the image registration algorithm can require significant computing power. Performing the image registration algorithm as a pre-processing step on a server computer rather than on a mobile, battery-powered telecommunications device has the advantage of conserving battery power and reducing delays experienced by the user.

[0032] According to an embodiment of the present invention, one or more fields of view are automatically defined in one or more of the images. A graphical symbol, such as a rectangular frame, may be displayed to indicate the location of the field of view in one of the images. A user may input an image transformation command with respect to the field of view by selecting the corresponding graphical symbol (such as by touching the graphical symbol on a touch-sensitive display). In response to the selection of the graphical symbol, a magnified image transformation may be performed with respect to the field of view and simultaneously with respect to aligned image portions in the other images.

[0033] Automatic definition of the field of view can also be performed by a server computer to reduce the computational burden on the telecommunication device, thereby increasing battery life and reducing latency. In this example, metadata describing the defined field of view is generated by the server computer and transmitted along with the image via a network to enable the telecommunication device to display a graphical symbol indicating the location of the field of view defined by the server computer.

[0034] According to another aspect of the present invention, there is provided an image processing system configured to perform the method of the present invention.

[0035] The present invention surprisingly effectively allows for the coordinated examination of multiple diagnostic images showing the same tissue region adjacent to each other on a single viewing screen. All images are aligned and scaled to a common reference frame, and can be panned and scaled together so that each shows an important aspect of histology. This enables more directive and definitive diagnosis of important conditions, where any single image might only support a more preliminary conclusion from an expert reader.

[0036] The present invention has at least the following advantageous features and robustness:

[0037] 1. Select a common display reference frame and use it for image visualization.

[0038] 2. Transforming the pre-processed images of the biological tissue sample into a common display reference frame by constructing a target view for each pre-processed image of the biological tissue sample.

[0039] 3. Accept user gestures to dynamically change the common display reference frame. For example, the image can be translated, rotated, or zoomed in or out simultaneously.

[0040] 4. When each image shows a different stain to highlight important aspects of a biological tissue sample, the simultaneous views provide a more certain diagnosis of the tissue condition than could be obtained by relying on the memory of an expert reader to review the same images sequentially.

[0041] The present invention is further applicable to images obtained from serial microtome sections, where they may require rotation in addition to translation to align common features of interest. Furthermore, the present invention may involve tagging images with metadata describing their location within the tissue section and using this information in the construction of an affine transformation to adjust the images to a common reference frame for display. Additionally, the present invention allows for simultaneous scaling of the magnification of all images at the same scale.

[0042] In one embodiment, the subject disclosure features a system for simultaneously displaying multiple views of the same region of a biological tissue sample. The system may include a processor and a memory coupled to the processor. The memory may store computer-readable instructions that, when executed by the processor, cause the processor to perform operations.

[0043] In another embodiment, the subject disclosure features a method for simultaneously displaying multiple views of the same region of a biological tissue sample. The method can be implemented by an imaging analysis system and can be stored on a computer-readable medium. The method can include logic instructions executed by a processor to perform operations.

[0044] In some embodiments, the operations may include receiving a plurality of pre-processed images of a biological tissue sample, selecting a common display reference frame to be used for image visualization, transforming the plurality of pre-processed images into the common display coordinate frame by constructing a target view for each of the plurality of pre-processed images to generate a plurality of displayable images, arranging the plurality of displayable images into a display pattern for viewing on a display screen, displaying the plurality of displayable images on the display screen, and accepting a user gesture to dynamically change the common display reference frame.

[0045] In yet other embodiments, the operation may further include uniformly translating, rotating, and zooming in and out of the plurality of images on the display screen in response to input gestures from the interface device to provide a desired view of the imaged biological tissue sample, removing one or more images from the plurality of images on the display screen to tidy up the display screen, adding a new mode image on the display screen, rearranging the display pattern to form an alternative display pattern, stacking two or more image modes to enhance image features, and saving the currently examined display pattern as a saved template for future examination.

[0046] In one embodiment of the present invention, a collection of pre-registered images may be provided through FOV analysis. An example of FOV analysis is described herein. The images are tagged with metadata describing their individual placement, rotation, and magnification with respect to a common reference frame. This metadata, along with any new reference frame, can define an affine mapping between the original reference frame of the image and the new frame.

[0047] Re-imaging the new frame can be accomplished by mapping the target pixel in the new frame back to its corresponding position in the source frame of the image and choosing that pixel value or an interpolation of the surrounding source pixel values ​​as the target pixel value. In this way, any image can be translated, rotated, stretched, or shrunk to a new reference frame shared by all other images in preparation for simultaneous display.

[0048] The decision of which arrangements are important to the diagnosing physician may be based solely on the best judgment of the expert reader. Some views may be deemed unimportant to the situation at hand, while others may be added to the selection because they are more important to the diagnosis.

[0049] Embodiments of the present invention are particularly advantageous because they provide an automated and reliable technique for identifying fields of view in multi-channel images while avoiding the tedious manual effort of pathologists or biologists to mark fields of view in multi-channel images and thereby eliminating subjectivity and human error. Due to spatial low-pass filtering, local maximum filtering and thresholding operations can be performed at high processing speeds, minimizing the computational overhead and latency experienced by the user. This is due to the fact that the definition of the field of view is not performed directly on the multi-channel image, but rather based on the filtered and thresholded image, achieving high processing speeds.

[0050] It is important to note that the analysis in step f is performed on the full-resolution multi-channel image and not on the spatially low-pass filtered deblended image. This ensures that all available image information can be used to perform the analysis, while the filtering operations (i.e., steps b, c, and d) are only used to identify the relevant fields of view where the full analysis is to be performed.

[0051] According to another embodiment of the present invention, one of the demixed images is processed for defining the field of view, while the other of the demixed images is segmented for identification of tissue regions. Figure 2 or generating a demixed image from multiple images (more than 2 channels).

[0052] Suitable segmentation techniques are those known from the prior art (cf. Digital Image Processing, 3rd Edition, Rafael C. Gonzalez, Richard E. Woods, Chapter 10, page 689 and Handbook of Medical Imaging, Processing and Analysis, Isaac N. Bankman, Academic Press, 2000, Chapter 2). Non-tissue areas are removed by means of segmentation because they are not of interest to the analysis.

[0053] This segmentation provides a mask by which those non-tissue areas are removed. The resulting tissue mask can be applied to the demixed image before or after spatial low-pass or local maximum filtering or thresholding operations and before or after the field of view is limited. It may be advantageous to apply the problem mask at an early stage so that the processing load may be further reduced, such as before performing spatial low-pass filtering.

[0054] According to one embodiment of the invention, another of the demixed images segmented for providing the tissue mask is obtained from a channel representing a stain being a counterstain to the stain represented by the demixed image processed according to step be. of claim 1 .

[0055] According to one embodiment of the present invention, a field of view is defined for at least two of the demixed images. Fields of view defined in two different demixed images can be merged if they are located at the same or almost identical image positions. This is particularly advantageous for stains that can be co-localized so that a single field of view produces co-localized stains that identify common biological structures. By merging such fields of view, the processing load is further reduced and the analysis in step f only needs to be performed once on the merged field of view. In addition, the cognitive burden on the pathologist or biologist is also reduced because only one analysis result is given instead of two related results. According to an embodiment, two fields of view can be merged if the degree of spatial overlap of the fields of view is above an overlap threshold.

[0056] According to an embodiment of the present invention, analysis of a field of view is performed by counting biological cells shown in a multi-channel image within the field of view. Cell counting can be performed using appropriate image analysis techniques applied to the field of view. In particular, cell counting can be performed using image classification techniques.

[0057] According to other embodiments of the present invention, analysis of the field of view is performed with the aid of a trained conventional neural network, such as by inputting the field of view or an image patch taken from the field of view into the conventional neural network for determining a probability that a biometric feature is present within the field of view or the image patch, respectively. Image patches from the field of view may be extracted for input into the conventional neural network by first identifying an object of interest within the field of view and then extracting the image patch containing the object of interest.

[0058] According to another embodiment of the invention, the analysis in step f is performed as a data analysis, such as a cluster analysis or a statistical analysis, on the field of view.

[0059] According to another aspect of the present invention, there is provided an image processing system for analyzing a multi-channel image obtained from a biological tissue sample stained with a plurality of stains, which is configured to perform the method of the present invention.

[0060] The subject disclosure features a preprocessing system and method for automatic field of view (FOV) selection based on the density of each cell marker in a whole slide image. The operations described herein include reading images for individual markers from demixed multiplex slides or from abnormally stained slides, and calculating tissue region masks based on the individual marker images.

[0061] The heat map of each marker can be determined by applying a low-pass filter on the individual marker image channels and selecting the top K highest intensity regions from the heat map as the candidate FOV for each marker. The candidate FOVs from the individual marker images are merged together. The merging may include one or both of the following: adding all FOVs together to the same coordinate system or adding only the FOVs from the selected marker images based on input preferences or selections by first registering all individual marker images to a common coordinate system and merging through morphological operations. After this, inverse registration is used to transfer all identified FOVs back to the original image to obtain the corresponding FOV image at high resolution.

[0062] In some embodiments, low-resolution images are used to speed up the calculation of the FOV. Because the image resolution is lower, it is computationally much faster to calculate the heat map and tissue region mask. This allows the selection of the FOV to be automated and rapid, which allows for faster analysis of tissue samples.

[0063] Tissue slide images contain many features, only some of which are of interest to any particular study. These regions of interest may have the specific colors brought about by selective staining agent uptake. They may also have a wide spatial range. Importantly, regions of interest may have certain specific spatial frequencies that are removed from the image by means of spatial frequency filtering. Such filters include, but are not limited to, low-pass, high-pass, and band-pass filters. More carefully, tuned spatial frequency filters may be those that are referred to as matched filters. Non-limiting examples of spatial frequency filters include, but are not limited to, low-pass filters, high-pass filters, band-pass filters, multi-bandpass filters, and matched filters. Such filters may be statically defined, or adaptively generated.

[0064] In the process of locating the region of interest, it is therefore helpful to first select the appropriate color through a deblending process, which can be viewed as a linear operator applied to the primary color channels (R, G, and B of the image). Spatial frequency filtering is also applied to prioritize interesting features in the image. These operations can be applied in any order, as they are both linear operators.

[0065] In parallel with this region selection, a broader segmentation mask can be formed by using completely different tuned spatial frequency filters to select only the total area of ​​the slide image where, for example, tissue is present and discard empty areas. Thus, multiple different spatial frequency filters can be applied to the same tissue slide image.

[0066] Once filtered, the location of the region of interest can be determined by applying a local maximum filter (a morphological nonlinear filter), which produces an image by causing each pixel of the result to hold the value of the maximum pixel value from the source image located below the maximum filter kernel. The kernel is a geometric mask of arbitrary shape and size, but will be constructed for this purpose to have approximately the size of the feature of interest. The output image from the local maximum filter will tend to have an island shape like the kernel and a constant value equal to the maximum pixel value in the region.

[0067] In some embodiments, given the current configuration of a local maximum filter image, a threshold can be applied to convert the filter image into a binary mask by assigning a binary mask value of 1 to corresponding filter image pixels above the threshold and a value of 0 to corresponding filter image pixels below the threshold. The result will be a blob of 1s that can be labeled as regions and have a measurable spatial extent. These region labels, positions, and spatial extents together provide a record of the region of interest (ROI) or field of view (FOV). BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figures 1A-1B Depicted are a system and a workflow, respectively, for automatic FOV selection according to an exemplary embodiment of the subject disclosure.

[0069] Figure 2 Depicted is a heat map calculation according to an exemplary embodiment of the subject disclosure.

[0070] Figure 3 Depicted is a tissue mask calculation according to an exemplary embodiment of the subject disclosure.

[0071] Figure 4 Depicting candidate FOVs according to one exemplary embodiment of the subject disclosure.

[0072] Figures 5A-5B Depicted are the merge of FOVs from all markers and from selected markers, respectively, according to an exemplary embodiment of the subject disclosure.

[0073] Figures 6A-6B depicts integrating FOV according to an exemplary embodiment of the subject disclosure; Figure 6C Shows morphological operations.

[0074] Figure 7 Depicted is a user interface for image analysis using an all marker view according to an exemplary embodiment of the subject disclosure.

[0075] Figure 8 Depicted is a user interface for image analysis using individual marker views according to an exemplary embodiment of the subject disclosure.

[0076] Figure 9 Depicted is a digital pathology workflow for immune score calculation according to an exemplary embodiment of the subject disclosure.

[0077] Figure 10 A process flow diagram is depicted for an exemplary embodiment of the present invention.

[0078] 11a and lib depict process flow diagrams for an exemplary embodiment of the present invention starting with a single stain marker image.

[0079] Figure 12 A process flow diagram depicting an exemplary embodiment of the present invention starting with multiple slides.

[0080] Figure 13 A process flow diagram depicting an exemplary embodiment of the present invention starting with a single stain image.

[0081] Figure 14 Depicted is a flow diagram of an exemplary process for displaying multiple views simultaneously, according to one embodiment of the present invention.

[0082] Figure 15 Depicted is a flow chart of an exemplary process for selecting a common display reference frame according to one embodiment of the present invention.

[0083] Figure 16 Depicted is a flow chart of an exemplary process for converting a pre-processed image to produce a displayable image, according to one embodiment of the present invention.

[0084] Figure 17 Depicting a panned view of an image on a user interface according to an exemplary embodiment of the subject disclosure.

[0085] Figure 18 Depicts a rotated view of an image on a user interface according to an exemplary embodiment of the subject disclosure.

[0086] Figure 19 Depicts two images being deleted from a user interface according to an exemplary embodiment of the subject disclosure.

[0087] Figure 20A display depicting rearrangement of images on a user interface according to an exemplary embodiment of the subject disclosure.

[0088] Figure 21 Depicts a magnified view of an image on a user interface according to an exemplary embodiment of the subject disclosure.

[0089] Figure 22 Depicted is a stacked view of two images on a user interface according to an exemplary embodiment of the subject disclosure.

[0090] Figure 23 Depicted are schematic diagrams illustrating embodiments of the present invention.

[0091] Figure 24 One embodiment of the present invention is depicted using a pinch gesture to zoom in or out. DETAILED DESCRIPTION

[0092] The present invention features systems and methods for simultaneously displaying multiple views of the same region of a biological specimen (e.g., a tissue sample). In some embodiments, the system may include a processor and a memory coupled to the processor. The memory may store computer-readable instructions that, when executed by the processor, cause the processor to perform operations.

[0093] In other embodiments, the method may be implemented by an imaging analysis system and may be stored on a computer-readable medium.The method may include logic instructions executed by a processor to perform operations.

[0094] As in Figure 14 As shown in , operations for the systems and methods described herein may include, but are not limited to, receiving a plurality of pre-processed images of a biological tissue sample (2100), selecting a common display reference frame to be used for image visualization (2110), transforming the plurality of pre-processed images into the common display coordinate frame by constructing a target view for each of the plurality of pre-processed images to produce a plurality of displayable images (2120), arranging the plurality of displayable images into a display pattern for viewing on a display screen (2130), displaying the plurality of displayable images on the display screen (2140), and accepting user gestures to dynamically change the common display reference frame (2150). Without wishing to limit the present invention to a particular theory or mechanism, the present invention allows for coordinated examination of multiple images displayed adjacent to each other on a single viewing screen.

[0095] In some embodiments, displaying multiple displayable images (2140) can allow for simultaneous dynamic viewing of different aspects of the imaged biological tissue sample. Repeating the transformation process (2120) can cause all displayable images to simultaneously perform an apparent coordinated translation, rotation, or magnification change.

[0096] In some embodiments, each preprocessed image can show a view mode of the same area of ​​a biological tissue sample, and each preprocessed image can have metadata describing the image reference frame with respect to a global standard reference frame. The metadata for each preprocessed image can describe the preprocessed image of a local reference frame (P1-LRF) with respect to a global standard reference frame (GSRF). For example, the metadata can describe the spatial position, orientation, and magnification of the preprocessed image with respect to the global standard reference frame. As another example, the metadata describes the translation, rotation, and magnification of each image with respect to the standard reference frame. Given a common display reference frame, an affine transformation is created to associate source image pixels with pixels displayed for the image mode view. As used herein, an affine transformation, or alternatively, an affine mapping, can be defined as a linear transformation, expressible as a matrix operator on augmented position vectors that can represent arbitrary translations, rotations, and magnifications of these vectors. Affine transformations are known to those of ordinary skill in the art.

[0097] In some embodiments, the pre-processed image local reference frame (Pl-LRF) is a two-dimensional reference frame used to describe the positions of pixels in the pre-processed image.

[0098] In other embodiments, the global standard reference frame is a conventional fixed two-dimensional reference frame used to describe the space of pixel positions, and it allows understanding the spatial relationship between different images by defining an affine mapping between each image local reference frame (I-LRF) and the global standard reference frame. In some embodiments, the metadata for each pre-processed image describes the spatial position, orientation, and magnification of the pre-processed image with respect to the GSRF. For example, the metadata can define a first affine mapping between the image reference frame and the global standard reference frame.

[0099] In some embodiments, as Figure 15 As shown in , the operation of selecting a common display reference frame (2110) can further include creating a two-dimensional display image pixel grid (2111), constructing a two-dimensional display image local reference frame (DI-LRF) for describing pixel positions in the display image pixel grid (2112), selecting the position, orientation, and magnification of the DI-LRF with respect to the GSRF (2113), and calculating an affine transformation that maps pixel positions in the DI-LRF to positions in the GSRF (2114). The grid intersections can represent pixel positions. This construction can serve as a display image template and can provide an affine partial mapping for the production of the display image.

[0100] In some embodiments, such as Figure 16As shown in FIG, the operation of converting the plurality of pre-processed images to a common display coordinate system (2120) may further include constructing a working copy of a CDRF display image template and an affine partial map (2121), composing the affine partial map using a first affine map for the pre-processed image to produce a composite map that transforms pixel positions in the DI-LRF of the display image to positions in the PI-LRF of the pre-processed image (2122), and drawing the display image by performing an operation on each display image pixel (2123). In some embodiments, the working copy of the display image template includes a memory unit for storing pixel values ​​of the display image.

[0101] Operations for drawing the display image may include, but are not limited to, mapping using a composite affine transformation from the DI-LRF location of a display image pixel to a location in the PI-LRF of the pre-processed image (2124), interpolating pixel values ​​between adjacent pixels in the pre-processed image around the mapped location (2125), and delivering the interpolated pixel value as the pixel value used in the display image at the display image pixel (2126). By performing these operations for each display image pixel, each pre-processed image can be transformed into a display image for presentation on a display screen.

[0102] In some embodiments, interpolation between adjacent pixels may be performed by simply picking the nearest pixel for its value or by using bilinear interpolation between the four nearest neighbors (2125). In other embodiments, more sophisticated methods (such as spatial low-pass filtering) may be needed to avoid sample aliasing or imaging artifacts when the magnification between the source and target images changes, as this is equivalent to sample rate conversion.

[0103] In other embodiments, the operation of converting the plurality of pre-processed images (2120) may perform non-linear correction on the plurality of pre-processed images to remove optical distortion. Exemplary non-linear corrections may include removal of pincushion or barrel distortion, defocus, coma, or astigmatism.

[0104] In some embodiments, any of the two-dimensional reference frames mentioned herein, such as the two-dimensional local reference frames (PI-LRF and DI-LRF) and the conventional fixed two-dimensional reference frame (GSRF), can be an orthogonal Cartesian reference frame. In other embodiments, any of the two-dimensional reference frames mentioned herein can be a non-orthogonal and / or non-Cartesian reference frame.

[0105] In some embodiments, multiple images are generated by preprocessing an image of a biological tissue sample. The preprocessing of the image can utilize a method such as the FOV method described herein. However, it is understood that other appropriate methods can be used to preprocess the image.

[0106] In some embodiments, the display pattern may be in the form of rows and columns. The display pattern may represent "m" rows and "n" columns, where "m" and "n" may be any natural numbers. For example, the display pattern may have 2 rows and 3 columns. In other embodiments, the display pattern may be a ring or a square. In yet another embodiment, the display pattern may be a pyramid.

[0107] In other embodiments, the operation may further include translating the plurality of images in unison on the display screen in response to an input gesture from the interface device, rotating the plurality of images in unison on the display screen in response to an input gesture from the interface device, and zooming in and out of the plurality of images in unison on the display screen in response to an input gesture from the interface device. Figure 17-19 As shown in , operations of translating, rotating, and zooming the multiple images can provide a desired perspective view of the imaged biological tissue sample. For example, translating the multiple images can include sliding the images in a linear direction. Rotating the multiple images can be performed in a clockwise or counterclockwise direction. Zooming in on the multiple images can provide a closer view of an area of ​​the biological tissue sample. Zooming out on the multiple images can provide a distant view of the biological tissue sample.

[0108] In some embodiments, such as Figure 20 As shown in , the operation may further include removing one or more images from the plurality of images on the display screen to unclutter the display screen. For example, if an image depicts an undesirable or irrelevant view of a biological tissue sample, the image may be removed. In other embodiments, the operation may further include adding a new mode image to the display screen. The new mode image may be viewed in conjunction with other image modes.

[0109] Non-limiting examples of modes in which an image may be viewed may include various color channels, image filtering states, or edge detection states. Generally, there may be useful alterations to the original image that highlight certain features, which may provide a simultaneous view of important features of a diagnosis of interest to an expert reader.

[0110] In some embodiments, such as Figure 21 As shown in , the operation may further include rearranging the display pattern to form an alternative display pattern. The alternative display pattern may bring together image patterns for closer inspection. In other embodiments, such as in Figure 22 As shown in , the operation may further include stacking two or more modes to enhance image features. The stacking of two or more image modes may be responsive to an input gesture from an interface device. In some embodiments, the two or more image modes may be translucent.

[0111] In other embodiments, the operations may further include saving the display of the current examination as a saved template to facilitate display of additional multiple images in future examinations.

[0112] In one embodiment of the present invention, an expert reader can affect all images simultaneously by invoking an action on only one of the images so that all images respond jointly. Non-limiting exemplary input gestures and interface devices may include, but are not limited to, a mouse, a tactile sensor, an eye sensor, and an electronic camera. For example, an expert reader may use a mouse click to activate one of the images and then rotate the mouse wheel to affect the zoom magnification of the image. A mouse click and drag within the activated image may drag all images in the same direction. As another example, a tactile sensor may be used to perform changes to the selected image. The tactile sensor may provide rotation, translation, zooming, stacking, etc., which may be more sophisticated than a simple computer mouse.

[0113] Eye sensors can detect the expert reader's eye gestures, such as changing the center of visual attention, blinking, etc. Electronic cameras can witness specific gestures of the operator, such as hand movements, that indicate image translation, rotation, magnification, display rearrangement, image stacking and control of translucency during stacking, etc. In other embodiments, any sufficient and effective way of interacting with a device (such as a computer) can be used, with a preference for the simplest and most direct interaction to achieve the expert reader's goals.

[0114] In alternative embodiments, the method of simultaneously displaying multiple views of the same area may be used in the examination of multispectral surface imagery for remote sensing applications or for battlefield management.

[0115] A non-limiting example of a method for implementing the simultaneous display of multiple views of the same area of ​​a biological tissue sample on a display screen may be characterized as follows:

[0116] 1. Load data for biological tissue samples.

[0117] 2. Select a file from the file list.

[0118] 3. Display the six images from the selected file in a 3 column by 2 row display pattern.

[0119] 4. Select important markers.

[0120] 5. Display a heatmap of the markers for the image samples.

[0121] 6. Switch between original view, heatmap view, or individual marker view.

[0122] 7. Display hot spots of image samples.

[0123] 8. Align to the same coordinate system.

[0124] 9. Rotate, translate, or zoom in and out of the image.

[0125] 10. Combine FOV.

[0126] 11. Assign labels to areas of the sample being imaged.

[0127] 12. Rename the image.

[0128] 13. Add or delete images.

[0129] 14. Save the file.

[0130] Image preprocessing

[0131] In some embodiments, the present invention may utilize systems and methods for preprocessing biological slide images. It will be appreciated that any suitable system or method may be used to preprocess images. In one embodiment, a non-limiting example of a preprocessing system or method can be characterized as automatic field of view (FOV) selection based on the density of each cell marker in a whole-slide image. The operations described herein include, but are not limited to, reading individual marker images from demixed multiplex slides or from abnormally stained slides, and calculating tissue region masks from the individual marker images. A heat map for each marker can be determined by applying a low-pass filter to the individual marker image channels and selecting the top K highest intensity regions from the heat map as candidate FOVs for each marker. The candidate FOVs from the individual marker images can then be merged together. This merging can include either or both of: first registering all individual marker images to a common coordinate system and merging them via a morphological operation, based on input preferences or selections, to add all FOVs together into the same coordinate system, or adding only the FOVs from selected marker images. Subsequently, inverse registration is used to transfer all identified FOVs back to the original image to obtain a corresponding FOV image at high resolution. Without wishing to limit the present invention to any theory or mechanism, the systems and methods of the present invention may provide advantages such as being reproducible, unbiased to human readers, and more efficient.

[0132] In some embodiments, a system for quality control of automated whole-slide analysis includes an image acquisition system (102), a processor (105), and a memory (110) coupled to the processor. The memory is configured to store computer-readable instructions that, when executed by a processor, cause the processor to perform one or more of the following operations, including but not limited to: reading a high-resolution input image (231) from an image acquisition system (102); computing a low-resolution version of the high-resolution input image; reading a plurality of low-resolution image marker images from the image acquisition system (102), wherein each image marker image has a single color channel (232) of the low-resolution input image; computing a tissue region mask (233) corresponding to the low-resolution input image; computing a low-pass filtered image (234) of each image marker image (114); generating a mask-filtered image for each image marker image (113), where the mask-filtered image is the tissue region mask multiplied by the low-pass filtered image; identifying a plurality of candidate fields of view (FOVs) within each mask-filtered image (116); merging a subset of the plurality of candidate FOVs for each image marker image (117) into a plurality of merged FOVs; and depicting merged portions of the plurality of candidate fields of view on the input image.

[0133] In some embodiments, a heat map can be calculated for the masked filtered image. In some embodiments, the heat map includes applying colors to the masked filtered image, where low-intensity areas are designated as blue and higher-intensity areas are designated as yellow-orange and red. Any other suitable color or combination of colors can be used to designate low-intensity and high-intensity areas.

[0134] In some embodiments, generation of the tissue region mask includes one or more of the following operations (including but not limited to the following operations): calculating the illuminance (337) of the low-resolution input image (336), generating a illuminance image (338), applying a standard deviation filter to the illuminance image (339), generating a filtered illuminance image (340), and applying a threshold to the filtered illuminance image (341) such that pixels having an illuminance above a given threshold are set to 1 and pixels below the threshold are set to 0, thereby generating a tissue region mask (342).

[0135] In some embodiments, the tissue region mask is calculated directly from the high resolution input image. In this case, the tissue region mask can be converted to a lower resolution image before being applied to the filtered image mask image.

[0136] In some embodiments, the image marker image is obtained by demixing (111) multiple slides, where a demixing module uses a reference color matrix (112) to determine which colors correspond to individual color channels. In other embodiments, the image marker image is obtained from a single stain slide.

[0137] In some embodiments, the image registration process includes selecting an image marker image to serve as a reference image and calculating a transformation of each image marker to the coordinate system of the reference image. Methods for calculating the transformation of each image to the reference image are well known to those skilled in the art. In other embodiments, if the images are obtained by demixing multiple reference slides, no registration is required because all demixed images are already in the same coordinate system.

[0138] This subject disclosure provides systems and methods for automated field of view (FOV) selection. In some embodiments, the FOV selection is based on the density of each cell marker in a whole-slide image. The operations described herein include reading images for individual markers from demixed multiplex slides or from abnormally stained slides, and calculating tissue region masks based on the individual marker images. A masked image for each marker can be determined by applying a low-pass filter to the individual marker image channels and then applying the tissue region mask. The top K highest intensity regions from the masked image are selected as candidate FOVs for each marker. The candidate FOVs from the individual marker images are merged. This merging can include either or both of: first registering all individual marker images to a common coordinate system and merging them using a morphological operation, or adding only the FOVs from selected marker images, based on input preferences or selections. Subsequently, inverse registration is used to transfer all identified FOVs back to the original image to obtain corresponding FOV images at high resolution. Without wishing to be bound by any theory or mechanism, the systems and methods of the present invention may provide advantages such as being reproducible, unbiased to human readers, and more efficient. Thus, a digital pathology workflow for automated FOV selection according to the subject disclosure includes a computer-based FOV selection algorithm that automatically provides candidate FOVs that can be further analyzed by a pathologist or other evaluator.

[0139] For exemplary purposes, the operations described herein have been described in conjunction with the identification of immune cells and for use in immune score calculations. However, the systems and methods can be applied to any type of image of a cell or biological specimen, and to determining the type, density, and location of any type of cell or cell population. As used herein, the terms "biological specimen" and "biological tissue sample" can be used interchangeably. Furthermore, in addition to cancerous tissue and immune markers, the subject disclosure can be applied to any biological specimen or tumor of any disease or non-disease state, as well as images of biological specimens that have undergone any type of staining (such as images of biological specimens that have been stained with fluorescent and non-fluorescent stains). Moreover, one of ordinary skill in the art will recognize that the order of steps can be different from those described herein.

[0140] Figures 1A-1B A system 100 and a workflow for automatic FOV selection are respectively depicted according to an exemplary embodiment of the subject disclosure. Figure 1A , system 100 includes a memory 110 that stores a plurality of processing modules or logic instructions executed by a processor 105 coupled to a computer 101. Input from the image acquisition system 102 can trigger the execution of one or more of the plurality of processing modules. In addition to the processor 105 and the memory 110, the computer 101 also includes user input and output devices, such as a keyboard, a mouse, a stylus, and a display / touch screen. As will be explained in the discussion below, the processor 105 executes the logic instructions stored on the memory 110, including automatically identifying one or more FOVs in an image of a slice (containing a biological specimen, such as a tissue sample) that has been stained with one or more stains (e.g., fluorophores, quantum dots, reagents, tyramide, DAPI, etc.).

[0141] Image acquisition system 102 may include a detector system (such as a CCD detection system), a scanner or camera (such as a spectral camera), or a microscope or camera on a whole-slide scanner with a microscope and / or imaging components (the image acquisition system is not limited to the aforementioned examples). For example, a scanner may scan a biological specimen (which may be placed on a substrate such as a slide), and the image may be stored as a digitized image in the system's memory. Input information received from image acquisition system 102 may include information regarding the target tissue type or subject, as well as identification of the staining and / or imaging platform. For example, the sample may have been stained using a staining assay containing one or more different biomarkers associated with a chromogenic stain for brightfield imaging or a fluorophore for fluorescence imaging. The staining assay may use a chromogenic stain for brightfield imaging, an organic fluorophore, quantum dots, or an organic fluorophore in combination with quantum dots for fluorescence imaging, or any combination of stains, biomarkers, and a viewing or imaging device. Furthermore, a representative sample may be processed in a sample automated staining / assay platform to generate a stained sample. The input information can further include which and how many specific antibody molecules are bound to certain binding sites or targets on the tissue (such as tumor markers or biomarkers of specific immune cells). The selection of biomarkers and / or targets can be input into the system to determine the optimal combination of stains to be applied to the test. The additional information input to the system 100 can include any information related to the staining platform, including the concentration of the chemicals used in the staining, the reaction time of the chemicals applied to the tissue in the staining, and / or the pre-analysis conditions of the tissue (such as tissue age, fixation method, duration, how the sample is embedded, cut, etc.). Image data and other input information can be directly transmitted or can be provided via a network or via a user operating the computer 101.

[0142] For example, if the image is a multiplex image, a demixing module 111 may be executed to demix the image. The demixing module 111 demixes the image into individual marker color channels. The demixing module 111 may read from a reference color matrix database 112 to obtain a reference color matrix and use the reference color matrix to perform the demixing operation. If the image is a single-stained slide, the image may be used directly for FOV selection. In either case, a heat map calculation module 113 may be executed to evaluate a heat map for each individual marker image or single-stained image. The heat map maps the density of various structures or biomarkers on the whole slide image. To accomplish this, the heat map calculation module 113 may perform operations such as assigning colors to the low-pass filtered image processed by the low-pass filter module 114. A tissue region mask may also be applied to the low-pass filtered image. The heat map illustrates pixels according to their respective densities and therefore corresponds to the density of cell distribution in each image. For example, the heat map will distinguish high-density pixels from low-density pixels by illustrating higher-density pixels with warmer colors than those used for lower-density pixels. A local maximum filter module 115 may be executed to apply a local maximum filter to the low pass filtered image to obtain local maxima of the image. Subsequently, a front K FOV selection module 116 may be executed to select the front K regions with the highest density from the local maximum filtered image. The front K regions are designated as candidate FOVs for each image. For example, cells may be clustered together in high density regions, while they are more dispersed in low density regions. The FOVs from each image are merged together by a merge FOV module 117, which performs operations such as taking all FOVs or only taking FOVs from selected markers and merging them. A registration module 118 is called to transfer all images to the same coordinate system so that the coordinates of the FOVs in the same coordinate system can be directly added together.

[0143] As described above, modules comprise logic executed by processor 105. As used herein and throughout this disclosure, "logic" refers to any information in the form of instruction signals and / or data that can be applied to affect the operation of a processor. Software is an example of such logic. Examples of processors include computer processors (processing units), microprocessors, digital signal processors, controllers, and microcontrollers, among others. Logic can be formed by signals stored on a computer-readable medium, such as memory 110. In an exemplary embodiment, memory 110 can be random access memory (RAM), read-only memory (ROM), erasable / electrically erasable programmable read-only memory (EPROM / EEPROM), flash memory, and the like. Logic can also include digital and / or analog hardware circuits, such as hardware circuits that implement logical AND, OR, XOR, NAND, NOR, and other logical operations. Logic can be formed by a combination of software and hardware. On a network, logic can be programmed on a server or a complex of servers. A particular logic unit is not limited to a single logical location on the network. Furthermore, modules do not need to be executed in any specific order. Each module can call another module when it is needed to be executed.

[0144] exist Figure 1B An example workflow for FOV selection is depicted in . Figure 1B Where N represents the number of markers applied to the slide. For multiple slides 121, color unmixing 122 is performed, for example, according to the unmixing method disclosed in WO 2014 / 195193 A1, filed June 3, 2013, and entitled "Image Adaptive Physiologically Plausible Color Separation," the disclosure of which is incorporated herein by reference in its entirety. The method disclosed in patent application 61 / 943,265, filed February 21, 2014, and entitled "Group Sparsity Model for Image Unmixing," and PCT / EP2014 / 078392, filed December 18, 2014, which are incorporated herein by reference in their entirety, is an exemplary embodiment of the method utilized to obtain an image 123 for each marker. Otherwise, if the image is of a single stain slide, the scanned image 124 of the single stain slide for each marker is utilized as an automatic FOV selection system (such as in Figure 1A126 ). For example, a heat map calculation operation can be performed to calculate hot spots 125 from the image of each marker to generate a top candidate FOV 126 for each marker. The candidate FOVs 126 can be integrated 127 to generate a final FOV list 128. The final FOV list 128 includes a list of possible FOVs for selection by a pathologist to utilize for evaluating a biological specimen (e.g., immune cells).

[0145] As used herein and throughout this disclosure, a hotspot is an area containing a high density of labeled (i.e., stained) cells, for example, a hotspot can be cells from different types of images and markers (such as ISH, IHC, fluorescent, quantum dots, etc.). This subject disclosure uses immune cells in IHC images as an example to demonstrate this feature (as previously discussed, the present invention is not limited to immune cells in IHC images). In view of this subject disclosure, one of ordinary skill in the art can use various algorithms to find hotspots and use automatic hotspot selection as a module in the immune score calculation. An exemplary embodiment of this subject disclosure utilizes the automatic FOV selection operation described herein to solve the problem of avoiding biased manually selected FOVs. In order to automatically identify FOVs that may be of interest to a pathologist or other assessor, a heat map is calculated for each marker or image representing a single marker based on a low resolution image (e.g., a 5x magnification image).

[0146] Figure 2 Depicted is a heat map calculation according to an exemplary embodiment of the subject disclosure. Figure 2 The operations described in the figure illustrate how to use heat map calculation to identify hot spots. For example, given a single marker channel 232 of an input image 231, a low-pass filtered image 234 is used to generate a heat map 235, which essentially uses the low-pass filtered image 234 as input and applies a color map on top of it for visualization purposes. For example, red can correspond to high-intensity pixels in the low-pass filtered image and blue can correspond to low-intensity pixels. In view of this disclosure, other depictions of color and / or intensity may be obvious to those of ordinary skill in the art. A tissue region mask 233 can be created by identifying tissue regions and excluding background regions. This identification can be achieved through image analysis operations such as edge detection, etc. The tissue region mask 233 is used to remove non-tissue background noise in the image, such as non-tissue regions.

[0147] In About Figure 2 In the embodiment considered, the input image 231 is stained with the aid of a stain providing two channels, namely an FP3 channel and an HTX channel, and its corresponding counterstain. The two-channel image 231 is demixed, which provides demixed images 232 and 238 of the FP3 and HTX channels, respectively.

[0148] The demixed image 232 is then low pass filtered by means of a spatial low pass filter providing a low pass filtered image 234. Next, a heat map 235 may be added to the low pass filtered image 234 for visualization purposes.

[0149] Then pass Figure 3 The demixed image 238 is used to calculate the tissue region mask 233 using the method described in .

[0150] The low pass filtered image 234 with or without the added heat map 235 is then subjected to local maximum filtering, which provides a local maximum filtered image 236. The local maximum filtered image 236 includes many local maxima 239, such as Figure 2 The example considered here includes five local maxima 239.1-239.5 as depicted in . Next, a thresholding operation is performed on the local maximum filtered image 236, such as by applying a threshold on the local maximum filtered image 236 so that the thresholding operation only does not remove the local maxima 239.1 and 239.4 that exceed the threshold.

[0151] Alternatively, the local maxima 239 are ranked in a sorted list and only a number of K top local maxima are taken from the list, where K is 2 in the embodiment considered here for illustrative purposes, resulting in local maxima 239.1 and 239.4. Each of the local maxima 239 consists of a set of neighboring pixels.

[0152] This thresholding operation provides a thresholded image 237. Each of the local maxima 239.1 and 239.4 in the thresholded image 237 can define the location of a corresponding field of view 240.1 and 240.2, respectively. Depending on the embodiment, these fields of view 240.1 and 240.2 can be candidate fields of view for testing whether they can be merged with other fields of view in subsequent processing operations, such as those described below with respect to FIG. 6. The locations of the fields of view 240.1 and 240.2 are defined using the thresholded image 237 and its local maxima. However, the content of the fields of view is retrieved from the corresponding image region within the original multi-channel image 231 so that image analysis of the corresponding fields of view can be performed using the full picture information content.

[0153] Figure 3The following illustrates tissue mask calculation according to an exemplary embodiment of the subject disclosure, such as calculating tissue mask 233 from demixed image 238 using segmentation techniques. A linear combination 337 of the RGB channels 336 of the tissue RGB image is calculated to create a grayscale luminance image 338. The combination weights for the R, G, and B channels (e.g., 0.3, 0.6, 0.1 in 337) vary depending on the application. A 3-pixel by 3-pixel standard deviation filter 339 is applied to luminance image 338, resulting in a filtered luminance image 340. The filter size (e.g., 3 by 3, 5 by 5) varies depending on the application. Tissue mask 342 is a binary image obtained by thresholding 341 the filtered luminance image 340. For example, tissue mask 342 may include regions with pixel intensity values ​​greater than 1.5. The thresholding parameter MaxLum (e.g., 1.5, 2.0, 3.0) may vary depending on the application.

[0154] Figure 4 Depicts candidate FOVs according to an exemplary embodiment of the subject disclosure. Candidate FOVs 443 are selected from the top K highest density regions (also referred to as hotspots) of the heat map. For example, K may be selected from 5, 10, 15, 20, etc. A local maximum filter is applied to the low-pass filtered image 234 (refer to FIG. 2 ) with the added heat map 235. Figure 2 ) to provide a locally maximum filtered image 236. It should be noted that the heat map 235 is not necessary for processing but is used for visualization purposes. The local maximum filter is a function used to identify a region of constant value connected to a pixel and all of the outer boundary pixels have lower values. It can use 4 or 8 connected neighborhoods for 2D images. An implementation of this function can be obtained in Matlab (http: / / www.mathworks.com / help / images / ref / imregionalmax.html).

[0155] The local maximum is obtained as the average intensity within the connected area. The local maxima are classified to provide a classified list to generate a ranking of hotspots and the top K hotspots are reported. Thus, the image after the local maximum filter is thresholded. Alternatively, a predefined threshold is applied to the image after the local maximum filter so that all hotspots above the threshold are reported. The area returned by the local maximum filter calculation module is the location of the local maximum.

[0156] As described herein, different FOVs can be obtained for different marker images resulting from the demixing of multiple slides or from a single stain slide result. The FOVs are integrated to ensure that the same set of FOVs is referenced across different markers for each patient being diagnosed. There are several possible options for integrating the FOVs. Figures 5A-5B Depicts the merge of FOVs from all markers and from selected markers, respectively, according to an exemplary embodiment of the subject disclosure. Figure 5A As depicted in , all candidate FOVs from different marker images can be merged. In this alternative, different FOVs for different marker images can be selected and merged, as in Figure 5B As described in .

[0157] Furthermore, different FOVs for different marker images can be analyzed independently based on the user's needs. Figures 6A-6B Depicts integrating FOV according to an exemplary embodiment of the subject disclosure. Figure 6A , select all FOVs, and refer to Figure 6B , only the FOV corresponding to a specific marker is selected. Each circle 661 represents a possible FOV for a marker. Each point 662 in each circle 661 represents a local maximum point for each FOV. Each circle 661 can enclose a different marker. Line 663 corresponds to the interval between the tumor region and the non-tumor region. The FOV 664 outside the tumor region is excluded by morphological operations (such as union and intersection). The final FOV (i.e., the FOV selected for analysis) is the union of all FOVs from each marker, as given by Figure 5A and 5B The method described.

[0158] In some embodiments, the FOV can be a rectangle about the local maximum. In other embodiments, the FOV can be an arbitrary shape. In some embodiments, the FOV can be a boundary around the high intensity area.

[0159] Figure 6B Depicts specifying the most important markers for a given problem by the user and merging FOVs based on the selected markers. For example, suppose PF3 and CD8 are the most important markers. Image registration can be used to align all images of a single marker to the same coordinate system (e.g., the reference coordinate can be a slide slice in the middle of a tissue block or a slide with a specific marker). Each image can thus be aligned from its old coordinate system to a new reference coordinate system. The FOVs of selected markers (e.g., FP3 and CD8) from the individual marker images can be aligned to a common space and merged using morphological operations such as union and intersection to obtain a merged FOV ( Figure 6B FOV in 665). Figure 6CMorphological operations are shown. Assume that A is the FOV from the CD8 image and B is the FOV from FP3. We first superimpose A and B in the same coordinate system and obtain the superimposed area C by calculating the intersection of A and B. We then evaluate the ratio of the area of ​​C to the area of ​​A (or B). If the ratio is greater than a threshold of 9 (0.6, 0.8, etc.), we select that FOV; otherwise, we discard it. Inverse registration (i.e., aligning the registered image in the new coordinate system back to its original old coordinate system) can be used to map the merged FOV back to all single marker images for future analysis. FOV 664 outside the tumor area is excluded.

[0160] Figure 7 and 8 Depicts a user interface for image analysis using all marker views and individual marker views according to exemplary embodiments of the subject disclosure. In these exemplary embodiments, FOV selection can be performed using a user interface associated with a computing device. The user interface can have an all marker function ( Figure 7 ) and individual marker functions ( Figure 8 ). The marker functionality can be accessed by selecting from the tabs at the top of the user interface. Figure 7 When using the All Markers feature shown in , all markers can be viewed and heat map calculations, FOV selection, key marker selection, registration, and deregistration can be performed. In the All Markers view (i.e., a view that illustrates all markers side by side), options such as loading a list of image folders 771 (a) can be provided, where each folder contains all images, including multiple and single stains for the same case. Batch processing of all images in the list is allowed. Other options provided in the feature panel 772 may include linking axes for all images to zoom in and out simultaneously to view corresponding areas (b), selecting multiple FOVs (c), aligning images to a common coordinate system (d), and allowing the user to pick the most important markers for integrating FOVs (e). Colors can be drawn to indicate the marker from which the FOV is derived. Other options provided may include allowing the user to switch between heat map view and IHC view 774, and calculating 773 a heat map for each image.

[0161] Figure 8Depicted is an individual marker view or single marker view showing the final selected FOV for each marker. Features provided in this view may include a thumbnail 881 showing a whole slide image, with the FOV annotated with a box in the thumbnail image and a text number near the box indicating the index of the FOV. Other features may include allowing the user to select from a list of FOVs 883 to delete unneeded FOVs using a checkbox, displaying a high-resolution image of the selected FOV 882, saving the image of each FOV at its original resolution in a local folder (d), and allowing the user to assign a label to each FOV (e). The label may be a region associated with the FOV, such as a peripheral region, a tumor region, a lymphocyte, etc. One of ordinary skill in the art will recognize that these exemplary interfaces may vary for various applications and across various computing technologies, and that different versions of the interface may be used, provided that the novel features described herein can be implemented in light of this disclosure.

[0162] Thus, the systems and methods disclosed herein provide automated FOV selection, which has been found to be important for analyzing biological specimens and useful in calculating tissue analysis scores (e.g., in immune score calculations). The operations disclosed herein overcome known shortcomings in the prior art (such as non-reproducible FOV selection and bias in manual FOV selection by human readers) because automated FOV selection can provide the FOV via a computer without relying on manual selection by a human reader. When combined with automated immune cell counting and data analysis, the disclosed operations enable a fully automated workflow that takes one or more scanned images or image data as input and outputs a final clinical outcome prediction. The systems and methods disclosed herein provide automated FOV selection, which has been found to be important for analyzing biological specimens and useful in calculating tissue analysis scores (e.g., in immune score calculations). The operations disclosed herein overcome known shortcomings in the prior art (such as non-reproducible FOV selection and bias in manual FOV selection by human readers) because automated FOV selection can provide the FOV via a computer without relying on manual selection by a human reader. When combined with automated immune cell counting and data analysis, the disclosed operations allow for a complete automated workflow that takes one or more scanned images or image data as input and outputs a final clinical outcome prediction.

[0163] Figure 9A digital pathology workflow for immune score calculation according to an exemplary embodiment of the subject disclosure is depicted. This embodiment illustrates how the automated FOV selection method disclosed herein can be utilized in the immune score calculation workflow. For example, after a slide is scanned 991 and FOVs are selected 992 according to the operations disclosed herein, automated detection of different cell types in each FOV can be performed 993. Automated cell detection techniques, such as those disclosed in U.S. Patent Application Serial No. 62 / 002,633 filed on May 23, 2014, and PCT / EP2015 / 061226, entitled “Deep Learning for Cell Detection,” which are incorporated herein by reference in their entirety, are utilized to obtain the detected cells in one exemplary embodiment. Furthermore, features associated with one or more cells detected for each biological specimen (e.g., tissue sample, etc.) can be extracted 994 (e.g., features associated with the number and / or type of identified cells). Such features can include the number and ratio of different cell types in different FOVs associated with different regions in the tissue image, such as a tumor region and a peripheral region. These features can be used to train 995 classifiers (such as random forests and support vector machines) and classify each case into different outcome categories (such as the likelihood of recurrence or no recurrence).

[0164] Figure 10 A process flow diagram for an exemplary embodiment of the present invention is depicted. An input image is received from an image acquisition system (1001). In addition, a series of low-resolution marker images are received from the image acquisition system (1004). The marker images may be derived by unblending the high-resolution image or may be received as a single stain slide image. The low-resolution input image is used to calculate a tissue region mask (1003) that indicates which portions of the image contain tissue of interest. The low-resolution image marker image is low-pass filtered to produce a filtered image marker image (1005). The tissue region mask is then applied to the low-pass filtered image to block out regions of no interest (reducing them to zero). This produces a mask-filtered image for each marker (1006). A local maximum filter is applied to the maximum-filtered image to identify local maxima (1007). The top K local maxima are selected (1008), and a field of view is defined for each local maximum (1009). The FOV for each image is then merged (1010) by transferring all images to a common coordinate system and overlaying them, and combining any overlaid fields of view. The merged field of view is then transferred back to the original image coordinate system, and regions are extracted from the high-resolution input image for analysis.

[0165] FIG11 illustrates a different process flow for another exemplary embodiment of the present invention. The process flow is divided into a FOV generation step (1100), as shown in FIG11a, and a field of view merging step (1124), as shown in FIG11b. In the FOV generation step, a single stain image is received from an image acquisition system (1101). The image is low-pass filtered (1102). In some embodiments, the image can be converted to a lower resolution (1103), which speeds up processing. In some embodiments, if it has not already been reduced to a single color channel, a de-blending step (1104) can be applied to extract the color channel of interest from the single stain slide, thereby producing a single marker image (1108). In some embodiments, an HTX image (1105) can also be generated. The single marker image is then segmented to identify features of interest. A tissue region mask (1110) is generated from the segmented image. In some embodiments, the single marker image can be made visible using a heat map (1107) by assigning colors to regions of different intensities in the single marker image. A tissue region mask (1110) is then applied to the individual marker images (1111), resulting in a foreground image (1112) that displays the intensity of the marker image only in the tissue region of interest. This foreground image is passed through a local maximum filter (1113) to identify peaks in intensity. Candidate FOV coordinates are identified as the top K peaks of the local maximum filtered image (1114). Finally, a region is defined around each candidate FOV coordinate (1115) to obtain a list of candidate FOVs (1116). These operations are performed for each individual stain slide.

[0166] In the FOV merging step (1124), a list of all candidate FOVs for the various single stain slides is obtained (1117). The images are registered to a single coordinate system by selecting one image as the reference image and transforming the other images to match the reference image (1118). The candidate FOV coordinates are then transformed accordingly to obtain an aligned candidate FOV list (1119). The FOVs are then superimposed and merged (1120) to obtain a unified FOV list for all images (1121). An inverse registration is then performed (1122) to transform the unified FOV back into each of the original coordinate systems of the original single stain images (1123). The FOVs can then be displayed on the original single stain slide.

[0167] Figure 12A process flow of an alternative embodiment of the present invention is shown using multiple slides as input (1201). In the FOV generation step, the multiple slides are received from the image acquisition system (1201). The image is low-pass filtered (1202). In some embodiments, the image can be converted to a lower resolution (1203), which speeds up processing. In this embodiment, a deblending step (1204) is applied to extract the color channel of interest from the multiple slides, thereby generating multiple single marker images (1208). In some embodiments, an HTX image (1205) can also be generated. The first single marker image is then segmented (1209) to identify features of interest. A tissue region mask (1210) is generated from the segmented image. In some embodiments, the single marker images can be made visible (1265) by assigning colors to regions of different intensities in the single marker images using a heat map (1207). The tissue region mask (1210) is then applied to the single marker image (1210), thereby generating a foreground image (1212) that shows the intensity of the marker image only in the tissue region of interest. This foreground image is passed through a local maximum filter (1213) to identify peaks in intensity. Candidate FOV coordinates are identified as the top K peaks of the local maximum filtered image (1214). Finally, a region is defined around each candidate FOV coordinate (1215) to obtain a list of candidate FOVs (1216). These operations are performed sequentially for each individual stain slide. The FOV merging step is performed as in Figure 11b.

[0168] Figure 13Yet another process flow of an alternative embodiment of the present invention is shown using a single stain image (1301) as input. The image is low pass filtered (1302). In some embodiments, the image can be converted to a lower resolution (1303), which speeds up processing. In some embodiments, if it has not yet been reduced to a single color channel, a deblending step (1304) can be applied to extract the color channel of interest from the single stain slide, thereby producing a single marker image (1308). In some embodiments, an HTX image (1305) can also be generated. In other embodiments, the single marker image can be made visible (1306) using a heat map (1307) by assigning colors to areas of different intensities in the single marker image. In one embodiment, the lower resolution image is segmented (1309) to identify features of interest. A tissue region mask (1310) is generated from the segmented image, and the masking operation is then applied (1311) to the segmented image, thereby producing a foreground image (1312) that shows the intensity of the marker image only in the tissue region of interest. In another embodiment, a masking operation (1311) is applied to the single marker image (1308), resulting in a foreground image (1312). In either embodiment, this foreground image (1312) is passed through a local maximum filter (1313) to identify peaks in intensity. Candidate FOV coordinates are identified as the top K peaks of the local maximum filtered image (1314). Finally, a region around each candidate FOV coordinate is defined (1315) to obtain a list of candidate FOVs (1316). These operations are performed for each individual stain slide. The FOV merging step is performed as in Figure 11b.

[0169] For illustrative purposes, the computer-implemented method for automatic FOV selection according to the present invention has been described in conjunction with the identification of immune cells and for use in immune score calculation. However, the computer-implemented method for automatic FOV selection according to the present invention is applicable to any type of cell image or biological specimen image and is suitable for determining the type, density, and location of any type of cell or cell population. Furthermore, in addition to medical applications such as anatomical or clinical pathology, prostate / lung cancer diagnosis, and the like, the same method can be implemented to analyze other types of samples, such as remote sensing of geological or astronomical data. The operations disclosed herein can be ported to a hardware graphics processing unit (GPU), enabling multi-threaded parallel implementation.

[0170] Figure 23 A biopsy tissue sample 10 is shown that has been obtained from a tissue region of a patient. The tissue sample 10 is sliced ​​into adjacent tissue slices, such as in Figure 231 , 2, 3 and 4 are shown in FIG. The tissue slices may have a thickness in the micrometer range, such as between 1 μm - 10 μm, for example 6 μm.

[0171] The tissue section is stained with a single stain, a stain and a counterstain, or a plurality of stains. In this way, an image 231 stained with a stain and a counterstain is obtained (see Figure 2 ) and multi-channel images5.

[0172] can be obtained from a tissue section that has been stained with multiple stains (e.g. Figure 1B A multi-channel image 5 is obtained from stained tissue sections 1, 2, 3, and 4 of a multiple slide 121 of the image processing system. In addition, other images can be obtained from the stained tissue sections (such as single stain images 6 and 7). These images 231, 5, 6, and 7 can be stored in the electronic memory of the image processing system (such as the electronic memory of the computer 101 (refer to Figure 1A )), the image processing system can be a server computer.

[0173] Automatic field of view definition may be performed with respect to one or more of the plurality of images (such as with respect to image 231), which results in a field of view defined in accordance with Figure 2 The thresholded images 237 of the fields of view 240.1 and 240.2 are indicated by the corresponding rectangular boxes of the embodiment of FIG. 5. Demixing the image 5 provides a set of demixed images 5.1, 5.2 and 5.3, assuming without limiting the generality that N=3 (refer to FIG. Figure 1B ). It is noted that the demixed images 5.1, 5.2 and 5.3 share exactly the same coordinate system, since they are all obtained from the same multi-channel image 5 so that no image registration or alignment is required for the set of images. The additional images 6 and 7 may or may not have undergone image processing operations.

[0174] Images 231 / 237, 5, 6, and 7 are then registered and aligned using an image registration algorithm. For example, multi-channel image 5 is selected as the reference image for performing the image registration algorithm. The image registration algorithm generates a geometric transformation of each of the other images (i.e., images 231 / 237, 6, and 7) with respect to multi-channel image 5. Using multi-channel image 5 as the reference image for registration has the advantage that only three alignment operations need to be performed in the example considered here. In comparison, when, for example, Figure 7 When selected as the reference image, five alignment operations will be required to transform images 231 / 237, 5.1, 5.2, 5.3 and 6 to align with image 7. Therefore, selecting multi-channel image 5 as the reference greatly reduces the computational burden and lowers the latency for image alignment.

[0175] For example, a mapping is generated for each of the other images 231 / 237, 6, and 7 to the reference image 5, such as a mapping for mapping each pixel of the image 231 / 237 to a corresponding pixel in the image 5, a mapping for mapping each pixel of the image 6 to a corresponding pixel in the multi-channel image 5, and so on. In the example considered here, this results in three mappings. It is noted that the mapping for mapping the image 231 / 237 to the multi-channel image 5 can be obtained using either the image 231 or the image 237, because due to the Figure 2 The deblending step is performed so that both images share the same coordinate system.

[0176] The geometric transformation obtained as a result of the image registration (ie a mapping in the example considered here) is then used to align the images 237, 6 and 7 with respect to the reference image (ie the multi-channel image 5 / demixed images 5.1, 5.2 and 5.3).

[0177] These aligned images are displayed on a display 8, such as the display of computer 101 (see the embodiment of FIG. 1 ), or on a display of a mobile battery-powered telecommunication device (such as a smartphone) running, for example, an Android or iOS operating system. In the latter case, images 237, 5.1, 5.2, 5.3, 6, 7, along with the geometric transformation (e.g., a mapping) obtained from the image registration and metadata indicating fields of view 240.1 and 240.2, are transmitted to the mobile battery-powered telecommunication device via a telecommunication network (such as a mobile cellular digital telecommunication network), for example, based on the GSM, UMTS, CDMA, or Long Term Evolution standards. Display 8 may be touch-sensitive, enabling the input of commands via a graphical user interface of computer 101 or the telecommunication device using gesture recognition.

[0178] In one embodiment, the user can select one of the fields of view by touching the corresponding geometric object (i.e., rectangular box) that symbolizes the field of view. Figure 23 By way of example only, this may be the field of view 240.1 on which the user places one of his or her fingers 14. In response to this gesture, the field of view is magnified (also as in Figure 23 ) to perform magnification in image transformation.

[0179] An identical magnification transformation is performed synchronously with respect to the other images 5.1, 5.2, 5.3, 6, and 7. Field of view 240.1 corresponds to image portions 9, 10, 11, 12, and 13 in images 5.1, 5.2, 5.3, 6, and 7, respectively. These image portions 9 through 13 are given by corresponding geometric transformations (i.e., mappings) obtained from image registration. In response to a user gesture (i.e., touching field of view 240.1 with finger 14), the magnification image transformation performed with respect to field of view 240.1 is also performed synchronously with respect to image portions 9 through 13.

[0180] Figure 24 An alternative embodiment using a pinch gesture to zoom in or out is shown. A user can select a portion of one of the images (such as image 5.1) by placing two fingers 14 and 15 on display 8, thereby defining a rectangular area 16. This rectangular area 16 corresponds to co-localized image regions 17 to 21 in the other images 237, 5.2, 5.3, 6, and 7, respectively, given by the geometric transformation (i.e., mapping) obtained from image registration. Regions 18 and 19 are identical to region 16 because images 5.1, 5.2, and 5.3 share the same coordinate system.

[0181] By Figure 24 As illustrated in FIG, by moving fingers 15 and 14 further apart, zooming in is performed with respect to region 16 providing enlarged image portion 16′ and also synchronously with respect to other co-located regions 17-21 providing enlarged regions 17′, 18′, 19′, 20′, and 21′. Zooming out can similarly be performed by reducing the distance between fingers 14 and 15.

[0182] A computer typically includes known components such as a processor, an operating system, system memory, memory storage devices, input / output controllers, input / output devices, and a display device. Those skilled in the relevant art will also understand that there are many possible computer configurations and components and that a computer may also include cache memory, data backup units, and many other devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, and the like. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, and the like. A display device may include one that provides visual information, which may typically be logically and / or physically organized as an array of pixels. An interface controller may also be included, which may include any of a variety of known or future software programs for providing input and output interfaces. For example, an interface may include what is commonly referred to as a "graphical user interface" (often referred to as a GUI), which presents one or more graphical representations to a user. An interface typically enables user input to be accepted using selection or input methods known to those skilled in the relevant art. The interface may also be a touchscreen device. In the same or alternative embodiments, applications on a computer may employ interfaces, including those known as "command line interfaces" (often referred to as CLIs). A CLI typically provides text-based interaction between an application and a user. Typically, a command line interface presents output and receives input in the form of lines of text via a display device. For example, some implementations may include what are referred to as "shells" (such as the Unix Shell known to those skilled in the art or Microsoft Windows PowerShell, which employs an object-oriented programming framework such as the Microsoft .NET Framework).

[0183] Those of ordinary skill in the relevant art will recognize that this interface can comprise one or more GUI, CLI or its combination.Processor can comprise commercially available processor, such as Celeron, Core or Pentium processor manufactured by Intel Corporation, SPARC processor manufactured by Sun Microsystems, Athlon, Sempron, Phenom or Opteron processor manufactured by AMD Corporation, or it can be available or will become one of available other processors. Some embodiments of processor can comprise being referred to as multi-core processor and / or making it possible to adopt those of parallel processing technology in single or multi-core configuration. For example, multi-core architecture generally comprises two or more processor " execution cores ".In this example, each execution core can be executed as an independent processor realizing the parallel execution of multiple threads. In addition, those of ordinary skill in the relevant art will recognize that can configure processor in the architecture that is generally referred to as 32 or 64-bit architecture or other architecture configurations that are now known or can be developed in the future.

[0184] The processor typically executes an operating system, which may be, for example, a Windows-type operating system from Microsoft Corporation; the Mac OS X operating system from Apple Computer; a Unix or Linux-type operating system available from a number of vendors or referred to as open source; another or future operating system; or some combination thereof. The operating system interfaces with the firmware and hardware in a well-known manner and facilitates the processor in coordinating and executing the functions of various computer programs that may be written in a variety of programming languages. The operating system, typically in cooperation with the processor, coordinates and executes the functions of the other components of the computer. The operating system also provides scheduling, input-output control, file and data management, memory management, communication control, and related services, all in accordance with known techniques.

[0185] System memory can include any of a variety of known or future memory storage devices that can be used to store desired information and accessed by the computer. Computer-readable media can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Examples include any commercially available random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), digital versatile disks (DVDs), magnetic media (such as resident hard drives or magnetic tapes), optical media (such as read-and-write compact disks), or other memory storage devices. Memory storage devices can include any of a variety of known or future devices, including compact disk drives, magnetic tape drives, removable hard drives, USB or flash drives, or disk drives. These types of memory storage devices typically read from and / or write to program storage media, such as compact disks, magnetic tapes, removable hard drives, USB or flash drives, or floppy disks. Any of these program storage media, or other program storage media currently in use or that may be developed later, can be considered computer program products. As will be appreciated, these program storage media typically store computer software programs and / or data. Computer software programs (also referred to as computer control logic) are typically stored in system memory and / or program storage devices used in conjunction with memory storage devices. In some embodiments, a computer program product is described that includes a computer-usable medium having control logic (computer software program, including program code) stored therein. When executed by a processor, the control logic causes the processor to perform the functions described herein. In other embodiments, some functions are primarily implemented in hardware, using, for example, a hardware state machine. Implementing a hardware state machine to perform the functions described herein will be readily apparent to those skilled in the relevant art. An input-output controller may include any of a variety of known devices for receiving and processing information from a user (whether human or machine, whether local or remote). Such devices include, for example, modem cards, wireless cards, network interface cards, sound cards, or other types of controllers for any of a variety of known input devices. An output controller may include a controller for any of a variety of known display devices for presenting information to a user (whether human or machine, whether local or remote). In the presently described embodiments, the functional elements of the computer communicate with each other via a system bus. Some embodiments of the computer may communicate with some functional elements using a network or other type of remote communication.As will be apparent to those skilled in the relevant art, the instrument control and / or data processing application (if implemented in software) can be loaded into and executed from system memory and / or memory storage devices. All or part of the instrument control and / or data storage application can also reside in a read-only memory or similar device such as a memory storage device, eliminating the need to first load the instrument control and / or data processing application via an input-output controller. Those skilled in the relevant art will appreciate that, as advantageous for execution, the instrument control and / or data processing application or part thereof can be loaded by the processor into system memory or cache memory, or both, in a known manner. Furthermore, the computer can include one or more library files, experimental data files, and an internet client stored in the system memory. For example, the experimental data can include data related to one or more experiments or tests (such as detected signal values ​​or other values ​​associated with one or more sequencing by synthesis (SBS) experiments or processes). Furthermore, the internet client can include an application that enables access to remote services on another computer using a network, and can include, for example, what are commonly referred to as "web browsers." In this example, some commonly used web browsers include Microsoft Internet Explorer available from Microsoft Corporation, Mozilla Firefox from Mozilla Corporation, Safari from Apple Computer, Inc., Google Chrome from Google, Inc., or other types of web browsers currently known in the art or developed in the future. Moreover, in the same or other embodiments, the Internet client can include, or can be an element of, a specialized software application that enables access to remote information via a network (such as a data processing application for biological applications).

[0186] The network may include one or more of many various types of networks known to those of ordinary skill in the art. For example, the network may include a local or wide area network that may employ what is commonly referred to as the TCP / IP protocol suite for communication. The network may include a network comprising a worldwide system of interconnected computer networks commonly referred to as the Internet, or may also include various intranet architectures. Those of ordinary skill in the relevant art will also recognize that some users in a networked environment may prefer to employ what are commonly referred to as "firewalls" (sometimes also referred to as packet filters or boundary protection devices) for controlling information traffic to and from hardware and / or software systems. For example, a firewall may include hardware or software elements or a combination thereof and is typically designed to implement security policies implemented by users (such as, for example, network administrators, etc.).

[0187] The foregoing disclosure of exemplary embodiments of the subject disclosure has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the subject disclosure to the precise forms disclosed. In light of the above disclosure, many variations and modifications of the embodiments described herein will be apparent to those of ordinary skill in the art. The scope of the subject disclosure is limited solely by the claims appended hereto and their equivalents.

[0188] In addition, in the representative embodiment describing the present disclosure, the method and / or process of the present disclosure may be presented as a specific sequence of steps in this specification. However, insofar as the method or process does not rely on the specific sequence of the steps set forth herein, the method or process should not be limited to the specific sequence of the steps described. As those of ordinary skill in the art will recognize, other sequences of steps may also be possible. Therefore, the specific sequence of the steps set forth in this specification should not be interpreted as a restriction to the claims. In addition, the claims for the method and / or process of the present disclosure should not be limited to the steps performed in the order written, and those skilled in the art can easily recognize that the sequence can change and still remain within the spirit and scope of the present disclosure.

Claims

1. A system for simultaneously displaying multiple views of the same area of ​​an object of interest, the system comprising: processor; as well as a memory coupled to the processor, the memory storing computer-readable instructions that, when executed by the processor, cause the processor to perform operations comprising: receiving a plurality of pre-processed images, the pre-processed images depicting a plurality of views of a same region corresponding to at least a portion of the object of interest, wherein each pre-processed image is associated with: (i) an image viewing mode, and (ii) metadata describing the pre-processed image with respect to a Global Standard Reference Image (GSRF); determining a display image template with respect to a global standard reference image, said display image template facilitating simultaneous display of said plurality of pre-processed images; generating a set of displayable images, wherein each displayable image in the set of displayable images is generated based on the display image template to map pixel positions of the preprocessed image to corresponding pixel positions of the displayable image; arranging the set of displayable images into a display pattern for viewing on a display screen; as well as causing the arranged set of displayable images to be displayed on the display screen, The operation of generating the set of displayable images further comprises: constructing a copy of the display image template and an affine partial mapping of the display image template; processing the affine partial map with a first affine map of a pre-processed image to generate a composite map, wherein the composite map indicates pixel locations of the pre-processed image to corresponding pixel locations of the displayable image; and generating a displayable image from the set of displayable images, wherein the displayable image comprises a set of pixels, and wherein the displayable image is generated based on operations comprising: For each image pixel in a set of image pixels: mapping positions of pixels of the image to corresponding positions of pixels in the preprocessed image using the composite map; estimating pixel values ​​of adjacent pixels in the preprocessed image based on the corresponding positions; and The estimated pixel value of the neighboring pixel is assigned as the pixel value of the image pixel.

2. The system of claim 1 , wherein the display image template is a reference image frame indicating positions of pixels in a preprocessed image of the plurality of preprocessed images, and wherein the metadata of each preprocessed image comprises a preprocessed image local reference image frame (PI-LRF).

3. The system of claim 2 , wherein the GSRF is a fixed reference image system that indicates a spatial relationship between two or more of the plurality of pre-processed images by defining an affine mapping between each pre-processed image local reference system and the global standard reference image system, and wherein the metadata further includes a first affine mapping between the PI-LRF and the GSRF.

4. The system of claim 3, wherein the operations further comprise: receiving a user action to manipulate a displayable image in the arranged set of displayable images; and All other displayable images in the arranged set of displayable images are manipulated in unison with the displayable image based on the user action.

5. The system according to claim 3, wherein the operation of determining the display image template comprises: Create a pixel grid for displaying the image; constructing a display image local reference frame DI-LRF indicating pixel positions associated with the display image pixel grid, wherein the DI-LRF corresponds to the display image template; The position, orientation, and magnification of the DI-LRF were determined with reference to the GSRF; Compute the affine transformation that maps the pixel positions of the DI-LRF to the pixel positions of the GSRF; and An affine partial map is generated for the display image template based on the calculated affine transformation.

6. The system according to claim 3, wherein: The object of interest is a biological tissue sample.

7. The system according to claim 3, wherein: Generating the set of displayable images also includes performing a non-linear correction on each pre-processed image in the plurality of pre-processed images.

8. The system according to claim 3, wherein: The operations also include: (i) increasing a magnification level of the set of displayable images in unison on the display screen in response to a user action on the displayable images, (ii) moving the set of displayable images in unison on the display screen in response to a user action on the displayable images, (iii) rotating the set of displayable images in unison on the display screen in response to a user action on the displayable images, or (iv) a combination thereof.

9. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a system for simultaneously displaying multiple views of the same area of ​​an object of interest, cause the system to perform a method comprising: receiving a plurality of pre-processed images, the pre-processed images depicting a plurality of views of a same region corresponding to at least a portion of the object of interest, wherein each pre-processed image is associated with: (i) an image viewing mode, and (ii) metadata describing the pre-processed image with respect to a Global Standard Reference Image (GSRF); determining a display image template with respect to a global standard reference image, said display image template facilitating simultaneous display of said plurality of pre-processed images; generating a set of displayable images, wherein each displayable image in the set of displayable images is generated based on the display image template to map pixel positions of the preprocessed image to corresponding pixel positions of the displayable image; arranging the set of displayable images into a display pattern for viewing on a display screen; as well as causing the arranged set of displayable images to be displayed on the display screen, Wherein generating the set of displayable images further comprises: constructing a copy of the display image template and an affine partial mapping of the display image template; processing the affine partial map with a first affine map of the pre-processed image to generate a composite map, wherein the composite map indicates pixel locations of the pre-processed image to corresponding pixel locations of the displayable image; and generating a displayable image from the set of displayable images, wherein the displayable image comprises a set of pixels, and wherein the displayable image is generated based on operations comprising: For each image pixel in a set of image pixels: mapping positions of pixels of the image to corresponding positions of pixels in the preprocessed image using the composite map; estimating pixel values ​​of adjacent pixels in the preprocessed image based on the corresponding positions; and The estimated pixel value of the neighboring pixel is assigned as the pixel value of the image pixel.

10. The non-transitory computer-readable medium of claim 9, wherein the display image template is a reference image frame that indicates positions of pixels in a pre-processed image of the plurality of pre-processed images, and wherein the metadata of each pre-processed image includes a pre-processed image local reference image frame (PI-LRF).

11. The non-transitory computer-readable medium of claim 10 , wherein the GSRF is a fixed reference image system that indicates a spatial relationship between two or more of the plurality of pre-processed images by defining an affine mapping between each pre-processed image local reference system and the global standard reference image system, and wherein the metadata further includes a first affine mapping between the PI-LRF and the GSRF.

12. The non-transitory computer-readable medium of claim 11 , wherein determining the display image template comprises: Create a pixel grid for displaying the image; constructing a display image local reference frame DI-LRF indicating pixel positions associated with the display image pixel grid, wherein the DI-LRF corresponds to the display image template; The position, orientation, and magnification of the DI-LRF were determined with reference to the GSRF; Compute the affine transformation that maps the pixel positions of the DI-LRF to the pixel positions of the GSRF; and An affine partial map is generated for the display image template based on the calculated affine transformation.

13. The non-transitory computer-readable medium of claim 11, wherein: The object of interest is a biological tissue sample.

14. A computer-implemented method comprising: receiving a plurality of pre-processed images, the pre-processed images depicting a plurality of views of a same region corresponding to at least a portion of an object of interest, wherein each pre-processed image is associated with: (i) an image viewing mode, and (ii) metadata describing the pre-processed image with respect to a Global Standard Reference Image (GSRF); determining a display image template with respect to the global standard reference image, the display image template facilitating simultaneous display of the plurality of pre-processed images; generating a set of displayable images, wherein each displayable image in the set of displayable images is generated based on the display image template to map pixel positions of the preprocessed image to corresponding pixel positions of the displayable image; arranging the set of displayable images into a display pattern for viewing on a display screen; as well as causing the arranged set of displayable images to be displayed on the display screen, Wherein generating the set of displayable images further comprises: constructing a copy of the display image template and an affine partial mapping of the display image template; processing the affine partial map with a first affine map of the pre-processed image to generate a composite map, wherein the composite map indicates pixel locations of the pre-processed image to corresponding pixel locations of the displayable image; and generating a displayable image from the set of displayable images, wherein the displayable image comprises a set of pixels, and wherein the displayable image is generated based on operations comprising: For each image pixel in a set of image pixels: mapping positions of pixels of the image to corresponding positions of pixels in the preprocessed image using the composite map; estimating pixel values ​​of adjacent pixels in the preprocessed image based on the corresponding positions; and The estimated pixel value of the neighboring pixel is assigned as the pixel value of the image pixel.

15. The computer-implemented method of claim 14 , wherein the display image template is a reference image frame indicating positions of pixels in a preprocessed image of the plurality of preprocessed images, and wherein metadata of each preprocessed image comprises a preprocessed image local reference image frame (PI-LRF), and wherein the object of interest is a biological tissue sample.

16. The computer-implemented method of claim 15 , wherein the GSRF is a fixed reference image frame that indicates a spatial relationship between two or more of the plurality of pre-processed images by defining an affine mapping between each pre-processed image local reference frame and the global standard reference image frame, and wherein the metadata further comprises a first affine mapping between the PI-LRF and the GSRF.

17. The computer-implemented method of claim 16, wherein determining the display image template comprises: Create a pixel grid for displaying the image; A display image local reference frame DI-LRF is constructed indicating pixel positions associated with the display image pixel grid, wherein DI-LRF corresponds to the display image template; The position, orientation, and magnification of the DI-LRF were determined with reference to the GSRF; Compute the affine transformation that maps the pixel positions of the DI-LRF to the pixel positions of the GSRF; and An affine partial map is generated for the display image template based on the calculated affine transformation.

Images