Systems and methods for biological sample imaging and analysis

Abstract

Systems and methods for imaging and, in some instances, analyzing biological samples such as biological tissue are described. The systems and methods may be directed to the interrogation of biological samples labeled with multiple markers (e.g., fluorophores and / or chromophores) corresponding to various biomarkers and / or cellular or subcellular structures within the sample. Some such systems and methods may combine certain optics (e.g., configurationally-fixed optics for fluorescence imaging and / or actuatable filters for chromogenic imaging) with multi-color sensors (e.g., RGB sensors) to facilitate the imaging and analysis of a sample. The systems and methods may also promote advantageous techniques for segmenting images of components of the biological sample. In some instances, the methods can assist users (e.g., pathologists) in scoring samples for indications such as diseases. The systems and methods may, in some instances, promote economical, accurate, and / or higher-throughput imaging and analysis of the sample.

Description

[0001] SYSTEMS AND METHODS FOR BIOLOGICAL SAMPLE IMAGING AND ANALYSIS

[0002] RELATED APPLICATIONS

[0003] This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 562,146, filed March 6, 2024, and entitled “Multiplex Immunofluorescence Imaging Method,” which is incorporated herein by reference in its entirety for all purposes.

[0004] TECHNICAL FIELD

[0005] Systems and methods for imaging and, in some instances, analyzing biological samples such as biological tissue are described.

[0006] BACKGROUND

[0007] Various techniques have been developed for imaging and analyzing biological samples. For example, multiplexed immunofluorescence and multiplexed immunohistochemistry are methods for revealing the presence, density, arrangement, and / or functional state, among other things, of analytes such as cells and / or biomarkers in healthy and diseased tissue.

[0008] Accordingly, improved systems and methods for imaging and / or analyzing biological samples are desirable.

[0009] SUMMARY

[0010] Systems and methods for imaging and, in some instances, analyzing biological samples such as biological tissue are described. The subject matter of the present invention involves, in some cases, interrelated products, alternative solutions to a particular problem, and / or a plurality of different uses of one or more systems and / or articles.

[0011] According to some embodiments, systems for detecting light emitted from a sample are described. In some embodiments, the system comprises: a sensor; and optics comprising an optical block, wherein the optics are configured to: receive excitation light from an excitation source; direct, via the optical block, the excitation light in at least n excitation bands to at least a portion of a sample on a sample holder when the sample and the sample holder are present; receive fluorescence light emitted from the sample; filter, via the optical block, the received fluorescence light to produce filtered fluorescence light having at least n emission bands; and direct, via the optical block, the filtered fluorescence light to a detector comprising the sensor when the detector is present, such that the filtered fluorescence light is incident upon sensing elements of the sensor; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

[0012] In some embodiments, the system comprises a sensor; and optics comprising one or more filters, wherein the optics are configured to, in a single configuration of the one or more filters: receive excitation light from an excitation source; direct the excitation light in at least n excitation bands to at least a portion of a sample on a sample holder when the sample and the sample holder are present; receive fluorescence light emitted from the sample; filter the received fluorescence light to produce filtered fluorescence light having at least n emission bands; and direct the filtered fluorescence light to a detector comprising the sensor when the detector is present, such that the filtered fluorescence light is incident upon sensing elements of the sensor; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

[0013] In some embodiments, the system comprises: a sensor; and optics comprising an optical block, wherein the optics are configured to: receive excitation light from an excitation source; direct, via the optical block, the excitation light in at least n excitation bands to at least a portion of a sample on a sample holder when the sample and the sample holder are present; receive photo-induced emission light emitted from the sample; filter, via the optical block, the received emission light to produce filtered emission light having at least n emission bands; and direct, via the optical block, the filtered emission light to a detector comprising the sensor when the detector is present, such that the filtered emission light is incident upon sensing elements of the sensor; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

[0014] According to some embodiments, methods for detecting light emitted from a biological sample are described. In some embodiments, the method comprises: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially; directing, via an optical block, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the at least a portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optical block, the fluorescence light to produce filtered fluorescence light having at least n emission bands; and directing, via the optical block, the filtered fluorescence light to a detector comprising a sensor; and detecting, with sensing elements of the sensor, the filtered fluorescence light; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

[0015] In some embodiments, the method comprises: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially; directing, via optics comprising one or more filters, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the at least a portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optics, the fluorescence light to produce filtered fluorescence light having at least n emission bands, wherein the one or more filters are in a single configuration during at least the filtering the fluorescence light; and directing, via the optics, the filtered fluorescence light to a detector comprising a sensor; and detecting, with sensing elements of the sensor, the filtered fluorescence light; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response. In some embodiments, the method comprises: producing excitation light having at least n excitation bands; directing, via an optical block, the excitation light to at least a portion of a biological sample to excite labeling agents immobilized with respect to the at least a portion of the sample such that the labeling agents emit photo-induced emission light; filtering, via the optical block, the emission light to produce filtered emission light having at least n emission bands; and directing, via the optical block, the filtered emission light to a detector comprising a sensor; and detecting, with sensing elements of the sensor, the filtered emission light; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

[0016] According to some embodiments, methods of generating an image of a biological sample are described. In some embodiments, the method comprises: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially, wherein n is an integer; directing, via optics comprising one or more filters, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the at least a portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optics, the fluorescence light to produce filtered fluorescence light having at least n emission bands; directing, via the optics, the filtered fluorescence light to a detector; and detecting, with the detector, the filtered fluorescence light to acquire at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands; generating at least two spectral response images from each of the at least n fluorescence acquisition images, for a total of at least 2n spectral response images; and performing an unmixing of the detected fluorescence light from the fluorophores using at least some of the 2n spectral response images to generate a multilayer image set of the portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores, wherein the one or more filters are in a single configuration during at least the filtering the fluorescence light.

[0017] In some embodiments, the method comprises: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially, wherein n is an integer; directing, via optics comprising one or more filters, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optics, the fluorescence light to produce filtered fluorescence light having at least n emission bands without changing the configuration of the one or more filters; directing, via the optics, the filtered fluorescence light to a detector; and detecting, with the detector, the filtered fluorescence light to acquire at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands; generating at least two spectral response images from each of the at least n acquired fluorescence acquisition images, for a total of at least 2n spectral response images; and performing an unmixing of the detected fluorescence light from the fluorophores using at least some of the 2n spectral response images to generate a multilayer image set of the portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores.

[0018] In some embodiments, the method comprises: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially, wherein n is an integer greater than or equal to 4; directing, via an optical block, the excitation light to at least a portion of a biological sample to excite at least m different fluorophores immobilized with respect to the at least a portion of the sample such that the at least m different fluorophores emit fluorescence light, wherein m is an integer greater than or equal to n+1; filtering, via the optical block, the fluorescence light to produce filtered fluorescence light having at least n emission bands; and directing, via the optical block, the filtered fluorescence light to a detector; and detecting, with the detector, the filtered fluorescence light to acquire at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands; performing a determined or over-determined unmixing of the detected fluorescence light from the at least m different fluorophores using spectral response images generated from the at least n fluorescence acquisition images to generate a multilayer image set of the at least a portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores of the at least m different fluorophores.

[0019] According to some embodiments, devices for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers are described. In some embodiments, the device comprises: a single epi-cube, a multi-Light Emitting Diode excitation light source (LED excitation source); and a ‘Red-Green-Blue’ color imaging sensor (RGB color imaging sensor), wherein the epi-cube comprises a multi-bandpass emission filter and a dichroic, each with 5 or more bandpasses, and wherein the LED excitation source comprises 5 or more individually controlled LEDs.

[0020] According to some embodiments, systems for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers are described. In some embodiments, the system comprises: a single epi-cube; an LED excitation source, and an RGB color imaging sensor, wherein the epi-cube comprises multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source has 5 or more individually controlled LEDs.

[0021] According to some embodiments, methods for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers are described, the method comprising the steps of: preparing a tissue slide from tissue sample from a subject; staining the slide with up to 15 fluorophore markers; and processing the stained slide using a device for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, wherein the device comprises: a single epi-cube; an LED excitation source; and an RGB color imaging sensor; wherein the epi-cube comprises a multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source has 5 or more individually controlled LEDs.

[0022] According to some embodiments, systems for detecting light transmitted through a sample are described. In some embodiments, the system comprises: a detector, wherein when a sample holder and illumination source are present, the sample holder, illumination source, and detector establish an optical path for illumination light from the illumination source to travel from the illumination source, through at least a portion of a sample when the sample is present on the sample holder, and to the detector; and an actuatable filter having a first configuration in which the actuatable filter is not located in the optical path and a second configuration in which the actuatable filter is located in the optical path; wherein: the detector comprises a sensor and a color filter array, the presence of which results in the sensor comprising at least: first color pixels or subpixels having a first color spectral response; second color pixels or subpixels having a second color spectral response; and third color pixels or subpixels having a third color spectral response, wherein the first color spectral response, second color spectral response, and third color spectral response are different; and the actuatable filter is configured to reduce transmission of a portion of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response.

[0023] According to some embodiments, methods for generating an image of a biological sample are described. In some embodiments, the method comprises: directing, during a first period of time, illumination light along an optical path from an illumination source, through at least a portion of a biological sample comprising multiple different immobilized chromophores, and to pixels or subpixels of a detector, wherein the detector comprises a color filter array that results in the pixels or subpixels comprising: first color pixels or subpixels that have a first color spectral response; second color pixels or subpixels that have a second color spectral response; and third color pixels or subpixels that have a third color spectral response; detecting the illumination light that was directed to the detector during the first period of time to acquire a first color intensity acquisition image, the first color intensity acquisition image comprising color intensity spectral response images comprising: a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third intensity color intensity spectral response image corresponding to the third color pixels or subpixels; directing, during a second period of time, illumination light along the optical path to the pixels or subpixels of the detector, wherein a filter is located in the optical path during the second period of time but not the first period of time, wherein the filter reduces transmission of a portion of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response; detecting the illumination light that was directed to the detector during the second period of time to acquire a second color intensity acquisition image, the second color intensity acquisition image comprising color intensity spectral response images comprising: a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third color intensity spectral response image corresponding to the third color pixels or subpixels; generating a multilayer image set of the at least a portion of the biological sample, the multilayer image set comprising layers comprising two of the following: (1) one or more color optical density spectral response images calculated from at least some of the color intensity spectral response images of the first color intensity acquisition image, (2) one or more color optical density spectral response images calculated from at least some of the color intensity spectral response images of the second color intensity acquisition image, and (3) one or more color optical density spectral responses images calculated from: (i) subtraction of the second color acquisition intensity image from the first color intensity acquisition image, or (ii) subtraction of (a) a second optical density image calculated from the second color intensity acquisition image from (b) a first optical density image calculated from the first color intensity acquisition image; and performing an unmixing of the multilayer image set to generate unmixed images, each corresponding to one of the multiple different immobilized chromophores.

[0024] According to some embodiments, methods for performing a segmentation of an image of a biological sample are described. In some embodiments, the method comprises: obtaining a multilayer image set that was acquired by a detector, wherein the multilayer image set comprises: a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample; and a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample; generating a combination image based on the nuclei label image and the membrane label image; and segmenting images of nuclei in the biological sample based at least in part on the combination image to produce a nuclear segmentation map.

[0025] According to some embodiments, systems are described. In some embodiments, the system comprises: at least one processor; and at least one non-transitory computer- readable storage medium storing processor executable instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for performing a segmentation of an image of a biological sample, the method comprising: obtaining a multilayer image set that was acquired by a detector, wherein the multilayer image set comprises: a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample; and a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample; generating a combination image based on the nuclei label image and the membrane label image; and segmenting images of nuclei in the biological sample based at least in part on the combination image to produce a nuclear segmentation map.

[0026] According to some embodiments, non-transitory computer-readable storage media are described. In some embodiments, at least one non-transitory computer- readable storage medium storing processor executable instructions are described that, when executed by at least one processor, cause the at least one processor to perform a method for performing a segmentation of an image of a biological sample, the method comprising: obtaining a multilayer image set that was acquired by a detector, wherein the multilayer image set comprises: a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample; and a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample; generating a combination image based on the nuclei label image and the membrane label image; and segmenting images of nuclei in the biological sample based at least in part on the combination image to produce a nuclear segmentation map.

[0027] According to some embodiments, methods for processing a biological sample comprising biological tissue are described. In some embodiments, the method comprises: performing an immunohistochemistry procedure on at least a portion of a sample on a slide using a first labeling agent immobilized with respect to the sample; generating a first score associated with an indication for the at least a portion of the sample based on the immunohistochemistry procedure, wherein the performing of the procedure and the generating of the first score are conducted according to a clinical standard for the indication; and generating a fluorescence image of at least a portion of the sample on the same slide, the fluorescence image based, at least in part, on signal detected from a second labeling agent immobilized with respect to the sample; after determining that the first score is indeterminant, processing the fluorescence image to classify components of the at least a portion of the sample; and generating, from the classified components, (a) a second score associated with the indication and / or (b) a parameter indicative of the quantity, density, and / or level of an analyte in the sample.

[0028] Other advantages and novel features of the present invention will become apparent from the following detailed description of various non-limiting embodiments of the invention when considered in conjunction with the accompanying figures. In cases where the present specification and a document incorporated by reference include conflicting and / or inconsistent disclosure, the present specification shall control.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Non-limiting embodiments of the present invention will be described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. In the figures, each identical or nearly identical component illustrated is typically represented by a single numeral. For purposes of clarity, not every component is labeled in every figure, nor is every component of each embodiment of the invention shown where illustration is not necessary to allow those of ordinary skill in the art to understand the invention. In the figures:

[0031] FIG. 1A shows a flow chart for an example of a method for detecting light emitted from a sample, according to some embodiments;

[0032] FIG. IB shows a flow chart for an example of a method for generating a multilayer image set, according to some embodiments;

[0033] FIG. 1C shows a flow chart for a representative process as disclosed herein for staining and imaging a slide with 6 or more markers, including how RGB image tile planes are parsed into a 6- and 15-plane whole slide image and unmixed to produce isolated marker planes for image analysis, according to some embodiments;

[0034] FIG. 2A shows a schematic diagram of a system for detecting light emitted from a sample, comprising optics that include an optical block, according to some embodiments;

[0035] FIG. 2B shows a representative optical layout of a microscope, with an RGB camera used in conjunction with a 5-band epi-cube to image slides with 6 or more fluorophores, according to some embodiments;

[0036] FIG. 2C shows a schematic diagram of a sensor comprising sensing elements, according to some embodiments; FIG. 2D shows an example of a schematic plot of spectral sensitivities versus wavelength for examples of first and second sensing elements, according to some embodiments;

[0037] FIG. 2E shows a schematic illustration of a Bayer color filter array, according to some embodiments;

[0038] FIG. 2F shows an example of a schematic plot of spectral sensitivities versus wavelength for examples of red, green, and blue pixels or subpixels, according to some embodiments;

[0039] FIG. 3A shows a schematic diagram of a system for detecting light emitted from a sample, comprising optics that include an optical block, according to some embodiments;

[0040] FIG. 3B shows an optical layout of a microscope including an RGB camera using in conjunction with a 5-band epi-cube as shown in FIG. 2B, but with a single quintuple pass excitation filter incorporated into the epi-cube rather than 5 individual bandpass filters placed in front of an LED light source, according to some embodiments;

[0041] FIG. 3C shows an example of a schematic plot showing n different excitation bands for an excitation light directed to a sample, according to some embodiments;

[0042] FIG. 3D shows an example of a schematic plot showing n different emission bands for filtered fluorescence light directed to a detector, according to some embodiments;

[0043] FIG. 4 shows an example of a spectral response of the RGB imaging sensor, a typical quintuple bandpass emission filter, and emissions of 6 fluorophores, according to some embodiments;

[0044] FIG. 5A, FIG. 5B, FIG. 5C, FIG. 5D, FIG. 5E, and FIG. 5F show a representative example of a combined system spectral response to each of the 6 fluorophore emissions based in FIG. 4, according to some embodiments;

[0045] FIG. 6 shows a representative example of the relative signals from 6 fluorophores in 10 image planes selected from the 15 resultant image planes created by combining a quintuple-pass emission filter, a 5-LED light source with individual control, and an RGB image sensor. This combination provides an over-determined unmixing matrix facilitating more accurate signal isolation and quantitation compared to attempting to unmix 6 fluorophores with just 5 image planes using an underdetermined matrix. Note that the signals are mostly isolated optically, to be expected with the clear differences between the signals detected by the RGB image sensor as shown in FIGS. 5A-5F, alleviating dependence on mathematical unmixing to correct for excitation and spectral crosstalk between channels, in accordance with some embodiments;

[0046] FIG. 7A shows a flow chart for an example of a method for detecting light transmitted through a sample, according to some embodiments;

[0047] FIG. 7B shows a flowchart for an example of a method using a multi-bandpass filter to create 6 spectrally, substantially distinct spectral profiles with an RGB slide scanner, and then performing image analysis to measure biological parameters of interest, according to some embodiments;

[0048] FIG. 8A shows a schematic diagram of a system for detecting light transmitted through a sample, comprising an actuatable filter, according to some embodiments;

[0049] FIG. 8B shows a schematic diagram of a system for detecting light transmitted through a sample, comprising one or more actuatable filters, according to some embodiments;

[0050] FIG. 8C shows an optical configuration of an example of a slide scanner system, showing potential placements of a software-actuated motorized multi-bandpass filter, according to some embodiments;

[0051] FIG. 8D shows a schematic diagram of a combination system for detecting light emitted from a sample and / or for detecting light transmitted through a sample, comprising optics that include an optical block and / or one or more actuatable filters, according to some embodiments;

[0052] FIG. 8E shows an illustrative plot of examples of red, green, and blue spectral responses versus wavelength when an example of an actuatable filter is absent from the optical path (top) and when the actuatable filter is present in the optical path (bottom), according to some embodiments;

[0053] FIG. 9 shows representative spectral profiles of the R, G, and B channels of an example of an RGB imaging sensor, in accordance with some embodiments;

[0054] FIG. 10 shows a transmittance profile of an example multi-bandpass filter that approximately splits each R, G and B channels in half, to, in this case, create 5 distinct system spectral profiles, in accordance with some embodiments; FIG. 11 shows the resulting effective 5 spectral profiles created by an example process, according to some embodiments;

[0055] FIG. 12 shows example chromophore absorption spectra, according to some embodiments;

[0056] FIG. 13 shows example eigenvectors representing relative absorption for the chromophores across the 5 ‘spectral channels,’ according to some embodiments;

[0057] FIG. 14A shows a flowchart for an example of a method for segmenting components of an image of a biological sample, according to some embodiments;

[0058] FIG. 14B shows a flowchart describing steps for an example of a method for segmenting components of an image of a biological sample, including staining the slide with a multiplex protocol including a nuclear counterstain such as DAPI and at least one membrane counterstain, imaging on a suitable slide imaging system, optionally scaling nuclear and membrane counterstains to achieve a standard intensity ratio, subtracting the membrane image from the nuclear image, and performing cell segmentation on the difference image and membrane images, according to some embodiments;

[0059] FIG. 15 shows an illustration of cells that are sectioned in a tissue sample, which captures different parts of cells that may overlap when viewed above while imaging, according to some embodiments;

[0060] FIG. 16 shows example image of a DAPI nuclear counterstain, according to some embodiments;

[0061] FIG. 17 shows example image of a membrane counterstain created with a cocktail of antibodies, according to some embodiments;

[0062] FIG. 18 shows an example difference image created by subtracting a scaled version of the membrane image of FIG. 17 from the nuclear counterstain image of FIG. 16, creating an image that is easier to segment and focuses on cells captured well by the sectioning process, according to some embodiments;

[0063] FIG. 19 shows example segmentation of nuclei seen in difference image, in this case using a ‘stardist’ Al-based nuclear segmentation algorithm, according to some embodiments;

[0064] FIG. 20 shows a flow chart describing conventional immunohistochemical (IHC) testing, ; FIG. 21 shows a flow chart that describes conventional immunohistochemical (IHC) testing for a PD-L1 expression indication, ;

[0065] FIG. 22 shows flow chart that describes an enhanced immunohistochemical (IHC) testing procedure, according to some embodiments;

[0066] FIG. 23 shows a representative image of standard IHC imagery of a lung cancer sample, according to some embodiments;

[0067] FIG. 24 shows representative images for: (a) staining of additional proteins revealed by tumor and for macrophages fluorescence markers imaged on a scanner equipped to image fluorescence; (b) the same image as (a) but with image analysis used to identify individual cell nuclei; and (c) the same image as (a) but with cells classified into categories using machine learning algorithms, in this case tumor cells, macrophages, and other cell types, according to some embodiments;

[0068] FIG. 25 shows a representative display on the computer screen that a user can use to confirm accuracy of image analysis-based cell classifications, to support a calculation of the clinical test score, such as Tumor Proportion Score (TPS), Immune Proportion Score (IPS), and Combined Positivity Score (CPS), according to some embodiments;

[0069] FIG. 26 is a block diagram of an example system 2600 for processing image(s) of biological sample(s), according to some embodiments of the technology described herein;

[0070] FIG. 27 is a schematic diagram of an illustrative computing device with which embodiments described herein may be implemented;

[0071] FIG. 28 shows a composite image of a biological tissue sample, according to some embodiments;

[0072] FIG. 29 shows an unmixed nuclear stain layer of the composite image of FIG. 28, according to some embodiments;

[0073] FIG. 30 shows an unmixed membrane stain layer of the composite image of FIG. 28, according to some embodiments;

[0074] FIG. 31 shows an unmixed cytokeratin stain layer of the composite image of FIG. 28, according to some embodiments;

[0075] FIG. 32 shows an unmixed PD-L1 stain layer of the composite image of FIG. 28, according to some embodiments; FIG. 33 shows an unmixed CD8 stain layer of the composite image of FIG. 28, according to some embodiments; and

[0076] FIG. 34 shows the unmixed CD 163 stain layer of the composite image of FIG. 28, according to some embodiments.

[0077] DETAILED DESCRIPTION

[0078] Systems and methods for imaging and, in some instances, analyzing biological samples such as biological tissue are described. The systems and methods may be directed to the interrogation of biological samples labeled with multiple markers (e.g., fluorophores and / or chromophores) corresponding to various biomarkers and / or cellular or subcellular structures within the sample. Some such systems and methods may combine certain optics (e.g., configurationally-fixed optics for fluorescence imaging and / or actuatable filters for chromogenic imaging) with multi-color sensors (e.g., RGB sensors) to facilitate the imaging and analysis of a sample. The systems and methods may also promote advantageous techniques for segmenting images of components of the biological sample. In some instances, the methods can assist users (e.g., pathologists) in scoring samples for indications such as diseases. The systems and methods may, in some instances, promote economical, accurate, and / or higher-throughput imaging and analysis of the sample.

[0079] Certain embodiments described in this disclosure involve systems and methods for multiplexed fluorescence detection from the biological sample (e.g., for multiplex whole-slide immunofluorescence imaging of tissue or cells). In some embodiments, such systems include a multi-color sensor (e.g., an RGB sensor) and a single optical block (e.g., a single epi-cube comprising a dichroic filter and a multi-bandpass filter). The optical block may be configured to direct, sequentially, multiple (e.g., at least 4) bands of excitation light from an excitation source (e.g., a collection of individually- controllable light emitting diodes (LEDs)) to the sample and then filter the resulting fluorescence into multiple (e.g., at least 4) emission bands to be received by the sensor. These different emission bands may align spectrally with the emission spectra of one or more non-endogenous fluorophores associated with biomarkers within the sample. The combination of configurationally-fixed optics (e.g., a single optical block) and a multicolor sensor (e.g., an RGB sensor) may facilitate fast a relatively inexpensive independent analysis of a relatively large number of fluorophores and associated biomarkers (e.g., 6 or more) without requiring complex workflows or optoelectronic equipment.

[0080] Certain embodiments described in this disclosure involve systems and methods for multiplexed chromogenic detection from the biological sample (e.g., for multiplexed immunohistochemistry imaging of tissue or cells). In some embodiments, such systems employ a multi-color sensor (e.g., an RGB sensor) and an actuatable filter (e.g., a multibandpass filter) that can automatedly be placed in or removed from the optical path of the system. The actuatable filter may reduce the transmission of some of the wavelengths of light in one or more of the color spectral responses of the sensor (e.g., by splitting one or more of the R, G, or B responses). Imaging of the sample with and without the actuatable filter may, consequently, increase the number of available spectral response images usable for unmixing of chromogenic signal, permitting the use of a greater number of different chromogens than would be available without use of such a filter (e.g., permitting detection and unmixing of four or more chromogens with a simple RGB sensor).

[0081] Certain embodiments described in this disclosure involve methods for performing a segmentation of an image of a biological sample are generally described. In some embodiments, a multilayer image set of the biological is obtained, including a layer corresponding to a nuclei label image produced by nuclear labeling agents (e.g., a nuclear counterstain) and a layer corresponding to a membrane label image corresponding to one or more membrane labeling agents (e.g., a membrane counterstain). The nuclei label image and membrane label image may be used to generate a combination image (e.g., by subtracting the membrane label image (or scaled version thereof) from the nuclei label image (or scaled version thereof). The combination image may promote more effective segmentation of the images of the various nuclei captured by the imaging (e.g., with clearer delineation of nuclei and / or better co-location of nuclei images and analyte labeling). The resulting nuclear segmentation map may, in some instances be used, along with a membrane segmentation map from the membrane label image, for automated and accurate cellular image segmentation and, in some instances, more accurate cell classification (e.g., facilitated by more accurate biomarker expression information.

[0082] Certain embodiments described in this disclosure involve methods for processing a biological sample comprising a biological tissue are generally described. In some embodiments, an immunohistochemistry procedure (e.g., an immunohistochemistry procedure according to a clinical standard as described by an FDA in vitro diagnostics label and / or College of Anatomical Pathologists guidelines) is performed on at least a portion of the sample (e.g., a sample of biological tissue on a slide) using a first labeling agent (e.g., a stain). A first score for an indication (e.g., a disease such as a cancer) may be generated by a user (e.g., a pathologist) from the immunohistochemistry procedure. A fluorescence image of at least a portion of the sample may also be generated, based at least in part on a signal detected from a second labeling agent (e.g., a fluorophore). If the first score is determined to be indeterminant, the fluorescence image may be processed to classify (e.g., via machine-vision techniques) components of the sample, and a second score (e.g., a sample score associated with the indication) and / or a parameter (e.g., a parameter indicative of the quantity, density, and / or level of an analyte such as a cancer biomarker in the sample) may be generated from the classified components. In some embodiments, the second score and / or parameter may be used to determine a diagnosis, prognosis, and / or treatment plan for a subject (e.g., patient) from whom the biological tissue was obtained. This may advantageously reduce the uncertainty of a diagnosis, prognosis, and / or a treatment plan which is derived from the first score and / or second score when the first score is indeterminant.

[0083] Systems and methods employing any of a variety of combinations of the inventive systems and methods described above are also possible. For example, the system may include the multi-color sensor and both the single optical block (e.g., for multiplexed fluorescence imaging) and the actuatable filter (e.g., for multiplexed chromogenic imaging). As another example, certain methods may involve obtaining multiplexed fluorescence imaging and / or multiplexed chromogenic imaging on the sample with a cocktail that includes a nuclear counterstain and a membrane counterstain, and then processing (e.g., with a software on a processor of the system) a resulting multilayer image set of the sample to perform nuclear image segmentation enhanced by the membrane counterstain. The multiplexed imaging and analysis of the sample described above may be employed to assist a user in determining a more quantitative assessment of a sample for an indication (e.g., a disease such as cancer) if the user obtains an indeterminant score following a conventional clinical standard immunohistochemistry procedure. Other embodiments combining the inventive systems and methods described above are also possible. In some embodiments, elements of the inventive systems and methods may operate synergistically to improve sample processing, imaging processes, and image analysis, diagnostic processes, and other clinical and research processes.

[0084] Many tools commonly used to analyze biological samples (e.g., biological tissue samples) in diagnostic, clinical, and research settings rely on the analysis of images of such samples. In many cases, it is desirable to obtain information regarding the cellular and / or subcellular structures of a biological sample and / or or other biomarkers within the sample in order to perform diagnostic and / or other clinical assessments (e.g., prognoses and / or treatment plans) based on the sample. Conventional approaches often rely on the use of markers (e.g., chromophores and / or fluorophores) in biological samples in conjunction with imaging of the samples, followed by human visual and / or semiautomated analysis of the samples. However, such approaches are often slow, prone to error, low-throughput, and / or expensive. Accordingly, there is a need for improved analytical tools for analyzing biological samples using images.

[0085] It has been recognized in the context of this disclosure that systems and methods for analyzing biological samples leveraging, in some instances, multiplexing (e.g., multiplexed immunofluorescence procedures and / or multiplexed immunohistochemistry procedures) and multi-planar imaging may be used to achieve fast, inexpensive, accurate, and high-throughput imaging and analysis of biological samples (e.g., biological tissue). Various systems and methods that, in some instances, exhibit one or more of these advantages in imaging and analysis are described below.

[0086] In many applications, it is useful to analyze the properties of biological samples (e.g., biological tissue) such as the presence, density, arrangement, functional state and / or other properties of analytes such as biomarkers and / or cells in the sample. For example, it may be useful in basic science applications to understand these properties to better understand how organisms function. In some clinical and / or research applications, it is useful to understand these properties in order to better understand differences between healthy and diseased tissues. This can be used to better guide translational research in the development of drugs and other treatments, or in clinical settings to complement patient care and / or guide treatment decisions. To such ends, multiplexed immunofluorescence methods are used to study properties of biological samples. However, the complexity of biological samples such as tissues - which can contain hundreds or thousands of different proteins, DNAs, RNAs, and other molecules which may be labeled by the fluorophore markers commonly employed in immunofluorescence methods - can present significant challenges. Many existing platforms and technologies for using immunofluorescence methods to study biological samples are complicated, expensive, low throughput, and limited in their ability to allow multiple fluorescently labeled biomarkers efficiently. Accordingly, there is a need for improved multiplexed immunofluorescence systems and methods.

[0087] It has been recognized in the context of this disclosure that certain arrangements of components, some of which may be commonly found in multiplexed immunofluorescence equipment, may be leveraged to promote independent and accurate assessment of a relatively large number labeled biomarkers simultaneously. The optomechanical systems described in this disclosure may have any of a variety of several advantages, including the ability to be economically mass-produced, a small footprint that is compatible with many existing laboratory operating models, and allowing for the fast, accurate, and economical isolation and quantification of multiple labeled biomarkers in a single biological sample. It has also been recognized that methods for detecting light emitted from a biological sample leveraging this inventive system may be beneficially used to perform high-throughput characterization of biological samples.

[0088] Immunohistochemistry techniques for processing biological samples are commonly used in several clinical and research settings, including, for example, in clinical predictive testing in oncology. Such systems generally rely on the use of chromogen markers to stain different biomarkers within a biological sample (e.g., a tissue sample). However, many existing systems and methods used in immunohistochemistry analyses are only marginally predictive. As a non-limiting example, many existing immunohistochemistry analyses for predicting tumor response to anti-programmed cell death ligand 1 (anti-PDLl) therapy have a receiver operator curve area-under-the-curve (AUC) rating of only 0.6, where 0.5 is of no predictive value and 1.0 indicates perfectly accurate prediction. Such poor performances of traditional immunohistochemistry techniques can arise from the complex expressions of many biomarkers (such as the PD-L1 biomarker), which can be expressed in several cell types and / or serve different roles in healthy cells and in cells within the tumor microenvironment. Additionally, many traditional immunohistochemistry procedures rely on the visual assessment of samples in order to make a diagnosis or prediction, which introduces human error and variation. Digitized immunohistochemistry platforms often rely on simple modifications to RGB scanners to improve the visual assessment of these samples, which fundamentally limits the number of chromogens which can be independently assessed within a single sample. As such, improved systems and methods for immunohistochemistry are needed.

[0089] It has been recognized in the context of this disclosure that actuated filters, such as actuated multi-bandpass filters, can be placed in the optical path of a detector (e.g., an RGB sensor) when imaging a biological sample (e.g., a tissue sample) using an illumination source. It has been recognized that such a configuration may be used to obtain multiple color spectral responses of a sample. Certain systems and methods in this disclosure may advantageously allow for the accurate, cost-effective, quantitative assessment of a relatively higher number of chromogenic markers (e.g., 4 or more) in a single biological sample, including, in some instances, when such markers are colocalized within individual pixels of a detected image, and / or when the information about the expression level of a given chromogenic marker is desired rather than information solely about the presence of a given chromogenic marker.

[0090] In many applications using techniques such as spatial biology, images can be analyzed to extract information about the behavior of various components of a biological sample (e.g., a tissue sample). The extraction of useful information from these images may depend, for example, on accurate measurement of the abundance of proteins on and / or in subcellular structures. Typically, such measurements rely on the use of stains and / or counterstains or other markers that preferentially bind to certain proteins which correspond to certain subcellular structures. To determine these measurements using images, segmentation processes may be used. Segmentation processes involve selecting pixels to represent certain cells and their subcellular structures such as nuclei, cytoplasm, and / or membranes. However, several factors present complications for existing segmentation processes. First, many biological samples (e.g., tissue samples section) are comprised of overlapping cell fragments. This leads, in some cases, to partially sectioned or whole nuclei distributed throughout the thickness of the section.. This may be compounded by the fact that standard immunohistochemistry or immunofluorescence imaging protocols may deposit markers only on the top surface of a sample, leading to misalignment between imaged markers and nuclei deeper in the section, as viewed by the imaging sensor. Taken together, this can lead to significant inaccuracies in the association of pixels in an image with particular cells or structures. As such, improved systems and methods for cell segmentation are needed.

[0091] It has been recognized in the context of this disclosure that obtaining a multilayer image set of a sample in which one layer corresponds to nuclei labels and another layer corresponds to membrane labels may be leveraged to perform more accurate cell segmentation. For example, the layer corresponding to membrane labels may be subtracted from the layer corresponding to nuclear labels, to produce a combination image. In this combination image, nuclei that do not overlap with membrane signals may be more clearly visible from the surface of the sample, and thus more pronounced and easily detected and segmented, thus in some instances promoting cells that are well sectioned and represented on the top surface of the section. The improved visibility may result in an image more conducive to accurate segmentation of images of nuclear substructures and, ultimately cell images. This processing may facilitate automated, computer-implemented nucleus and / or cell image segmentation.

[0092] Immunohistochemistry (IHC) procedures may be employed in, for example, clinical practice and research. As a non-limiting example, IHC is widely used for diagnosis of cancers because specific tumor antigens are expressed de novo or up- regulated in certain cancers. IHC for such applications use tissue from biopsies. These are processed into sections with a microtome and then the sections are incubated with an appropriate antibody. The site of antibody binding is visualized and / or imaged with an ordinary or fluorescent microscope by a labeling agent. The labeling agent may comprise, for example, a chromogen, fluorescent dye, enzyme, radioactive element, and / or colloidal gold. The labeling agent may be directly linked to the primary antibody or to an appropriate secondary antibody. However, IHC is associated with a number of limitations, including high inter- observer variability and its limitation to labeling only one marker per tissue section under the prevailing clinical standards for most in vitro diagnostics. The most important of these is that this technique only permits, under prevailing clinical standards, the labelling of a single marker per tissue section. This results in missed opportunities to improve precision and accuracy of assessments when assessments depend on accurately identifying cell types expressing the antigen of interest. Another limitation is the inconsistency of human visual perception when assessing biopsy tissue sections stained with IHC staining protocols, especially when the parameter of interest is at or near scoring thresholds. Mistaken assessments can lead to a) unproductive treatment with side effects or b) missing patients that would have responded to treatment.

[0093] Multiplexing techniques have emerged to circumvent these constraints, allowing simultaneous detection of multiple markers on a single tissue section. Among these techniques, multiplex Immunohistochemistry / Immunofluorescence (mIHC / IF) has emerged to be particularly promising. This technique has immediate potential for translational research and clinical practice, particularly including in the era of cancer immunotherapy .

[0094] Predicting response of tumors to immune-oncology drugs targeting particular biomarkers (for example, the programmed cell death- 1 (PD-l) / programmed cell death ligand-1 (PD-L1) immune checkpoint) using immunohistochemistry procedures, as shown in FIG. 21, has been found to be limited by difficulties in terms of assessing biomarker expression (for example, PD-L1 expression) by IHC. Further, mistaken assessments such as false negatives can lead to patients who could have benefitted missing out on a treatment that could have benefitted them. False positives are also possible. Such mistaken assessments, which can lead to unnecessary treatments and / or high levels of stress for a patient, are common when using conventional IHC procedures to predict response (e.g., tumor response) to drugs targeting particular biomarkers. Accordingly, there is a need for better quantitation to address inconsistencies that can occur when using just visual assessment of conventional single-marker IHC for detection of biomarkers (such as, for example, PD-1 and / or PD-L1) in tissue or biopsy samples from subjects (for example, cancer patients) and for subsequent treatment prediction based on such biomarker identification and / or quantification in subjects (e.g., cancer patients). Certain embodiments described in this disclosure involve systems for detecting light emitted and / or light transmitted through a sample (e.g., for imaging and / or analysis). The system may comprise a sensor and optics configured to direct and / or manipulate light to a sample and detect resulting light emanating from or through the sample. For example, in FIG. 2A system 100 comprises sensor 102 configured to detect light directed and / or manipulated by optics 110 to interrogate sample 111 on sample holder 109, while in FIG. 8A system 200 similarly comprises sensor 202 and optics 210 to interrogate sample 211 on sample holder 209. The system may be comprise, for example, components for a slide- scanning microscope system for interrogating samples (e.g., tissue samples or cells) mounted on a sample holder such as a slide. Various portions of this disclosure are directed to combinations of certain arrangements of the optics with certain types of sensors that promote enhanced multispectral imaging analysis (e.g., with determined or overdetermined unmixing) without requiring complex optoelectronic or optomechanical equipment.

[0095] Sensor and Detector

[0096] In some embodiments, the system comprises a sensor. The sensor may be responsive to incident light. For example, the sensor may be configured to respond to stimulus from incident light (e.g., fluorescence light from a sample or illumination light transmitted through a sample) and generate an electrical signal indicative of a property of the incident light (e.g., wavelength and / or intensity). In some embodiments, the sensor is a solid state imaging sensor.

[0097] In some embodiments, the sensor comprises sensing elements as individual components that directly interact with incident light and convert it into a measurable electrical signal. In some embodiments, the sensing elements are photosensitive regions or structures of the sensor. Accordingly, the sensing elements may comprise a photosensitive material (in some instances combined with an electrical component such as a transistor) and be configured to convert light into electrical current and / or charge. Non-limiting examples of sensing elements or components thereof include, but are not limited to, photodiodes, phototransistors, photogates, and / or capacitors. In some embodiments, the sensing elements are pixels or subpixels comprising a photodiode (e.g., a pinned photodiode), phototransistor, photogate, and / or capacitor. For example, in some embodiments, the sensor is a complementary metal-oxide semiconductor (CMOS) sensor comprising a one- or two-dimensional array of pixels or subpixel as the sensing elements. One non-limiting example of a type of CMOS sensor that the systems of this disclosure may comprise is a scientific CMOS (sCMOS). In some embodiments, the sensor is a charge-couple device (CCD) comprising a one- or two-dimensional array of pixels or subpixels as the sensing elements.

[0098] In some embodiments, the system comprises a detector that comprises the sensor. Referring back to FIGS. 2A and 8A, sensor 102 and sensor 202, respectively, may be housed on and / or in a detector. For example, the sensor may be mounted on or within a housing with other electronics components configured to store, convert, and / or transmit the electrical signals generated by the sensing elements of the sensor. Any of a variety of detectors may be employed, such as a CCD detector (comprising a CCD sensor) and / or a CMOS detector (comprising a CMOS sensor).

[0099] As noted above, in some embodiments the sensor is a multi-color sensor. More generally, the sensor may comprise multiple sets of sensing elements that have different spectral responses to incident light. That is, the different sensing elements may have different sensitivities to different wavelengths of incident light. Such a sensor configuration stands in contrast to monochromatic sensors often used in microscope imaging, including multiplexed immunofluorescence or immunohistochemistry imaging, due to the perceived benefit of a uniform spectral response across wavelengths. But, as discussed above, it has been realized in the context of this disclosure that employing a sensor with multiple different spectral responses, such as multi-color sensors (e.g., RGB sensors) can permit the acquisition of a greater amount of spectral data for later analysis and / or manipulation (e.g., unmixing) without requiring complicated optical and / or optomechanical configurations for the system.

[0100] In some embodiments, the sensing elements of the sensor comprise first sensing elements and second sensing elements. For example, as shown in the schematic illustration in FIG. 2C, sensor 102 comprises first sensing elements 112 and second sensing elements 113 arranged as a two-dimensional array, in accordance with some embodiments. While first sensing elements 112 and second sensing elements 113 are uniformly distributed in their array in FIG. 2C, other arrangements are possible, such as those in which the first sensing elements and second sensing elements are segregated into separate regions of sensor 102. In some embodiments, the first sensing elements and the second sensing elements are arranged on a single chip.

[0101] In some embodiments, the first sensing elements have a first spectral response, while the second sensing elements that have a second spectral response that is different from the first spectral response. FIG. 2D shows an example of a schematic plot of spectral sensitivities versus wavelength for examples of first sensing elements 112 and second sensing elements 113, where the first sensing elements 112 have first spectral response 114 and second sensing elements have second spectral response 115.

[0102] The first spectral response and the second spectral response may partially overlap in some embodiments, while in other embodiments there may be no overlap between the first spectral response and the second spectral response (e.g., with neither sensing element having any detectable spectral sensitivity for at least one wavelength between the first spectral response and the second spectral response within the error of detection of the sensing elements).

[0103] The different spectral responses of the different sensing elements of the sensor may accomplished via any of a variety of configurations. For example, the different sensing elements may comprise different types of components (e.g., different photoactive materials) with different absorption profiles. As another example, the different sensing elements may be otherwise identical but are coupled to different filters having different spectral transmission profiles. As one example, a sensor may comprise or be coupled to a filter array. The color filter array may comprise a mosaic of different individual color filters located over the individual sensing elements. One non-limiting example of a color filter array is an RGB color filter array. For example, the color filter array may be a Bayer color filter array, a schematic illustration of which is shown in FIG. 2E.

[0104] The sensor may comprise any of a variety of numbers of different sensing elements having different spectral responses. For example, in some embodiments, the sensor comprises third sensing elements having a third spectral response that is different than the first spectral response and the second spectral response. As noted above, one type of sensor comprising three different sensing elements is an RGB sensor. RGB detectors are commonly available, including commercially, and so systems employing RGB detectors may be assembled relatively inexpensively, which may make the systems of such embodiments more economical than more complex systems. In the RGB sensor, the first sensing elements may correspond to red pixels or subpixels, the second sensing elements may correspond to green pixels or subpixels, and the third sensing elements may correspond to blue pixels or subpixels. FIG. 2F shows an illustrative example of a schematic plot of spectral sensitivities versus wavelength for examples of red pixels or subpixels (R), green pixels or subpixels (G), and blue pixels or subpixels (B). As noted above, the RGB sensor’s pixels or subpixels may be arranged in a Bayer pattern in some embodiments, but other arrangements are possible.

[0105] More generally, the different spectral responses of the different sensing elements may correspond to different colors of visible light. In this context, a color generally refers to a region of wavelengths of visible (or near-infrared or near-ultraviolet light) rather than one specific wavelength. For example, the sensor may comprise first color pixels or subpixels having a first color spectral response, second color pixels or subpixels having a second color spectral response; and third color pixels or subpixels configured having a third color spectral response, where the first color spectral response, second color spectral response, and third color spectral response are different. In terms of an RGB detector, the first color would be red, the second color would green, and the third color would be blue.

[0106] In some embodiments, the system comprises more than three different sensing elements. A non-limiting example of a color filter array sensor configuration with four sensing elements is a cyan, magenta, yellow, and gray (e.g., flat spectral response) sensor.

[0107] While FIGS. 2C-2F illustrate examples of configurations of sensor 102 of system 100, it should be understood that sensor 202 of system 200 may, in some instances, have any of the aforementioned configurations. For example, sensor 202 may be an RGB sensor or a different type of multi-color sensor. Further description of combinations of multi-color sensors such as RGB sensors with optics and / or image analysis techniques that may contribute to one or more of the above-mentioned advantages are described below.

[0108] Optics

[0109] As noted above, the system for detecting light emitted from and / or light emitted through the sample may comprise optics. For example, system 100 comprises optics 110, while system 200 comprises optics 210, in accordance with some embodiments. Any of a variety of optics may be present, depending on the desired application and / or type of imaging to be performed. The optics may include standard microscopy optics, including, but not limited to one or more of an objective lens, a dichroic mirror (e.g., for fluorescence imaging), an emission filter (e.g., for fluorescence imaging), and / or a condenser lens (e.g., for brightfield imaging). Other filters, such as excitation filters for fluorescence imaging, may also be included. The filters may comprise, for example, interference filters. Further specific arrangements and configurations of optics that may contribute to one or more of the above-mentioned advantages are described in more detail below.

[0110] Sample Holder

[0111] In some embodiments, a sample holder is present in the system. The sample holder may be a solid substrate capable of supporting at least a portion of the sample. The sample holder may be located such that light may be incident upon the sample and / or be transmitted through at least a portion of the sample when the sample is on the sample holder. In the embodiments shown in FIG. 2A and FIG. 8A, sample 111 is on sample holder 109 and sample 211 is on sample holder 209, respectively, in accordance with certain embodiments. In some embodiments, the sample holder is or comprises a slide (e.g., a microscope slide). It should be understood that when a sample is described as being on the sample holder, any of a variety of relative configurations with respect to other components of the system may be possible. For example, while sample 211 is shown as being between sample holder (e.g., slide) 209 and sensor 202 in FIG. 8A, in other arrangements sample 211 and sample holder 209 are inverted such that sample holder 209 is between sample 211 and sensor 202.

[0112] Other components associated with the sample holder when the sample holder is present may also be included in the system. For example, the system may comprise a stage (e.g., an x-,y- stage). In FIG. 2A and FIG. 8A, for example, system 100 comprises stage 108 while system 200 comprises stage 208, respectively. The stage may be configured to permit translation of the sample and / or sample holder in one or both of the x- and y-directions. Any of a variety of configurations of system components (e.g., a stage, the sample holder, and / or the optics) may facilitate a slide- scanning ability, where different sub-regions of a sample on a slide are sequentially imaged (e.g., via fluorescence and / or chromogenic imaging), thereby ultimately producing a whole- slide image of the sample. For example, in some embodiments, the sample holder is kept stationary, while the optics are moved to scan different portions of the sample on the sample holder and / or to focus. In other embodiments, the optics are kept stationary while the sample holder is moved (e.g., via the stage) such that the stationary optics scan different portions of the sample on the sample holder and / or focus. In yet other embodiments, the system is configured to scan different portions of a sample on the sample holder and to focus via movement by both (a) components of the optics and (b) the sample holder (e.g., via the stage). For example, the stage may be configured to scan different portions of the sample on the sample holder by translating the sample holder in the x-direction and by moving the optics in the y-direction, and one or both of the sample holder and the optics may be configured to move in the z-direction to facilitate focusing. The stage and / or optics positioning may be controlled by a controller, e.g., associated with a computer system comprising at least one processor and computer-readable storage media, examples of which are described below.

[0113] Fluorescence Imaging

[0114] Certain embodiments described in this disclosure involve systems and methods for detecting light emitted from a sample (e.g., biological sample) and, in some instances, generating one or more unmixed images of the sample using a relatively large number of different fluorophores (e.g., greater than or equal to 5, greater or equal to 6, or greater). In some such embodiments, the unmixing can be accomplished in a determined or overdetermined manner without relying on complex opto-electronic and / or complex opto-mechanical components. For example, the systems and methods may employ optics with configurationally-fixed filters (as opposed complex optical block turrets, filter wheels, and / or liquid crystal tunable filters) such as a single optical block. In some such embodiments, such optics may be employed in conjunction with the sensor with multiple spectral responses (e.g., an RGB sensor) described above. The system’s components (e.g., the excitation source, sample holder, detector with sensor, and / or the optics) may be part of a fluorescence microscope. The fluorescence microscope may be, for example, a slide-scanning microscope. In some embodiments, the system comprises optics for fluorescence imaging of samples (e.g., including a dichroic and one or more emission filters). In some embodiments, the optics are configured to receive excitation light from an excitation source and direct the light to at least a portion of a sample on a sample holder. For example, referring to FIG. 2A and FIG. 3A, optics 110 receives excitation light in the form of individual light excitations k=l, k=2, k=3,...k=n (e.g., each being in a different wavelength band) from excitation source 105 and directs the excitation light as light 103 to sample 111 on sample holder 109. The optics may be configured to perform the above-described functions by including appropriate light-directing optics and / or lightconditioning optics such as mirrors (e.g., a dichroic mirror) and lenses (e.g., an objective lens). Such optics are known and available commercially. One non-limiting example is the TRF89902-EM ET - 405 / 488 / 56 l / 647nm Laser Quad Band set (Chroma Technology Corp, Bellows Falls, VT) employable with, for example, a 9100 series filter cube (Chroma Technology Corp). For example, in FIGS. 2A and 3A, optics 110 include dichroic 116 arranged at an appropriate angle to receive incident light and direct it as light 103 to sample 111. Light 103 may be focused using objective lens 107. Objective lens 107 may be adjustable to vary the Z-focus on the sample as represented by arrows 120. The excitation light is employed to electronically excite fluorophores (e.g., non- endogenous fluorophores) in the sample to produce emitted fluorescence light.

[0115] The excitation light and / or fluorescence light may be in any of a variety of regions of the electromagnetic spectrum. In some embodiments, the excitation light and / or fluorescence light comprises wavelengths in the visible region of the electromagnetic spectrum (e.g., wavelengths greater than or equal to 400 nm and less than or equal to 700 nm). In some embodiments, the excitation light and / or fluorescence light comprises wavelengths in the near-infrared region of the electromagnetic spectrum (e.g., wavelengths greater than 700 nm and less than or equal to 1400 nm). In some embodiments, the excitation light and / or fluorescence light comprises wavelengths in the near-ultraviolet region of the electromagnetic spectrum (e.g., wavelengths greater than or equal to 200 nm and less than 400 nm). In some embodiments, the excitation light and / or fluorescence light comprises wavelengths in each of the near-ultraviolet, visible, and near-infrared regions of the electromagnetic spectrum. In some embodiments, the optics are configured to receive the received fluorescence light emitted from the sample. The optics may be configured to filter the received fluorescence light to produce filtered fluorescence light. For example, in FIG. 2A and FIG. 3A, optics 110 receive fluorescence light 104 emitted by sample 111 and filter the light with emission filter 117 to produce filtered fluorescence light 118. The optics may be arranged via appropriate placement of the emission filter and the dichroic.

[0116] In some embodiments, the optics are configured to direct the filtered fluorescence light to a detector comprising the sensor. For example, in FIG. 2A and FIG. 3A, optics 110 are configured to direct filtered fluorescence light 118 to the detector comprising sensor 102. The direction of the filtered fluorescence light may be performed such that the filtered fluorescence light is incident upon the sensing elements of the sensor (e.g., the pixels of a CCD sensor or a CMOS sensor). The sensor, in the detector, may then generate electrical signal responsive to the incident fluorescence light, as discussed above. The system may be further configured, using electronics components controlled by one or more processors, to transmit the electrical signal for processing (e.g., image generation and / or analysis).

[0117] The excitation light directed by the optics to the sample may be in various discrete excitation bands. In some embodiments, the excitation light directed by the optics to the sample are in at least n excitation bands, where n is an integer. For example, as noted above, in FIG. 2A and FIG. 3A, optics 110 direct excitation light in individual excitations, each in different bands corresponding to k=l, =2, k=3, and up to k=n. Each excitation band may be a distinct range of wavelengths. The use of multiple different excitation bands can facilitate the separate excitation of different fluorophores in the sample having different excitation spectra (e.g., due to having different light absorption spectra). Accordingly, the multiple different excitation bands may permit multiplexed fluorescence imaging of the sample. FIG. 3C shows an illustrative example of a plot showing in different excitation bands for the excitation light directed to the sample. In some embodiments, at least some of the at least n excitation bands are nonoverlapping. For example, as shown in FIG. 3C, the non-overlapping excitation bands 1, 2, 3,..., and n are separated by at least 1 nm (e.g., at least 2 nm, at least 10 nm, at least 20 nm, at least 50 nm, and / or up to 100 nm, or more) of wavelength. In some embodiments, all of the at least n excitation bands are non-overlapping. At least some of the excitation light in the various of various different excitation bands may be produced sequentially. Therefore, at least some of the excitation light in the various different excitation bands may be incident up on the sample sequentially, and then portions of the filtered fluorescence light may then be detected sequentially. The production of some or all of the excitation light in the various different excitation bands may be produced sequentially such that each exposure of the detector during fluorescence acquisition image acquisition detects emission light induced by excitation light in only one of the excitation bands. For example, in some embodiments, in which n individual LEDs are used to sequentially produce excitation light in n different excitation bands, only one of the n individual LEDs is on during each fluorescence acquisition image acquisition by the detector. In some embodiments, there is complete temporal separation between the time periods during which the sequentially-produced excitation light in the at least n excitation bands are initiated and directed to the sample. The sequential production of the excitation light in the different excitation bands and according sequential excitation of the different fluorophores in the sample may permit the at least partial (or complete) temporal separation of signal produced by different excitation bands. For example, in some embodiments, the system is configured to individually capture, using the sensor, a separate fluorescence acquisition image corresponding to fluorescence light produced by excitation from each of the at least n different excitation bands. In some embodiments where at least n excitation bands are employed, the system is configured to produce light from the excitation source in at least two (e.g., at least three, at least four, at least five, and / or up to all n) of the at least n excitation bands sequentially. In some embodiments, the system is configured to produce light from the excitation source in each of the at least n excitation bands sequentially. The sequential production of the light in the at least n different excitation bands can be accomplished in any of a variety of manners. For example, the excitation source, described in more detail below, may comprise multiple different individually- controllable light sources. The system may include, for example, a controller configured to individually actuate each of the individual lights sources to initiate light production. As another example, the excitation source may include just a single light source, but the system’s optics may include components such as temporally-varying filters (e.g., an optical chopper and / or an actuatable filter wheel) that can alter the spectral profile of excitation light directed through the optical path of the system during different periods of time.

[0118] In some embodiments, the excitation light directed by the optics to the sample are in a relatively high number of excitation bands. In other words, in some embodiments, n is a relatively high integer. For example, in some embodiments where the excitation light is in at least n different excitation bands, n is an integer greater than or equal to 4, greater than or equal to 5, greater than or equal to 6, or higher. In some embodiments, n is an integer less than or equal to 7. Combinations of these ranges are possible. For example, in some embodiments where the excitation light is in at least n different excitation bands, n is an integer greater than or equal to 4 and less than or equal to 7. As another example, in some embodiments, n is an integer greater than or equal to 5 and less than or equal to 7. In some embodiments, the number of excitation bands (and in some embodiments the number of emission bands) is equal to n, where n is an integer in one of the ranges described above. In some embodiments, the number of excitation bands (and in some embodiments the number of emission bands) is equal to 5.

[0119] The different excitation bands may be established in any of a variety of ways. In some embodiments, the excitation source individually produces light with wavelengths within the distinct excitation bands. For example, the excitation source may comprise a plurality of different monochromatic or narrow-band light sources such as lasers or lightemitting diodes with wavelengths selected to be sufficiently distinct to be in the separate excitation bands. However, in some embodiments, the excitation source produces light in a relatively broad spectrum, but the optics of the system include filters or other components (e.g., prisms) able to spectrally separate out the produced excitation light into the respective different excitation bands. In some embodiments, the excitation light is produced by individual light sources with different wavelength outputs, with the excitation light further cleaned up using one or more excitation filters to establish the different excitation bands. For example, the system may comprise multiple individual light emitting diodes (LEDs), each coupled to an individual bandpass filter, with the transmission profiles of the bandpass filters determining the resulting excitation bands. Alternatively or additionally, the multiple individual LEDs may each send their light output to a single multi-bandpass filter, with the transmission profiles of the different bands of the multi-bandpass filter determining the resulting excitation bands. As noted above, in some embodiments, the system comprises an excitation source configured to produce the excitation light. For example, system 100 in FIG. 2A and FIG. 3 A comprises excitation source 105 configured to produce the excitation light represented by arrows with k=l up to k=n. The excitation source may be a single light source or a plurality of individual light sources that collectively constitute the excitation source. For example, in some embodiments in which the system is configured to direct, with the optics, excitation light in at least n excitation bands to the sample, the excitation source comprises at least n individual light sources. As a more specific example, the excitation source may comprise at least n different light emitting diodes (LEDs). In some such embodiments, each of the at least n LEDs may be configured to individually produce light, with each LED corresponding to one of the at least n excitation bands (before or after excitation filter cleanup). As noted above, in some embodiments, the system comprises a controller configured to initiate light production from each of the individual light sources (e.g., by transmitting a signal to the individual light sources the initiates light production). The system may, in accordance with some embodiments, be configured to produce light from the excitation source in each of the at least n excitation bands individually, such that the production of light from the light in the different excitation bands is not dependent on the production of light from another of the different excitation bands (e.g., different initiation signals may be sent to different individual light sources).

[0120] The excitation source may be any of a variety of light sources appropriate for fluorescence imaging of biological samples. Examples of light sources that may be employed include but are not limited to LEDs, lasers (e.g., pulsed lasers and / or continuous-wave lasers), laser-diodes, incandescent light sources (e.g., tungsten-halogen lamps), arc-lamps (e.g., mercury arc-lamps, xenon arc-lamps), and / or flash lamps (e.g., xenon flash lamps).

[0121] As noted above, in some embodiments, the excitation light is filtered prior to reaching the sample. For example, in some embodiments, the optics are further configured to filter the received excitation light to produce filtered excitation light having the at least n excitation bands and direct the filtered excitation light to at least a portion of a sample. The filtering may promote relatively tight excitation bands, which may assist with better selectivity in exciting the various different fluorophores in the sample. Any of a variety of configurations for filtering the excitation light may be employed. For example, in some embodiments, the optics comprise at least n individual single-bandpass excitation filters configured to filter the received excitation light to produce the filtered excitation light. For example, in FIG. 2A, optics 110 comprise n individual singlebandpass filters 106, each single-bandpass filter 106 being configured to filter incident light from excitation source 105 into one of the n excitation bands. As another example, in some embodiments in which the optics include an optical block (see below), the optical block comprises a multi-bandpass excitation filter configured to produce the filtered excitation light. For example, in FIG. 3 A, optical block 119 comprises multibandpass excitation filter 121, with the different bands of multi-bandpass filter 121 being configured to filter incident light from excitation source 105 into one of the n excitation bands. As one example, in some embodiments in which n=4, the optical block comprises a quadruple-bandpass excitation filter. As another example, in some embodiments in which n=5, the optical block comprises a quintuple-bandpass excitation filter.

[0122] In some embodiments, the sample comprises a relatively large number of different fluorophores immobilized with respect to the at least a portion of the sample to be excited by the excitation light. In some embodiments the number of fluorophores exceeds the number of excitation bands. In some embodiments in which there are at least n excitation bands, the number of different immobilized fluorophores is at least m, where m is an integer greater than or equal to n + 1. In some embodiments in which there are at least n excitation bands, the number of different immobilized fluorophores is equal to m, where m is an integer greater than or equal to n + 1. As a non-limiting example, in some embodiments in which n = 5, m may be greater than or equal to 6 (e.g., greater than or equal to 7, greater than or equal to 8, greater than or equal to 10, and / or up to 12, up to 14, up to 15, or greater). It should be understood that immobilized fluorophores include any that are not free to move away from the sample, and need not necessarily be covalently or noncovalently bound to the sample.

[0123] As noted above, the optics of the system may produce filtered fluorescence light. In some embodiments, the filtered fluorescence light has at least n emission bands, where n is the same integer described above for the excitation bands. For example, as noted above, in FIG. 2A and FIG. 3A, optics 110 direct filtered fluorescence light 118 to sensor 102. Filtered fluorescence light 118 may be composed of light, all wavelengths of which are within the at least n emission bands. Each emission band may be a distinct range of wavelengths. In some instances the filtered fluorescence light having the at least n emission bands includes wavelengths in only a subset of the at least n emission bands (e.g., just a single of the at least n emission bands, or multiple of the emission bands but not all of the emission bands). In other instances, the filtered fluorescence light having the at least n emission bands includes wavelengths in each of the at least n emission bands. The use of multiple different emission bands can facilitate the detection of desirable fluorescence light from multiple different fluorophores having different fluorescence spectra while blocking undesirable light such as traces of excitation light and / or other stray light. Accordingly, the multiple different emission bands may facilitate multiplexed fluorescence imaging of the sample. FIG. 3D shows an illustrative example of a plot showing n different emission bands for the filtered fluorescence light directed to the detector. In some embodiments, at least some of the at least n emission bands are non-overlapping. For example, as shown in FIG. 3D, the non-overlapping emission bands 1, 2, 3,..., and n are separated by at least 1 nm (e.g., at least 2 nm, at least 10 nm, at least 20 nm, at least 50 nm, and / or up to 100 nm, or more) of wavelength. In some embodiments, all of the at least n emission bands are non-overlapping.

[0124] The at least n emission bands may be produced via any of a variety of filter configurations. For example, in some embodiments, the optics comprise a multibandpass filter, where the transmission profile of each bandpass corresponds to one of the at least n emission bands. Alternatively, in some embodiments, the optics comprise a plurality of single bandpass filters, each having a transmission profile corresponding to one of the at least n emission bands. A combination of such filters may also be employed in the system.

[0125] The at least n excitation bands and the least n emission bands created by the system (e.g., using the optics and / or the excitation source) may be selected based on the excitation and emission spectral profiles of the fluorophores in the sample. In some embodiments, the at least n excitation bands are interlaced with the at least n emission bands. FIG. 3C and FIG. 3D, which show n excitation bands and n emission bands, respectively, on the same wavelength axis demonstrate an example of interlaced excitation and emission bands. In these figures, the first emission band is at longer wavelengths than the first emission band but at shorter wavelengths of the second excitation band, and the second emission band is at longer wavelengths than the second excitation band but at shorter wavelengths than the third excitation band, and so on. The wavelengths of the excitation bands and the emission bands may be selected (e.g., via judicious choice of light source, excitation filter, and / or emission filters) such that the kth excitation band of the at least n excitation bands and the kth emission band of the at least n emission bands correspond to suitable excitation wavelengths and emission wavelengths, respectively, of one or more of the fluorophores in the sample being imaged, while also ensuring that excitation light is substantially or completely blocked from reaching the sensor. Such an arrangement may facilitate effective multiplexed fluorescent imaging of the sample.

[0126] In some embodiments, the optics are configured to perform the functions described above involving the directing of the excitation light and / or the filtering and directed of the fluorescence light without requiring complex operation of the optics. In some embodiments, the optics are configured to filter the fluorescence light to produce the filtered fluorescence light having at least n excitation bands while the one or more filters present in the optics are in a single configuration. The single configuration refers to the set of filter components in the optical path from the excitation source to the sample and from the sample to the sensor as well as their relative arrangement. This stands in contrast to systems that change configurations of filters by, for example, replacing filters (e.g., using a filter wheel or optical block turrets) and / or tuning filters in order obtain relatively high numbers of different emission bands (e.g., greater than or equal to 4, greater than or equal to 5, or greater). For example, if a filter wheel switches which filter is in the optical path, then more than one configuration of the filters is being used. Similarly, if an optical block turret rotates to switch which optical block is in the optical path, then more than one configuration is being used. But moving a filter’s position along the optical path without ultimately changing how the light is filtered is not considered a change in configuration of the filters in this context. By performing the filtering and, in some instances, other acts such as directing the excitation light and / or fluorescence light without changing filter configurations, the system may avoid high costs associated with such complex equipment and / or achieve improved accuracy by limiting or avoiding errors in matching image positions associated with moving optics. In some embodiments, the optics of the system comprise an optical block that is configured to perform the directing and filtering described above. For example, in system 100 in FIG. 2A and FIG. 3A, optics 110 comprise optical block 119. Optical blocks for fluorescence imaging are known and generally comprise a dichroic and an emission filter. A non-limiting example of a commercially-available optical block is the TRF89902-EM ET - 405 / 488 / 56 l / 647nm Laser Quad Band set employable with, for example, a 9100 series filter cube described above. In some embodiments, the optical block comprises a dichroic and a multi-bandpass emission filter. The multi-bandpass emission filter of the optical block my permit transmission of light with wavelengths in the at least n excitation bands. For example, in some embodiments where n=4, the optical block comprises a tetra-pass filter where the four bandpasses correspond to the 4 emission bands. As another example, in some embodiments where n=5, the optical block comprises a penta-pass filter where the five bandpasses correspond to the 5 emission bands. The optical block may be configured to be readily removable by the user from the optical path of the system. For example, in some embodiments, the optical block is a drop-in filter cube. In some embodiments, the optical block is an epi-cube. The epi-cube may comprise a dichroic and a multi-bandpass filter configured to filter the received fluorescence light into the filtered fluorescence light (e.g., having the at least n emission bands).

[0127] In some embodiments in which the optics comprise an optical block for performing the directing and filtering discussed above, the optics comprise a single (i.e., only one) optical block. The single optical block may be configured to filter the fluorescence light into the at least n excitation bands (e.g., where n may be at least 4, at least 5, or greater). As noted above, such a configuration stands in contrast to systems employing multiple optical blocks (e.g., multiple epi-cubes) for achieving relatively high numbers of emission bands. The use of a single optical block in the system (e.g., in combination with a multi-color sensor) is one non-limiting example of a configuration in which the optics can direct and filter the light across all of the at least n excitation bands and at least n emission bands without changing the filter configurations. In some embodiments, the system is configured to excite a sample on the sample holder with at least n excitation bands of the excitation light and detect the at least n emission bands of the filtered fluorescence light without replacing the optical block (e.g., where n is greater than or equal to 4, greater than or equal to 5, and / or up to 7). In some embodiments, the system is configured to excite a sample on the sample holder with at least n excitation bands of the excitation light and detect the at least n emission bands of the filtered fluorescence light without replacing any of the optics (e.g., where n is greater than or equal to 4, greater than or equal to 5, and / or up to 7). In some embodiments, the system is configured to excite a sample on the sample holder with at least n excitation bands of the excitation light (e.g., sequentially) and detect the at least n emission bands of the filtered fluorescence light without changing a configuration of any filters of the optics (e.g., where n is greater than or equal to 4, greater than or equal to 5, and / or up to 7).

[0128] While the systems and methods are described above in terms of fluorescence excitation and emission, it should be understood that the systems and methods may be employed with any of a variety of types of photo-induced emission in which emission light is emitted from the sample and filtered and directed to the sensor (e.g., a multi-color sensor).

[0129] The detector may be configured to produce at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands. For example, an excitation in each of the different excitation bands may be performed sequentially, with filtered fluorescence light from each sequential excitation being captured sequentially as different fluorescence acquisition images by the detector. The system may be configured such that a processor can obtain the at least n fluorescence acquisition images (e.g., as data transmitted from the detector, from one or more data stores, from a user). For example, the processor may be communicatively coupled to the detector such that the processor may obtain the at least n fluorescence acquisition images via any suitable wired or wireless communication network.

[0130] In some embodiments, the processor is configured to generate a plurality of spectral response images from the fluorescence acquisition images. In some embodiments, the plurality of spectral response images comprise multiple (e.g., at least two, at least three, or more) spectral response images from each of the at least n fluorescence acquisition images. The spectral response images refer to images produced or derived from a single type of sensing element in the sensor. For example, in embodiments where the sensor comprises first sensing elements (corresponding to a first type of sensing element such as pixels with a first spectral response) and second sensing elements (corresponding to a second type of sensing element such as pixels with a second, different spectral response), a fluorescence acquisition image captured by the sensor using all of the sensing elements may be decomposable into a first spectral response image corresponding to fluorescence light detected by the first sensing elements and a second spectral response image corresponding to fluorescence light detected by the second sensing elements. Because the first sensing elements and second elements have different spectral responses, as discussed above, the different spectral response images will contain different spectral information despite the spectral response images having been obtained simultaneously and superimposed in a composite fluorescence acquisition image when captured by the sensor. Accordingly, the number of spectral response images generated from each such fluorescence acquisition image captured by the sensor is equal to the number of different types of sensing elements. Accordingly, if the sensor comprises two different types of sensing elements, then each fluorescence acquisition image can be decomposed into two spectral response images, and so the at least n fluorescence acquisition images would produce at least 2n spectral response images. As another example, if the sensor comprises three different types of sensing elements (e.g., an R, G, and B sensor in a typical “RGB” sensor), then each fluorescence acquisition image can be decomposed into three spectral response images corresponding to the R, G, and B sensing elements, and so the at least n fluorescence acquisition images would produce at least 3n spectral response images.

[0131] As a more specific example, in embodiments where the sensor is an RGB sensor comprising red color pixels, green color pixels, and blue color pixels, a fluorescence acquisition image captured by the RGB sensor using all of the pixels may be decomposable into a red color image using just signal detected by the red pixels, a green image using just signal detected by the green pixels, and a blue color image using just signal detected by the blue pixels. Because the red, green, and blue pixels have different spectral responses, as discussed above, the different spectral response images will contain different spectral information despite the spectral response images being obtained simultaneously.

[0132] In some embodiments, the processor is configured to perform an unmixing of detected fluorescence light from the fluorophores excited in the sample. In such a way, the contributions to the fluorescence acquisition images of the spectral response images from the different fluorophores can be partially or completely deconvoluted (e.g., to reduce effects of crosstalk and spectral overlap of fluorescence spectra). The unmixing may therefore facilitate an ability to more accurately isolate signals from specific fluorophores (each of which may be associated with a different analyte (e.g., biomarker) in the sample. Details of how an unmixing procedure can be conducted are described in more detail below.

[0133] In some embodiments, the processor uses at least some of the plurality of spectral response images generated from the fluorescence acquisition images to perform the unmixing. In some embodiments, the processor all of the plurality of spectral response images to perform the unmixing. It has been realized in the context of this disclosure that sensors with multiple spectral responses (e.g., multi-color sensors such as RGB sensors) produce a greater number of spectral response images with which the unmixing can be performed compared to monochromatic sensors. Moreover, it has been realized in the context of this disclosure that the large number of spectral response images produced can facilitate determined or overdetermined unmixing, which increases the accuracy of the unmixing by avoiding underfitting. This can be particularly helpful when relatively large numbers of fluorophores are employed.

[0134] As a specific example, in embodiments in which at least 2n spectral response images are generated by the processor from the at least n fluorescence acquisition images (e.g., due to having at least two different types of sensing elements in the sensor), the processor may be configured to perform an unmixing of the detected fluorescence light from the fluorophores using at least some of the 2n spectral response images. In embodiments in which 3n spectral response images are generated by the processor from n fluorescence acquisition images using an RGB sensor, the processor may be configured to perform an unmixing of the detected fluorescence light from the fluorophores using at least some of the 3n spectral response images.

[0135] In some embodiments, the unmixing of the spectral response images results in the generation of a multilayer image set of the portion of the biological sample imaged. The multilayer image set may comprise layers corresponding to unmixed signals from the individual fluorophores that contributed fluorescence light to the detected signal. That is, an individual layer of the multilayer image set may show an image of the portion of the sample with signal intensities corresponding to the intensity of fluorescence produced by a single type of fluorophore in the sample. In some embodiments in which the sample comprises at least m different fluorophores (e.g., m different non-endogenous fluorophores), the multilayer image set comprises layers corresponding to unmixed signals from individual fluorophores of the at least m different fluorophores.

[0136] In some embodiments, the multilayer image set is a multilayer composite image of the at least a portion of the biological sample, where the multilayer composite image comprises the layers corresponding to unmixed signals from individual fluorophores. Each layer of the multilayer composite image may correspond to the unmixed signal from one of the individual fluorophores. The multilayer composite image may be, for example, a superimposed image of a stack of the layers (e.g., in a single data file).

[0137] In some embodiments, the multilayer image set is a plurality of individual layers of images of the at least a portion of the sample, where the plurality of individual layers comprise the layers corresponding to unmixed signals from individual fluorophores. Each layer of the plurality of layers may correspond to the unmixed signal from one of the individual fluorophores. The plurality of layers may be, for example, a collection of the layers (e.g., each in an individual data file).

[0138] In some embodiments, the system may be further configured, e.g., using the processor, to analyze the multilayer image set. For example, the processor may be configured to analyze one or more of the layers to determine the presence, density, and / or level of at least one analyte (e.g., a biomarker and / or a type of cell) in the at least a portion of the biological sample. Additionally or alternatively, the processor may be configured to analyze one or more layers to identify cellular objects and / or tissue regions, classify cells into cell types and / or functional states, and / or measure various parameters that capture biology of interest, which may include spatial information such as density and / or proximity. Any of a variety of image analysis techniques may be employed to perform such an analysis. For example, machine learning or artificial intelligence techniques may be performed. Alternatively or additionally, deterministic approaches such as watershed methods may be employed by the processor to perform the image analysis Further details of examples of such techniques are described below.

[0139] In some embodiments, at least some of the processes described above for multiplexed fluorescence detection using the systems and methods of this disclosure is performed on a first sub-region of the sample on the sample holder (e.g., a first tile of a slide), and then the system is adjusted to perform the processes again on a different subregion of the sample (e.g., a different tile of the slide). As described above, the system may, for example, (a) translate the sample and / or sample holder using the stage in one or more of the x-, y-, and z- directions and / or (b) move the optics in one or more of the x-, y-, and z- directions so that the system is configured to focus on and image a different sub-region (e.g., a different tile of the slide). This process may be repeated numerous times, e.g., until the whole sample (e.g., the whole slide) is imaged. It should be understood that various image processing steps describe above, such as the unmixing and / or generation of the multilayer image set, may be performed before, during, and / or after all of the different sub-regions of the sample are imaged.

[0140] The following is a non-limiting example of one way in which system 100 for detecting light emitted from a sample may be operated. In method 130a described by the flowchart of FIG. 1A, excitation light having at least n excitation bands is produced (as shown by act 130). In some embodiments, n is greater than or equal to 4 (e.g., at least 5, and / or up to 7). In some embodiments, the excitation light is produced by excitation source 105, as shown in FIG. 2A and FIG. 3A. In some, but not necessarily all embodiments, the excitation light is filtered into n excitation bands. For example, as shown in FIG. 2A, in some embodiments the excitation light is filtered into n excitation bands using n individual single-bandpass filters 106. In some embodiments, as shown in FIG. 3A, the excitation light is filtered into n excitation bands using a single multibandpass filter 121. According to act 131 shown in FIG. 1A, the excitation light is directed via an optical block (e.g., optical block 110 as shown in FIGS. 2A and 3A) to at least a portion of a biological sample (e.g., light 103 directed to the biological sample 111 on sample holder 109, supported by stage 108 as shown in FIGS. 2A and 3A).

[0141] According to act 132 as shown in FIG. 1A, the excitation light directed to the at last a portion of the biological sample excites fluorophores immobilized with respect to at least a portion of the sample such that the fluorophores emit fluorescence light (e.g., fluorescence light 104 as shown in FIGS. 2A and 3A). According to act 133 of FIG. 1A, the fluorescence light is filtered (e.g., via emission filter 117 in optical block 119 as shown in FIGS. 2A and 3 A) to produce filtered fluorescence light having at least n emission bands (e.g., filtered fluorescence light 118 as shown in FIGS. 2A and 3A). In act 134 of FIG. 1A, the filtered fluorescence light may then be directed, via the optical block, to a detector comprising a sensor (e.g., sensor 102 as shown in FIGS. 2A and 3A). Sensing elements of the sensor may then be used, according to act 135 of FIG. 1A, to detect the filtered fluorescence light. In some embodiments, the sensing elements of the sensor comprise first sensing elements that have a first spectral response and second sensing elements that have a second, different spectral response. For example, sensor 102 may be an RGB detector.

[0142] In some embodiments, the filtered fluorescence light detected by the detector may be used to acquire at least n fluorescence acquisition images. Act 136 in method 130b in FIG. IB corresponds to obtaining the acquired fluorescence acquisition images. In some embodiments, the fluorescence acquisition images are acquired using sensor 102 as shown in FIGS. 2A and 3 A.

[0143] Excitation light in the respective at least n excitation bands in act 130 may directed to sample 111 sequentially and the fluorescence light produced by each respective sequential excitation may be separately captured sequentially in different fluorescence acquisition images. For example, a first band of excitation light (k= 1 ) from a first individual light source of excitation source 105 may be directed by optics 110 to excite fluorophores immobilized in sample 111 and produce first fluorescence light that is filtered by optics 110 and directed to sensor 102 and detected as a first fluorescence acquisition image. Subsequently, a second, different band of excitation light (k=2) from a second individual light source of excitation source 105 may be directed by optics 110 to excite fluorophores immobilized in sample 111 and produce second fluorescence light that is filtered by optics 110 and directed to sensor 102 and detected as a second fluorescence acquisition image. This may be repeated, e.g., until an nth (e.g., 4th, or 5th, or more), different band of excitation light (k=n) from an nth individual light source of excitation source 105 is directed by optics 110 to excite fluorophores immobilized in sample 111 and produce nth fluorescence light that is filtered by optics 110 and directed to sensor 102 and detected as an nth (e.g., 4th, or 5th, or more) fluorescence acquisition image.

[0144] In some embodiments, according to act 137 of FIG. IB, a plurality (e.g., at least two, at least three, or more) spectral response images may be generated, e.g., using a processor, from each of the at least n fluorescence acquisition images. Accordingly, at least 2n spectral response images are generated. According to act 138 as shown in FIG. 1B, an unmixing (such a determined or overdetermined unmixing) of the detected filtered fluorescence light from the fluorophores may be performed using the spectral response images. In some embodiments, per act 139 of FIG. IB, a multilayer image set of the portion of the biological sample may be performed, the layers of the multilayer image set corresponding to the unmixed signals from the individual fluorophores.

[0145] Example Fluorescence Imaging Embodiment

[0146] Some embodiments of the systems and methods of this disclosure relate to multiplex immunofluorescence techniques, systems, and devices for independent analysis of 6 or more fluorophores on tissue sections obtained from a subject using a slide scanner equipped with one epi-cube consisting of quintuple pass emission filter and dichroic mirror, and an LED excitation source with 5 or more individually controlled LEDs, each of which has a single bandpass excitation filter. Some embodiments of the systems and methods of this disclosure provide devices, systems, and methods for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the device comprising a single epi-cube, an Light Emitting Diode referred to as LED excitation source, and an RGB color imaging sensor, wherein the epi-cube consists of multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with excitation filters designed to work with the emission filter and dichroics to excite fluorophores sequentially. Alternatively, a quintuple-pass excitation filter can be incorporated into the epi-cube rather than having 5 individual single bandpass filters in front of the LED light sources.

[0147] Some embodiments of the systems and methods of this disclosure relate generally to multiplexed immunofluorescence techniques. More specifically, some embodiments of the systems and methods of this disclosure relate to multiplexed immunofluorescence techniques, devices, systems, and methods for independent and accurate assessment of 6 or more fluorophore markers.

[0148] Multiplexed immunofluorescence is a popular method to reveal presence, density, arrangement, and functional state, among other things, of cells in healthy and diseased tissue. It has important applications in basic science to understand how organisms function, in translational research to develop better drugs and treatment options, and in clinical patient care to assist and guide treatment decisions. Since the biology of interest is often very complex, represented by 1000s of proteins, RNA and DNA molecules, most of which can be labelled with fluorescence markers, it is often helpful to increase the number of fluorophore markers in an experiment to facilitate a more comprehensive understanding of the biology in question, thus making information more specific to functional states and types of disease and revealing opportunities to intervene, among many other scientific or clinical goals. The scientific, academic, clinical and economic drivers behind these endeavors have led to the formation of the ‘Spatial Biology’ industry, which essentially means multiplex analyte imaging of tissue combined with image analysis and data analysis to glean biological information from the sample.

[0149] While goals are certainly worthwhile, platforms and technologies developed to deliver spatial biology information are generally overly complicated, expensive, and low throughput to provide practical economical and broadly adopted use, especially in the clinical setting often requiring simple and lower cost solutions.

[0150] Accordingly, there is a need in the art for better technical approaches to support simple and cost-effective and analytically robust multiplexed immunofluorescence analysis for tissue sections. Some embodiments of the systems and methods of this disclosure address this need by leveraging existing components commonly found in instruments and approaches, but arranged in a novel way that facilitates independent and accurate assessment of 6 or more fluorophore markers, resulting in an opto -mechanic al platform that can be mass-produced economically, that fits into laboratory operating models, and that effectively isolates and quantitates all fluorophore channels.

[0151] Some embodiments of the systems and methods of this disclosure provide multiplex immunofluorescence technology and methodology for independent analysis of 6 or more fluorophores on tissue sections with a single fluorescence epi-cube. Some embodiments of the systems and methods of this disclosure facilitate determined or overdetermined linear unmixing of 6 and up to 15 fluorophores using a slide scanner equipped with an epi-cube consisting of quintuple-pass emission filter and dichroic mirror, and an LED excitation source with 5 or more individually controlled LEDs, each of which has a single bandpass excitation filter. An alternative embodiment uses an epi- cube consisting of quintuple-bandpass emission filter, quintuple-pass dichroic mirror and quintuple- bandpass excitation filter, thus obviating the need for individual single bandpass filters in front of each LED, but potentially causing a larger degree of cross excitation among fluorophores.

[0152] Some embodiments of the systems and methods of this disclosure provides a device for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the device comprising a single epi-cube, a multi-Light Emitting Diode excitation light source referred to as LED excitation source, and a ‘Red-Green- Blue’ color imaging sensor, referred to as RGB color imaging sensor, wherein the epi- cube comprises a multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with an excitation filter to filter LED output to create a more narrow spectral range of excitation aligning spectrally with the bandpasses of the quintuple emission filter and dichroic, so that fluorophores are substantially excited individually and sequentially. In some embodiments, the epi-cube consists of the multibandpass emission filter and the dichroic. An alternative configuration replaces the 5 individual bandpass filters placed in front of an LED light sources with a single quintuple bandpass excitation filter located within the epi-cube.

[0153] Some embodiments of the systems and methods of this disclosure provide a system for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the system comprising a single epi-cube, a multi- Light Emitting Diode excitation light source referred to as LED excitation source, and an RGB color imaging sensor, wherein the epi-cube comprises a multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with an excitation filter to filter LED output to a single bandpass for each LED, each single bandpass aligning spectrally with one of the bandpasses of the quintuple emission filter and dichroic, so that fluorophores are substantially excited sequentially. In some embodiments, the epi-cube consists of the multi-bandpass emission filter and the dichroic. As described above, an alternative configuration replaces the 5 individual bandpass filters placed in front of an LED light source with a single quintuple bandpass excitation filter located within the epi-cube.

[0154] Some embodiments of the systems and methods of this disclosure provides a method for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the method comprising the steps of preparing a tissue slide from tissue sample from a subject, staining the slide with up to 15 fluorophore markers, processing the stained slide using a device for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, wherein the device comprises a single epi-cube, a multi- Light Emitting Diode excitation light source referred to as LED excitation source, and an RGB color imaging sensor, wherein the epi-cube comprises a multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with an excitation filter to filter LED output to a single bandpass for each LED, each single bandpass aligning spectrally with one of the bandpasses of the quintuple emission filter and dichroic, so that fluorophores are substantially excited sequentially. In some embodiments, the epi-cube consists of the multi-bandpass emission filter and the dichroic. As described above, an alternative configuration replaces the 5 individual bandpass filters placed in front of an LED light source with a single quintuple bandpass excitation filter located within the epi-cube.

[0155] FIG. 1C illustrates an embodiment of the methods of this disclosure and illustrates a flowchart describing a representative process as disclosed herein for staining and imaging a slide with 6 or more markers, including how RGB image tile planes are parsed into a 6- to 15-plane whole slide image and unmixed to produce isolated marker planes for image analysis.

[0156] FIG. 2B illustrates an embodiment of the systems of this disclosure and illustrates a representative optical layout of the microscope, with the distinguishing features being an RGB camera used in conjunction with a 5-band epi-cube to image slides with 6 or more fluorophores as disclosed herein.

[0157] FIG. 3B illustrates an embodiment of the systems of this disclosure and illustrates a representative alternative optical layout of the microscope, with the distinguishing features being an RGB camera using in conjunction with a 5-band epi-cube as shown in FIG. 2B, but with a single quintuple pass excitation filter incorporated into the epi-cube rather than 5 individual bandpass filters placed in front of an LED light source.

[0158] FIG. 4 illustrates an embodiment of the systems and methods of this disclosure and illustrates an example of a spectral response of the RGB imaging sensor, a typical quintuple bandpass emission filter, and emissions of 6 fluorophores. FIG. 5 illustrates an embodiment of the systems and methods of this disclosure and illustrates a representative example of a combined system spectral response to each of the 6 fluorophore emissions based in FIG. 4.

[0159] FIG. 6 illustrates an embodiment of the systems and methods of this disclosure and illustrates a representative example of the relative signals from 6 fluorophores in 10 image planes selected from the 15 resultant image planes created by combining a quintuple-pass emission filter, a 5-LED light source with individual control, and an RGB image sensor. This combination provides an over-determined unmixing matrix facilitating more accurate signal isolation and quantitation compared to attempting to unmix 6 fluorophores with just 5 image planes using an underdetermined matrix. Note that the signals are mostly isolated optically, to be expected with the clear differences between the signals detected by the RGB image sensor as shown in FIG. 5, alleviating dependence on mathematical unmixing to correct for excitation and spectral crosstalk between channels.

[0160] As discussed above, there exists an unaddressed need in the art for a more practical and cost-efficient and effective multiplexed immunofluorescence approach that avoids overly complicated and expensive opto-mechanical approaches for translational research and medical use in clinical settings. Some embodiments of the systems and methods of this disclosure address the need and provides a solution to the aforesaid problem by disclosing devices, systems, techniques, and methods for multiplex immunofluorescence slide scanning that leverages existing components but arranged in a novel way that facilitates independent and accurate assessment of 6-15 fluorophore markers with an opto-mechanical platform that can be mass-produced economically, that fits into laboratory operating models, and that effectively isolates and quantitates all fluorophore channels. Some embodiments of the systems and methods of this disclosure accomplish this with a single multi-bandpass epi-cube by utilizing an RGB color imaging sensor rather than a monochrome imaging sensor, to effectively triple the inputs into the unmixing matrix.

[0161] It is believed that all existing platforms and approaches designed and produced to image and unmix more than 5 fluorophores use multiple epi-cubes and monochrome imaging sensors. More than 1 epi-cube is used because a single epi-cube used with a monochrome imaging sensor cannot support more than 5 fluorophore excitation and emission spectral bands, due to practical limitations and ability to isolate signals from more than 5 fluorophores, thus limited determined linear unmixing to only 5 fluorophore channels. Monochrome imaging sensors are used because of their flat spectral response, for flexibility across fluorophore emission wavelengths and for perceived benefit of perfect registration of parameters within each image pixel. This may to some extent be a result of a legacy belief since early scientific CCD cameras for multiplex immunofluorescence of more than 3 fluorophores were exclusively monochrome sensors. Because of these established ways and misconceptions, platform developers have missed an opportunity to develop platforms capable of analyzing more than 5 fluorophores with a single epi-cube and have thus developed systems that are overly complicated and very expensive, due to the complexity needed to register images acquired with multiple epi-cubes. A novelty of this approach is the use of an RGB sensor rather than a monochrome sensor, coupled to a quintuple bandpass epi-cube and sequential 5-LED excitation. The RGB sensor provides sufficient spatial resolution despite sub-micron level pixel displacements between R, G, and B spectral bands (each imaging sensor pixel is dedicated to one of these three bands).

[0162] In some embodiments, with each LED excitation, an RGB image is acquired rather than a monochrome image, thus giving more spectral information and effectively 3 unmixing planes for each LED illumination. In the case of a 5-bandpass system with 5 individually controlled LEDs, imaging with and RGB sensor effectively yields 15 spectral channels to support linear unmixing. Not all planes need to be used, especially if some contain no useful information. This is important because the practical limit for number of fluorophores that can be effectively unmixed with a monochrome imaging sensor is 5 using a quintuple multi-bandpass epi-cube plus sequential LED illumination. Commercial systems that are capable of imaging more than 5 fluorophores require moving filter epi-cubes in and out of the optical path during scanning to provide more than 5 excitation-emission bandpass pairs. Opto-mechanical hardware capable of doing this and producing sub-pixel image registration across images acquired with different epi-cubes is substantially more complicated and expensive compared to systems that utilize just one epi-cube for a slide scan because of mechanical requirements for speed, precision, and durability. Additionally, moving epi-cubes in and out of the optical path during image acquisition adds significantly lengthens scanning time. Some embodiments of the systems and methods of this disclosure provide devices, systems, techniques, and methods to facilitate independent analysis of 6 or more fluorophores on tissue sections with a single epi-cube. Said differently and with more detail, it facilitates determined or over-determined linear unmixing of 6 to 15 fluorophores using a slide scanner equipped with a single epi- cube consisting of quintuple pass emission filter and dichroic mirror, and an LED excitation source with 5 or more individually controlled LEDs, each of which has a single bandpass excitation filter. Alternatively, a quintuple pass excitation filter can be incorporated into the epi- cube rather than having 5 individual single bandpass filters in front of the LED light sources.

[0163] In an embodiment of the systems and methods of this disclosure, a device is described for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the device comprising a single epi-cube, an Light Emitting Diode referred to as LED excitation source, and an RGB color imaging sensor, wherein the epi- cube consists of multi-bandpass emission filter and dichroic, each with 5 or more bandpass, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with excitation filters designed to work with the emission filter and dichroics to excite fluorophores sequentially. Alternatively, a quintuple pass excitation filter can be incorporated into the epi-cube rather than having 5 individual single bandpass filters in front of the LED light sources.

[0164] In another embodiment of the systems and methods of this disclosure providing the device as disclosed herein, wherein the RGB color imaging sensor allows the use of individual image planes (R, G, or B) associated with each LED excitation to perform linear decomposition, or ‘unmixing’, using a cross talk compensation matrix that is either determined or over- determined, rather than undetermined which can lead to inaccurate or ambiguous results.

[0165] In an embodiment of the systems and methods of this disclosure, a system is described for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the system comprising a single epi-cube, an Light Emitting Diode referred to as LED excitation source, and an RGB color imaging sensor, wherein the epi- cube consists of multi-bandpass emission filter and dichroic, each with 5 or more bandpass, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with excitation filters designed to work with the emission filter and dichroics to excite fluorophores sequentially. Alternatively, a quintuple pass excitation filter can be incorporated into the epi-cube rather than having 5 individual single bandpass filters in front of the LED light sources.

[0166] In another embodiment of the systems and methods of this disclosure providing the system as disclosed herein, wherein the RGB color imaging sensor allow the use of individual image planes (R, G, and B) associated with each LED excitation to perform linear decomposition, or ‘unmixing’, using a cross talk compensation matrix that is either determined or over- determined, rather than undetermined, which can lead to inaccurate or ambiguous results.

[0167] In an embodiment of the systems and methods of this disclosure, a method is described for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the method comprising the steps of preparing a tissue slide from tissue sample from a subject, staining the slide with up to 15 fluorophore markers, processing the stained slide using a device for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, wherein the device comprises a single epi-cube, an Light Emitting Diode referred to as LED excitation source, and an RGB color imaging sensor, wherein the epi-cube consists of multi-bandpass emission filter and dichroic, each with 5 or more bandpass, and wherein the LED excitation source with 5 or more individually controlled LEDs, each with excitation filters designed to work with the emission filter and dichroics to excite fluorophores sequentially. Alternatively, a quintuple pass excitation filter can be incorporated into the epi-cube rather than having 5 individual single bandpass filters in front of the LED light sources.

[0168] In another embodiment of the systems and methods of this disclosure providing the method as disclosed herein, wherein the RGB color imaging sensor allow the use of individual image planes (R, G, or B) associated with each LED excitation to perform linear decomposition, or ‘unmixing’, using a cross talk compensation matrix that is either determined or over-determined, rather than undetermined, which can lead to inaccurate or ambiguous results.

[0169] FIG. 1C provides an example of an embodiment of the methods of this disclosure and provides the overall method of the an embodiment of this disclosure, including the steps consisting of staining or labeling a tissue slide obtained from a subject with 6 or more fluorophores, up to 15 fluorophores, which can utilize a variety of immunofluorescence detection methods including direct, indirect, TSA-amplified, oligoconjugated, and oligo-conjugated plus rolling circle or branch DNA amplification. Staining can be performed manually on a lab bench or on commercial auto staining robots, such as those from Danaher, Roche, and Agilent.

[0170] FIG. 2B provides an example of an embodiment of the systems of this disclosure and provides the procedure of imaging a tissue slide obtained from a subject with the invented slide scanning system as disclosed herein, which consists of: A) LED excitation source 305 with at least 5 LEDS, each individually controllable and with excitation clean-up filters 306 to make each excitation light limited to a small wavelength range to support isolation of excitation light from detected emission wavelength range. B) An epicube 319 consisting of a quintuple emission bandpass filter 317 that transmits only emission light when LED excitation is occurring and blocks all excitation light, and a quintuple pass dichroic mirror 316 with selectively reflects excitation light and transmits emitted fluorescence light. C) A microscope objective lens 307 that focuses excitation light 303 on the tissue area to be imaged and collects emission light 304 that is then delivered to and spatially resolved on the RGB imaging sensor as filtered fluorescence light 318. D) Slide 309 and objective mechanical automation 308 to move the slide in horizontal X and Y directions and objective mechanical automation 320 that moves the objective relative to the slide surface to bring images on the imaging sensor into focus. E) An RGB imaging sensor that captures digital red, green, and blue image planes, according to the spectral response profiles shown in FIG. 4 described below.

[0171] FIG. 3B provides an alternative example of an embodiment of the systems and methods of this disclosure and provides the procedure of imaging a tissue slide obtained from a subject with the invented slide scanning system as disclosed herein, which consists of: A) LED excitation source 305 with at least 5 LEDS, each individually controllable. B) An epi-cube 319 consisting of a quintuple emission bandpass filter 317 that transmits only emission light when LED excitation is occurring and blocks all excitation light, a quintuple pass dichroic mirror 316 with selectively reflects excitation light 303 and transmits emitted fluorescence light 304, and a quintuple pass excitation filter 321 that selects wavelengths from the LED light source to illuminate the sample, to minimize excitation light from reaching the imaging sensor so that acquired images are of fluorescence emission rather than excitation light. C) A microscope objective lens 307 that focuses excitation light on the image area and collects emission light that is then delivered to and spatially resolved on the RGB imaging sensor. D) Slide 109 and objective mechanical automation 108 to move the slide in horizontal X and Y directions and objective mechanical automation 320 that moves the objective relative to the slide surface to bring images on the imaging sensor into focus. E) An RGB imaging sensor 102 that captures digital red, green, and blue image planes, according to the spectral response profiles shown in FIG. 4 described below.

[0172] FIG. 4 provides an image acquisition system and method as disclosed herein, which consists of: a) capturing rectangular areas, often called tiles, one at a time, on a rectangular grid that covers the area of interest across the slide; b) for each tile, individual FED excitation sources are illuminated and RGB images are captured, which produces R, G, and B image planes for each FED excitation; c) individual image planes (R, G, and / or B) across FED excitations are selected for optimum information gathering and with sufficient number to avoid under-determined linear unmixing, and combined into a single multilayer composite image, with as many as 15 image planes, based on 5 FED excitations and 3 (R, G, and B) image planes per FED excitation; and d) these composite tiles are either first unmixed to create fluorophore- specific image planes and then merged into a single whole-slide or whole-tissue section image for analysis, or first merged into a single whole-slide image for unmixing and analysis. Unmixing consists of a linear decomposition of the signals at each image pixel of the image according to an unmixing matrix determined either through mathematical modelling or empirically using single- stained slides. FIG. 4 shows a spectral profile of the 6 examples fluorophores, with the sensor red response 401, green response 402, and blue response 403 as well as the transmission from the example fluorophores #1 (411), #2 (412), #3 (413), #4 (414), #5 (415), and #6 (416). In FIG. 4, the quintuple bandpass filter transmission 420 is also shown. FIG. 5 and FIG. 6 show the characteristic RGB spectral profiles of the 6 example fluorophores, and representative eigenvectors for cross talk compensation, demonstrating that the additional spectral information facilitated by the use of an RGB sensor with a quintuple pass filter cube facilitates mathematically determined linear decomposition of 6 or more fluorophores. The unmixed composite image is then ready for image analysis, which typically uses algorithms such as those based on Artificial Intelligence (Al) and deep learning or conventional deterministic approaches such as watershed methods, to identify cellular objects and tissue regions, classifies cells into cell types and functional states, and measures various parameters that capture biology of interest, which usually includes spatial information such as density and proximity.

[0173] Chromogenic Imaging

[0174] Certain embodiments described in this disclosure involve systems and methods for detecting light transmitted through a sample (e.g., biological sample) and, in some instances, generating one or more unmixed images of the sample using a relatively large number of different chromophores (e.g., greater than or equal to 4, greater or equal to 5, greater than or equal to 6, or greater). In some such embodiments, the unmixing can be accomplished in a determined or overdetermined manner without relying on complex opto-electronic and / or complex opto-mechanical components. For example, the systems and methods may employ optics with one or more actuatable filters (e.g., actuatable bandpass filters) in the optical path of the system. In some such embodiments, such optics may be employed in conjunction with the sensor with multiple spectral responses (e.g., an RGB sensor) described above. The system’s components (e.g., an illumination source, sample holder, detector with sensor, and / or the optics such as actuatable filter) may be part of a microscope (e.g., brightfield imaging microscope). The microscope may be, for example, a slide-scanning microscope.

[0175] In some embodiments, the system comprises optics for chromogenic imaging (e.g., multiplexed immunohistochemistry imaging) of samples. In some embodiments, the optics are configured to direct illumination light from an illumination source (e.g., a broadband illumination source), direct the light through at least a portion of a sample on a sample holder, and to the sensor. For example, referring to FIG. 8A, optics 210 direct illumination light 212 from illumination source 201 through sample 211 on sample holder 209 and to sensor 202 (which may be on and / or within a detector). The optics may be configured to perform the above-described function by including appropriate light-directing optics and / or light-conditioning optics such as mirrors and lenses (e.g., an objective lens). For example, in FIG. 8B, optics 210 include objective lens 207 for focusing light. Objective lens 207 may be adjustable to vary the Z-focus on the sample as represented by arrows 220. The sample’s chromophores may absorb some or all light at various wavelengths of the illumination light, thereby reducing transmission of the light at those various wavelengths. The reduction in transmission of light varies spatially depending on the spatial distribution of the chromophores, so that the transmitted light directed to the sensor can be detected to form an image of the portion of the sample interrogated. The intensity of signal at each sensing element (e.g., each pixel) of the sensor will vary based on the amount of light transmitted and directed to that sensing element. This permits the calculation of an optical density at each sensing element, to form an optical density image.

[0176] The sample holder, illumination source, and detector may establish an optical path for illumination light from the illumination source to travel from the illumination source, through at least a portion of a sample holder, and to the detector. For example, in FIGS. 8A-8B, optical path 213 is established by illumination source 201, sample holder 209, and sensor 202, along which illumination light 212 from illumination source 201 travels when sample 211 is to be imaged by system 200. The transmitted illumination light may be directed to the detector such that it is incident upon the sensing elements (e.g., pixels or subpixels) of the sensor of the detector (e.g., the pixels of a CCD sensor or a CMOS sensor). The sensor, in or on the detector, may then generate electrical signal responsive to the incident transmitted light, as discussed above. The system may be further configured, using electronics components controlled by one or more processors, to transmit the electrical signal for processing (e.g., image generation and / or analysis).

[0177] The illumination light may be in any of a variety of regions of the electromagnetic spectrum. In some embodiments, the illumination light comprises wavelengths in the visible region of the electromagnetic spectrum (e.g., wavelengths greater than or equal to 400 nm and less than or equal to 700 nm). In some embodiments, the illumination light comprises wavelengths in the near-infrared region of the electromagnetic spectrum (e.g., wavelengths greater than 700 nm and less than or equal to 1400 nm). In some embodiments, the illumination light comprises wavelengths in the near-ultraviolet region of the electromagnetic spectrum (e.g., wavelengths greater than or equal to 200 nm and less than 400 nm). In some embodiments, the illumination light comprises wavelengths in each of the near-ultraviolet, visible, and near-infrared regions of the electromagnetic spectrum. In some embodiments, the illumination light is in the form of broadband electromagnetic radiation (e.g., from a white light source).

[0178] Any of a variety of illumination sources may be employed. For example, the illumination source may be any commonly employed in chromogenic imaging of biological samples (e.g., using microscopy). In some embodiments, the illumination source is configurated to produce broadband electromagnetic radiation (e.g., white light). For example, the illumination source may comprise a lamp (e.g., a halogen lamp, a tungsten lamp, a xenon lamp) and / or one or more LEDs.

[0179] As noted above, the sensor (e.g., sensor 202) may be a multi-color sensor. As discussed above, one such implementation of a multi-color sensor is where the detector comprises a color filter array, the presence of which results in the sensor comprising at least first color pixels or subpixels having a first color spectral response, second color pixels or subpixels having a second color spectral response, and third color pixels or subpixels configured having a third color spectral response, where the first color spectral response, second color spectral response, and third color spectral response are different. For example, the detector may be configured such that the sensor is an RGB sensor. In terms of an RGB detector, the first color would be red, the second color would green, and the third color would be blue.

[0180] In some embodiments, the system comprises an actuatable filter. The actuatable filter is induced to change one or more configurations (e.g. properties, position, state of motion, etc.) of the filter in response to an actuating signal such as an electrical signal from a controller (e.g., of a computing system). Such different configurations of the actuatable filter may correspond, for example, to different locations of the actuatable filter within the system. In some embodiments, the transmission profile of the filter is unchanged in its various configurations, but the location of the filter is changed upon actuation (e.g., based on a signal from a controller). In some embodiments, the actuatable filter has a first configuration in which the actuatable filter is not located in the optical path and a second configuration in which the actuatable filter is located in the optical path. For example, referring again to FIGS. 8 A, actuatable filter 203a may be translatable according to the double arrows shown such that actuatable filter 203a is in optical path 213 when in its second configuration (which is shown in FIGS. 8A-8B) but not in optical path 213 when in its first configuration (not shown). As such, when the actuatable filter is in its first configuration, the illumination light does not pass through the actuatable filter, but when the actuatable filter is in its second configuration, the illumination light does pass through the actuatable filter.

[0181] The actuatable filter may, in its different configurations, permit an ability to vary the spectral responses expected for the various sensing elements of the sensor (e.g., by reducing or eliminating certain ranges of wavelengths of illumination light from reaching the sensor). This variance in spectral responses between the different filter configurations may provide an ability to produce a greater number of spectral response images for a given multi-color sensor (e.g., an RGB sensor). The greater number of spectral response images may permit for more accurate unmixing for imaging the chromophores, as discussed in more detail below.

[0182] Movement of the actuatable filter into and out of the optical path may be controlled using any of a variety of components. For example, the position of the actuatable filter may be mechanically-controlled. In some embodiments, the position of the actuatable filter is automatically-controlled. For example, in some embodiments, the actuatable filter is coupled to a servomechanism that can permit precise control of the angular or linear position of the filter (e.g., based on signal from a controller). Other non-limiting implementations for automatically actuating a change in position of the filter include, but are not limited to a solenoid-actuated filter, a stepper motor- actuated filter, a filter with a continuous motor with detents, a filter actuated with a limit switch, and / or a filter actuated with a sensor switch. However, in some embodiments, the position of the actuatable filter is manually controlled.

[0183] The actuatable filter may be positioned in any of a variety of locations in the system such that it can toggle between being in the optical path and not being in the optical path. In some embodiments, when in its second configuration, the actuatable filter is located in a portion of the optical path from the illumination source to the sample holder. Actuatable filter 203a in FIGS. 8A-8B is an example of such a configuration, where illumination light 212 from illumination source 201 passes through filter 203a prior to reaching sample 211 on sample holder 209. Alternatively or additionally, in some embodiments, when in its second configuration, the actuatable filter is located in a portion of the optical path from the sample holder to the detector. Actuatable filter 203b in FIG. 8B is an example of such a configuration, where illumination light 212 from illumination source 201 passes through filter 203b after passing through sample 211 on sample holder 209 but before reaching sensor 202. While FIG. 8B shows both actuatable filter 203a and actuatable filter 203b in the same figure, in some embodiments the system comprises only a single actuatable filter (e.g., only filter 203a or only filter 203b). However, in some embodiments, the system comprises multiple actuatable filters (e.g., a first actuatable filter and a second actuatable filter, each with the two aforementioned configurations). In some embodiments, the second actuatable filter has a different transmission profile than the first actuatable filter, such that the resulting color spectral responses for the sensing elements of the sensor are different. Such an embodiment may provide for the generation of an even greater number of different spectral response images from a sample in a given experiment, for even more accurate unmixing and / or analysis and / or for an ability for determined or overdetermined unmixed multiplexed imaging of an even greater number of chromophores.

[0184] In some embodiments, the actuatable filter is configured to reduce (e.g., decrease or eliminate) transmission of a portion of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response. The filter may be so configured via, for example, appropriate selection of the transmission profile for the actuatable filter such that a portion of the wavelengths in the spectral response(s) are filtered. For example, in some embodiments where the sensor is an RGB sensor, the actuatable filter is configured to reduce (e.g., decrease or eliminate) transmission of a portion of wavelengths in the red color response, a portion of wavelengths in the green color response, and / or a portion of wavelengths in the blue color response. FIG. 8E, for example, shows an illustrative plot of red, green, and blue spectral responses when the actuatable filter is absent from the optical path (top) and when the actuatable filter is present in the optical path (bottom), according to some embodiments. In FIG. 8E, the actuatable filter is a bandpass filter configured to block transmission of approximately half of the wavelengths in the red, green, and blue spectral responses as indicated by the blocked regions shown as dashed rectangles.

[0185] In some embodiments, the actuatable filter is configured to reduce transmission of greater than or equal to 30%, greater than or equal to 40%, greater than or equal to 45%, greater than or equal to 50%, and / or up to 55%, up to 60%, up to 70%, or more of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response. Combinations of these ranges are possible (e.g., greater than or equal to 30% and less than or equal to 70%, greater than or equal to 40% and less than or equal to 60%, or greater than or equal to 45% and less than or equal to 55%). In some embodiments, the actuatable filter is configured to reduce transmission of greater than or equal to 30%, greater than or equal to 40%, greater than or equal to 45%, greater than or equal to 50%, and / or up to 55%, up to 60%, up to 70%, or more of the wavelengths in each of the first color spectral response, the second color spectral response, and the third color spectral response. Combinations of these ranges are possible (e.g., greater than or equal to 30% and less than or equal to 70%, greater than or equal to 40% and less than or equal to 60%, or greater than or equal to 45% and less than or equal to 55%). As an illustrative example, if the blue spectral response for the sensor spans from 400 nm to 460 nm and the filter reduces transmission of wavelengths from 430 nm to 460 nm, then the filter would be considered to reduce transmission of 50% of the wavelengths in the blue spectral response because 30 nm is 50% of the 60-nm width of the spectral response. In some embodiments, the wavelengths for which transmission is reduced by the actuatable filter undergo an average reduction in transmission of at least 30%, at least 50%, at least 75%, at least 80%, at least 90%, at least 95%, at least 99%, or 100% compared to the transmission at those wavelengths in the absence of the filter.

[0186] In some embodiments, the actuatable filter is a bandpass filter. The bandpass filter may have one or more bands that permit transmission of a first portion of wavelengths of some or all of the color spectral responses of the sensor but reduce (e.g., eliminate) transmission of a second portion of the wavelengths in some or all of the color spectral responses of the sensor. In some embodiments, the filter is a multi-bandpass filter (e.g., with at least three passbands, at least four passbands, or more).

[0187] In operation, the system may be configured to direct the illumination light during two different periods of time with two different configurations of the actuatable filter to acquire color intensity acquisition images of the sample. For example, in some embodiments, the system is configured to direct, during a first period of time, the illumination light along the optical path from the illumination source, through at least a portion of the sample (e.g., biological sample), and to the detector such that the transmitted illumination light is incident upon the sensing elements (e.g., pixels or subpixels) of the sensor for detection of a first color intensity acquisition image of the sample. In some such embodiments, the filter is not located in the optical path during the first period of time (e.g., the filter may be in its first configuration). In some such embodiments, the sample comprises multiple different immobilized chromophores (e.g., non-endogenous chromophores such as histochemistry stains). In some embodiments, the system is further configured to direct, during a second, different period of time, illumination light along the optical path to pixels or subpixels of the detector for detection of a second image of the sample. The sample may be in the same position during both the first period of time and the second period of time such that the same portion of the sample is imaged during both periods of time. In contrast to the first period of time, the actuatable filter may be located in the optical path during the second period of time (e.g., the filter may be in its second configuration). The terms “first” period of time and “second” period of time are used for convenience in differentiating the periods of time, but are not meant to imply any particular order. For example, in some embodiments the first period of time occurs before the second period of time, while in other embodiments the first period of time occurs after the second period of time. It should be understood that immobilized chromophores include any that are not free to move away from the sample or are inhibited from moving away from the sample under prevailing sample conditions.

[0188] In some embodiments, the system is configured to acquire color intensity acquisition images from the illumination light that was directed to the detector during the respective periods of time (e.g., with the actuatable filter in the respective configurations). For example, the system may detect the illumination light that was directed to the detector during the first period of time to acquire a first color intensity acquisition image. Additionally, the system may also detect the illumination light that was directed to the detector during the second period of time to acquire a second color intensity acquisition image. These color intensity acquisition images refer to the raw images (e.g., the photon counts or charge / current intensity detected by the sensor pixels or subpixels).

[0189] In some embodiments in which the sensor comprises first color pixels or subpixels, second color pixels or subpixels, and third color pixels or subpixels, each of the color intensity acquisition image comprises respective color intensity spectral response images corresponding to the different types of color pixels or subpixels.

[0190] For example, the first color intensity acquisition image may comprise (e.g., be a composite of) a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third color intensity spectral response image corresponding to the third color pixels or subpixels. Similarly, the second color intensity acquisition image may also comprise (e.g., be a composite of) a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third color intensity spectral response image corresponding to the third color pixels or subpixels.

[0191] Using an RGB sensor as an illustrative example, the system may be configured to acquire a first color intensity acquisition image that is a composite of a red spectral response image, a green spectral response image, and a blue spectral response image during a first period of time when the filter is absent from the optical path, and the system may further be configured to acquire a second color intensity acquisition image that is a composite a red spectral response image, a green spectral response image, and a blue spectral response image during a second period of time when the filter is present in the optical path. Because the actuatable filter changes the spectral response of one or more of the red, green, or blue pixels or subpixels, then the constituent color spectral response images of the first color intensity acquisition image and the second color intensity acquisition image will be different, thereby resulting in a greater total number of different color intensity spectral response images of the imaged sample than if the actuatable filter were not used. The greater number of spectral response images may facilitate determined or overdetermined unmixing of the acquired images of the sample for a greater number of different chromophore labels in the sample than would be available without the actuatable filter.

[0192] In some embodiments, the system is configured to generate optical density spectral response images from color intensity acquisition images and / or their constituent color intensity spectral response images. An optical density image may be generated from a color intensity acquisition image by converting the transmitted light signals (e.g., intensity) detected by each sensing element (e.g., each color pixel) that makes up the color intensity acquisition images to an optical density value for that sensing element (e.g., that color pixel) in the image. The conversion to an optical density value for each sensing element (e.g., pixel) can be performed, for example, using reference images taken of a blank (e.g., a blank slide with no sample present) to calculate percent transmission (%T). The %T may be calculated by dividing the measured signal intensity by the reference value intensity at each sensing element (e.g., pixel) and multiplying by 100. Then, the equation optical density = logio (100 / %T) may be used to calculate optical density from the % T values. Optical density may be approximately proportional to quantity of chromophore.

[0193] In some embodiments, a first optical density image is calculated from the first color intensity acquisition image (e.g., acquired when the actuatable filter was absent from the optical path). That is, color optical density spectral response images may be calculated from the respective color intensity spectral response images of the first color acquisition image. For example, the first color intensity spectral response image of the first color intensity acquisition image may be converted into a first color optical density spectral response image corresponding to the first color (e.g., red) pixels or subpixels. Similarly, the second color intensity spectral response image of the first color intensity acquisition image may be converted into a second color optical density spectral response image corresponding to the second color (e.g., green) pixels or subpixels. Further, the third color intensity spectral response image of the first color intensity acquisition image may be converted into a third color optical density spectral response image corresponding to the third color (e.g., blue) pixels or subpixels. The overall first optical density image is the composite of the color optical density spectral response images calculated from the first color intensity acquisition image (e.g., as a single superimposed image or as a collection of the individual color optical density spectral response images).

[0194] In some embodiments, a second optical density image is calculated from the second color intensity acquisition image (e.g., acquired when the actuatable filter was present in the optical path). That is, color optical density spectral response images may be calculated from the respective color intensity spectral response images of the second color acquisition image. For example, the first color intensity spectral response image of the second color intensity acquisition image may be converted into a first color optical density spectral response image corresponding to the first color (e.g., red) pixels or subpixels. Similarly, the second color intensity spectral response image of the second color intensity acquisition image may be converted into a second color optical density spectral response image corresponding to the second color (e.g., green) pixels or subpixels. Further, the third color intensity spectral response image of the second color intensity acquisition image may be converted into a third color optical density spectral response image corresponding to the third color (e.g., blue) pixels or subpixels. The overall second optical density image is the composite of the color optical density spectral response images calculated from the second color intensity acquisition image (e.g., as a single superimposed image or as a collection of the individual color optical density spectral response images). The color optical density spectral response images may be two-dimensional arrays of optical density values, each element of the array corresponding to a sensing element (e.g., a color pixel) of the detector.

[0195] The system may be configured to facilitate unmixing for relatively highly multiplexed chromogenic imaging of samples (e.g., to interrogate for a relatively high number of different biomarkers in the sample simultaneously). In some embodiments, the multiple different immobilized chromophores in the sample comprise at least 4, at least 5, and / or up to 6, up to 7, up to 8, up to 10, up to 12, or more different immobilized chromophores (e.g., each having a different absorption spectrum from each other). Using multiple different actuatable filters with different passbands may facilitate use of higher numbers of different immobilized chromophores. In some embodiments, the number of different immobilized chromophores (e.g., each having a different absorption spectrum from each other) interrogated with determined or overdetermined unmixing may be greater than the number of different types of color pixels or subpixels in the sensor and may be less than or equal to the number of different types of color pixels or subpixels in the sensor multiplied by the quantity f + 1, where f is equal to the number of different actuatable filters used. For example, in some embodiments where an RGB sensor and a single actuatable filter (f = 1) are used, the number of different immobilized chromophores interrogated may be greater than or equal to 4 (which is greater than the 3 colors of the RGB sensor) and less than or equal to 6 (which is 3 times the quantity (1+1)). As another example, in some embodiments where an RGB sensor and two different actuatable filters (f = 2) are used, the number of different immobilized chromophores interrogated may be greater than or equal to 4 (which is greater than the 3 colors of the RGB sensor) and less than or equal to 9 (which is 3 times the quantity (2+1)). Increasing the number of actuatable filters and / or the number of different types of color pixels in the sensor may permit greater multiplexed imaging of chromophores.

[0196] In some embodiments, the system is configured (e.g., using a processor) to generate a multilayer image set of the at least a portion of the biological sample using at least some of the color optical density spectral responses images from or derived from the color optical density spectral response images described above. The number of color optical density spectral response images in the multilayer image set may be equal to or exceed the number of different chromophores to be analyzed in the sample. For example, in some embodiments in which four different chromogen labels are employed in the sample, the multilayer image set is constructed to have at least four total color optical density spectral response images. For an RGB detector, then at least some of the color optical density spectral response images would may be obtained or derived from the first optical density image and at least some of the color optical density spectral response images may be obtained or derived from the second optical density spectral response images in order for at least four color optical density spectral response images to be obtained given that one RGB image can only produce three color spectral response images (the red color spectral response, the green color spectral response, and the blue color spectral response).

[0197] In some embodiments, the multilayer image set is a multilayer composite image of the at least a portion of the biological sample, where the multilayer composite image comprises the color optical density spectral response images as layers. The multilayer composite image may be, for example, a superimposed image of a stack of the layers (e.g., in a single data file).

[0198] In some embodiments, the multilayer image set is a plurality of individual layers of images of the at least a portion of the sample, where the plurality of individual layers comprise the color optical density spectral response images as layers. The plurality of layers may be, for example, a collection of the layers (e.g., each in an individual data file).

[0199] In certain specific exemplary embodiments, some of the layers of the multilayer image set are generated by a subtraction procedure performed using the color intensity acquisition images or their respective optical density images (that is, taking a difference or any equivalent linear combination to taking a difference of the color intensity acquisition images or their respective optical density images). For example, in some embodiments, the second color acquisition intensity image is subtracted from the first color intensity acquisition image. Such a subtraction can be performed by directly subtracting these composite color intensity acquisition images from each other, or by individually taking the differences of their respective color intensity spectral response images corresponding to the first color pixels or subpixels, second color pixels or subpixels, and third color pixels or subpixels. The resulting “difference” color intensity acquisition image may then be converted to a “difference” optical density image according to the equation provided above. Alternatively, in some embodiments, the second optical density image (calculated from the second color intensity acquisition image as described above) is subtracted from the first optical density image (calculated from the first color intensity acquisition image as described above). Such a subtraction can be performed by directly subtracting these composite optical density images from each other, or by individually taking the differences of their respective color optical density spectral response images corresponding to the first color pixels or subpixels, second color pixels or subpixels, and third color pixels or subpixels. At least some of the resulting “difference” color optical density spectral response images may be used as layers in the multilayer image set. In some embodiments, subtracting optical density images (or their constituent color optical density spectral response images) comprises subtracting the optical density values of pixels of one optical density image (or its constituent color optical density spectral response images) from the optical density values of corresponding pixels of the other optical density image (or other the constituent color optical density spectral response images of the other optical density image). It has been realized in the context of this disclosure that the aforementioned generation of a “difference” optical density image via subtraction may create a greater separation of color spectral response images used for subsequent unmixing than when using those from just the first color intensity acquisition image and the second color intensity acquisition image, which may improve unmixing accuracy

[0200] In some embodiments, the multilayer image set comprises layers comprising two of the following: (1) one or more (at least two, at least three, or more) color optical density spectral responses images calculated from at least some of the color intensity spectral response images of the first color intensity acquisition image,

[0201] (2) one or more (at least two, at least three, or more) color optical density spectral responses images calculated from at least some of the color intensity spectral response images of the second color intensity acquisition image, and

[0202] (3) one or more (at least two, at least three, or more) color optical density spectral responses images calculated from:

[0203] (i) subtraction of the second color acquisition intensity image from the first color intensity acquisition image, or

[0204] (ii) subtraction of (a) a second optical density image calculated from the second color intensity acquisition image from (b) a first optical density image calculated from the first color intensity acquisition image.

[0205] The multilayer image set comprising layer comprising two of (1), (2), and (3) above means that the multilayer image set may comprise (1) and (2), (2) and (3), or (1) and (3). In some such embodiments, the multilayer image set comprises layers comprising (1) and (2). In some embodiments, the layers consist of (1) and (2). In embodiments, the multilayer image set comprises layers comprising (1) and (3). In some embodiment, the layers consist of (1) and (3). In some embodiments, the multilayer image set comprises layers comprising (2) and (3). In some embodiments, the layers consist of (2) and (3). A sufficient number of layers from two of (1), (2), and (3) may be used such that the total number of layers in the multilayer image set is greater than or equal to the number of chromophores to be analyzed with unmixing. In some embodiments in which the multilayer image set comprises layers comprising (3), the color optical density spectral responses images of (3) are calculated from (i). In some embodiments in which the multilayer image set comprises layers comprising (3), the color optical density spectral responses images of (3) are calculated from (ii).

[0206] In some embodiments, the processor is configured to perform an unmixing of the chromogenic signal (e.g., optical density values) detected from the chromophores in the sample. For example, the processor may be configured to performing an unmixing of the multilayer image set to generate unmixed images, each corresponding to one of the multiple different immobilized chromophores in the sample. In such a way, the contributions to the optical density images of the spectral response images from the different chromophores can be partially or completely deconvoluted (e.g., to reduce or eliminate effects of crosstalk and spectral overlap of absorption spectra). The unmixing may therefore facilitate an ability to more accurately isolate signals from specific chromophores (each of which may be associated with a different analyte (e.g., biomarker) in the sample. Details of how an unmixing procedure can be conducted are described in more detail below.

[0207] It has been realized in the context of this disclosure that sensors with multiple spectral responses (e.g., multi-color sensors such as RGB sensors) used in combination with an actuatable filter that can alter the spectral profile of light directed to the sensors can produce a greater number of color optical density spectral response images with which the unmixing can be performed compared to monochromatic sensors or systems that do not employ such an actuatable filter. Moreover, it has been realized in the context of this disclosure that the large number of color optical density spectral response images produced can facilitate determined or overdetermined unmixing, which increases the accuracy of the unmixing by avoiding underfitting. This can be particularly helpful when relatively large numbers of chromophores are employed.

[0208] As a specific example, in embodiments in which at least 6 total color optical density spectral response images are generated by the processor from two or more of the first optical density image, the second optical density image, and the “difference” optical density image described above (e.g., due to the sensor having at least three different color pixels or subpixels and the color spectral responses being altered by the actuatable filter), the processor may be configured to perform an unmixing of the chromogenic signal from the chromophores using at least some of the 6 color optical density spectral response images.

[0209] In some embodiments, the unmixing of the color optical density spectral response images results in the generation of an unmixed multilayer image set of the portion of the biological sample imaged. The unmixed multilayer image set may comprise layers corresponding to unmixed signals from the individual chromophores that contributed to the absorption of the illumination light and the optical density values detected by the sensor. That is, an individual layer of the unmixed multilayer image set may show an image of the portion of the sample with signal intensities corresponding to the absorption produced by a single type of chromophore in the sample. In some embodiments in which the sample comprises at least 4 different chromophores (e.g., m different non- endogenous chromophores), the unmixed multilayer image set comprises layers corresponding to unmixed signals from individual chromophores of the at least 4 different chromophores.

[0210] In some embodiments, the system is further configured, e.g., using the processor, to analyze the unmixed multilayer image set. For example, the processor may be configured to analyze one or more of the layers to determine the presence, density, and / or level of at least one analyte (e.g., a biomarker and / or a type of cell) in the at least a portion of the biological sample. Additionally or alternatively, the processor may be configured to analyze one or more layers to identify cellular objects and / or tissue regions, classify cells into cell types and functional states, and / or measure various parameters that capture biology of interest, which may include spatial information such as density and / or proximity. Any of a variety of image analysis techniques may be employed to perform such an analysis. For example, machine learning or artificial intelligence techniques may be performed. Alternatively or additionally, deterministic approaches such as watershed methods may be employed by the processor to perform the image analysis. Further details of examples of such techniques are described below

[0211] In some embodiments, at least some of the processes described above for multiplexed chromogenic detection using the systems and methods of this disclosure is performed on a first sub-region of the sample on the sample holder (e.g., a first tile of a slide), and then the system is adjusted to perform the processes again on a different subregion of the sample (e.g., a different tile of the slide). As described above, the system may, for example, (a) translate the sample and / or sample holder using the stage (e.g., stage 208) in one or more of the x-, y-, and z- directions and / or (b) move the optics in one or more of the x-, y-, and z- directions so that the system is configured to focus on and image a different sub-region (e.g., a different tile of the slide). This process may be repeated numerous times, e.g., until the whole sample (e.g., the whole slide) is imaged. It should be understood that various image processing steps described above, such as the subtraction of images, unmixing, and / or generation of the unmixed multilayer image set, may be performed before, during, and / or after all of the different sub-regions of the sample are imaged. In some embodiments, the acquisition of the first and second color intensity acquisition images (that is, with and without the actuatable filter in the optical path) is performed in a respective sub-region of the sample before the system is adjusted to interrogate the next sub-region. However, in other embodiments, the system acquires all first color intensity acquisition images without the actuatable filter in the optical path and then re-scans the sample with the actuatable filter in the optical path to acquire the second color intensity acquisition images with the actuatable filter in the optical path. The system and methods described in this portion of the disclosure may advantageously facilitate an ability to image each sub-region of the sample (e.g., each tile) and combine the images into whole slide images acquired with and without the actuated filter in the optical path. This may promote an effective and economical way to get perfect or nearperfect pixel registration for both sets of color intensity spectral response images (e.g., both sets of three raw images corresponding to red, green, and blue for RGB detectors).

[0212] In some embodiments, a system as described herein can be used to perform a method for detecting light transmitted through a sample. For example, in the method summarized by the flow chart 770a of FIG. 7A, act 700 of the method comprises directing, during a first period of time, illumination light (e.g., illumination light 212 as shown in FIGs. 8A and 8B) along an optical path (e.g., optical path 213 as shown in FIGs. 8A and 8B) from an illumination source (e.g., an illumination source 201 as shown in FIGs. 8A and 8B), through at least a portion of a biological sample comprising multiple different immobilized chromophores (e.g., biological sample 211 as shown in FIGs. 8A and 8B), and to pixels or subpixels of a detector (e.g., detector 202 as shown in FIGs. 8A and 8B). In some embodiments, the detector comprises a color filter array that results in the pixels or subpixels comprising first color pixels or subpixels that have a first color spectral response, second color pixels or subpixels that have a second color spectral response, and third color pixels or subpixels that have a third color spectral response. In some embodiments, according to act 701 as shown in FIG. 7A, the method comprises detecting the illumination light that was directed to the detector during the first period of time to acquire a first color intensity acquisition image comprising color spectral response images (e.g., a first color spectral response image, a second color spectral response image, and a third color spectral response image). In some embodiments, the first color intensity acquisition image comprises a first color spectral response image corresponding to the first color pixels or subpixels, a second color spectral response image corresponding to the second color pixels or subpixels, and a third color spectral response image corresponding to the third color pixels or subpixels.

[0213] In some embodiments, according to act 702 as shown in FIG. 7A, the method comprises directing, during a second period of time, illumination light along the optical path, with a filter located in the optical path, to filters or subpixels of a detector. In some embodiments, the filter may be located in a portion of the optical path from the illumination source to a sample holder holding the biological sample (e.g., sample holder 209 as shown in FIGs. 8A and 8B), such as filter 203a as shown in FIGs. 8A and 8B. In some embodiments, the filter may be located in a portion of the optical path from the sample holder to the detector, such as filter 203b as shown in FIG. 8B. In some embodiments, according to act 703 as shown in FIG. 7B, the method comprises detecting the illumination light that was directed to the detector during the second period of time to acquire a second color intensity acquisition image comprising color spectral response images (e.g., a first color spectral response image, a second color spectral response image, and a third color spectral response image). In some embodiments, the second color intensity acquisition image comprises a first color spectral response image corresponding to the first color pixels or subpixels, a second color spectral response image corresponding to the second color pixels or subpixels, and a third color spectral response image corresponding to the third color pixels or subpixels.

[0214] In some embodiments, according to act 704 as shown in FIG. 7A, the method comprises generating a multi-layer image of the at least a portion of the biological sample comprising at least two of (1) one or more color optical density spectral response images calculated from at least some of the color intensity spectral response images of the first color intensity acquisition image, (2) one or more color optical density spectral response images calculated from at least some of the color intensity spectral response images of the second color intensity acquisition image, and (3) one or more color optical density spectral responses images calculated from the color acquisition intensity images. In some embodiments, the color optical density spectral response images of (3) are calculated from the color acquisition intensity images by: (i) subtraction of the second color acquisition intensity image from the first color intensity acquisition image, or (ii) subtraction of (a) a second optical density image calculated from the second color intensity acquisition image from (b) a first optical density image calculated from the first color intensity acquisition image; and performing an unmixing of the multilayer image set to generate unmixed images. In some embodiments, according to act 705 as shown in FIG. 7A, the method comprises performing an unmixing of the multi-layer image to generate unmixed images, each corresponding to one of the multiple different immobilized chromophores.

[0215] Example Chromogenic Imaging Embodiment

[0216] The following is a non-limiting description of an example of an embodiment of a system and method for detecting light transmitted through a sample. Some embodiments of the systems and methods of this disclosure facilitate the accurate quantitative assessment of more than 3 chromogenic markers on the same tissue section with a relatively simple modification to an RGB scanner. This is particularly important when markers co-localize within individual image pixels, and / or when important information is revealed by looking at expression level, not just whether the marker is present or not.

[0217] Some embodiments of the systems and methods of this disclosure avoid undetermined linear unmixing of 4 or more chromophores in multicolor immunohistochemistry, which occurs when these slides are scanned on ‘RGB’ slides scanners without the modifications discussed in this disclosure. Some embodiments of the systems and methods of this disclosure provide this benefit with a relatively low cost and simple modification to RGB scanner designs, without the need to implement expensive and complicated opto-mechanical approaches involving monochrome cameras, filter wheels, and / or discrete ‘narrow band’ illumination channels.

[0218] Some embodiment of the systems and methods of this disclosure accomplish this by adding to the slide illumination optical path a simple actuated multi-bandpass filter that has bandpasses that cut-on and cut-off in a way that, in some embodiments, approximately splits each of the red, green, blue spectral profiles of the RGB imaging sensor in the scanner into roughly halves. Operation of the enhanced scanner to analyze 4 to 6 chromophores may comprise a) scanning without the added multi-bandpass filter in the illumination path, thus operating as a conventional RGB scanner, b) moving the multi-bandpass filter into the illumination path of the scanner, c) scanning the slide again with roughly half of each spectral profiles of the R, G, and B channels due to the multi- bandpass filter, d) subtracting the second scan from the first to create an ‘RGB’ difference image, e) creating a 6-plane image stack using the 3 image planes from the second scan ‘R’, ‘G’, and ‘B’ planes and the 3 planes from the difference RGB image, and f) analyzing the 6-plane image as a spectral image consisting of 6 nearly independent spectral channels, facilitating spectral unmixing of up to 6 chromophores with a determined or over-determined unmixing matrix.

[0219] Some embodiments of the systems and methods of this disclosure effectively double the inputs into the spectral unmixing matrix. It is believed that all commercial RGB scanners for chromogenic brightfield imaging utilize standard RGB sensors, thus limiting the number of input unmixing parameters to the three R, G, and B spectral profiles of the sensor. This means that if the slide is stained with more than 4 chromophores, attempting to perform a linear unmixing will be ‘underdetermined’, thus results can be ambiguous and erroneous.

[0220] It is believed that existing platforms that expand beyond standard RGB imaging for chromogenic brightfield multiplex applications involving more than three chromophores deploy complicated opto-mechanical designs that deploy narrow band filters and monochrome sensors, using white-light LEDs and individually controlled discrete color LEDs, which adds significant complexity and cost, making those platforms a challenge for adoption in, for example, the price sensitive clinical testing market.

[0221] In some embodiments, a system is described for detecting light transmitted through a sample. In some embodiments, the system comprises an RGB brightfield slide scanner. In some embodiments, the scanner comprises a software controlled actuated multi-bandpass filter in the optical path of the scanner, on either the illumination optical path or imaging optical path of the scanner (or both). One advantage of putting it in the illumination path is that it does not have optical effects on the performance of the imaging path. In some embodiments, the multi-bandpass filter is selected to at least approximately split each of the R, G, B spectral peaks in half. In some embodiments, software may be present to control two scans of the slide, one with and one without the multi-bandpass filter in place in the illumination path. In some embodiments, software is included to register the two scans with at least close to sub-pixel accuracy, which may involve localized relative different translocation and rotation of one of the images, to effectively create a 6-plane image. One advantage of this is to accommodate imprecise stage movement between the two scans. In some embodiments, software is included to perform a linear unmixing to create images that are substantially specific to only one chromophore, and, in some instances, perform image analysis to quantitate biological parameters of interest. In some embodiments, the software may also include a digital pathology interface that provides users (e.g., pathologists) with opportunity to annotate areas for analysis, c) image analysis algorithms to segment tissue areas (e.g., , tumor, stroma, necrosis, etc.) and cells (including cell compartments), classify cells based on information associated with each cell, and d) reduce image analysis output data to summary statistics and / or scores that are of use to the pathologist when assessing the patient sample. The multiplexing level could be extending to 7-9 chromophores by adding an additional software actuated filter, with a different multi-bandpass configuration.

[0222] The process by which increased quantitative multiplexed analysis beyond 3 colors is achieved, may comprise staining a slide with 4 or more chromophores plus a hematoxylin nuclear counterstain. In some embodiments, the method comprises scanning the slide with the modified RGB scanner but with the added multi-bandpass filter not inserted in the illumination optical path, thereby acquiring a standard RGB image. Examples of typical RGB sensor spectral channels are shown in FIG. 9. In some embodiments, the method comprises inserting the multi-bandpass filter, which might have a spectral transmission function shown in FIG. 10. In some embodiments, the method comprises scanning the slide a second time. The method may comprise converting both RGB transmitted light signals to optical density values with in pixel, in each of the R, G, and B planes, using reference images taken of a blank slide to calculate % transmission, and using the following equation to calculate optical density for each R, G, and B band in each pixel of each image, where optical density = logio 100 / %T. Optical density may be approximately proportional to quantity of pigment, and thus of target antigen. In some embodiments, the method comprises using algorithms that use common features between the two scans to distort and rotate one image relative to the other, to various degrees and within local regions, to achieve subpixel co-registration between the images. This may be done to compensate for imprecision in slide motion during image acquisition. The method may comprise subtracting the second image from the first, thus creating a third image, called the difference image. In some embodiments, the method comprises combining the 3 planes from the difference image with the 3 planes from the second image to create a 6-plane image. Example spectral response functions for each of the 6 planes are shown in FIG. 11 as blue spectral responses 1101 and 1102, green spectral responses 1103 and 1104, and red spectral responses 1105 and 1106. The method may comprise using signatures of each chromophore, generated with models or empirically with single- stained samples, to perform a linear decomposition of signals to create image planes that are substantially representative of only one chromophore. A typical set of signature profiles for 5 common chromophores are shown in FIG. 12 as first chromophore signature 1201, second chromophore signature 1202, third chromophore signature 1203, fourth chromophore signature 1204, and fifth chromophore signature 1205. The corresponding set of eigenvectors for performing linear decomposition are shown in FIG. 13 as first eigenvector 1301, second eigenvector 1302, third eigenvector 1303, fourth eigenvector 1304, and fifth eigenvector 1305. In some embodiments, the method comprises displaying images on the computer screen in user-friendly ways, since having 4 or more chromophores on a slide can create very complex scenes that make it hard for human perception to understand what cells are expressing which markers. User-friendly ways include showing on the screen not all of the markers at once, providing flexibility to select which markers to show, or by showing images inverted and in pseudo-fluorescence mode, since this mode can make it easier to see lower signals. In some embodiments, the method comprises performing image analysis on unmixed images to measure parameters of interest. In some embodiments, the method comprises performing image analysis on unmixed images when spectral absorption profiles overlap, which may be particularly important when parameters of interest depend on independent and accurate quantitation of each marker and standard imagery would not support reliable signal isolation and unmixing.

[0223] FIG. 7B shows a flowchart describing the use of a multi-bandpass filter to create 6 spectrally, substantially distinct spectral profiles with a conventional RGB slide scanner, and then performing image analysis to measure biological parameters of interest, according to this example embodiment.

[0224] FIG. 8C shows a non-limiting example of an optical configuration of a slide scanner system, showing potential placements of a software-actuated motorized multibandpass filter, according to this example embodiment. The configuration shown in FIG. 8C includes: A) illumination source 301. B) Actuatable filters 333a and 333b, such that illumination light 312 from illumination source 301 passes through filter 333a before passing through a sample on a sample holder 309 but before reaching sensor 302, and illumination light 312 from illumination source 301 passes through filter 333b after passing through a sample on a sample holder 309 but before reaching sensor 302 (however it should be understood that in some embodiments only one of filter 333a and filter 333b is employed in system 200). C) A microscope objective lens 307 that collects transmitted illumination light that is then delivered to and spatially resolved on RGB sensor 302. D) Slide 309 and objective mechanical automation 308 configured to move the slide in horizontal x- and / or y- directions and objective mechanical automation 320 configured to move the objective relative to the slide surface in the Z direction to bring images on the imaging sensor into focus. E) An RGB imaging sensor 302 that captures digital red, green, and blue image planes (red spectral response images, green spectral response images, and blue spectral response images, respectively) according to the spectral response profiles shown in FIG. 9 described below.

[0225] FIG. 9 shows representative spectral profiles of the R, G, and B channels of an RGB imaging sensor, in accordance with some embodiments.

[0226] FIG. 10 shows an example multi-bandpass filter that approximately splits each R, G and B channels in half, to, in this case, create 5 distinct system spectral profiles, in accordance with some embodiments.

[0227] FIG. 11 shows the resulting effective 5 spectral profiles created by the described process, according to this example embodiment.

[0228] FIG. 12 shows example chromophore absorption spectra, according to this example embodiment

[0229] FIG. 13 shows example eigenvectors representing relative absorption for the chromophores across the 5 ‘spectral channels,’ according to this example embodiment.

[0230] Image Segmentation, including Nuclear Segmentation

[0231] Certain embodiments described in this disclosure involve systems and methods for performing a segmentation of an image of a biological sample. In some such embodiments, the segmentation of the images includes segmenting images of cellular nuclei obtained from a sample using a process that includes signal from labeling agents bound to the cell membranes. The process may further involve segmenting cellular membranes and, in some instances segmenting entire cells via image analysis. The image-analysis may be computed-implemented and conducted using a processor with instructions from software. In some embodiments, the segmentation process can be performed automatedly using the software (e.g., without human intervention once the segmentation process is initiated). The segmentation process may be useful for analyzing images of biological samples (e.g., tissue or cells) obtained from any of a variety of sources, including but not limited to images from microscopes (e.g., slidescanning microscopes). The images may be, for example, fluorescence images and / or optical density images (e.g., from immunofluorescence and / or immunohistochemistry imaging).

[0232] In some embodiments, the method comprises obtaining a multilayer image set. The multilayer image set may comprise layers of images of at least a portion of the biological sample. In some embodiments, obtaining the multilayer image set includes obtaining, by a processor, a multilayer image set that was previously-acquired by a detector. For example, the multilayer image set may be obtained from the detector, from a data store, from a user (e.g., by the user uploading the image), or from any other suitable source, as aspects of the technology are not so limited. In some embodiments, obtaining the multilayer image set includes acquiring the multilayer image set using a detector. Any of a variety of detectors and imaging systems may be suitable. For example, while the segmentation process may be performed on multilayer image sets obtained using various of the specific imaging systems in this disclosure (e.g., involving multi-color detectors), though the process is not so limited and can also be used with any of a variety of other systems, including commercially available systems capable of multiplexed fluorescence and / or chromogenic imaging of samples.

[0233] Prior to detection, the sample may have been labeled with multiple different types of labeling agents. For example, in some embodiments, the biological sample is exposed to nuclear labeling agents (that is, labeling agents specific to the cells’ nuclei) such that the nuclear labeling agents are immobilized with respect to the nuclei of the cells. The nuclear labeling agents may comprise, for example, a nuclear counterstain. In some embodiments, the biological sample is exposed to membrane labeling agents such that the membrane labeling agents are immobilized with respect to the plasma membranes of the cells. The membrane labeling agents may comprise, for example, one or more membrane counterstains and / or any of a variety of other labeling agents capable of being localized at the plasma membrane. The labeling of the nuclei with the nuclear labeling agents and the labeling of the plasma membranes of the cells with the membrane labeling agents may be performed simultaneously or sequentially. Any of a variety of labeling techniques, such as immunolabeling techniques and / or amplification techniques (e.g., with tyramide signal amplification or other amplification techniques) may be employed.

[0234] Following the labeling of the biological sample with the labeling agents (e.g., including the nuclear labeling agents and membrane labeling agents), the sample may be imaged. The image may be acquired by detecting optical signal (e.g., fluorescence light for fluorescence imaging and / or transmitted illumination light for chromogenic imaging). Accordingly, the multilayer image set may be acquired by detecting, with the detector, the optical signal produced by the nuclear labeling agents and detecting, with the detector, the optical signal produced by the membrane labeling agents. The detection of the optical signal produced by the nuclear labeling agents and the membrane labeling agents may be performed simultaneously or sequentially.

[0235] In some embodiments, the multilayer image set comprises a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample. The layer may be an unprocessed image or may be generated as a result of an unmixing procedure, as described elsewhere in this disclosure. As one non-limiting example, a layer of the multilayer image set be a nuclei label image showing optical signal produced by a DAPI labeling agent (e.g., following unmixing).

[0236] In some embodiments, the multilayer image set comprises a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample. The layer may be an unprocessed image or may be generated as a result of an unmixing procedure, as described elsewhere in this disclosure. As one non-limiting example, a layer of the multilayer image set be a membrane label image showing optical signal produced by one or more membrane counterstains or any other suitable membrane labeling agent that label (e.g., selectively label) the plasma membranes of the cells in the sample. In some embodiments, the membrane label image corresponds to optical signal produced by only a single type of membrane labeling agent. However, in some embodiments, the membrane label image is generated from optical signal produced by multiple different membrane labeling agents. For example, the membrane labeling agents may comprise first membrane labeling agents and second, different membrane labeling agents. In some such embodiments, the membrane label image is generated by combining a first membrane-labeling agent image associated with an optical signal produced by the first membrane labeling agents and a second membrane-labeling agent image associated with an optical signal produced by the second membrane labeling agents. The combination (e.g., merging) of different membrane labeling- agent images may be performed, e.g., by normalizing the images to 1 and using a maximum pixel exercise to set the maximum pixel intensity in the combined (e.g., merged) image equal to the maximum pixel intensity of the different membrane labeling- agent images being combined using an adjustable scaling factor on the combined image. It has been realized in the context of this disclosure that the use of multiple different membrane labeling agents and, in some instances, merging the associated images to produce the membrane label image of the multilayer image set may, in some instances, generate a more accurate membrane label image for later processing to enhance nuclear and, in some instances, membrane segmentation.

[0237] In some embodiments, a combination image based on the first nuclei label image and the membrane label image is generated, e.g., using the processor. The first nuclei label image and the membrane label image may be combined in any of a variety of manners. In some embodiments, the combination image is based on performing a mathematical operation (e.g., subtraction, addition, multiplication, division) involving the corresponding pixel intensity values of the nuclei label image and the membrane label image (or values derived from those pixel intensity values). For example, in some embodiments, the processor generates the combination image by subtracting the membrane label image or an image derived from the membrane label image from the nuclei label image or an image derived from the nuclei label image. The subtraction of the membrane label image from the nuclei label image may result in the combination image being a “difference image”. Such a difference image may show nuclei of interest brighter and more representative of nuclear area on the top surface of the sample, and thus more conducive of accurate cell segmentation of areas on the top surface of the section (where labeling may be concentrated). The nuclei not overlapping with membrane signal may tend to be more complete and represent cells substantially captured by the section, rather than cell and nuclear fragments not captured well by the sectioning process. Nuclear areas in the image that are under membrane signal or are split by membrane signal may tend to be suppressed or diminished because of the image subtraction in the combination image.

[0238] In some embodiments, the processor scales the intensities of at least some of the pixels in the nuclei label image to generate a scaled nuclei label image and / or scales the intensities of at least some of the pixels in the membrane label image to generate a scaled membrane label image. One way in which the scaling of the respective images may be performed, e.g., by the processor, is by normalizing the intensity of each of the pixels in an image such that the maximum pixel intensities in the image are equal to 1. This normalization process may make the combination (e.g., subtraction) of the nuclei label image and the membrane label image more accurate. In some embodiments, the combination image is generated by the processor by subtracting the scaled membrane label image from the scaled nuclei label image.

[0239] In some embodiments, images of the nuclei in the biological sample are segmented (e.g., using the processor). The segmentation of the images of the nuclei may be performed based at least in part on the combination image to produce a nuclear segmentation map. For example, the segmentation of the images of the nuclei may be performed based on the difference image generated by subtracting the scaled membrane label image from the scaled nuclei label image. Details of procedures and techniques for segmenting the images of the nuclei (e.g., to determine which portions of the image correspond to distinct nuclei of cells) are described in more detail below. In some embodiments, a nuclear segmentation map is produced from the nuclear segmentation process. The nuclear segmentation map may indicate, for each of at least some (e.g., all) of the pixels in the image, a class label assigned to that pixel based on a result of the segmentation. For example, the class label assigned to a particular pixel may indicate whether or not the particular pixel depicts a nucleus. The nuclear segmentation map may include a visual representation that divides the image of the biological sample into distinct regions corresponding to different nuclei. In some embodiments, the segmentation of the images of the nuclei is performed using additional information, such as from a membrane segmentation map, such as in the Example Embodiment described in more detail below.

[0240] In some embodiments, the processor is configured to segment images of plasma membranes in the biological sample based at least in part on the membrane label image to produce a membrane segmentation map. The membrane segmentation map may indicate, for each of at least some (e.g., all) of the pixels in the image, a class label assigned to that pixel based on a result of the segmentation. For example, the class label assigned to a particular pixel may indicate whether or not the particular pixel depicts a membrane. The membrane segmentation map may include a visual representation that divides the image of the biological sample into distinct regions corresponding to different plasma membranes. As noted above, the membrane segmentation map may also be employed, together with the nuclei label image, to generate the nuclear segmentation map.

[0241] In some embodiments, the processor is configured to segment images of cells in the biological sample based on the nuclear segmentation map and the membrane segmentation map. The resulting segmentation map may indicate, for each of at least some (e.g., all) of the pixels in the image, a class label assigned to that pixel based on a result of the segmentation. For example, the class label assigned to a particular pixel may indicate whether or not the particular pixel depicts a cell. The resulting segmentation map may include a visual representation that divides the image of the biological sample into distinct regions corresponding to different cells in the sample.

[0242] In some embodiments, the nuclear segmentation is performed as part of an experiment that further involves imaging of an analyte of interest (e.g., a biomarker). For example, the analyte of interest may be a protein biomarker. In some embodiments, the multilayer image set obtained by the processor further comprises a layer corresponding to an analyte image associated with an optical signal produced by analyte labeling agents immobilized with respect to or in proximity to an analyte in the biological sample. The analyte labeling agents may be different from the nuclei labeling agents and the membrane labeling agents.

[0243] In some embodiments, at least some of the segmented images of the cells described above are analyzed. For example, the segmented images of the cells may be analyzed to determine the presence, density, and / or level of at least one analyte in the biological sample (e.g., in some instances the presence, density, and / or level of at least one analyte at a particular cell or type of cell).

[0244] In some embodiments, a classification procedure may be performed on the segmented images of the cells. It has been realized in the context of this disclosure that cell classification (e.g., based on biomarker expression such as protein expression) may be more accurate following the above-described segmentation process. For example, in the case of protein expression, protein expression information about segmented cells may be more accurate when this segmentation procedure is performed, at least because the appropriate pixels representing the cell membrane may have been detected more accurately.

[0245] In some embodiments, the multilayer image set obtained from the detector is a multilayer composite image of the at least a portion of the biological sample, where the multilayer composite image comprises the layers that include the nuclei label image, and the membrane label image (and in some instances at least one analyte label image). The multilayer composite image may be, for example, a superimposed image of a stack of the layers (e.g., in a single data file).

[0246] In some embodiments, the multilayer image set is a plurality of individual layers of images of the at least a portion of the sample, where the plurality of individual layers comprise the layers that include the nuclei label image, and the membrane label image (and in some instances at least one analyte label image). The plurality of layers may be, for example, a collection of the layers (e.g., each in an individual data file).

[0247] In some embodiments, a system, e.g., comprising a processor as described herein can be used to perform the method for segmenting images of a biological sample. For example, in the example method summarized by the flow chart 1440a of FIG. 14A, act 1400 comprises obtaining a multilayer image set, acquired by a detector, comprising a layer comprising a nuclei label image and a layer comprising a membrane label image. In act 1401, a combination image (e.g., a difference image) based on the nuclei label image and the membrane label image is generated. In some such instances a difference image is generated following scaling of the nuclei label image and the membrane label image (e.g., to normalize values). In act 1402, the processor segments images of nuclei in the biological sample based at least in part on the combination image (e.g., difference image) to produce a nuclear segmentation map. In some embodiments, the processor further segments the membrane label image based to generate a membrane segmentation map, and in some instances the processor further segments the cells in the biological image based on the nuclear segmentation map and the membrane segmentation map.

[0248] Example Image Segmentation Embodiment

[0249] The following is a non-limiting description of an example of an embodiment of a system and method for segmenting an image of a sample. Some embodiments of the systems and methods of this disclosure facilitate accurate automated segmentation of cells and subcellular compartments in ways that can be standardized. This may advantageously substantially increase the consistency of and reduce the labor associated with spatial biology measurements. Some embodiments of the systems and methods of this disclosure automate cell segmentation so that: a) pixel selection to represent cell compartments is standardized and accurate without need for human intervention to tune and adjust parameters for each study, assay or tissue type, b) segmentation favors cell compartments on the top surface of the section so that detected markers are more closely aligned with cell compartments and cells as a unit, and c) image analysis computation is fast so that processing slide images through to a reliable and robust score is time efficient to support rapid delivery of results.

[0250] Some embodiments of the systems and methods of this disclosure accomplish this by: a) integrating a standard membrane counterstain comprising a cocktail of antibodies into multiplex panels and b) combining the information from the nuclear counterstain image and the membrane counterstain image to yield a third image better suited to support nuclear segmentation. For example, if the nuclear counterstain and the membrane counterstain images are each scaled and then the membrane image is subtracted from the nuclear image, the resultant image has nuclei of interest brighter and more representative of nuclear area on the top surface, and thus more conducive of accurate cell segmentation of areas on the top surface of the section. The nuclei not overlapping with membrane signal will tend to be more complete and represent cells substantially captured by the section, rather than cell and nuclear fragments not captured well by the sectioning process. Nuclear areas in the image that are under membrane signal or are split by membrane signal will tend to be suppressed or diminished because of the image subtraction.

[0251] Performing nuclear segmentation on this third, difference image, which highlights well-sectioned cells and depresses cell fragments and cells deeper in the section, may lead to more accurate cell characterization and classification data. Examples of advantages of some embodiments of the systems and methods of this disclosure that facilitate performing nuclear segmentation on this third image include a) producing a new nuclear signal image easier to segment and split into individual nuclei aligned with membrane signal, b) naturally filtering out cells less of interest and nuclei not associated with protein marker stain that resides on the top surface of the tissue section, and c) being very fast and not taking much CPU power.

[0252] Some embodiments of the systems and methods of this disclosure facilitate the automated and accurate quantitation of protein markers to yield accurate classification and characterizations of individual cells, to support scientific and translational research and clinical applications.

[0253] In some embodiments of the systems and methods of this disclosure, a spatial biology assay is performed. In some embodiments, performing a spatial biology assay starts with a multi-marker staining protocol, typically applied to a tissue section mounted on a microscope slide using a staining robot, such as the Leica Bond Rx. The protocol may include markers such as DAPI nuclear counterstain, cytokeratin to indicate epithelial cells, CD8 to indicate cytotoxic T cells, CD163 to indicate macrophages, PD- L1 to indicate programmed cell death ligand 1, and Foxp3 to indicate regulatory T cells.

[0254] In some embodiments of the systems and methods of this disclosure, a standard membrane cocktail marker is added to staining protocols so that cell membrane is revealed. For example, the membrane antibody cocktail might include ATPase and CD45LCA.

[0255] In some embodiments of the systems and methods of this disclosure, slides are imaged on an appropriate slide scanning platform with the hardware and software to resolve individual markers to create a multi-plane whole slide image of the tissue section. Typically, one of image planes indicates the nuclear counterstain, typically DAPI or Hoechst. If the membrane cocktail or counterstain is included in the staining protocol, there may be a dedicated image plane for that as well. In some embodiments of the systems and methods of this disclosure, image analysis is performed to identify individual cells in the multilayer whole slide image to classify and characterize identified cells by measuring proteins associates with each cell and performing classifications based on staining spatial patterns. In some embodiments, accurate cell segmentation is advantageous to avoid assigning protein expressions to the wrong cells, thus interfering with classification of cell types which depend on multiple protein expression.

[0256] In some embodiments of the systems and methods of this disclosure, once cells have been accurately classified and characterized, spatial parameters can be determined, such as density, proximity, gradients, occurrence or frequency of cell arrangement or neighborhood events, and / or other cellular arrangements suggesting certain biological interactions. Further, in some embodiments, biological parameters can be surmised which, in some instances, correlate with scientific and translational goals, such as determination of the prevalence of a patient type, confirmation of drug method of action, prognosing outcome, or predicting response to therapies. Thus, one advantageous analytical performance attribute of a spatial biology platform is accurately and precisely identifying individual cells in images of sections.

[0257] To help guide cell segmentation algorithms in image analysis, a universal membrane counterstain may be used in a multiplex staining panel (e.g., a multiplex immunofluorescence staining panel), in order to develop a standard image analysis algorithm that can be used on any panel or assay, as long as it has this membrane counterstain.

[0258] FIG. 15 shows an illustration of cells that are sectioned in a tissue sample, which captures different parts of cells that may overlap when viewed above while imaging, according to some embodiments.

[0259] Combining information from the nuclear counterstain image (for example, a nuclear counterstain image as shown in FIG. 16) and the membrane counterstain image (For example, a membrane counterstain image as shown in FIG. 17) to produce a third image for nuclear segmentation (For example, the third images shown in FIGs. 18 and 19) and then performing membrane detection of the membrane counterstain image to find membrane pixels associated with each detected nucleus may facilitate more efficient nuclear segmentation analyses than what was previously achievable. In some embodiments of the systems and methods of this disclosure, detecting membrane in the membrane counterstain image may or may not include information derived from the nuclear counterstain image or the nuclear segmentation map derived from the difference image described above.

[0260] In some embodiments, a system or method for segmenting an image of a sample as described in this disclosure comprises performing a multiplex immunofluorescence staining protocol. FIG. 14B shows a flow chart of one example of how the method may be conducted. In some embodiments, the multiplex immunofluorescence reagents and protocol include both a DAPI or other nuclear counterstain and a membrane counterstain, which may be comprised of a cocktail of antibodies specific to proteins common in membrane compartments of cells of interest. In some embodiments, the systems or methods of this disclosure comprise a slide imaging system capable of creating multilayer images of slides stained with the multiplex reagents and protocol, where at least two of the image planes are substantially of either the nuclear counterstain or the membrane counterstain. In some embodiments, the systems or methods of this disclosure comprise an image analysis process or algorithm whereby the nuclear and membrane counterstain images are scaled to achieve an approximate relative intensity of brightest pixels in each plane, and the scaled membrane image is subtracted from the scaled nuclear image to decrease intensity where membrane signal is, thus creating a third image for nuclear segmentation. In some embodiments, the systems and methods of this disclosure comprise applying a segmentation algorithm to the third image to identify nuclei. In some embodiments, the segmentation algorithm could comprise standard image analysis methods or could be based on modem Al-based approaches such as ‘deep learning’, or ‘stardist’. In some embodiments, the systems and methods of this disclosure comprise applying a second segmentation algorithm on the membrane counterstain image to detect membrane pixels, which could be based on standard image analysis approaches like dilation, smoothing, watershed, etc. or Al-based. In some embodiments, the systems or methods of this disclosure comprise combining output of both algorithms to determine cells each comprised of nuclear, cytoplasm, and membrane compartments and / or pixels. In some embodiments, the systems and methods of this disclosure facilitate an optimum pixel selection and sampling strategy so that measured protein expressions correlate strongly with individual cells as biological units. Example of Combinations of Features

[0261] In some embodiments, various combinations of the systems and methods described above are also possible. In some embodiments, a system is configured to be capable of both detecting light emitted from a sample and detecting light transmitted through a sample. For example, such a combination system may comprise: a sensor (e.g., sensor 102 as shown in FIG. 8D), optics (e.g., optics 110 as shown in FIG. 8D) comprising an optical block (e.g., optical block 119 as shown in FIG. 8D), an excitation source (e.g., excitation source 105 as shown in FIG. 8D), and an illumination source (e.g., illumination source 201 as shown in FIG. 8D) forming an optical path (e.g., optical path 213, as shown in FIG. 8D) from the illumination source 201, through at least a portion of the biological sample 111, and to the sensor 102. In some embodiments, the system may comprise an actuatable filter (e.g., actuatable filter 203a and / or actuatable filter 203b as shown FIG. 8D) having a first configuration in which the actuatable filter is not located the optical path and a second configuration in which the actuatable filter is located in the optical path, one or more single-bandpass filters (e.g. single-bandpass filters 106 as shown in FIG. 8D), and / or a multi-bandpass filter (e.g., multi-bandpass filter 119 as shown in FIG. 3A). Such a combination system may be configured to (1) perform fluorescence detection (e.g., using the optical block) in one configuration (e.g., when the actuatable filter is not in the optical path and the illumination source is not actuated to produce illumination light) and (2) to perform chromogenic imaging (e.g., brightfield imaging) in another configuration where excitation light from the excitation source is not actuated.

[0262] In some embodiments, the any of the systems described in this disclosure for fluorescence imaging and / or chromogenic imaging of biological systems may further be configured to perform the image segmentation process described in this disclosure, including nuclear segmentation. For example, a processor of a computing system of the overall system may be configured to obtain a multilayer image set following the fluorescence imaging and / or chromogenic imaging described in this disclosure and perform the image segmentation process described in this disclosure, including nuclear segmentation. Enhanced Immunohistochemistry Procedures

[0263] Methods for processing biological samples comprising biological tissue are useful in a broad range applications. For example, such methods are useful in any of a variety of diagnostic capacities in the medical field. For example, one common method for processing biological samples comprising biological tissue is the use of immunohistochemistry procedures. Immunohistochemistry (IHC) procedure may involve the application of monoclonal and / or polyclonal antibodies (or fragments thereof) to determine the distribution of an antigen of interest in a tissue section.

[0264] Immunohistochemistry procedures may be employed in, for example, clinical practice and research. As a non-limiting example, IHC is widely used for diagnosis of cancers because specific tumor antigens are expressed de novo or up-regulated in certain cancers. IHC for such applications use tissue from biopsies. These are processed into sections with a microtome and then the sections are incubated with an appropriate antibody. The site of antibody binding is visualized and / or imaged with an ordinary or fluorescent microscope by a labeling agent. The labeling agent may comprise, for example, a chromogen, fluorescent dye, enzyme, radioactive element, and / or colloidal gold. The labeling agent may be directly linked to the primary antibody or to an appropriate secondary antibody. Conventional IHC, comprising the steps shown in FIG. 20, is a widely used diagnostic technique in tissue pathology. FIG. 21 shows a flow chart that describes conventional immunohistochemical (IHC) testing for a PD-L1 expression indication.

[0265] As noted above, Immunohistochemistry (IHC) procedures are a standard for clinical predictive testing in oncology. As discussed above, there remains a limitation in the art caused by the inconsistency with human visual assessment of IHC stained slides when assessing tissue sections (e.g., biopsy tissue sections) labeled (e.g., stained with a single labeling agent (e.g., a single chromogenic marker). This is particularly problematic when the parameter of interest is at or near scoring thresholds. Accordingly, there is a problem and need for protocols, methods, and systems that can help users of such procedures (e.g., pathologists) avoid mistaken assessments that could lead to (a) unproductive treatment with side effects, and / or (b) missing patients that would have responded to treatment. A recent non-limiting example is in immuno-oncology (IO) for predicting response to anti-PDl therapy with PDL1 IHC testing. PD-L1 IHC assessment typically involves a pathologist assessing staining patterns in cancer biopsy tissue sections stained with standardized IHC staining protocols that reveal location and amount of the target protein with a dark brown pigment on top of a blue hematoxylin nuclear counterstain which provides the pathologist with context and a visual way to determine where cells are and their type. Despite the availability of digital pathology solutions which create high fidelity images of stained slides for display on the computer screen for assessment remotely, it is estimated that 80% of PD-L1 IHC assessments are still performed worldwide by looking at slides through the eyepieces of conventional brightfield microscopes. PD-L1 IHC is the clinical standard and is thoroughly integrated into clinical practice and used as a companion diagnostic with FDA-approved IO drugs.

[0266] A significant challenge with biomarker IHC visual assessment (for example, PD- L1 IHC visual assessment) is that it can be inconsistent, especially when biomarker expression levels are near scoring thresholds, due to a) the limitations of visual perception which is not well- suited to counting large numbers of cells in complicated scenes in practical time periods, and b) the variability among those making the visual assessment (e.g., pathologists). Visual assessment of expression of biomarkers (for example, PD-L1 tumor cell expression) can be particularly challenging with certain conditions in which tissue (e.g., tumors) may be infiltrated with cells that naturally express the biomarkers of interest. For example, with lung cancer, tumors can be highly infiltrated by macrophages which naturally express PD-L1 protein. This becomes a significant issue when expression levels are near cellular positivity thresholds, which are typically 1%, 5%, or 10%.

[0267] It is believed that multiplexed IHC or immunofluorescence (IF) allow for the recovery of more detailed information about tissue (e.g., tumor) microenvironment biology that correlates better with response to treatments such as immunotherapy. For example, in the case of cancer, additional markers can reveal presence of cytotoxic T cells and PD1 positive T cells, which are cell types central to the mechanism-of-action of anti-PDl drugs. However, there are significant challenges to adoption of these complicated and expensive approaches, including pathologist acceptance, fit with standard clinical workflows, regulatory approval, platform cost, reimbursement, and worldwide deployment.

[0268] There have been numerous research studies attempting to use artificial intelligence, or machine learning methods, to extract predictive information from conventional immunohistochemistry procedures to identify biomarkers (such as the analysis of PD-L1 IHC stained slides and / or standard histology hematoxylin and eosin (H&E) stained slides) which leverage morphological features of tissue. These studies, although promising, rarely prove robust across the vast heterogeneity of tissue and staining. Accordingly, improved methods for processing biological samples comprising biological tissue are needed.

[0269] The present disclosure describes methods for processing a biological sample comprising biological tissue. One advantage of some of the inventive methods described herein is to provide an analysis workflow that uses existing practice for cases with which the user (e.g., pathologist) is comfortable and confident scoring but provides an opportunity for users of the analysis workflow (such as pathologists) to further perform an photo-induced emission (e.g., fluorescence) imaging procedure that produce accurate quantitative inputs in to scoring calculations. The fluorescence imaging procedure may facilitate, for example, robust software-based classification (e.g., with robust machine vision-based cell classification). Non-limiting examples of such scoring calculations include Tumor Proportion Score (TPS), Immune Proportion Score (IPS), and / or Combined Positivity Score (CPS). Such an analysis workflow may be particularly advantageous when tissue staining patterns and morphologies pose challenges for visual identification using traditional methods, especially in marginal cases.

[0270] In some embodiments, as shown in FIG. 22, the method for processing a biological sample comprising biological tissue comprises performing an immunohistochemistry procedure on at least a portion of a sample on a slide. The immunohistochemistry procedure performed may use a first labeling agent immobilized with respect to the sample. In some embodiments, the method comprises generating a first score associated with an indication for the at least a portion of the sample based on the immunochemistry procedure. The score may be, for example, a TPS, IPS, and / or CPS. In some embodiments, the performance of the immunohistochemistry procedure and the generation of the first score may be conducted according to an existing clinical standard for the indication. For example, in some embodiments, the clinical standard corresponds to clinical test instructions from (a) an in vitro diagnostic product label from the U.S. Food and Drug Administration and / or (b) guidelines from the College of Anatomical Pathology.

[0271] In some embodiments, the performance the immunohistochemistry procedure may be conducted by a user via a visual inspection of at least a portion of the sample using a microscope. However, in some embodiments, the immunohistochemistry procedure comprises assessment based on a digital image acquired from the biological sample. In some embodiments, a method comprises performing an immunohistochemistry procedure comprising a staining protocol that creates slides that appear in a standard microscope to be identical to the existing clinical standard. This may permit users to visually assess the slides as per clinical and approved standards. In some embodiments, the first labeling agent may be a labeling agent targets a specific biomarker or be associated with a targeting entity (e.g., an antibody or fragment thereof) that targets a specific biomarker. For example, in some embodiments, the first labeling agent may be a PD-L1 labeling agent.

[0272] As noted above, in some embodiments, the method comprises generating a fluorescence image of at least a portion of the sample on the same slide used for the immunohistochemistry procedure. In some embodiments, the fluorescence image is based at least in part on a signal detected from a second labeling agent which is immobilized with respect to the sample. The second labeling agent may be different from the first labeling agent. The fluorescence image may be based on multiple fluorescence signals produced by multiple different labeling agents. For example, in addition to the signal detected from the second labeling agent, the fluorescence image may also be based at least in part on a signal detected from a third labeling agent which is immobilized with respect to the sample. The third labeling agent may be different from both the first labeling agent and the second labeling agent. In some embodiments, the generation of the fluorescence image (e.g., via a fluorescence microscope) is performed before the completion of the performance of the immunohistochemistry procedure (e.g., before the generation of the first score). In some embodiments, the generation of the fluorescence image (e.g., via a fluorescence microscope) is performed simultaneously with some or all of the immunohistochemistry procedure. In some embodiments, the generation of the fluorescence image (e.g., via a fluorescence microscope) is performed after the completion of the immunohistochemistry procedure (e.g., after the generation of the first score).

[0273] In some embodiments, the method comprises determining whether the first score is indeterminant. In some embodiments, the method comprises, after the determining that the first score is indeterminant, processing the fluorescence image to classify components of at least a portion of the sample. In some embodiments, the classification of the components comprises classifying the cells of the sample as tumor cells or nontumor cells. In some embodiments, processing the fluorescence image comprises machine vision-based cell classification.

[0274] In some embodiments, the method comprises generating from the classified components a second score associated with the indication of interest (e.g., the disease of interest) and / or generating a parameter indicative of the quantity, density, and / or level of an analyte (e.g., biomarker such as PD-L1) in the sample. In some embodiments, processing the fluorescence image may have several advantages, for example by allowing the provision of machine vision and image analysis algorithms with information that, for example, promote accurate quantitation of biological parameters that a user may not be able to assess visually and / or that the user may benefit from confirmation and / or verification of the results from such a visual assessment via the processing of the fluorescence image. In some embodiments, the first score and / or the second score may comprise a Tumor Proportion Score (TPS), Immune Proportion Score (IPS), and Combined Positivity Score (CPS).

[0275] In some embodiments, the second labeling agent and / or third labeling agent are not visible under a normal microscope viewing conditions or in views created in typical digital pathology workflows. In some embodiments, a second labeling agent and / or the third labeling agent may comprise an immunofluorescence (IF) marker. In some embodiments, the second labeling agent and / or the third labeling agent may comprise a fluorophore.

[0276] In some embodiments, the second labeling agent can, for example, comprise labeling agents that target or are associated with targeting entities (e.g., antibodies or fragments thereof) that target tissue of interest, such as tumor and / or macrophage cells. In some embodiments, this may allow for more accurate assessment of the expression of particular biomarkers (for example, PD-L1) on sample cells of interest (e.g., tumor cells) and avoid visual confusion from high infiltration from other cells (e.g., macrophages) which may naturally express the same biomarkers. The second labeling agent may comprise or be associated with, for example, markers for cellular structures, such as for membrane to assist with image analysis cell segmentation. This may allow for the use of cell classification algorithms by providing marker information that is consistent and accurate, allowing for the use of classification algorithms that are more robust and not compromised by training sets that contain noise which can dilute or even obscure biological signatures.

[0277] In some embodiments, the signal detected from the second labeling agent is detected from a system configured to perform both brightfield and fluorescence imaging of biological samples on slides. For example, in some embodiments, detecting the signal may comprise the use of a slide scanner which is configured to image both chromogenic and fluorogenic markers. In some embodiments, detecting the signal using a system that is configured to perform both brightfield and fluorescence imaging may allow for processing the image using an image analysis process that includes registering both brightfield and fluorescence images (e.g., chromogenic and fluorogenic images) and classifying components of the sample (e.g., using classification algorithms and, in some instances segmentation algorithms) to identify cell types (e.g., within a tumor microenvironment). In some embodiments, classifying components of the same may comprise extracting quantitative measures of analyte (e.g., protein expression) in different cell types, and calculating scores that can be used in place of visual estimates. In this context, ‘quantitative’ means determining a level of expression relative to a standard.

[0278] The processing method is particularly helpful when the analyte (e.g., protein) being marked by IHC is in multiple cell types, such as in the case with PD-L1, where its expression in different cell types is hard to decern visually based on hematoxylin nuclear counterstain, which can be confounding and can have different implications for the associated drug mechanism-of-action. The method addresses the previously discussed limitation and need and provides a solution to the problem by providing a method and protocol that adds an elective step to routine testing IHC workflow. If a user is uncertain of their score after performing the immunohistochemistry procedure on at least a portion of a sample on a slide using a first labeling agent (e.g., after generating an indeterminant first score associated with an indication for the at least a portion of the sample based on the immunohistochemistry procedure by performing a visual assessment the IHC-stained sample using a microscope), the user may have the slide scanned on a scanner capable of both brightfield and fluorescence scanning, to reveal additional information that is provided by adding labeling agents (e.g., fluorescence markers) to the conventional brightfield IHC staining. These markers may not be visible during routine visual assessments through eyepieces or as viewed on computer screens after scanning on a conventional color slide scanner. But they may become visible, or imageable, with a fluorescence slide scanner.

[0279] In some embodiments, the method comprises performing a slide staining protocol that includes an existing clinical standard staining step (e.g., adding a first labeling agent labeling agent to at least a portion of a sample on a slide) and additional steps to add at least a second labeling agent (e.g., a fluorescence marker) to classify cells. For example, the cells may be classified into tumor cells and immune cells of various types. The present method may also provide a sample analysis workflow that supports a user to first perform an immunohistochemistry procedure on at least a portion of a sample on a slide using a first labeling agent to generate a first score associated with an indication for the at least a portion of the sample and then, if the first score is indeterminant, perform one or more additional steps comprising generating and processing a fluorescence image of at least a portion of the sample, as described above. Further, if the one or more additional steps are performed, a software workflow is described that may a) register brightfield and fluorescence images according to fiducials that are detectable in both imaging modes, or some other method, b) include a digital pathology interface that provides users (e.g., pathologists) with opportunity to annotate areas for analysis, c) include image analysis algorithms to detect and segment cells in the brightfield imagery to create a cell table that includes x-y coordinates of each nucleus, d) include image analysis algorithms that classifies cells according to stain patterns in brightfield and fluorescence images, d) include scoring algorithms that reduce image analysis output data to summary statistics and / or scores associated with the original clinical test protocol, and e) be configured to present brightfield imagery on a computer screen with cell classification overlay ed that can be toggled on and off, to give the user an opportunity to review classification results and accept or reject scoring based on image analysis result.

[0280] In some embodiments, a method described herein comprises a method of staining a slide, the method comprising the steps of an existing clinical standard staining step using a first labeling agent (e.g., an immunohistochemical marker) and additional steps to add at least a second labeling agent (e.g., a fluorophore) to classify cells of various types. In some embodiments, the method may comprise adding a third labeling agent.

[0281] In some embodiments, a method as described herein further comprises determining a diagnosis, prognosis, and / or treatment plan for a subject from whom the biological sample was obtained based on the second score and / or the parameter indicative of the quantity, density, and / or level of an analyte in the sample. In some embodiments, the subject (e.g., a patient) is subsequently administered a treatment specific to the diagnosis, prognosis, and / or treatment plan in response to the determination of the diagnosis, prognosis, and / or treatment plan. An alert may be transmitted to the user based on the determination, or a report indicating the diagnosis, prognosis, and / or treatment may be produced, and in some instances transmitted and / or displayed (e.g., to the user).

[0282] The method is for staining a at least a portion of a sample for diagnosis and / or predictive testing in oncology, the method comprising the steps of performing an immunohistochemistry staining procedure using a first labeling agent (e.g., an existing clinical standard staining step with an immunohistochemical marker and for a cancer diagnosis and / or predictive testing in oncology) and additional steps to add a second labeling agent (e.g., a fluorescence marker) to classify components of the at least a portion of the sample (e.g., to classify cells into tumor cells and immune cells of various types).

[0283] In some embodiments, as shown in FIG. 22, a method for processing a biological sample comprising biological tissue, comprises the steps of performing an immunohistochemistry procedure on at least a portion of a sample on a slide using a first labeling agent (e.g., performing an existing clinical standard staining step with an immunohistochemical marker for a cancer diagnosis and / or predictive testing in oncology), generating a first score associated with an indication for the at least a portion the sample based on the immunohistochemistry procedure, wherein the performing of the procedure and the generating of the first score are conducted according to an existing clinical standard for the indication, generating a fluorescence image of at least a portion of the sample on the same slide based at least in part on a signal detected from a second labeling agent (e.g., a fluorescence marker such as a fluorophore), determining that the first score is indeterminant, and processing the fluorescence image to classify components of the at least a portion of the sample (e.g., to classify cells into tumor cells and immune cells of various types by visualizing at least one labeling agent on the immunohistochemical marker- stained slide with a bright-field microscope through the eyepieces or on the screen of a digital pathology system for the immunohistochemical marker and visualizing the staining on the same said slide with a fluorescent microscope or on the screen of a digital pathology system for the fluorescence markers), and generating, based on the classified components, a second score associated with the indication and / or a parameter indicative of the quantity, density, and / or level of an analyte in the sample. In some embodiments, such a method may aid a user in making an assessment of the indication and help avoid mistaken assessments that could lead to unproductive treatment with side effects and / or missing cancer patients that would have responded to a treatment.

[0284] In some embodiments, the method comprises assessing the staining of a slide sample. In some such embodiments, the assessing the staining comprises, (not necessarily in this order): (i) performing standard visual assessment through eyepieces of a microscope or on the screen of a computer in a digital pathology workflow; (ii) scanning the slide with conventional bright-field color imaging hardware to create an accurate digital color image of the slide for display on a computer screen when the user is uncertain of their visual assessment; (iii) performing image analysis, including AI- based algorithms developed with approaches, examples of which include but are not limited to stardist and mezmer, on the brightfield imagery to identify x-y coordinates of the centers of cell nuclei, wherein the nuclei are identified by staining selected from a group consisting of markers for hematoxylin stain and nuclear immunohistochemistry markers, (iv) scanning the slide, a second time with fluorescence imaging optics to capture images of the fluorescence markers, which have been selected to assist image analysis algorithms, including artificial intelligence (Al)-based algorithms, to classify cells into categories needed to calculate the score of the test, wherein these categories are what the user (e.g., pathologist) is attempting to discern and count visually based on views through the eyepieces or on the computer screen that include tumor and immune cells; (v) using fiducials on the slide that are visible in both bright- field and fluorescence imaging modes to register bright-field and fluorescence images coordinates; (vi) using nuclei x-y coordinates from the bright-field image analysis to identify regions in the fluorescence imagery, one for each nuclei, to provide input patterns into Al-based cell classification algorithms that classify cells that were detected in the bright-field imagery into cell types that the user was attempting to do visually; (vii) using measured optical density of the brightfield IHC marker to determine positivity of cells classified into cell types in step (v) and then using this information as input into scoring calculations, examples of which include but are not limited to, TPS, IPS, and CPS.

[0285] In some embodiments, the method comprises assessing the staining comprises: (i) performing standard visual assessment through eyepieces of a microscope or on the screen of a computer in a digital pathology workflow after conventional color scanning; (ii) if the user is uncertain of their visual assessment, scanning the slide with imaging hardware capable of imaging both chromogen and Anorogenic markers in the same scan to create an accurate digital color image of the chromogenic markers for display on a computer screen while retaining Auorescence imagery for use with image analysis algorithms (iii) using image analysis algorithms, including artificial intelligence (AI)- based algorithms, to classify cells into categories needed to calculate the score of the test using information from either or both chromogenic and Anorogenic imagery, wherein these cell categories are what the user is attempting to discern and count visually based on views through the eyepieces or on the computer screen that include tumor and immune cells; (iv) using measured optical density of the brightAeld IHC marker to determine positivity of cells classified into cell types in step (iii) and then using this information as input into scoring calculations, including, TPS, IPS, and CPS.

[0286] FIG. 23 shows a representative image of standard IHC imagery of a lung cancer sample, according to some embodiments; FIG. 24 shows representative images for: (a) staining of additional proteins revealed by tumor and for macrophages fluorescence markers imaged on a scanner equipped to image fluorescence; (b) the same image as (a) but with image analysis used to identify individual cell nuclei; and (c) the same image as (a) but with cells classified into categories using machine learning algorithms, in this case tumor cells, macrophages, and other cell types, according to some embodiments;

[0287] FIG. 25 shows a representative display on the computer screen that a user can use to confirm accuracy of image analysis-based cell classifications, to support a calculation of the clinical test score, such as Tumor Proportion Score (TPS), Immune Proportion Score (IPS), and Combined Positivity Score (CPS), according to some embodiments;

[0288] Examples of Biological Samples and Biomarkers

[0289] The biological sample imaged and / or analyzed in the systems and methods described above may comprise any of a variety of suitable biological materials. For example, in some embodiments, a sample (e.g., a biological sample) comprises biological tissue and / or biological cells. In some embodiments, the biological tissue may be obtained via biopsy (e.g., needle biopsy, endoscopic biopsy, and / or punch biopsy), surgical removal, and / or autopsy. In some embodiments, the biological tissue may comprise diseased tissue (or tissues suspected of being diseased), abnormal tissue (e.g., tumor tissue) and / or healthy tissue. In some embodiments, the biological material comprises biological cells. For example, the biological material may comprise tissue comprising biological cells. As another example, the biological material may comprise cells that are not collected as part of a tissue. In some embodiments, the biological cells are collected via swab (e.g., swabs of a mucosal tissue such as a nasal swab or throat swab), phlebotomy, fine needle aspiration, or from bodily fluids (e.g., urine, cerebrospinal fluid, pleural fluid, etc.). In some embodiments, a biological sample may comprise a combination of tissue sample and cells which are not collected as part of a tissue.

[0290] The biological material may be placed on a portion of the sample holder (e.g., a slide). The sample holder may comprise a material that is chemically and biologically inert. In some embodiments, such as those in which illumination light such as brightfield light is to be transmitted through the sample, at least a portion of the sample holder (e.g., the slide) is optically transparent. For example, in some embodiments, the sample holder (e.g., slide) comprises glass (e.g., optical quality glass, such as soda lime or borosilicate glass) and / or plastic. In some embodiments, the biological material may be fixed to the sample holder. The biological material may be physically fixed to the sample holder. In some, but not necessarily all embodiments, paraffin is used to support and protect the tissue in a ‘block’ for storage (e.g., the biological material may be a paraffin-embedded biological material such as a paraffin-embedded tissue). In some embodiments, the biological sample has been supported with paraffin but the paraffin is removed, for example in slide processing before labeling (e.g., staining). In some embodiments, the biological material is chemically fixed, for example by using formalin (e.g., the biological material may be a formalin-fixed biological material) and / or any of a variety of other fixation agents such as glutaraldehyde and / or osmium tetroxide.

[0291] In some embodiments, the biological sample (e.g., a tissue sample, a cell sample) comprises one or more biomarkers that may be detected, quantified, and / or characterized using labeling and imaging. In some embodiments, the biomarkers comprise proteins and / or peptides. In some embodiments, the biomarkers comprise proteins that are expressed by certain cells of clinical (e.g., diagnostic, treatment planning) or research interest). For example, in some embodiments, the biological sample comprises one or more biomarkers associated with one or more types of cancer (e.g., prostate-specific antigen (PSA), p63, cytokeratin (CK) 7 (CK7), CK20, thyroid transcription factor (TTF) 1 (TTF-1), Cyclin DI, neural cell adhesion molecule (NCAM), cancer antigen (CA) 125 (CA125), matrix metalloproteinase (MMP) 9 (MMP-9), vascular endothelial growth factor (VEGF), estrogen receptor (ER), human epidermal growth factor 2 (HER2), paired box protein pax-5 (pax5), paired box gene 8 (pax8), S100 proteins, SRY-box transcription factor 10 (Sox 10), and others). In some embodiments, the biological sample may comprise one or more biomarkers which are associated with a biological response such as an immune response (e.g., cluster differentiation (CD) 8 (CD8), alkaline phosphatase (ALP), programmed cell death ligand 1 (PD-L1), and others).

[0292] Examples of Labeling Agents

[0293] As noted above, some embodiments involve use of one or more labeling agents (e.g., a first labeling agent, a second labeling agent). The labeling agents may comprise a chemical marker, isotope, nanocrystal, small molecule, quantum dot, and / or other compound that can be localized to a component within a biological sample (e.g., a cell, a protein, a cellular substructure, etc.) by modifying and / or associating to the component directly (e.g., via a covalent bond, a non-covalent interaction and / or deposition / precipitation). In some embodiments, labeling agents may be localized to a component within a biological sample via association with (e.g., binding to) a binding agent (e.g., a biomolecule such an antibody and / or a fragment thereof and / or a nucleotide) that modifies and / or associates directly to a component of the sample (e.g., via a covalent bond or non-covalent conjugation). A labeling agent may be visually and / or optically identifiable such that the component to which the labeling agent is localized can be identified. For example, in some embodiments, a labeling agent is a nuclei labeling agent, a membrane labeling agent, a protein labeling agent (e.g., a labeling agent that selectively binds to a particular protein), a nucleic acid labeling agent, or other labeling agent which can be localized to a component within a biological sample.

[0294] In some embodiments, the labeling agent comprises a fluorophore. In some embodiments, the fluorophore may be directly or indirectly bound to a binding agent (e.g., an antibody). In some embodiments, the labeling agent comprises a fluorophore that is localized directly to a target analyte (e.g., a target biomarker) within the biological sample (e.g., the fluorophore is not bound to a binding agent but instead directly binds to the analyte or another sample component in proximity to the analyte). In some embodiments, the fluorophore is a stain. A fluorophore can absorb and re-emit light upon excitation. The fluorophore can absorb light at a specific wavelength (e.g., an excitation wavelength) or within a specific range of wavelengths (e.g., within an excitation range). The fluorophore may emit light at a longer wavelength than the excitation wavelength (e.g., an emission wavelength) or at a range of wavelengths which are longer than those in the excitation range (e.g., an emission range). In some instances, at least some of the fluorophores are small molecules. In some embodiments, the fluorophores comprise one or more dyes. Examples of small molecule fluorophores include, but are not limited to, Alexa Fluor dyes, fluorescein, rhodamine, cyanine, coumarin, fluorescein isothiocyanate (FITC), 4',6-diamidino-2-phenylindole (DAPI), 4,4-difluoro-4-bora-3a,4a-diaza-s-indacene (BODIPY), or others, amino acids (e.g., tryptophan, tyrosine, or phenylalanine), long-Stokes-shift fluorophores (e.g., DyLight™ Long Stoke Shift dyes from Thermo Scientific ™), and / or Forster Resonance Energy Transfer (FRET) fluorophore pairs (e.g., R-phycoerythirin (R-PE)-allophycocyanin (APC pair, R-PE-Invitrogen Cyanine5 (Cy5) pair, R-PE-Cyanine 5.5 (Cy5.5) pair). In some embodiments, the fluorophore is a protein (e.g., green fluorescent protein (GFP) or red fluorescent protein (RFP)). In some embodiments, the fluorophore is a solid (e.g., a solid particle), such as a semiconducting nanocrystal (e.g., a quantum dot). Any of a variety of fluorophores may be suitable for particular applications, and may be selected based on the target analyte(s) within the biological sample, desired user-friendliness, spectral compatibility with other labeling agents being used (e.g., to reduce or eliminate overlap in excitation or emission spectra), or other considerations. In some embodiments, the fluorophores are chosen from commonly used fluorophores in the fields of immunohistochemistry and / or immunofluorescence microscopy imaging.

[0295] In some embodiments, the labeling agent comprises a chromophore. In some embodiments, the chromophore may be directly or indirectly bound to a binding agent (e.g., an antibody). In some embodiments, the labeling agent comprises a chromophore which may be localized directly to a target analyte (e.g., a target biomarker) within the biological sample (e.g., the chromophore is not bound to a binding agent but instead directly binds to the analyte or another sample component in proximity to the analyte). In some embodiments, the chromophore is a stain. In some embodiments, at least some of the chromophores are small molecules. In some embodiments, the chromophores comprise one or more dyes. In some embodiments, the chromophores comprise one or more proteins (e.g., one or more chromoproteins).

[0296] In some embodiments, the chromophore may be present in the sample via a chromogen (e.g., in some embodiments, a chromogen may be used to label the sample). The chromogen may be directly or indirectly bound to the sample in any of the ways as described above for chromophores (e.g., via a binding agent, via direct binding to the analyte, etc.). In some embodiments, a chromogen is a compound which can be converted via chemical reaction to a chromophore. For example, a chromogen may be substantially colorless (by visual inspection) until it has been activated by a chemical reaction. In some embodiments, for example a chromogen is used to label the sample, and the chromogen is then activated by a chemical reaction to generate a chromophore which can be detected by visual and / or optical detection. In some embodiments, a chromogen reacts with a specific activating compound to convert to a chromophore. For example, in some embodiments, a chromogen may comprise 3'-diaminobenzidine (DAB), aminoethyl carbazole (AEC), or 3,3'5,5'-tetramethylbenzidine (TMB), which are activated by a reaction with horseradish peroxidase. A chromogen may comprise nitro blue tetrazolium chloride (NBT) and / or 5-bromo-4-chloro-3-indolyl phosphate (BCIP), which are activated by alkaline phosphatase (AP). A variety of chromogens may be suitable for particular applications, and may be selected based on the target analyte(s) within the biological sample, desired user-friendliness, spectral compatibility with other labeling agents being used, or other considerations. In some embodiments, the chromophores and / or chromogens are chosen from commonly used chromophores and / or chromogens in the field of immunohistochemistry.

[0297] In some embodiments, one or more labeling agents may be immobilized with respect to a sample (e.g., a biological sample, a biological sample comprising biological tissue). In some embodiments, a labeling agent immobilized with respect to a sample is associated with the sample. In some embodiments, the labeling agent is immobilized with respect to the sample without being directly bound and / or conjugated to the sample or any part of the sample. For example, in some embodiments, the labeling agent is restricted from free transport throughout the sample via kinetic limits on the speed of diffusion of the labeling agent throughout the sample. In some embodiments, the labeling agent that is immobilized with respect to the sample is directly bound and / or conjugated to the sample and / or a part of the sample. In some embodiments, the labeling agent is covalently bound to the sample or non-covalently conjugated to the sample. In some embodiments, the labeling agent is covalently bound to a binding agent that is covalently bound to the sample or non-covalently conjugated to the sample. For example, in some embodiments, a labeling agent is immobilized with respect to a sample via immunolabeling. In immunolabeling, one or more antibodies and / or fragments thereof may be used as a binding agent. The antibody and / or antibody fragment that binds to the target component may be called the primary antibody and / or antibody fragment. For example, in some embodiments, the primary antibody and / or antibody fragment binds to a target antigen within the sample comprising a protein, a sugar, and / or a ligand. The primary antibody and / or antibody fragment may be bound covalently to a labeling agent (e.g., a chromophore, a fluorophore). In some embodiments the primary antibody and / or antibody fragment is covalently or noncovalently bound to a secondary antibody and / or antibody fragment. In some embodiments, the secondary and / or antibody fragment is noncovalently or covalently bound to a labeling agent. In some embodiments, the primary antibody and / or antibody fragment and / or the secondary antibody and / or antibody fragment is noncovalently or covalently bound to an activating agent (e.g., an activating enzyme) that may bind to, react with, and / or catalyze a reaction with a precursor labeling agent that is not otherwise associated with the primary antibody and / or antibody fragment or secondary antibody and / or antibody fragment to produce the labeling agent. The labeling agent may be bound to the primary antibody and / or antibody fragment and / or the secondary antibody and / or antibody fragment or may be localized within the sample (e.g., via a conjugation, chemical reaction, catalyzed reaction, and / or deposition or precipitation)., thus becoming immobilized with respect to the sample. The labeling agent may then be identified visually or optically, allowing for identification of the target component within the biological sample.

[0298] In some embodiments, amplification techniques may be used (e.g., in conjunction with immune labeling, as described above) to increase the signal produced in the biological sample. For example, in some embodiments, tyramide signal amplification is performed on a sample. For example, a primary antibody and / or antibody fragment and / or a secondary antibody and / or antibody fragment may be bound to horseradish peroxidase (HRP) as the activating agent. The sample may then be exposed to an inactive tyramide substrate covalently bound to a labeling agent (e.g., a fluorophore) in the presence of hydrogen peroxide. The HRP catalyzes activation of the tyramide substrate, which then may bond to portions of the sample (e.g., tyrosine residues) on or in proximity to the antigen to which the primary antibody is bound. This may result in highly localized, high-density binding of the tyramide or near at the target analyte, at which point the labeling agent bound to the tyramide substrate can be detected by the visual and / or optical detection. Other signal amplification techniques for forming large numbers of labeling agents include, for example, nucleic acid amplification techniques (e.g., rolling circle amplification (RCA)).

[0299] In some embodiments, a labeling agent is immobilized with respect to a sample by selectively binding to a target analyte of a sample due to a particular chemical affinity. For example, in some embodiments, the labeling agent comprises a nuclei stain. In some such embodiments, the nuclei stain may selectively bind to nucleic acids, nuclear proteins, and / or other analytes which are specific to the nucleus within the sample, thus becoming immobilized with respect to the sample. In some embodiments, a nuclei stain comprises a fluorescent nuclei stain (e.g., a fluorophore) such as 4', 6- diamidino-2-phenylindole (DAPI), propidium iodide, thiazole dyes (e.g., thiazole red, thiazole orange, thiazole green, oxazole gold, bisbenzimide, or others), and / or DRAQ5™ dye. In some embodiments, a nuclei stain comprises a visually detectable non- fluorescent nuclei stein (e.g., hematoxylin, carmine, methylene blue, toluyene red, Nile blue, or others). A variety of nuclei stains may be suitable for particular applications, and may be selected based on the target nuclei analyte within the biological sample, desired user-friendliness, spectral compatibility with other labeling agents, or other considerations. In some embodiments, the nuclei stain(s) is chosen from commonly used nuclei stains in the fields of immunohistochemistry and / or immunofluorescence microscopy imaging.

[0300] In some embodiments, the labeling agent comprises a membrane stain. In some such embodiments, the membrane stain may selectively bind to membranes within the sample, thus becoming immobilized with respect to the sample. In some embodiments, a membrane stain comprises a fluorescent membrane stain (e.g., lipophilic dyes such as DiO, DiA, Dil, DiD, CellBrite , MemBrite, PlasMem, CellMask, SynaptoGreen, SynaptoRed). In some embodiments, a membrane stain comprises a non-fluorescent membrane stain (e.g., Oil Red O, Nile red, and others). A variety of membrane stains may be suitable for particular applications, and may be selected based on the target membrane analyte within the biological sample, desired user-friendliness, spectral compatibility with other labeling agents, or other considerations. In some embodiments, the membrane stain(s) is chosen from commonly used membrane stains in the fields of immunohistochemistry and / or immunofluorescence microscopy imaging.

[0301] Unmixing Methodology

[0302] In some embodiments, as noted above, at least n acquisition images (e.g., fluorescence acquisition images or color intensity acquisition images) are acquired (e.g., each corresponding to one of at least n different excitations, or each corresponding to one of at least n different absorption spectral bands) and at least two spectral response images are generated from each acquisition image (e.g., using a processor). In some embodiments, three spectral response images may be generated from each acquisition image (e.g., when spectral response images corresponding to a red spectral response image, a green spectral response image, and a blue spectral response image are generated from an RGB sensor). In such embodiments, at least 3n spectral response images can be generated from n acquisition images. In some embodiments, more than three spectral response images may be generated from each acquisition image (e.g., when a CMYK sensor is employed).

[0303] In some embodiments, the method comprises performing an unmixing of the detected light from the labeling agents using at least some of the generated spectral response images. For example, in some embodiments, the method comprises performing an unmixing of the detected fluorescence light from one or more fluorophores using at least some of the generated spectral response images. In some embodiments, the method comprises performing an unmixing of the optical density values from one or more chromogens using at least some of the generated spectral response images. Unmixing is a procedure that allows for the deconvolution of multiple spectral signals (e.g., fluorescence signals or chromogenic signals) from different sources (e.g., labeling agents) contained in a spectral response image. The unmixing may comprise a linear unmixing procedure. The linear unmixing procedure may comprise adjusting the parameters of a model (e.g., adjusting the coefficients of a model comprising a linear combination of the amount of each spectrum corresponding to a known labeling agent within the acquisition image) using a least-squares fit. A least-squares fit can be represented in matrix form as Eqn. 1:

[0304] (XTX) p = XTy Eqn. 1 where X is a data matrix in which each element Xtj in the data matrix represents the spectra from an individual sensing element (e.g., pixel) in the spectral response image; XTis the transpose of X and y is a dependent variable vector. This represents a set of linear equations that can be solved for the vector p, which represents the leastsquares estimator of coefficient values, using Eqn. 2:

[0305] P = (XTX)~1XTy Eqn. 2 While Eqn. 2 can be used to solve directly for the vector fl, doing so can result in difficulty evaluating the coefficient values due to possible numerical instabilities. As such, a QR decomposition method or a singular value decomposition method may also be used.

[0306] An additional consideration for a linear unmixing procedure is that, typically, an optical spectrum is available at each sensing element of the acquisition image (corresponding to an element of X), but there is no directly measured dependent variable (corresponding to an element of y) for each sensing element. In such cases, the linear unmixing procedure may comprise performing a linear fit of the coefficients in a model to minimize the residual between the spectrum data S and the fitted model, as shown in Eqn. 3: wherein Stj is the spectrum at sensing element of the acquisition image, the terms (A, B, ... ) are members of a basis set (e.g., a spectral library) being used for the unmixing procedure, and the terms bij,...) are the least-squares fit coefficients at the sensing element The members of the basis set may correspond to spectra associated with the labeling agents used in the sample being characterized by the acquisition image (e.g., the basis set may comprise a spectral library comprising the spectra of each of the fluorophores used to obtain a fluorescence acquisition image, or the basis set may comprise a spectral library comprising the absorption spectra associated with each of the chromogens used to obtain a chromogenic acquisition image). For example, the term A of a basis set may correspond to the absorption spectrum of a first chromophore used in a sample from which a chromogenic acquisition image is obtained, or the term A of a basis set may correspond to the fluorescence emission spectrum of a first fluorophore used in a sample from which a fluorescence acquisition image is obtained. Each member of the basis set combined with its respective coefficient (e.g., a^A, b^B, etc.) corresponds to a layer of the spectral response image, meaning that each layer corresponds to a given member of the basis set (e.g., a particular fluorophore or a particular chromogen). These layers correspond to the “unmixed” signals of a particular labeling agent (e.g., fluorophore or chromogen) by separating the contribution of each component of the spectral response image corresponding to a particular member of the basis set (e.g., a particular labeling agent such as a particular fluorophore or chromogen) from the others. In some embodiments, these layers can be used to form a multilayer image set of the portion of a biological sample from which the acquisition image (e.g., the fluorescence acquisition image or the chromogenic acquisition image) was generated. These can be used in further processing and analysis of the fluorescence images or chromogenic images.

[0307] Additional constraints may be used in the unmixing procedure described above in order to prevent poor fitting. For example, in some instances, non-negativity constraints of the fit coefficients are used. However, a linear unmixing will typically, but not necessarily, follow the basic procedure outlined in Eqns. 1-3 above. Further details about unmixing can be found in, for example, the journal article published by J. R. Mansfield, C. Hoyt, and R. M. Levenson entitled “Visualization of Microscopy-based Spectral imaging data from Multi-Label Tissue Sections” and published in Current Protocols in Molecular Biology Unit 14.19 in 2008, which is incorporated herein by reference in its entirety.

[0308] Example Image Segmentation Techniques

[0309] As discussed above, some embodiments of the technology described in this disclosure involve segmenting image(s) of cells in a biological sample. In some embodiments, segmenting an image of cells involves identifying and distinguishing between portions (e.g., groups of neighboring pixels) of an image that depict different cells and / or cellular components. Identifying a portion of an image that depicts a cell may, for example, involve segmenting the image to identify portions of the image that depict one or more cellular component(s), such as the membrane and / or nuclei, and using the results of the segmentation to identify the cell. For example, the segmentation of the nuclei may be used to identify different instances of cells, while the segmentation of the membrane may be used to identify the boundaries of the different cells around the nuclei. Thus, in some embodiments, segmenting image(s) of cells in a biological sample may involve (i) segmenting image(s) of nuclei in the biological sample to produce a nuclear segmentation map, (ii) segmenting image(s) of plasma membranes in the biological sample to produce a membrane segmentation map, and (iii) segmenting image(s) of cells in the biological sample based on the nuclear and membrane segmentation maps. The image(s) may be segmented using semantic segmentation, instance segmentation, and / or panoptic segmentation techniques, as aspects of the technology described herein are not limited in this respect.

[0310] In some embodiments, image segmentation is performed using one or more deep learning models. For example, a deep learning model may include a neural network such as a convolutional neural network (CNN), a transformer, or any other suitable type of neural network, aspects of the technology described herein are not limited in this respect. For example, a CNN-based image segmentation model may include a U-Net, a fully connected network (FCN), a SegNet, mask R-CNN, DeepLab (vl-v3), pyramid scene parsing network (PSPNet), StarDist, HRNet, Mesmer, Cellpose, and Cellpose 2 or any other suitable CNN-based image segmentation model, as aspects of the technology described herein are not limited in this respect. A transformer-based segmentation model may include a vision transformer (ViT), a detection transformer (DETR), a detection transformer with improved denoising anchor (DINO), MaskDino, a segmentation transformer (SETR), Segmenter, a masked- attention mask transformer (Mask2Former), CelloType, or any other suitable transformer, as aspects of the technology described herein are not limited in this respect.

[0311] In some embodiments, the deep learning model(s) may be trained using any suitable training technique(s), including supervised techniques, semi- supervised techniques, unsupervised techniques, or any suitable combination thereof without limitation.

[0312] In some embodiments, a CNN-based image segmentation model is used to performed image segmentation. In some embodiments, a U-Net is used to perform image segmentation. U-Net is described by Ronneberger, O., et al. ("U-net: Convolutional networks for biomedical image segmentation." Medical image computing and computer- assisted intervention-MICCAI 2015 : 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 11118. Springer international publishing, 2015.), which is incorporated by reference herein in its entirety. In some embodiments, an FCN is used to perform image segmentation. FCNs are described by Long, J., et al. ("Fully convolutional networks for semantic segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.), which is incorporated by reference herein in its entirety. In some embodiments, SegNet is used to perform image segmentation. SegNet is described by Badrinarayanan, V., et al. ("Segnet: A deep convolutional encoder-decoder architecture for image segmentation." IEEE transactions on pattern analysis and machine intelligence 39.12 (2017): 2481-2495.), which is incorporated by reference herein in its entirety. In some embodiments, mask R-CNN is used to perform image segmentation. Mask R-CNN is described by He, K., et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017.), which is incorporated by reference herein in its entirety. In some embodiments, DeepLab is used to perform image segmentation. For example, any version of DeepLab may be used. DeepLab is described by Chen, L., et al. ("Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs." IEEE transactions on pattern analysis and machine intelligence 40.4 (2017): 834-848.), which is incorporated by reference herein in its entirety. In some embodiments, PSPNet is used to perform image segmentation. PSPNet is described by Zhao, H., et al. ("Pyramid scene parsing network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.), which is incorporated by reference herein in its entirety. In some embodiments, StarDist is used to perform image segmentation. StarDist is described by Schmidt, U., et al. ("Cell detection with star-convex polygons." Medical image computing and computer assisted intervention-MICCAI 2018: 21st international conference, Granada, Spain, September 16-20, 2018, proceedings, part II 11. Springer International Publishing, 2018.), which is incorporated by reference herein in its entirety. In some embodiments, HRNet is used to perform image segmentation. HRNet is described by Wang, J., et al. ("Deep high-resolution representation learning for visual recognition." IEEE transactions on pattern analysis and machine intelligence 43.10 (2020): 3349-3364.), which is incorporated by reference in its entirety. In some embodiments, Mesmer is used to perform image segmentation. Mesmer is described by Greenwald, N. F. et al. (“Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning.” Nat. Biotechnol. 40, 555-565 (2022).), which is incorporated by reference herein in its entirety. In some embodiments, Cellpose is used to perform image segmentation. Cellpose is described by Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. (“Cellpose: a generalist algorithm for cellular segmentation.” Nat. Methods 18, 100-106 (2021).), which is incorporated by reference herein in its entirety. In some embodiments, Cellpose 2 is used to perform image segmentation. Cellpose 2 is described by Pachitariu, M. & Stringer, C. (“Cellpose 2.0: how to train your own model.” Nat. Methods 19, 1634-1641 (2022).), which is incorporated by reference herein in its entirety.

[0313] In some embodiments, a transformer-based image segmentation model is used to perform image segmentation. In some embodiments, a ViT is used to perform image segmentation. ViT is described by Dosovitskiy, A., et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).), which is incorporated by reference herein in its entirety. In some embodiments, SETR is used to perform image segmentation. SETR is described by Zheng, S., et al. ("Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021.), which is incorporated by reference herein in its entirety. In some embodiments, Segmenter is used to perform image segmentation. Segmenter is described by Strudel, R., et al. "Segmenter: Transformer for semantic segmentation." Proceedings of the IEEE / CVF international conference on computer vision. 2021.), which is incorporated by reference herein in its entirety. In some embodiments, Mask2Former is used to perform image segmentation. Mask2Former is described by Cheng, B., et al. ("Masked-attention mask transformer for universal image segmentation." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022.), which is incorporated by reference herein in its entirety. In some embodiments, CelloType is used to perform image segmentation. CelloType is described by Pang, M., et al. "CelloType: a unified model for segmentation and classification of tissue images." Nature methods (2024): 1-10.), which is incorporated by reference herein in its entirety.

[0314] In some embodiments, the image segmentation is performed using one or more traditional segmentation techniques (i.e., non-deep learning techniques). For example, image segmentation may be performed using thresholding, edge detection, region-based segmentation, one or more clustering algorithms (e.g., Gaussian, k-means, etc.), the Watershed algorithm, or any other suitable techniques as aspects of the technology described herein are not limited in this respect.

[0315] Examples of Classification Techniques Embodiments of the technology described herein may employ machine learning techniques to classify subject matter depicted in one or more images. For example, machine learning techniques may be used to classify one or more components of a biological sample depicted in an image (e.g., a fluorescence image and / or an optical density image).

[0316] In some embodiments, classification is performed using the image segmentation techniques described herein. For example, in addition to the image segmentation itself, the output of the image segmentation techniques may indicate, for each of at least some of the pixels in an image, a classification of that pixel into one or more categories. For example, the output may indicate, for a particular pixel, the likelihood that the particular pixel depicts a particular component of a cell or a cell of a particular type (e.g., a tumor cell, an immune cell of a particular type, etc.). CelloType is an example of a combined cell segmentation and classification model.

[0317] In some embodiments, a separate classification model may be used to classify subject matter depicted in one or more images based on the segmentation of the one or more images. For example, the output of an image segmentation model may be provided as input to a classification model trained to classify the subject matter depicted in the image(s). The classification model may include a machine learning model. For example, the machine learning model may include a neural network, such as a CNN. Examples of classification models include CellSighter and CEEESTA. Cellsighter is described by Amitay, Y. et al. (“CellSighter: a neural network to classify cells in highly multiplexed images. Nat. Commun. 14, 4302 (2023).), which is incorporated by reference herein in its entirety. CEEESTA is described by Zhang, W. et al. (“Identification of cell types in multiplexed in situ images by combining protein expression and spatial information using CELESTA. Nat. Methods 19, 759-769 (2022).), which is incorporated by reference herein in its entirety.

[0318] As described above, embodiments of the technology described herein may involve performing image analysis. For example, embodiments of the technology described herein may involve analyzing layer(s) of multilayer image sets (e.g., unmixed multilayer image sets). For example, an image (or layer thereof) may be analyzed to determine the presence, density, and / or level of at least one analyte (e.g., a biomarker and / or a type of cell) in the at least a portion of the biological sample. Additionally or - I l l - alternatively, an image (or layer thereof) may be analyzed to identify cellular objects and tissue regions, classify cells into cell types and functional states, and / or measures various parameters that capture biology of interest, which may include spatial information such as density and proximity.

[0319] It should be appreciated that any of the above-described image segmentation and / or classification techniques may be used to perform the analysis of such image(s) and / or layer(s) thereof.

[0320] Example System Including Computer and Software Modules

[0321] FIG. 26 is a block diagram of an example system 2600 for processing image(s) of biological sample(s), according to some embodiments of the technology described herein. System 2600 includes one or multiple computing devices 2650. Software 2610 is configured to execute on computing device(s) 2650 to perform various functions in connection with processing image(s) of biological sample(s). In some embodiments, software 2610 includes a plurality of modules. A module may include processorexecutable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform function(s) of the module. Such modules are sometimes referred to herein as “software modules,” each of which includes processor-executable instructions configured to perform one or more acts of one or more processes.

[0322] In some embodiments, when computing device(s) 2650 includes multiple computing devices, the multiple computing devices may be configured to communicate via at least one communication network such as the Internet or any other suitable communication network(s), as aspects of the technology described herein are not limited in this respect. For example, the multiple computing devices may be part of a cloud computing environment. The cloud computing environment may be a public cloud computing environment, a private computing environment or a hybrid computing environment operating using a combination of publicly accessible and private infrastructure.

[0323] The computing device(s) 2650 may be operated by one or more user(s) 2630. In some embodiments, the user(s) 2630 provides input to computing device(s) 2650. For example, the user(s) 2630 may provide image data, data for training machine learning model(s), and / or any other suitable information, without limitation. Additionally or alternatively, user(s) 2630 may provide input specifying processing or other methods to be performed on image data, training data, and / or data obtained as a result of using detection system 2640. User(s) 2630 my provide input by uploading one or more files, interacting with a user interface, or using any other suitable technique for providing input, without limitation.

[0324] In some embodiments, the image processing module 2602 is configured to process image data obtained from the light detection system 2640, user(s) 2630, and / or data store(s) 2660. For example, the image processing module 2602 may be configured to generate spectral response image(s) from fluorescence acquisition image(s). Additionally or alternatively, image processing module 2602 may be configured to generate multi-layer image(s) from spectral response image(s). Additionally or alternatively, the image processing module 2602 may be configured to generate a combination image based on at least a nuclei label image and a membrane label image.

[0325] In some embodiments, the unmixing module 2604 is configured to perform unmixing of detected fluorescence light from fluorophores using spectral response images. In some embodiments, the unmixing module 2604 is configured to perform unmixing of detected chromogenic signals from chromophores using spectral response images.

[0326] In some embodiments, the image segmentation and classification module 2606 is configured to segment image(s) of biological sample(s). For example, this may involve (i) segmenting image(s) of nuclei in the biological sample to produce a nuclear segmentation map, (ii) segmenting image(s) of plasma membranes in the biological sample to produce a membrane segmentation map, and (iii) segmenting image(s) of cells in the biological sample based on the nuclear and membrane segmentation maps. The image(s) may be segmented using semantic segmentation, instance segmentation, and / or panoptic segmentation techniques, as aspects of the technology described herein are not limited in this respect. Example segmentation techniques are described herein in more detail.

[0327] Additionally or alternatively, the image segmentation and classification module 2606 is configured to classify portions of image(s) of biological samples. For example, the output of the segmentation techniques may indicate, for each of at least some of the pixels in an image, a classification of the pixel. The classification may indicate, for example, that the pixel depicts a cell or cell component of a particular type. For example, the classification techniques may be used to classify cells as tumor cells or immune cells of a particular type. Example segmentation and classification techniques are described herein in more detail.

[0328] In some embodiments, the image analysis module 2614 is configured to analyze image data and / or results of the segmentation and / or classification to determine the presence, density, and / or level of at least one analyte in a portion of a biological sample.

[0329] As shown in FIG. 26, software 2610 also includes user interface module 2612.

[0330] User interface module 2612 may be configured to generate a graphical user interface (GUI) through which user(s) 2630 may provide input and view information generated by software 2610. For example, in some embodiments, the user interface module 2612 may be a webpage or web application accessible through an Internet browser. In some embodiments, the user interface module 2612 may generate a GUI of an app executing on a user’s mobile device. For example, computing device(s) 2650 may be the user’s mobile device, and the user interface module 2612 may generate a GUI of an app executing thereon. In some embodiments, the user interface module 2612 may generate a number of selectable elements through which a user may interact. For example, the user interface module 2612 may generate dropdown lists, checkboxes, text fields, or any other suitable element. In some embodiments, the user interface module 2612 generates a GUI that includes one or more images, segmentation maps, and / or results of classifying portions of one or more images.

[0331] In some embodiments, the machine learning model training module 2616 is configured to train one or more machine learning models to (i) segment image(s), and / or (ii) classify portions of the image(s). For example, the machine learning model training module 2616 may obtain training data and / or validation data from data store(s) 2660, light detection system 2640, and / or user(s) 2630. For example, the training and / or validation data may include images of biological samples. The image(s) may be labeled. For example, pixels of the images may be labeled to indicate one or more classes to which the pixels belong. The machine learning model training module 2616 may be configured to use the obtained training and / or validation data to train and / or validate one or more machine learning models to segment images and / or classify portions of the images (e.g., classify cells and / or components of cells in the images). In some embodiments, the machine learning model training module 2616 may provide the trained machine learning model(s) to data store(s) 2660 for storage thereon. For example, the machine learning model training module 2616 may provide the values of parameters of the machine learning model(s) to the data store(s) 2660 for storage thereon.

[0332] As shown in FIG. 26, exemplary system 2600 also includes data store(s) 2660. The data store(s) 2660 may store image data obtained from light detection system 2640 and / or user(s) 2630. Additionally or alternatively, the data store(s) 2660 may store output(s) from one or more of the modules (e.g., image processing module 2602, unmixing module 2604, image segmentation and classification module 2606, and / or image analysis module 2614. Additionally or alternatively, the data store(s) 2660 may store training and / or validation data used by the machine learning model training module 2616 to train one or more machine learning models. For example, the data store(s) 2660 may store parameters of one or more trained machine learning models. It should be appreciated that data store(s) 2660 may be configured to store any other suitable information, without limitation. Each of the data stores may include any suitable type of data store (e.g., a flat file, a database system, a multi-file, etc.) and may store data in any suitable format, without limitation. The data store(s) 2660 may be part of software 2610 (not shown) or excluded from software 2610, as shown in FIG. 26.

[0333] Computer Implementation

[0334] An illustrative implementation of a computer system 00 that may be used in connection with any of the embodiments of the technology described herein is shown in FIG. 27. The computer system 2700 includes one or more processors 2710 and one or more articles of manufacture that comprise non-transitory computer-readable storage media (e.g., memory 2720 and one or more non-volatile storage media 2730). The processor 2710 may control writing data to and reading data from the memory 2720 and the non-volatile storage media 2730 in any suitable manner, as the aspects of the technology described herein are not limited to any particular techniques for writing or reading data. To perform any of the functionality described herein, the processor 2710 may execute one or more processor-executable instructions stored in one or more non- transitory computer-readable storage media (e.g., the memory 2720), which may serve as non-transitory computer-readable storage media storing processor-executable instructions for execution by the processor 2710.

[0335] Computing system 2700 may include a network input / output (I / O) interface 2740 via which the computing device may communicate with other computing devices. Such computing devices may be interconnected by one or more networks in any suitable form, including a local area network or a wide area network, such as an enterprise network, and intelligent network (IN) or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.

[0336] Computing system 2700 may also include one or more user I / O interfaces 2750, via which the computing device may provide output to and receive input from a user. The user I / O interfaces may include devices such as a keyboard, a mouse, a microphone, a display device (e.g., a monitor or touch screen), speakers, a camera, and / or various other types of I / O devices. In some embodiments, the computing system comprises a display device, where the display device is configured to display (e.g., to a user) an image generated according to one or more of the embodiments of this disclosure, and / or a report with an analysis thereof (e.g., comprising information regarding a measured parameter related to a biomarker and / or a diagnosis, prognosis, and / or recommended method of treatment related to an indication such as a disease).

[0337] Further, it should be appreciated that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, as examples. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smartphone, a tablet, or any other suitable portable or fixed electronic device.

[0338] The above-described embodiments can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor (e.g., a microprocessor) or collection of processors, whether provided in a single computing device or distributed among multiple computing devices. It should be appreciated that any component or collection of components that perform the functions described above can be generically considered as one or more controllers that control the above-described functions. The one or more controllers can be implemented in numerous ways, such as with dedicated hardware, or with general purpose hardware (e.g., one or more processors) that is programmed using microcode or software to perform the functions recited above.

[0339] In this respect, it should be appreciated that one implementation of the embodiments described herein comprises at least one computer-readable storage medium (e.g., RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible, non- transitory computer-readable storage medium) encoded with a computer program (i.e., a plurality of executable instructions) that, when executed on one or more processors, performs the above-described functions of one or more embodiments. The computer- readable medium may be transportable such that the program stored thereon can be loaded onto any computing device to implement aspects of the techniques described herein. In addition, it should be appreciated that the reference to a computer program which, when executed, performs any of the above-described functions, is not limited to an application program running on a host computer. Rather, the terms computer program and software are used herein in a generic sense to reference any type of computer code (e.g., application software, firmware, microcode, or any other form of computer instruction) that can be employed to program one or more processors to implement aspects of the techniques described herein.

[0340] The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects as described above. Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computer or processor but may be distributed in a modular fashion among a number of different computers or processors to implement various aspects of the present disclosure.

[0341] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0342] Also, data structures may be stored in computer-readable media in any suitable form. For simplicity of illustration, data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a computer-readable medium that convey relationship between the fields. However, any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements.

[0343] When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.

[0344] The foregoing description of implementations provides illustration and description but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the implementations. In other implementations the methods depicted in these figures may include fewer operations, different operations, differently ordered operations, and / or additional operations. Further, non-dependent blocks may be performed in parallel.

[0345] It will be apparent that example aspects, as described above, may be implemented in many different forms of software, firmware, and hardware in th...

Claims

CLAIMSWhat is claimed is:

1. A system for detecting light emitted from a sample, comprising: a sensor; and optics comprising an optical block, wherein the optics are configured to: receive excitation light from an excitation source; direct, via the optical block, the excitation light in at least n excitation bands to at least a portion of a sample on a sample holder when the sample and the sample holder are present; receive fluorescence light emitted from the sample; filter, via the optical block, the received fluorescence light to produce filtered fluorescence light having at least n emission bands; and direct, via the optical block, the filtered fluorescence light to a detector comprising the sensor when the detector is present, such that the filtered fluorescence light is incident upon sensing elements of the sensor; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

2. A system for detecting light emitted from a sample, comprising: a sensor; and optics comprising one or more filters, wherein the optics are configured to, in a single configuration of the one or more filters: receive excitation light from an excitation source;direct the excitation light in at least n excitation bands to at least a portion of a sample on a sample holder when the sample and the sample holder are present; receive fluorescence light emitted from the sample; filter the received fluorescence light to produce filtered fluorescence light having at least n emission bands; and direct the filtered fluorescence light to a detector comprising the sensor when the detector is present, such that the filtered fluorescence light is incident upon sensing elements of the sensor; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

3. The system of any one of claims 1-2, wherein the optics comprise a single optical block.

4. The system of any one of claims 1 and 3, wherein the optical block is an epi-cube comprising a dichroic and a multi-bandpass filter configured to filter the received fluorescence light into the filtered fluorescence light.

5. The system of any one of claims 1-4, wherein the at least n excitation bands are interlaced with the at least n emission bands.

6. The system of any one of claims 1-5, wherein the at least n excitation bands are non-overlapping .

7. The system of any one of claims 1-6, wherein the at least n emission bands are non-overlapping .

8. The system of any one of claims 1-7, wherein the sensor further comprises third sensing elements configured to have a third spectral response that is different from the first spectral response and the second spectral response.

9. The system of claim 8, wherein the sensor is a Red, Green, Blue (RGB) sensor, such that the first sensing elements correspond to red pixels or subpixels, the second sensing elements correspond to green pixels or subpixels, and the third sensing elements correspond to blue pixels or subpixels.

10. The system of claim 9, wherein the sensor comprises or is coupled to a Bayer color filter array.

11. The system of any one of claims 1-10, wherein the first sensing elements and the second sensing elements are arranged on a single chip.

12. The system of any one of claims 1-11, further comprising the excitation source, wherein the system is configured to produce light from the excitation source in at least two of the at least n excitation bands sequentially.

13. The system of any one of claims 1-12, further comprising the excitation source, wherein the system is configured to produce light from the excitation source in each of the at least n excitation bands sequentially.

14. The system of any one of claims 1-13, further comprising the excitation source, wherein the system is configured to produce light from the excitation source in each of the at least n excitation bands individually.

15. The system of any one of claims 1-14, further comprising the excitation source, wherein the excitation source comprises at least n individual light sources.

16. The system of any one of claims 1-15, further comprising the excitation source, wherein the excitation source comprises at least n light emitting diodes (LEDs).

17. The system of any one of claims 1-16, wherein the optics are further configured to filter the received excitation light to produce filtered excitation light in the at least n excitation bands and direct the filtered excitation light to the at least a portion of a sample on a sample holder when the sample and the sample holder are present.

18. The system of any one of claims 1-17, wherein the optics comprise at least n individual single-bandpass excitation filters configured to filter the received excitation light to produce the filtered excitation light.

19. The system of any one of claims 1-18, wherein the optical block comprises a multi-bandpass excitation filter configured to produce the filtered excitation light.

20. The system of any one of claims 1-19, wherein the system is configured to excite a sample on the sample holder with excitation light in at least n excitation bands sequentially and detect the resulting filtered fluorescence light from each excitation sequentially.

21. The system of any one of claims 1-20, wherein the system is configured to excite a sample on the sample holder with excitation light in at least n excitation bands and detect the at least n emission bands of the resulting filtered fluorescence light without replacing the optical block.

22. The system of any one of claims 1-21, wherein the system is configured to excite a sample on the sample holder with excitation light in at least n excitation bands and detect the at least n emission bands of the resulting filtered fluorescence light without replacing any of the optics.

23. The system of any one of claims 1-22, wherein the system is configured to excite a sample on the sample holder with excitation light in at least n excitation bands anddetect the at least n emission bands of the resulting filtered fluorescence light without changing a configuration of any filters of the optics.

24. The system of any one of claims 1-23, wherein the system comprises the excitation source, the sample holder, and the detector.

25. The system of claim 24, wherein the excitation source, sample holder, detector, and optics are part of a fluorescence microscope.

26. The system of claim 25, wherein the fluorescence microscope is a slide- scanning microscope.

27. The system of any one of claims 1-26, wherein the number of excitation bands and the number of emission bands are each equal to n.

28. The system of any one of claims 1-27, wherein n is an integer greater than or equal to 5.

29. The system of any one of claims 1-28, wherein n is an integer less than or equal to 7.

30. The system of any one of claims 1-29, wherein n is an integer equal to 5.

31. The system of any one of claims 1-30, further comprising: at least one processor; and at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for computationally generating an image of the biological sample, the method comprising: obtaining, from the detector, at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands;generating a plurality of spectral response images, the plurality of spectral response images comprising multiple spectral response images from each of the at least n fluorescence acquisition images, the multiple spectral response images from each of the at least n fluorescence acquisition images comprising a first spectral response image corresponding to fluorescence light detected by the first sensing elements and a second spectral response image corresponding to fluorescence light detected by the second sensing elements; and performing an unmixing of detected fluorescence light from fluorophores using at least some of the plurality of spectral response images to generate a multilayer image set of the at least a portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores.

32. The system of claim 31, wherein the multilayer image set is a multilayer composite image of the at least a portion of the biological sample, the multilayer composite image comprising layers corresponding to unmixed signals from individual fluorophores.

33. The system of claim 31, wherein the multilayer image set is a plurality of individual layers of images of the at least a portion of the sample, the plurality of individual layers comprising layers corresponding to unmixed signals from individual fluorophores.

34. A system for detecting light emitted from a sample, comprising: a sensor; and optics comprising an optical block, wherein the optics are configured to: receive excitation light from an excitation source; direct, via the optical block, the excitation light in at least n excitation bands to at least a portion of a sample on a sample holder when the sample and the sample holder are present; receive photo-induced emission light emitted from the sample; filter, via the optical block, the received emission light to produce filtered emission light having at least n emission bands; anddirect, via the optical block, the filtered emission light to a detector comprising the sensor when the detector is present, such that the filtered emission light is incident upon sensing elements of the sensor; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

35. A method for detecting light emitted from a biological sample, comprising: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially; directing, via an optical block, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the at least a portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optical block, the fluorescence light to produce filtered fluorescence light having at least n emission bands; and directing, via the optical block, the filtered fluorescence light to a detector comprising a sensor; and detecting, with sensing elements of the sensor, the filtered fluorescence light; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

36. A method for detecting light emitted from a biological sample, comprising:producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially; directing, via optics comprising one or more filters, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the at least a portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optics, the fluorescence light to produce filtered fluorescence light having at least n emission bands, wherein the one or more filters are in a single configuration during at least the filtering the fluorescence light; and directing, via the optics, the filtered fluorescence light to a detector comprising a sensor; and detecting, with sensing elements of the sensor, the filtered fluorescence light; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

37. A method of generating an image of a biological sample, comprising: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially, wherein n is an integer; directing, via optics comprising one or more filters, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the at least a portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optics, the fluorescence light to produce filtered fluorescence light having at least n emission bands; directing, via the optics, the filtered fluorescence light to a detector; and detecting, with the detector, the filtered fluorescence light to acquire at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands;generating at least two spectral response images from each of the at least n fluorescence acquisition images, for a total of at least 2n spectral response images; and performing an unmixing of the detected fluorescence light from the fluorophores using at least some of the 2n spectral response images to generate a multilayer image set of the portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores, wherein the one or more filters are in a single configuration during at least the filtering the fluorescence light.

38. A method of generating an image of a biological sample, comprising: producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially, wherein n is an integer; directing, via optics comprising one or more filters, the excitation light to at least a portion of a biological sample to excite fluorophores immobilized with respect to the portion of the sample such that the fluorophores emit fluorescence light; filtering, via the optics, the fluorescence light to produce filtered fluorescence light having at least n emission bands without changing the configuration of the one or more filters; directing, via the optics, the filtered fluorescence light to a detector; and detecting, with the detector, the filtered fluorescence light to acquire at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands; generating at least two spectral response images from each of the at least n acquired fluorescence acquisition images, for a total of at least 2n spectral response images; and performing an unmixing of the detected fluorescence light from the fluorophores using at least some of the 2n spectral response images to generate a multilayer image set of the portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores.

39. A method of generating an image of a biological sample, comprising:producing excitation light having at least n excitation bands, the light in at least two of the at least n excitation bands being produced sequentially, wherein n is an integer greater than or equal to 4; directing, via an optical block, the excitation light to at least a portion of a biological sample to excite at least m different fluorophores immobilized with respect to the at least a portion of the sample such that the at least m different fluorophores emit fluorescence light, wherein m is an integer greater than or equal to n+1; filtering, via the optical block, the fluorescence light to produce filtered fluorescence light having at least n emission bands; and directing, via the optical block, the filtered fluorescence light to a detector; and detecting, with the detector, the filtered fluorescence light to acquire at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands; performing a determined or over-determined unmixing of the detected fluorescence light from the at least m different fluorophores using spectral response images generated from the at least n fluorescence acquisition images to generate a multilayer image set of the at least a portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores of the at least m different fluorophores.

40. The method of any one of claims 35-39, wherein the fluorophores comprise at least m different fluorophores, wherein m is an integer greater than or equal to n+1.

41. The method of any one of claims 35-40, wherein the number of different fluorophores is greater than or equal to 6.

42. The method of any one of claims 39-41, wherein m is an integer greater than or equal to n+1 and less than or equal to 15.

43. The method of any one of claims 35-42, wherein the number of excitation bands and the number of emission bands are each equal to n.

44. The method of any one of claims 37-38 and 40-43, wherein n is an integer greater than or equal to 4.

45. The method of any one of claims 35-44, wherein n is an integer greater than or equal to 5.

46. The method of any one of claims 35-45, wherein the n is an integer less than or equal to 7.

47. The method of any one of claims 35-46, wherein n is an integer equal to 5.

48. The method of any one of claims 35-47, wherein the biological sample comprises biological tissue.

49. The method of any one of claims 35-36, further comprising using at least one processor to perform: obtaining, from the detector, at least n fluorescence acquisition images, each corresponding to one of the at least n excitation bands; generating a plurality of spectral response images, the plurality of spectral response images comprising multiple spectral response images from each of the at least n fluorescence acquisition images, the multiple spectral response images from each of the at least n fluorescence acquisition images comprising a first spectral response image corresponding to fluorescence light detected by the first sensing elements and a second spectral response image corresponding to fluorescence light detected by the second sensing elements; and performing an unmixing of detected fluorescence light from the fluorophores using at least some of the plurality of spectral response images to generate a multilayer image set of the at least a portion of the biological sample, the multilayer image set comprising layers corresponding to unmixed signals from individual fluorophores.

50. The method of any one of claims 37-49, further comprising analyzing the multilayer image set to determine the presence, density, and / or level of at least one analyte in the at least a portion of the biological sample.

51. The method of any one of claims 37-50, wherein the multilayer image set is a multilayer composite image of the at least a portion of the biological sample, the multilayer composite image comprising layers corresponding to unmixed signals from individual fluorophores.

52. The method of any one of claims 37-50, wherein the multilayer image set is a plurality of individual layers of images of the at least a portion of the sample, the plurality of individual layers comprising layers corresponding to unmixed signals from individual fluorophores.

53. The method of any one of claims 37-48 and 50-52, wherein the detector detects the filtered fluorescence light with sensing elements of a sensor, wherein the sensing elements comprise the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

54. The method of any one of claims 35-53, wherein the sensor further comprises third sensing elements that have a third spectral response that is different from the first spectral response and the second spectral response.

55. The method of claim 54, wherein the sensor is a Red, Green, Blue (RGB) sensor, such that the first sensing elements correspond to red pixels or subpixels, the second sensing elements correspond to green pixels or subpixels, and the third sensing elements correspond to blue pixels or subpixels.

56. The method of claim 55, wherein the sensor comprises or is coupled to a Bayer color filter array.

57. The system of any one of claims 35-56, wherein the first sensing elements and the second sensing elements are arranged on a single chip.

58. The method of any one of claims 35-57, wherein the optics comprise a single optical block.

59. The method of any one of claims 35-58, wherein the optical block is an epi-cube comprising a dichroic and a multi-bandpass filter that filters the received fluorescence light into the filtered fluorescence light.

60. The method of any one of claims 35-59, wherein the at least n excitation bands are interlaced with the at least n emission bands.

61. The method of any one of claims 35-60, wherein the at least n excitation bands are non-overlapping.

62. The method of any one of claims 35-61, wherein the at least n emission bands are non-overlapping .

63. The method of any one of claims 35-62, wherein the excitation light in each of the at least n excitation bands is produced sequentially.

64. The method of any one of claims 35-63, wherein the excitation light in each of the at least n excitation bands is produced individually.

65. The method of any one of claims 35-64, wherein the excitation light in the at least n excitation bands is produced by an excitation source comprising at least n individual light sources.

66. The method of any one of claims 35-65, wherein the excitation light in the at least n excitation bands is produced by an excitation source comprising. at least n light emitting diodes (LEDs).

67. The method of any one of claims 35-66, further comprising filtering the received excitation light to produce filtered excitation light in the at least n excitation bands and directing the filtered excitation light to the at least a portion of the sample.

68. The method of claim 67, wherein the optics comprise at least n individual singlebandpass excitation filters that filter the received excitation light to produce the filtered excitation light.

69. The method of claim 67, wherein the optical block comprises a multi-bandpass excitation filter that produces the filtered excitation light.

70. The method of any one of claims 35-69, comprising exciting the sample with the excitation light in at least n excitation bands sequentially and detecting the resulting filtered fluorescence light from each excitation sequentially.

71. The method of any one of claims 35-70, comprising exciting the sample with the excitation light in at least n excitation bands and detecting the at least n emission bands of the resulting filtered fluorescence light without replacing the optical block.

72. The method of any one of claims 35-71, comprising exciting the sample with the excitation light in at least n excitation bands and detecting the at least n emission bands of the resulting filtered fluorescence light without replacing any of the optics.

73. The method of any one of claims 35-72, comprising exciting the sample with the excitation light in at least n excitation bands and detecting the at least n emission bands of the resulting filtered fluorescence light without changing a configuration of any filters of the optics.

74. The method of any one of claims 35-73, wherein the method is performed at least in part using a fluorescence microscope.

75. The method of claim 74, wherein the fluorescence microscope is a slide- scanning microscope.

76. A method for detecting light emitted from a biological sample, comprising: producing excitation light having at least n excitation bands; directing, via an optical block, the excitation light to at least a portion of a biological sample to excite labeling agents immobilized with respect to the at least a portion of the sample such that the labeling agents emit photo-induced emission light; filtering, via the optical block, the emission light to produce filtered emission light having at least n emission bands; and directing, via the optical block, the filtered emission light to a detector comprising a sensor; and detecting, with sensing elements of the sensor, the filtered emission light; wherein: n is an integer greater than or equal to 4; and the sensing elements comprise: first sensing elements that have a first spectral response; and second sensing elements that have a second spectral response, wherein the first spectral response is different from the second spectral response.

77. A device for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the device comprising: a single epi-cube, a multi-Light Emitting Diode excitation light source (LED excitation source); and a ‘Red-Green-Blue’ color imaging sensor (RGB color imaging sensor), wherein the epi-cube comprises a multi-bandpass emission filter and a dichroic, each with 5 or more bandpasses, and wherein the LED excitation source comprises 5 or more individually controlled LEDs.

78. A system for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the system comprising: a single epi-cube; an LED excitation source, and an RGB color imaging sensor, wherein the epi-cube comprises multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source has 5 or more individually controlled LEDs.

79. A method for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, the method comprising the steps of: preparing a tissue slide from tissue sample from a subject; staining the slide with up to 15 fluorophore markers; and processing the stained slide using a device for multiplex immunofluorescence slide scanning for assessment of up to 15 fluorophore markers, wherein the device comprises: a single epi-cube; an LED excitation source; and an RGB color imaging sensor; wherein the epi-cube comprises a multi-bandpass emission filter and dichroic, each with 5 or more bandpasses, and wherein the LED excitation source has 5 or more individually controlled LEDs.

80. The device of claim 77, system of claim 78, or method of claim 79, wherein each of the 5 or more individually controlled LEDs has an excitation filter to filter LED output to create a more narrow spectral range or a single bandpass of excitation aligning spectrally with the bandpasses of the quintuple emission filter and dichroic, so that fluorophores are substantially excited individually and sequentially.

81. The device of any one of claims 77 and 80, system of any one of claims 78 and 80, or method of any one of claims 79 and 80, wherein the epi-cube consists of the multibandpass emission filter and the dichroic.

82. The device of claim 77, system of claim 78, or method of claim 79, wherein the device or system comprises a single quintuple bandpass excitation filter located within the epi-cube.

83. A system for detecting light transmitted through a sample, comprising: a detector, wherein when a sample holder and illumination source are present, the sample holder, illumination source, and detector establish an optical path for illumination light from the illumination source to travel from the illumination source, through at least a portion of a sample when the sample is present on the sample holder, and to the detector; and an actuatable filter having a first configuration in which the actuatable filter is not located in the optical path and a second configuration in which the actuatable filter is located in the optical path; wherein: the detector comprises a sensor and a color filter array, the presence of which results in the sensor comprising at least: first color pixels or subpixels having a first color spectral response; second color pixels or subpixels having a second color spectral response; and third color pixels or subpixels having a third color spectral response, wherein the first color spectral response, second color spectral response, and third color spectral response are different; and the actuatable filter is configured to reduce transmission of a portion of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response.

84. The system of claim 83, wherein the detector comprises a Red, Green, Blue (RGB) sensor such that the first color is red, the second color is green, and the third color is blue.

85. The system of any one of claims 83-84, wherein the system further comprises the illumination source.

86. The system of claim 85, wherein the illumination source is configured to produce broadband electromagnetic radiation.

87. The system of any one of claims 83-86, wherein, when in its second configuration, the actuatable filter is located in a portion of the optical path from the illumination source to the sample holder.

88. The system of any one of claims 83-86, wherein, when in its second configuration, the actuatable filter is located in a portion of the optical path from the sample holder to the detector.

89. The system of any one of claims 83-88, wherein the actuatable filter is configured to reduce transmission of greater than or equal to 30% and less than or equal to 70% of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response.

90. The system of any one of claims 83-89, wherein the actuatable filter is configured to reduce transmission of greater than or equal to 30% and less than or equal to 70% of the wavelengths in each of the first color spectral response, the second color spectral response, and the third color spectral response.

91. The system of any one of claims 83-90, wherein the actuatable filter is a bandpass filter.

92. The system of any one of claims 83-91, wherein the actuatable filter is a multibandpass filter.

93. The system of any one of claims 83-92, wherein the system comprises the illumination source, the sample holder, and the detector.

94. The system of claim 93, wherein the illumination source, sample holder, detector, and actuatable filter are part of a microscope.

95. The system of claim 94, wherein the microscope is a slide- scanning microscope.

96. The system of any one of claims 83-95, further comprising: at least one processor; and at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for computationally generating an image of the sample, the method comprising: obtaining a first color intensity acquisition image acquired when the actuatable filter is in the first configuration, the first color intensity acquisition image comprising color intensity spectral response images comprising: a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third intensity color intensity spectral response image corresponding to the third color pixels or subpixels; obtaining a second color intensity acquisition image acquired when the actuatable filter is in the second configuration, the first color intensity acquisition image comprising color intensity spectral response images comprising: a first color intensity spectral response image corresponding to the first color pixels or subpixels,a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third intensity color intensity spectral response image corresponding to the third color pixels or subpixels; generating a multilayer image set of the at least a portion of the sample, the multilayer image set comprising layers comprising two of the following:(1) color optical density spectral response images calculated from at least some of the color intensity spectral response images of the first color intensity acquisition image,(2) color optical density spectral response images calculated from at least some of the color intensity spectral response images of the second color intensity acquisition image, and(3) color optical density spectral responses images calculated from:(i) subtraction of the second color acquisition intensity image from the first color intensity acquisition image, or(ii) subtraction of (a) a second optical density image calculated from the second color intensity acquisition image from (b) a first optical density image calculated from the first color intensity acquisition image; and performing an unmixing of the multilayer image set to generate unmixed images, each corresponding to one of the multiple different immobilized chromophores.

97. The system of claim 96, wherein the multilayer image set comprises layers comprising (2) and (3).

98. The system of any one of claims 96-97, wherein the color optical density spectral responses images of (3) are calculated from (i).

99. The system of any one of claims 96-97, wherein the color optical density spectral responses images of (3) are calculated from (ii).

100. The system of any one of claims 96-99, wherein the multilayer image set is a multilayer composite image comprising the layers.

101. The system of any one of claims 96-99, wherein the multilayer image set is a plurality of individual layers.

102. A method for generating an image of a biological sample, comprising: directing, during a first period of time, illumination light along an optical path from an illumination source, through at least a portion of a biological sample comprising multiple different immobilized chromophores, and to pixels or subpixels of a detector, wherein the detector comprises a color filter array that results in the pixels or subpixels comprising: first color pixels or subpixels that have a first color spectral response; second color pixels or subpixels that have a second color spectral response; and third color pixels or subpixels that have a third color spectral response; detecting the illumination light that was directed to the detector during the first period of time to acquire a first color intensity acquisition image, the first color intensity acquisition image comprising color intensity spectral response images comprising: a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third intensity color intensity spectral response image corresponding to the third color pixels or subpixels; directing, during a second period of time, illumination light along the optical path to the pixels or subpixels of the detector, wherein a filter is located in the optical path during the second period of time but not the first period of time, wherein the filter reduces transmission of a portion of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response;detecting the illumination light that was directed to the detector during the second period of time to acquire a second color intensity acquisition image, the second color intensity acquisition image comprising color intensity spectral response images comprising: a first color intensity spectral response image corresponding to the first color pixels or subpixels, a second color intensity spectral response image corresponding to the second color pixels or subpixels, and a third color intensity spectral response image corresponding to the third color pixels or subpixels; generating a multilayer image set of the at least a portion of the biological sample, the multilayer image set comprising layers comprising two of the following:(1) one or more color optical density spectral response images calculated from at least some of the color intensity spectral response images of the first color intensity acquisition image,(2) one or more color optical density spectral response images calculated from at least some of the color intensity spectral response images of the second color intensity acquisition image, and(3) one or more color optical density spectral responses images calculated from:(i) subtraction of the second color acquisition intensity image from the first color intensity acquisition image, or(ii) subtraction of (a) a second optical density image calculated from the second color intensity acquisition image from (b) a first optical density image calculated from the first color intensity acquisition image; and performing an unmixing of the multilayer image set to generate unmixed images, each corresponding to one of the multiple different immobilized chromophores.

103. The method of claim 102, wherein the multiple different immobilized chromophores comprises at least 4 different immobilized chromophores.

104. The method of any one of claims 102-103, wherein the multilayer image set comprises layers comprising (2) and (3).

105. The method of any one of claims 102-104, wherein the one or more color optical density spectral responses images of (3) are calculated from (i).

106. The method of any one of claims 102-104, wherein the one or more color optical density spectral responses images of (3) are calculated from (ii).

107. The method of any one of claims 102-106, further comprising analyzing at least some of the unmixed images to determine the presence, density, and / or level of at least one analyte in the at least a portion of the biological sample.

108. The method of any one of claims 102-107, wherein the multilayer image set is a multilayer composite image comprising the layers.

109. The method of any one of claims 102-107, wherein the multilayer image set is a plurality of individual layers.

110. The method of any one of claims 102-109, wherein the biological sample comprises biological tissue.

111. The method of any one of claims 102- 110, wherein the detector comprises a Red, Green, Blue (RGB) sensor such that the first color is red, the second color is green, and the third color is blue.

112. The method of any one of claims 102-111, wherein illumination light is broadband electromagnetic radiation.

113. The method of any one of claims 102-112, wherein, during the second period of time, the actuatable filter is located in a portion of the optical path from the illumination source to the at least a portion of the sample.

114. The method of any one of claims 102-112, wherein, during the second period of time, the actuatable filter is located in a portion of the optical path from the at least a portion of the sample to the detector.

115. The method of any one of claims 102-114, wherein the actuatable filter reduces transmission of greater than or equal to 30% and less than or equal to 70% of the wavelengths in one or more of the first color spectral response, the second color spectral response, and the third color spectral response.

116. The method of any one of claims 102-115, wherein the actuatable filter reduces transmission of greater than or equal to 30% and less than or equal to 70% of the wavelengths in each of the first color spectral response, the second color spectral response, and the third color spectral response.

117. The method of any one of claims 102-116, wherein the actuatable filter is a bandpass filter.

118. The method of any one of claims 102-117, wherein the actuatable filter is a multibandpass filter.

119. The method of any one of claims 102-118, wherein the method is performed at least in part using a microscope.

120. The method of claim 119, wherein the microscope is a slide- scanning microscope.

121. A method for performing a segmentation of an image of a biological sample, the method comprising: obtaining a multilayer image set that was acquired by a detector, wherein the multilayer image set comprises:a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample; and a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample; generating a combination image based on the nuclei label image and the membrane label image; and segmenting images of nuclei in the biological sample based at least in part on the combination image to produce a nuclear segmentation map.

122. The method of claim 121, wherein the segmenting images of nuclei comprises segmenting at least a portion of the combination image to produce the nuclear segmentation map.

123. The method of any one of claims 121-122, wherein the combination image is generated by subtracting the membrane label image or an image derived from the membrane label image from the nuclei label image or an image derived from the nuclei label image.

124. The method of claim 123, wherein the combination image is generated by: scaling the intensities of the nuclei label image to generate a scaled nuclei label image; scaling the intensities of the membrane label image to generate a scaled membrane label image; and subtracting the scaled membrane label image from the scaled nuclei label image.

125. The method of any one of claims 121-124, further comprising: segmenting images of plasma membranes in the biological sample based at least in part on the membrane label image to produce a membrane segmentation map; and segmenting images of cells in the biological sample based on the nuclear segmentation map and the membrane segmentation map.

126. The method of any one of claims 121-125, wherein the membrane labeling agents comprise first membrane labeling agents and second membrane labeling agents, wherein the first membrane labeling agents are different from the second membrane labeling agents, and wherein the membrane label image is generated by combining a first membrane-labeling agent image associated with an optical signal produced by the first membrane labeling agents and a second membrane-labeling agent image associated with an optical signal produced by the second membrane labeling agents.

127. The method of any one of claims 121-126, further comprising exposing the biological sample to the nuclear labeling agents such that the nuclear labeling agents are immobilized with respect to the nuclei of the cells; exposing the biological sample to the membrane labeling agents such that the membrane labeling agents are immobilized with respect to the plasma membranes of the cells; acquiring, with the detector, the multilayer image set by: detecting, with the detector, the optical signal produced by the nuclear labeling agents; and detecting, with the detector, the optical signal produced by the membrane labeling agents.

128. The method of any one of claims 121-127, wherein the multilayer image set further comprises a layer corresponding to an analyte image associated with an optical signal produced by analyte labeling agents immobilized with respect to or in proximity to an analyte in the biological sample.

129. The method of any one of claims 121-128, further comprising analyzing at least some of the segmented images of the cells to determine the presence, density, and / or level of at least one analyte in the biological sample.

130. The method of any one of claims 121-129, further comprising performing a classification procedure on at least a portion of at least some of the segmented images ofthe cells to determine the presence, density, and / or level of at least one analyte in the biological sample.

131. The method of any one of claims 121-130, wherein the multilayer image set is a multilayer composite image comprising the layers.

132. The method of any one of claims 121-130, wherein the multilayer image set is a plurality of individual layers.

133. The method of any one of claims 121-132, wherein the biological sample comprises biological tissue.

134. A system, comprising: at least one processor; and at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for performing a segmentation of an image of a biological sample, the method comprising: obtaining a multilayer image set that was acquired by a detector, wherein the multilayer image set comprises: a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample; and a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample; generating a combination image based on the nuclei label image and the membrane label image; and segmenting images of nuclei in the biological sample based at least in part on the combination image to produce a nuclear segmentation map.

135. At least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for performing a segmentation of an image of a biological sample, the method comprising: obtaining a multilayer image set that was acquired by a detector, wherein the multilayer image set comprises: a layer comprising a nuclei label image associated with an optical signal produced by nuclear labeling agents immobilized with respect to nuclei of cells in the biological sample; and a layer comprising a membrane label image associated with an optical signal produced by membrane labeling agents immobilized with respect to plasma membranes of the cells in the biological sample; generating a combination image based on the nuclei label image and the membrane label image; and segmenting images of nuclei in the biological sample based at least in part on the combination image to produce a nuclear segmentation map.

136. A method for processing a biological sample comprising biological tissue, comprising: performing an immunohistochemistry procedure on at least a portion of a sample on a slide using a first labeling agent immobilized with respect to the sample; generating a first score associated with an indication for the at least a portion of the sample based on the immunohistochemistry procedure, wherein the performing of the procedure and the generating of the first score are conducted according to a clinical standard for the indication; and generating a fluorescence image of at least a portion of the sample on the same slide, the fluorescence image based, at least in part, on signal detected from a second labeling agent immobilized with respect to the sample; after determining that the first score is indeterminant, processing the fluorescence image to classify components of the at least a portion of the sample; andgenerating, from the classified components, (a) a second score associated with the indication and / or (b) a parameter indicative of the quantity, density, and / or level of an analyte in the sample.

137. The method of claim 136, wherein the clinical standard corresponds to clinical test instructions from (a) an in vitro diagnostic product label from the U.S. Food and Drug Administration and / or (b) guidelines from the College of Anatomical Pathology.

138. The method of any one of claims 136-137, wherein the performing the procedure is conducted by a user via visual inspection of the at least a portion of the sample using a microscope.

139. The method of any one of claims 136-138, wherein the classification of the components comprises classifying cells of the sample as tumor cells or non-tumor cells.

140. The method of any one of claims 136-139, wherein the processing the fluorescence image comprises machine vision-based cell classification.

141. The method of any one of claims 136-140, wherein the first labeling agent is a PD-L1 labeling agent.

142. The method of any one of claims 136-141, further comprising determining a diagnosis, prognosis, and / or treatment plan for a subject from whom the biological sample was obtained based on the second score and / or the parameter indicative of the quantity, density, and / or level of an analyte in the sample.

143. The method of any one of claims 136-142, wherein the signal is detected from a system configured to perform both brightfield and fluorescence imaging of biological samples on slides.

144. The method of any one of claims 136-143, wherein the fluorescence image is further based, at least in part, on signal detected from a third labeling agent immobilized with respect to the sample.

Images