Method and storage medium for determining the abundance of multiple non-endogenous spectral contributors

Abstract

The disclosure features a method comprising: exposing a biological sample to illumination light and measuring light emission from the sample to obtain N sample images (104), wherein each sample image corresponds to a different combination of a wavelength band of the illumination light and one or more wavelength bands of light emission, wherein the one or more wavelength bands of light emission define a wavelength range, and wherein N>1; and exposing the sample to illumination light in a background excitation band and measuring light emission from the sample in a background spectral band to obtain a background image (106) of the sample, wherein the background spectral band corresponds to wavelengths within the wavelength range. An autofluorescence image is obtained from the background image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent No. 62 / 666,697, filed May 3, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to multispectral imaging of biological specimens, particularly by fluorescence microscopy. Background Art

[0004] Fluorescence microscopy is used in pathology to provide information about disease and patient response, to advance human understanding in research settings, and to guide individual patient treatment in clinical settings. Most fluorescence microscopy utilizes dyes designed or selected to provide strong fluorescent signals with known excitation and emission properties. Immunofluorescence labeling techniques allow individual epitopes in a sample to be specifically targeted, allowing proteins, antibodies, nucleic acids, and other tissue and cellular components to be examined even when present in relatively small amounts. Summary of the invention

[0005] In multispectral imaging, multiple dyes or stains are applied to the sample, which is then imaged over a wide range of wavelengths to produce a combined set of spatial and spectral measurements. The dye or stain can be applied to the sample so that each stain specifically binds to the sample component of interest. The contribution of each sample component is separated by analyzing the set of spatial and spectral measurements to determine the quantity and spatial distribution of each component of interest within the sample. Certain fluorescent dyes have relatively well-defined excitation and emission bands. Therefore, for some samples, several dyes (usually 3 or 4) with sufficiently clear emission bands can be selected so that they can be detected via spectral filtering without generating too much crosstalk between emission bands.

[0006] Tissue specimens exhibit a degree of endogenous fluorescent response, which is called autofluorescence. Autofluorescence may be stronger in formalin-fixed paraffin-embedded (FFPE) specimens than in fresh-frozen specimens, which may be a result of the fixation chemicals or the fixation process. Unlike engineered dyes, autofluorescence is a natural phenomenon and does not have well-defined excitation and emission bands. Instead, autofluorescence has a broad emission that spans most of the visible range, making its emission difficult to distinguish from fluorescence emission due to dyes that specifically bind to components of interest in the specimen.

[0007] The present disclosure features methods and systems for measuring autofluorescence in a sample to which one or more dyes or stains have been applied. Typically, for a sample to which multiple dyes or stains have been applied, fluorescence emission is measured in multiple spectral wavelength bands, each of which can correspond largely to fluorescence emission from only a single one of the dyes or stains. Fluorescence emission is also measured in an emission band that does not correspond to significant emission from any applied dye or stain. The emission in this band corresponds largely to autofluorescence from the sample. Based on information derived from the measurement of fluorescence emission in this band, the autofluorescence of the entire sample can be quantified. The quantified autofluorescence can then be used to correct fluorescence emission measurements of some or all dyes or stains, and further used to quantitatively measure some or all dyes or stains at different locations within the sample.

[0008] In one aspect, the present disclosure features a method comprising the steps of: exposing a biological sample to illumination light and measuring light emission from the sample to obtain N sample images, wherein each sample image corresponds to a different combination of a wavelength band of the illumination light and one or more wavelength bands of light emission, wherein the one or more wavelength bands of light emission define a wavelength range, and wherein N>1; and exposing the sample to illumination light in a background excitation band, and measuring light emission from the sample in a background spectral band to obtain a background image of the sample, wherein the background spectral band corresponds to a wavelength within the wavelength range, and wherein for each of one or more non-endogenous spectral contributors in the sample exposed to the illumination light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 10% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after the sample is excited in each wavelength band of the illumination light and the background excitation band.

[0009] Embodiments of the method may include any one or more of the following features.

[0010] The method may include obtaining an autofluorescence image of the sample from a background image. For each of the one or more non-endogenous spectral contributors in the sample exposed to illumination light in a background excitation band, the spectral emission intensity at each wavelength within the background spectral band may be 4% or less (e.g., 2% or less) of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after the sample is excited in each wavelength band of the illumination light and the background excitation band. The background spectral band may include a spectral width Δλ having a full-width at half maximum (FWHM) and a central wavelength λ c The wavelength distribution of the background spectrum band can correspond to λ c -Δλ / 2 to λ c Wavelengths within the range of +Δλ / 2.

[0011] N may be greater than 3 (eg, greater than 5). The sample may include M non-endogenous spectral contributors, and wherein M≤N. M may be greater than 4 (eg, greater than 6). The method may include displaying the autofluorescence image on a display device.

[0012] The method may include determining an amount of autofluorescence emission from the sample at each of a plurality of locations in the sample. The method may include, at each of the plurality of locations in the sample, and for one or more of the N sample images, adjusting a value corresponding to the sample emission intensity to correct for the autofluorescence emission from the sample based on the amount of autofluorescence emission at each of the plurality of locations and at least one pure spectrum of autofluorescence emission from the sample. The at least one pure spectrum of autofluorescence emission may include a plurality of pure spectra of autofluorescence emission, and the plurality of pure spectra of autofluorescence emission may each correspond to a different subset of the plurality of locations. The method may include decomposing at least some of the N sample images to obtain M spectral contributor images based on the amount of autofluorescence emission from the sample at each of the plurality of locations, wherein each of the M spectral contributor images corresponds to light emission from only a different one of the non-endogenous spectral contributors; and determining, at each of the plurality of locations, an amount of the M non-endogenous spectral contributors in the sample.

[0013] The method may include decomposing at least some of the N sample images based on at least one pure spectrum of autofluorescence emissions from the sample. The at least one pure spectrum of autofluorescence emissions may include a plurality of pure spectra of autofluorescence emissions, and the plurality of pure spectra of autofluorescence emissions may each correspond to a different subset of the plurality of locations.

[0014] The sum of the spectral emission intensity of each non-endogenous spectral contributor in the sample at each wavelength within the background spectral band may be 10% or less of the total fluorescence emission intensity in the background spectral band.

[0015] The method may include classifying pixels of one or more of the sample images into different categories based on information derived from the autofluorescence image. The different categories may correspond to different cell types in the sample.

[0016] The M non-endogenous spectral contributors may include one or more fluorescent species that selectively bind to different chemical moieties in the sample. The one or more fluorescent species may include one or more immunofluorescent probes. The M non-endogenous spectral contributors may include one or more counterstains.

[0017] Embodiments of the method may also include any other features described herein, including any combination of features described separately in connection with different embodiments, unless explicitly stated otherwise.

[0018] In another aspect, the disclosure features a computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to: decompose a plurality of sample images of the sample using autofluorescence information of the biological sample to obtain one or more non-endogenous spectral contributor images of the sample; determine amounts of one or more non-endogenous spectral contributors at a plurality of locations in the sample based on the one or more non-endogenous spectral contributor images; and generate an output display including at least one of the non-endogenous spectral contributor images on a display device connected to the processing device, wherein each sample image corresponds to an illumination image used to illuminate the sample. different combinations of wavelength bands of irradiated light and one or more wavelength bands of emitted light from the sample, the one or more wavelength bands of emitted light defining a wavelength range, wherein a background image corresponds to irradiation of the sample with light in a background excitation band and measurement of light emission from the sample in a background spectral band, and wherein for each of the one or more non-endogenous spectral contributors in the sample illuminated with light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 10% or less of a maximum measured spectral emission intensity of the non-endogenous spectral contributor after exciting the sample in each wavelength band of the irradiated light and the background excitation band.

[0019] Embodiments of the storage medium may include any one or more of the following features.

[0020] The storage medium may include instructions that, when executed by the processing device, cause the processing device to obtain autofluorescence information of the sample from a background image of the sample. Each non-endogenous spectral contributor image corresponds only to light emission from a different one of the non-endogenous spectral contributors in the sample.

[0021] For each of one or more non-endogenous spectral contributors in a sample illuminated with light in a background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 4% or less (e.g., 2% or less) of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after exciting the sample in each wavelength band of the illuminating light and the background excitation band.

[0022] The sample may include M non-endogenous spectral contributors, where M ≤ N. M may be greater than 4 (eg, greater than 6).

[0023] The autofluorescence information may include an amount of autofluorescence emission from the sample at each of a plurality of locations in the sample. The storage medium may include instructions that, when executed by a processing device, cause the processing device to decompose a plurality of sample images using the autofluorescence information and at least one pure spectrum of autofluorescence emission from the sample. The storage medium may include instructions that, when executed by a processing device, cause the processing device to determine at least one pure spectrum of autofluorescence from a background image.

[0024] The at least one pure spectrum of autofluorescence emissions from the sample includes two or more different pure spectra of autofluorescence emissions from the sample, and each of the different pure spectra corresponds to a different subset of the plurality of spatial locations.

[0025] The storage medium may include instructions that, when executed by a processing device, cause the processing device to adjust a value corresponding to a sample emission intensity in each of the sample images to correct for the autofluorescence emission from the sample based on an amount of autofluorescence emission at each of a plurality of locations and at least one pure spectrum of the autofluorescence emission from the sample.

[0026] The sum of the spectral emission intensity of each non-endogenous spectral contributor in the sample at each wavelength within the background spectral band is 10% or less of the total fluorescence emission intensity in the background spectral band.

[0027] The storage medium may include instructions that, when executed by the processing device, cause the processing device to classify pixels of one or more of the sample images into different categories based on the autofluorescence information. The different categories may correspond to different cell types in the sample.

[0028] The M non-endogenous spectral contributors may include one or more fluorescent species that selectively bind to different chemical moieties in the sample. The M non-endogenous spectral contributors may include one or more counterstains.

[0029] Unless explicitly stated otherwise, embodiments of the storage medium may also include any other features described herein, including any combination of features described in connection with different embodiments.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those of ordinary skill in the art to which the present disclosure belongs. Although methods and materials similar or equivalent to the methods and materials described herein can be used for the practice or testing of the subject matter of this article, suitable methods and materials are described as follows. All publications, patent applications, patents and other references mentioned herein are incorporated herein by reference in their entirety. In the event of a conflict, this specification (including definitions) shall prevail. In addition, materials, methods and examples are illustrative only, not restrictive.

[0031] The details of one or more embodiments are set forth in the drawings and the description. Other features and advantages will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart showing example steps for measuring autofluorescence of a sample.

[0033] Figure 2 is a schematic diagram of an example system for measuring sample autofluorescence.

[0034] Figure 3 is a schematic diagram illustrating a set of example spectral emission bands that can be used to measure fluorescence emission due to non-endogenous spectral contributors in a sample and sample autofluorescence.

[0035] Figure 4 is a schematic diagram showing fluorescence emission from non-endogenous spectral contributors in a sample.

[0036] Figure 5A-5D is a graph showing the spectral characteristics of four epi-filter cubes that can be used to measure sample fluorescence.

[0037] Fig. 6A The graph shows the dye excitation response of the counterstain and six dyes used as immunofluorescence labels.

[0038] Figure 6B It is shown Fig. 6A A diagram of the fluorescence emission response of the counterstain and dye.

[0039] Figure 6C is a graph showing the autofluorescence emission of a formalin-fixed paraffin-embedded lung cancer sample when excited at 387 nanometers and when excited at 425 nanometers (as 604).

[0040] Figures 7A-7H These are sample images and background images obtained after exciting the sample in different combinations of spectral excitation wavelength bands and spectral emission wavelength bands and measuring the sample fluorescence, each of which shows a portion of the sample.

[0041] Figures 8A-8H is obtained by unmixing the sample image and the background image. Figures 7A-7H A pure spectral contributor image and an autofluorescence image of a sample, each of which shows a portion of the sample.

[0042] Figures 9A-9H is obtained by unmixing the sample image and the background image, Figures 7A-7HPure spectral contributor images and autofluorescence images of the sample, each of these images showing the entire sample.

[0043] Fig. 10A and 10B are image collections for samples prepared with various non-endogenous dyes and unstained samples, respectively, each collection includes an autofluorescence image, a binary classification mask, a FoxP3 abundance image, a threshold segmentation mask, and an image showing only FoxP3-positive cells.

[0044] The same reference symbols in different drawings identify the same elements. DETAILED DESCRIPTION

[0045] Various techniques have been developed to accurately measure immunofluorescent targets in the presence of autofluorescence. One approach involves multispectral imaging, in which sample images are acquired at different wavelengths to distinguish between the contribution of dyes associated with immunofluorescent labeling and the fluorescence emission due to sample autofluorescence. Although these signals overlap spectrally, the spectral shape of each signal is different. Spectral unmixing or other spectral decomposition steps are performed to separate the contribution to the measured fluorescence signal into separate spectral components corresponding to individual dyes, so that measurement of multiple dyes can be achieved despite the presence of spectrally broad autofluorescence emissions. Spectral unmixing techniques can distinguish the contributions of multiple dyes in a sample, providing key multiplex sample labeling advantages even if the fluorescence emissions from the dyes overlap spectrally.

[0046] Multiplexed immunofluorescence labeling refers to the targeting of multiple primary antibodies in one sample, and it can be achieved in a variety of ways. Conceptually, the simplest is direct labeling, in which the primary antibody molecule is conjugated to a dye, and different dyes are conjugated to different primary antibodies. This labeling process usually results in only one dye molecule per target, so the signal level may be low unless the target itself is abundant. Indirect labeling provides a method for coupling multiple dye molecules per target, allowing weaker targets to be studied. To avoid cross-labeling, different secondary antibody isotypes or species are usually used for each target, making the labeling process more complicated.

[0047] Another approach to introduce multiple specific dyes is to conjugate the primary antibody to a nucleotide sequence that is recognized by a complementary sequence conjugated to a dye polymer. This technique is not limited by the number of secondary antibody species available.

[0048] In order to increase the amount of each fluorescent dye that is selectively bound to the sample, the dyes can be selectively applied sequentially to the sample using tyramide signal amplification (TSA). In TSA, a single primary antibody is used with a secondary antibody coupled to horseradish peroxidase to catalyze the covalent binding of the dye molecules to the surrounding tissue through cinnamamide or tyramide molecules. Because the dye molecules are covalently bound to the tissue, the primary and secondary antibodies can then be removed, and the process can be repeated with other primary antibodies and dye molecules, allowing higher concentrations of dye molecules to be introduced into the sample in a multiple, specific manner.

[0049] In some TSA-based staining protocols, further amplification can be achieved by selective covalent binding of small molecule ligands to sample tissue via cinnamamide or tyramide molecules, followed by incubation of the sample with ligand-binding proteins conjugated to multiple dye molecules. For example, biotin can be introduced as a ligand, and streptavidin can be used as a ligand-binding protein, but other ligand-protein combinations can also be used. Because the dye molecules are not covalently bound to the tissue, incubation of the ligand-bound sample with the ligand-binding protein usually occurs at the end of the multiplex staining protocol.

[0050] Methods and systems for performing multispectral imaging on whole slide samples have been developed. Specifically, such methods and systems can generate mosaic images of multispectral images with diffraction-limited, multi-centimeter fields of view with two dimensions across samples. Such systems use multi-band filters and optional light sources such as LEDs or lasers, multiple epitaxial filter sets, or a combination of the two. When multiple epitaxial filters are used, the registration of multispectral images is important because it allows images corresponding to different parts of the sample to be combined to form seamless stitching. U.S. Patent Application Publication US2014 / 0193061 describes a method for using multiple filters, wherein each filter images samples in a common band such as a counterstain band (e.g., a DAPI counterstain band) and other bands, and a method for combining image sets into a mosaic image of a sample by using images measured at the common band to register images corresponding to all sample imaging bands. In this way, samples can be imaged in 10 or more different emission bands and can be imaged and co-registered. The entire contents of U.S. Patent Application Publication No. US2014 / 0193061 are incorporated herein by reference.

[0051] The aforementioned methods for performing multispectral sample imaging can determine the contribution from sample autofluorescence during spectral unmixing. For example, using an estimate of the pure spectrum of sample autofluorescence, the spectral unmixing process can produce an autofluorescence abundance image that represents the spatial distribution of autofluorescence within the sample. The autofluorescence image can be viewed as a type of "residual" image that accounts for sample fluorescence that is not attributable to the dye applied to the sample.

[0052] However, for some samples, the abundance measurement of autofluorescence derived purely from spectral unmixing may not be completely accurate. In particular, such measurements are derived using estimates of pure autofluorescence spectra (which may not correspond exactly to the autofluorescence in the particular sample of interest), especially when the sample also includes multiple fluorescent dyes and is preserved according to a particular fixation protocol. In addition, the autofluorescence in a sample may vary among different sample regions (and especially among different structures within the sample, such as different types of cells and different types of cellular features (e.g., stroma, cytoplasm, cell membrane, cell nucleus)), and this variation may not be adequately represented by a single assessment of the pure autofluorescence spectrum.

[0053] The methods and systems described herein allow autofluorescence in a sample to be measured directly, rather than being simply extracted during spectral unmixing. By directly measuring autofluorescence, sample-specific autofluorescence responses can be appropriately measured and interpreted during analysis of a multispectral image representing one or more dyes applied to the sample to target specific antibodies, proteins, nucleic acids, and other sample structures and cellular components. Specifically, before or during image analysis, fluorescence signals measured in spectral emission bands corresponding to individual applied dyes can be corrected to account for sample autofluorescence, thereby providing a more accurate abundance measurement of each of the individual applied dyes. To allow for these improvements, both the abundance and spatial distribution of sample autofluorescence at multiple locations within the sample are determined prior to spectral decomposition of the multispectral image corresponding to the individual applied dyes.

[0054] Sample imaging systems typically include one or more displays on which sample images representing the spectral contributions corresponding to individual applied dyes are displayed to a pathologist or other physician or technician for diagnosis of disease and other conditions within the sample. Dyes are typically selected as reporters for specific antibodies, proteins (e.g., transmembrane proteins), nucleic acid fragments, and / or cellular structures of interest that can indicate the presence or absence of certain conditions, and can provide complex physiological information about cell migration, protein expression, regulatory networks, mutations, and other events in the sample. The methods and systems described herein allow for the display of more accurate spectral contributor images, thereby significantly improving traditional sample imaging and display systems, which in turn improves the diagnostic information provided by such systems.

[0055] The (multiple) autofluorescence images measured according to the methods and systems described herein may also be used for other purposes. For example, a trained pixel-based classifier may be used to analyze the measured autofluorescence images to identify different types of cells or cell structures in a sample and exclude areas of the sample corresponding to certain types of cells or cell structures that are not of interest. For example, a trained classifier may be used to identify red blood cells in a sample that may not be of interest, but they may emit fluorescence in a spectral region corresponding to one or more dyes that have been applied to the sample to target a specific sample target of interest. The trained classifier may also be used to identify other types of cells, including (but not limited to) different types of white blood cells, such as lymphocytes, neutrophils, eosinophils, monocytes, and basophils; reticulocytes; and tumor cells.

[0056] Regions of no interest in the sample can be excluded from subsequent analysis (e.g., spectral unmixing of the sample image), thereby excluding regions from the sample that would otherwise represent false positive detections of one or more applied dyes, and thus excluding (multiple) cellular components targeted by those applied dyes. By excluding certain sample regions from analysis via classification operations on measured autofluorescence images, the methods and systems described herein allow for the display of more accurate spectral contributor images, thereby further significantly improving conventional sample imaging and display systems, and thereby improving the quality and utility of diagnostic information presented by such systems.

[0057] The methods and systems described herein enable relatively rapid, whole-slide multispectral imaging of multiply stained samples for accurate and quantitative analysis in applications including immuno-oncology analysis, cell signaling studies, and multiplex immunofluorescence pathology experiments. The methods and systems can be applied to a wide variety of samples that have been labeled using a variety of different methods, including biological samples, such as (but not limited to) formalin-fixed paraffin-embedded (FFPE) samples.

[0058] Figure 1 1 is a flow chart 100 showing a series of example steps for imaging a sample. In a first step 102, the sample is prepared for imaging by applying one or more non-endogenous dyes to the sample. As used herein, a "dye" is a non-endogenous substance that binds to a structural / chemical moiety within the sample and fluoresces when exposed to irradiating light. The term "dye" may be used interchangeably with the term "stain"; for purposes of this disclosure, "dye" and "stain" correspond to the same substance.

[0059] Typically, in step 102, one or more dyes are applied to the sample. For example, the number of dyes applied can be two or more (e.g., three or more, four or more, five or more, six or more, seven or more, eight or more, nine or more, ten or more, 12 or more, 15 or more, 20 or more, or even more). The dye applied can be bound to a specific type of antibody, protein, nucleic acid, vesicle, lipid or other substance in the sample. The dye applied can also be bound to a specific cell type (e.g., erythrocytes, lymphocytes, T cells, B cells) in the sample. In addition, the dye applied can be bound to a specific cell structure or compartment (e.g., interstitial, cell membrane, cytoplasm, nucleus, mitochondria, Golgi apparatus) in the sample.

[0060] In some embodiments, the one or more dyes applied to the sample may include one or more counterstains that bind to multiple structures, regions, or components in the sample cells. Examples of suitable counterstains include, but are not limited to, DAPI, DRAQ5, Hoechst 33258, Hoechst 33342, and Hoechst 34580.

[0061] In some embodiments, one or more dyes applied to the sample have spectrally separated emission bands. By selecting dyes with such properties, the fluorescence emission from each dye can be easily isolated from the emission caused by the other dyes and measured using appropriate emission filters, thereby minimizing interference caused by spectral emission cross-talk among the dyes. For example, in some embodiments, the dyes are selected so that for a given pair of dyes D1 and D2, each dye has an emission spectrum and a wavelength λ within the emission spectrum of the dye, respectively. D1 and λ D2 The maximum emission intensity is D1 max and D2 max , dye D1 in λ D2 The emission intensity at D1 max10% smaller than λ (e.g., smaller than 8%, smaller than 6%, smaller than 4%, smaller than 2%, smaller than 1%, smaller than 0.5%, smaller than 0.25%), and the dye D2 is smaller than λ D1 The emission intensity at D2 max 10% smaller than (e.g., 8% smaller than, 6% smaller than, 4% smaller than, 2% smaller than, 1% smaller than, 0.5% smaller than, 0.25% smaller than). Dyes can be selected so that the above pairwise relationship holds among all members of a group of three or more dyes, four or more dyes, five or more dyes, six or more dyes, eight or more dyes, ten or more dyes, 12 or more dyes, 15 or more dyes, 20 or more dyes, or even more dyes.

[0062] As described above, a variety of different sample preparation protocols may be used in step 102. One example of a suitable preparation protocol is described below. However, it should be understood that many different protocols for applying many different dyes to the sample may be used.

[0063] Can be used Samples were prepared for multispectral imaging using multiplex immunohistochemical (IHC) reagents (available from Akoya Biosciences, Menlo Park, CA), which can be used to label multiple molecular targets within the sample. FFPE tissue samples were prepared by baking at 60°C for 1 hour, followed by three 10-minute washes in xylene to remove paraffin. Samples were then rehydrated into deionized water through an ethanol gradient, fixed with 10% neutral buffered formalin for 20 minutes, and subsequently washed in deionized water.

[0064] The multiplex IHC staining process starts with a round of antigen retrieval, which can be performed via microwave treatment using buffer AR6 (antigen retrieval pH 6) or AR9 (antigen retrieval pH 9). After antigen retrieval, each sample target is labeled sequentially in a cycle that includes five steps: blocking, primary antibody incubation, secondary antibody incubation, tyramide deposition, and antibody stripping.

[0065] By keeping the samples at room temperature Blocking is achieved by incubating in antibody diluent for 10 minutes. The primary antibody can be incubated at different times and temperatures depending on the target. For example, for lung cancer samples, Table 1 lists examples of primary antibodies and dilution ratios for various sample targets.

[0066]

[0067]

[0068] Table 1

[0069] Each antibody was allowed to incubate for approximately 30 minutes at room temperature. The polymer HRPMs+Rb solution (Akoya Biosciences, Menlo Park, CA) was incubated at room temperature for 10 min, followed by a solution consisting of 0.1 M TRIS-HCl, pH 7.5, 0.15 M NaCl, and 0.05% 20 (Sigma-Aldrich, St. Louis MO). Tyramide deposition was performed at room temperature for 10 minutes, followed by 3 rinses with TBST. Different tyramide reagents were paired with each primary antibody depending on the target of choice and the dye selected to interrogate the selected target. The staining cycle was completed by antibody stripping via microwave treatment in AR6 or AR9. The aforementioned 5-step cycle can be repeated for each target in the sample. In direct analogy to the above process, if a ligand-based tyramide reagent is used during sample preparation, incubation with the dye-conjugated ligand-binding protein occurs after the final antibody stripping step.

[0070] After applying each dye corresponding to a specific target within the sample, one or more counterstains may optionally be applied to the sample by incubating the sample with the counterstain. For example, to apply a DAPI counterstain to a sample, the sample may be incubated in a DAPI solution (4 drops / ml) for 5 minutes at room temperature, followed by one wash in deionized water and one wash in TBST.

[0071] Stained specimens (to which one or more dyes associated with a particular specimen target have been applied, and optionally one or more counterstains) can be mounted by using a mounting medium (e.g. Diamond, available from ThermoFisherScientific, Waltham MA) and coverslips were further prepared for imaging.

[0072] In some embodiments, sample preparation can also include the preparation of an unstained sample that can be used to perform autofluorescence measurements. Typically, unstained samples can be prepared by a process similar to that for preparing stained samples, and simply omitting the incubation steps involving fluorescent dyes. In the above example, the incubation steps with the various dyes shown in the fifth column of Table 1 were omitted, but the steps were otherwise identical. This preparation protocol produces an unstained sample that has been treated with the same temperature changes, antibody additions, and pH changes that the stained slides have experienced, and to the extent that these processing steps affect the endogenous autofluorescence emission from the sample, the effects should be similar for both stained and unstained samples.

[0073] In general, a variety of different sample preparation techniques are available that use different combinations of dyes, counterstains, and other sample labeling moieties that are specific to certain sample targets. Development of sample preparation protocols for specific multiplex immunofluorescence for a particular purpose typically includes the selection of primary antibodies, refinement / optimization of antibody and dye dilutions, and other immunohistochemistry process development steps to maintain sample integrity and enhance selective dye binding to the sample. All of these protocols are generally applicable to the methods and systems described subsequently, unless explicitly stated otherwise.

[0074] Back to Figure 1 After sample preparation is completed in step 102, the prepared sample is imaged to obtain one or more sample images in step 104. Each sample image obtained in step 104 corresponds to a selected combination of illumination light within an excitation band and fluorescence emission from the sample measured in one or more emission bands in response to the illumination light.

[0075] A variety of different sample imaging systems can be used to obtain sample images. Figure 2 An example of a sample imaging system 200 implemented as a fluorescence microscope is shown in FIG. The system 200 includes a light source 202, a dichroic mirror 204, an optical filter 206 (implemented as an excitation filter 206a and an emission filter 206b), a lens 208, a stage 210, and a detector 212. Each of these components is coupled to a controller 214 including a processing device 216.

[0076] Controller 214 sends and receives control and data signals from each system component and can control each component accordingly (more specifically, processing device 216 controls each system component). In some embodiments, all steps and / or control functions described in conjunction with system 200 are performed by controller 214. Alternatively, in some embodiments, certain steps may be performed by an operator of system 200.

[0077] The light source 202 is an adjustable light source that can generate light with a variable illumination wavelength distribution. In some embodiments, for example, the light source 202 includes a plurality of LEDs of different wavelengths that can be selectively activated by the controller 214 to generate illumination light with desired spectral characteristics. In some embodiments, the light source 202 includes one or more laser diodes, lasers, incandescent light sources, and / or fluorescent light sources, each of which can be controlled by the controller 214.

[0078] The illumination light generated by the light source 202 is reflected from the dichroic mirror 204 and incident on the optical filter 206. The filter 206 generally includes a plurality of filters, each of which can be selectively inserted into the path of the illumination light. Each filter has an associated excitation spectral band and one or more emission spectral bands. The controller 214 adjusts the filter 206 based on the illumination light generated by the light source 202 to allow light of a suitable wavelength distribution to be incident on the sample.

[0079] The filter 206 can be implemented in a variety of ways. For example, in some embodiments, the filter 206 includes a plurality of different epitaxial filter cubes, each of which can be selectively rotated into the path of the illumination light by the controller 214. In certain embodiments, the filter 206 includes an adjustable filter element (e.g., a liquid crystal-based element) for which the controller 214 can selectively select the excitation and emission spectral bands. Other embodiments of the filter 206 can also be used in the system 200.

[0080] The filtered illumination light emitted from the filter 206 is then focused by the lens 208 onto the surface of the sample 250, which is supported by a slide 252 mounted on a stage 210. The stage 210 allows the sample 250 to be moved in each of the x and y directions, and can be controlled by the controller 214. The movement of the stage 210 in the x and y directions allows the filtered illumination light to be directed to different areas of the sample. By moving the sample relative to the focal area of ​​the illumination light, the illumination light can be directed to multiple different areas of the sample, allowing whole slide imaging of the sample 250.

[0081] The filtered illumination light produces fluorescent emission from the sample 250, and the fluorescent light emitted in the direction of the lens 208 is collimated by the lens 208 and passes through the filter 206. As described above, the filter 206, which can be adjusted by the controller 214, defines one or more emission spectral bands. The fluorescent emission from the sample 250 is filtered by the filter 206 so that only light within the one or more emission spectral bands is transmitted by the filter 206. The filtered fluorescent emission light is transmitted by the dichroic mirror 204 and detected by the detector 212.

[0082] Detector 212 can be implemented in various ways. In some embodiments, for example, detector 212 includes a CCD-based detection element. In certain embodiments, detector 212 includes a CMOS-based detection element. Detector 212 may also optionally include spectrally selective optical elements, such as one or more prisms, gratings, diffraction elements and / or filters, to allow wavelength-selective detection of filtered fluorescent emission light. In response to incident filtered fluorescent emission light, detector 212 generates one or more electronic signals representing a quantitative measurement of the filtered fluorescent emission light. The signal is sent to a controller 214 that processes the signal to extract measurement information corresponding to sample 250.

[0083] During operation, system 200 typically captures N different sample images, each of which corresponds to a different combination of an excitation wavelength band of filter 206 (which controls the spectral distribution of illumination light incident on sample 250) and one or more emission wavelength bands of filter 206 (which controls the spectral distribution of fluorescent emission from sample 250 detected by detector 212). N is 1 or greater, and can generally be any number (e.g., 2 or greater, 3 or greater, 4 or greater, 5 or greater, 6 or greater, 7 or greater, 8 or greater, 9 or greater, 10 or greater, 12 or greater, 15 or greater, 20 or greater, or even greater).

[0084] In some embodiments, N can be related to the number of non-endogenous spectral contributors in the sample. As used herein, a "non-endogenous spectral contributor" is a component of the sample 250 that has been added to the sample and emits fluorescence when excited by the illumination light from the light source 202. As described above, the non-endogenous spectral contributors include one or more dyes applied to the sample 250. For example, N can be equal to or greater than the number M of non-endogenous spectral contributors in the sample.

[0085] Back to Figure 1 In step 106, the system 200 also typically captures one or more background images of the sample. The background image(s) may be captured after, before, or interleaved with the capture of the spectral image of the sample in step 104. The background image(s) correspond to a combination of the excitation wavelength band of the filter 206 and the background spectral band of the filter 206. An important aspect of the background spectral band is that it is selected so that non-endogenous spectral contributors in the sample 250 do not generate significant fluorescence emission light within the background spectral band.

[0086] An example of a set of fluorescence emission spectrum wavelength bands and background spectrum wavelength bands is schematically shown in Figure 3 In Figure 3In FIG. 1 , seven different fluorescence emission spectrum wavelength bands B1-B7 and a background spectrum wavelength band X are shown. Each band corresponds to a respective wavelength λ 1 -λ 7 and λ x , such wavelengths respectively represent the center wavelengths of the spectral bands determined from the full width at half maximum (FWHM) spectral shape of each band. It should be noted that the following discussion refers to seven emission spectral wavelength bands B1-B7 for illustrative purposes only. In general, as described above, any number N of spectral wavelength bands may be used when capturing sample images.

[0087] Wavelength 1 -λ 7 The wavelength range of the fluorescence emission spectrum wavelength band B1-B7 is defined. In some embodiments, the background spectrum band X is selected so that the wavelength λ x falls within the wavelength range of the fluorescence emission spectrum wavelength band B1-B7. Alternatively, in some embodiments, λ x In some embodiments, λ x Being within the wavelength range of the fluorescence emission spectral wavelength bands B1-B7 is advantageous because selecting the background spectral wavelength band in this manner allows a larger spectral range to be dedicated to the fluorescence emission spectral wavelength band, making isolation of various fluorescence emission signals from the sample 250 easier.

[0088] It should be noted that although Figure 3 Background spectral band X is spectrally separated from spectral wavelength bands B1-B7, but this is not always the case. In some embodiments, background spectral band X overlaps with a portion of one or more of spectral wavelength bands B1-B7. Generally, background spectral band X can be selected so that it does not spectrally overlap with any spectral wavelength band corresponding to non-endogenous spectral contributors, or alternatively overlaps with one or more (e.g., two or more, three or more, four or more, or even more) spectral wavelength bands corresponding to non-endogenous spectral contributors. As described below, even when spectral overlap occurs between background spectral band X and one or more of spectral wavelength bands B1-B7, endogenous sample autofluorescence can still be measured by exciting the sample in a selected wavelength band and then measuring the sample autofluorescence in background spectral band X.

[0089] Typically, a background image of a sample is obtained by detecting fluorescence emission from the sample in a background spectral band X. In certain embodiments, some or all of the background images of the sample are obtained by detecting fluorescence emission from the sample in more than one background spectral band. Each background spectral band used to detect fluorescence emission from the sample generally shares a common property: each non-endogenous spectral contributor in the sample does not generate significant fluorescence emission in the background spectral band when the sample is excited in a certain excitation wavelength band. Therefore, by exciting the sample in a selected excitation wavelength band and measuring sample fluorescence in the background spectral band, a background image of the sample can be obtained, which background image of the sample includes spectral contributions substantially only from sample autofluorescence, without significant contributions from non-endogenous spectral contributors (such as applied dyes and counterstains).

[0090] Figure 4 is a schematic diagram showing an example of the characteristics of the relationship between the fluorescence emission from non-endogenous spectral contributors and the background spectral band. To image the sample, the sample is exposed to a band W exc,i Under the illumination light, and in the emission band W emi,j The fluorescence emission from the excited sample is measured in . This process is repeated to obtain N sample images, each corresponding to the excitation band W exc,i and emission band W emi,j In order to obtain the background image of the sample, the sample is exposed to the background excitation band W back,exc The illumination light in the background spectral band W back,emi Measure sample fluorescence.

[0091] For each non-endogenous spectral contributor in the sample, the spectral contributor will be represented by the emission band W. emi,j The maximum fluorescence intensity I is shown at a specific wavelength within the defined wavelength range. max . From the use of excitation band W exc,i and W back,exc The maximum fluorescence intensity I is obtained during the excitation of light in one of the max In the methods and systems described herein, the background excitation band W is selected back,exc and background spectral band W back,emi , so that for the background spectral band W back,emi At any wavelength in , the intensity of the fluorescence emission from each non-endogenous spectral contributor is the maximum fluorescence intensity I of the non-endogenous spectral contributor. max 10% or less (e.g., 8% or less, 6% or less, 5% or less, 4% or less, 3% or less, 2% or less, 1% or less, 0.5% or less, 0.25% or less, 0.1% or less, 0.05% or less, 0.01% or less, or even less).

[0092] exist Figure 4 In the figure, the background excitation band W is shown. back,exc The fluorescence emission intensity spectrum E(λ) of the non-endogenous spectral contributors excited by the illumination light is shown in Figure 1. The background spectral band W is also shown. back,emi (corresponding to spectral band X). Background spectral band W back,emi Corresponding to wavelength λ x (as described above), and includes the wavelength λ X1 -λ X2 To avoid fluorescence emission crosstalk in the background spectral band, for the range λ X1 -λ X2 At each wavelength in the spectrum, the fluorescence emission intensity E(λ) of the non-endogenous spectral contributor is exc,i and W back,exc The maximum fluorescence emission intensity of non-endogenous spectral contributors excited in max In some embodiments, the background spectral band X is selected from the group consisting of 10% or less of the autofluorescence emission of the sample (e.g., 8% or less, 6% or less, 5% or less, 4% or less, 3% or less, 2% or less, 1% or less, 0.5% or less, 0.25% or less, 0.1% or less, 0.05% or less, 0.01% or less, or even less). By selecting the background spectral band X in this manner, the background image of the sample captured by the system corresponds almost entirely (or entirely) to the autofluorescence emission from the sample.

[0093] Typically, background excitation and background spectral bands can be selected so that the above relationship applies to some or all non-endogenous spectral contributors in the sample. In addition, when capturing a background image of a sample by detecting fluorescence emission in multiple background spectral bands, multiple background spectral bands can be selected so that the above relationship applies to some or all background spectral bands (and some or all non-endogenous spectral contributors).

[0094] Figure 4 The background spectral band in has a spectral shape that corresponds approximately to a square or "top hat" distribution. Thus, the edge of the distribution defines the wavelength range λ associated with the background spectral band X1 -λ X2 When the background spectral band has a more complex shape, the wavelength range associated with the background spectral band is determined based on the FWHM spectral range of the background spectral band. Specifically, for a FWHM spectral range of Δλ and a wavelength of λ c The wavelength range associated with the background spectral band is from λ X1 =λ c -Δλ / 2 extends to λ X2 =λ c+Δλ / 2. The background spectral band can be selected so that the above relationship between the fluorescence emission from non-endogenous spectral contributors in the sample and the background spectral band remains within the wavelength range λ X1 To X2 Inside.

[0095] The background excitation band and the background spectral band may also be selected based on other criteria to ensure that fluorescence emission crosstalk into the background spectral band remains relatively low. In some embodiments, for example, the background excitation band and the background spectral band may be selected so that for some or all non-endogenous spectral contributors, the integrated intensity of the non-endogenous spectral contributors in the background spectral band after excitation in the background excitation band (i.e., at wavelength λ) is X1 With λ X2 The sum of the emission intensities between the two) is greater than when the non-endogenous spectral contributors are in the excitation band W exc,i When excited, it crosses the emission band W emi,j Less than 5% (e.g., less than 4%, less than 3%, less than 2%, less than 1%, less than 0.5%, less than 0.3%, less than 0.2%, less than 0.05%, less than 0.01%) of the maximum integrated fluorescence emission intensity of the non-endogenous spectral contributors for all wavelengths in the defined wavelength range.

[0096] In some embodiments, the background excitation band and the background spectral band can be selected so that when the sample is illuminated with light in the background excitation band, the integrated intensity of fluorescence emission from all non-endogenous spectral contributors in the sample is 10% or less (e.g., 8% or less, 6% or less, 4% or less, 2% or less, 1% or less, 0.5% or less, 0.25% or less, 0.1% or less, 0.05% or less, or even less) of the integrated intensity of all measured fluorescence emission in the background spectral band (i.e., due to non-endogenous spectral contributors and sample autofluorescence).

[0097] It should also be appreciated that in some embodiments, the background excitation band and the background spectral band may be selected such that more than one of the aforementioned conditions is met.

[0098] Back to Figure 1 In the next step 108, an autofluorescence image of the sample is obtained. In some embodiments, due to the selection of the background spectral band X, the background image of the sample substantially only effectively corresponds to the autofluorescence from the sample. Therefore, the sample autofluorescence image directly corresponds to the sample background image without any further processing.

[0099] In certain embodiments, a sample autofluorescence image can be extracted from a background image by decomposing the background image using techniques such as spectral unmixing. In spectral unmixing, a pure estimate of the sample autofluorescence spectrum and the fluorescence emission spectrum of each non-endogenous spectral contributor in the sample can be used to unmix the background image (which can be a multispectral image that includes emission intensity measurements at each wavelength within the background spectral band X and at each sample position of interest to form a multispectral image cube) to obtain an autofluorescence image of the sample. Because the background image of the sample contains only a small contribution from each non-endogenous spectral contributor in the sample, the unmixing process is typically very effective in isolating sample autofluorescence and generating an autofluorescence image. Suitable methods of spectral unmixing are described, for example, in U.S. Patent No. 7,321,791 and PCT Patent Application Publication WO 2005 / 040769, the entire contents of which are incorporated herein by reference.

[0100] Back to Figure 1 , the sample image obtained in step 104 can be decomposed in step 110 to obtain a set of pure spectral contributor images, wherein each spectral contributor image substantially only includes the contribution from one of the non-endogenous spectral contributors in the sample. It should be noted that in Figure 1 , step 108 may occur sequentially (i.e., the autofluorescence image may be acquired first, followed by the pure spectral contributor image), or alternatively, steps 108 and 110 may occur simultaneously (i.e., both the autofluorescence image and the pure spectral contributor image may be acquired simultaneously, such as via a single spectral imaging process).

[0101] By performing the decomposition step 110, the distribution of each applied dye within the sample can be determined, and therefore the distribution of molecular targets associated with each dye can be determined. Spectral unmixing can be used to perform the decomposition in step 110. In some embodiments, when the sample autofluorescence distribution is already known from step 108, the spectral unmixing process used in step 110 can take into account the autofluorescence abundance information. A two-step spectral unmixing process using previously determined autofluorescence abundance information is discussed below. Alternatively, steps 108 and 110 can be performed simultaneously in a one-step spectral unmixing process.

[0102] Next, in step 112, the amount of one or more non-endogenous spectral contributors in the sample is determined. In practice, this step corresponds to quantitatively determining the distribution of one or more non-endogenous spectral contributors, and thus determining the quantitative distribution of one or more targets in the sample. The amount of non-endogenous spectral contributors can be determined directly as entries in the abundance matrix using spectral unmixing. Therefore, in some embodiments, steps 110 and 112 are performed together because spectral unmixing produces both the spatial distribution and the quantitative amount of non-endogenous spectral contributors at some or all locations within the sample.

[0103] In step 114, the autofluorescence image obtained in step 108 can optionally be displayed on a display interface or device 218 connected to the controller 214. In addition, in step 114, any pure spectral contributor images obtained in step 110 can optionally be displayed on the interface 218. As described above, a system that displays all or some of these images represents a significant improvement over conventional fluorescence microscopes and other multi-spectral sample imaging devices that are not able to determine sample autofluorescence and obtain pure spectral contributor images in the same manner. Specifically, a system that determines sample autofluorescence and obtains pure spectral contributor images as discussed herein is able to obtain this information faster than conventional sample imaging devices and has reduced data storage requirements relative to conventional devices. For example, the number of spectral bands and / or wavelengths in which sample fluorescence is measured to obtain an accurate quantitative determination of endogenous autofluorescence is significantly reduced relative to conventional sample imaging devices. As a result, the amount of measured spectral information from the sample is reduced, and computationally intensive operations (such as spectral unmixing) are performed faster.

[0104] The process in flowchart 100 then terminates at step 116 .

[0105] It should be noted that the steps shown in flowchart 100 are not the only steps that may be performed in a sample analysis workflow. Figure 1 Other steps are performed before, simultaneously with, or after any step shown. For example, between steps 110 and 116, Figure 1 The process shown may include an additional analysis step in which some or all of the pure spectral contributor images are quantitatively analyzed to determine one or more characteristics of the sample and / or its molecular targets. In general, any analysis process may be performed, including, for example, quantitative determination of statistical properties associated with the distribution of various molecular targets.

[0106] generally, Figure 1 The process described in allows for quantitative determination of sample autofluorescence. The autofluorescence can be determined separately from the determination of the quantitative distribution of each non-endogenous spectral contributor in the sample. Furthermore, in some embodiments, the autofluorescence can be determined prior to determining the quantitative distribution of each non-endogenous spectral contributor in the sample. Alternatively, in some embodiments, the autofluorescence and the quantitative distribution of each non-endogenous spectral contributor are determined simultaneously.

[0107] As a result, in some embodiments, sample autofluorescence can be used to correct the raw spectral fluorescence emission measurements for each non-endogenous spectral contributor. In certain embodiments, such correction occurs during decomposition (e.g., spectral unmixing), as described above, where the sample autofluorescence distribution is used in a two-stage unmixing process.

[0108] Alternatively, in some embodiments, sample autofluorescence can be used to directly adjust measured intensity values ​​in sample images prior to unmixing of the sample images. For example, in some or all of the original multispectral sample images (and optionally, at some or all spatial locations within the image), the measured spectral emission intensity values ​​can be adjusted based on the amount of autofluorescence emission at the corresponding location in the sample and at least one pure spectrum of the sample autofluorescence emission. For example, the adjustment can be performed by subtracting the autofluorescence contribution from the measured spectral emission intensity values. In samples in which there is more than one sample autofluorescence emission spectrum (e.g., due to differences in autofluorescence emission in different sample regions, as will be discussed below), the different pure spectra of sample autofluorescence emission can be used to perform correction in the corresponding sample regions.

[0109] In general, the methods and systems described herein can use a variety of different filters to select appropriate spectral excitation bands and emission bands for obtaining autofluorescence images of pure contributors and samples. In addition, in some embodiments, a sample background image can be obtained based on fluorescence emission measurements in more than one background spectral band. For example, a sample background image can be obtained from fluorescence emission measurements in two or more (e.g., three or more, four or more, five or more) background spectral bands. Typically, when a sample background image is obtained from fluorescence emission measurements in more than one background spectral band, some or all of the background spectral bands meet the above criteria.

[0110] For spectral unmixing operations, pure spectral estimates for dye and sample autofluorescence can be obtained in various ways. For example, dye spectra can be modeled based on measured properties of dyes and filters, which will be discussed below, but they can also be measured directly. For example, U.S. Patent No. 10,126,242 describes a method for extracting pure component spectra from a single-stained artifact sample despite the presence of autofluorescence in the sample used for measurement. The entire contents of U.S. Patent No. 10,126,242 are incorporated herein by reference.

[0111] In some embodiments, the classification of pixel types can be combined with adaptive unmixing to improve unmixing accuracy. In other words, the pixels in the autofluorescence image can be classified according to the type of sample material at each pixel (e.g., interstitial, extracellular matrix (matrix), red blood cells, collagen), and then unmixed at each pixel with a type-specific pure spectrum estimate to improve unmixing accuracy. Alternatively or additionally, based on the type of sample material corresponding to certain pixels in the sample image (i.e., corresponding to the classification type determined for the pixel), certain pixels in the sample image can be excluded from further analysis. For example, pixels classified as corresponding to red blood cells, and / or collagen and / or extracellular matrix may not be of interest. These pixels can be designated as being outside the region of interest of the sample, and the measurement information corresponding to these pixels (e.g., spectral information) can be subsequently ignored. This can significantly reduce the analysis time for the sample. It can also be selected to delete the information measured for these pixels, thereby reducing data storage requirements.

[0112] Conversely, in certain embodiments, pixels corresponding to sample structures such as collagen, interstitium, extracellular matrix, and red blood cells may be preferentially analyzed and visualized by displaying an autofluorescence image in which pixels corresponding to sample structures of no interest are excluded (i.e., displayed as dark pixels).

[0113] As part of spectral unmixing, the autofluorescence spectrum of each sample material type can be measured to obtain a type-specific autofluorescence spectrum of the sample. These measurements can be performed on adjacent series of sections of the same sample to provide the most representative data. Alternatively, the measurements can be based on sections of different samples that share the same organ type, or the same fixation conditions, or the same disease state, or some other characteristic that recommends it as representative of the sample.

[0114] The spectra may be obtained by a human operator selecting pixel regions in an autofluorescence image of the sample, and then obtaining spectra of the selected pixel groups for each type. Alternatively, a trained machine classifier such as a neural network (e.g., implemented in processing device 216) may be used to perform pixel region selection, and material type-specific autofluorescence spectra may be extracted in a fully automated manner. Using the type-specific autofluorescence spectra, an S matrix may be created for each type, and inverted to produce a set of S + Matrix to unmix the pixels of each type of sample material.

[0115] In order to process sample images using type-specific autofluorescence spectra, the sample's autofluorescence image is classified and the type information associated with each pixel is used to select the appropriate S for that type of sample material. +matrix, and the pixels are unmixed to create an abundance vector A. To the extent that the actual sample autofluorescence is better matched by its type-specific spectrum than the ensemble average, the resulting unmixed image will be a more accurate indication of the true sample abundance.

[0116] The foregoing process can be used when pixels in a sample image correspond to different types of sample structures, with variations in endogenous autofluorescence associated with the different structures. A segmentation mask (or other spatial filtering technique) can be used in conjunction with the foregoing process to exclude specific sample structures identified from the autofluorescence image. For example, adaptive unmixing can be combined with the exclusion of pixels corresponding to red blood cells and / or collagen from downstream analysis. Identifying and excluding red blood cells can be beneficial in situations where red blood cells are preferably omitted from the analysis. In addition, adaptive unmixing can provide more accurate quantitative information for viable pixels (such as pixels corresponding to interstitial cells or corresponding to extracellular matrix) that have autofluorescence spectra that are different from the average sample autofluorescence spectrum.

[0117] One or more autofluorescence images can also be used for other applications. For example, in some embodiments, an autofluorescence image of a sample obtained according to the methods described herein can be used to generate a composite H&E image (or another type of composite image) of the sample. In certain embodiments, where different regions of a sample are scanned according to a scanning pattern and the sample and background images are then assembled to form a larger image (e.g., a whole slide image), the autofluorescence spectra corresponding to different regions of the sample can be used to register a set of multispectral sample images corresponding to the different regions. That is, the autofluorescence spectra can be used for pixel-based registration of different sets of sample images.

[0118] In the case of examining multiple samples, it should be understood that while the methods and systems described herein allow for measurement of intrinsic autofluorescence for each sample, it is also possible to measure intrinsic autofluorescence spectra for only a subset of the samples, or from one or more witness or reference samples, and the measured intrinsic autofluorescence spectra can then be used in a spectral unmixing operation performed on a collection of sample images from other samples whose intrinsic autofluorescence is not independently measured.

[0119] Method and system described herein are compatible with serial staining and imaging scheme, and can improve the detection sensitivity in such scheme.In scheme of this type, by selectively engaging dye and antibody with DNA barcode technology, sample is incubated with n antibodies once, and they are detected m times at a time.These schemes can attempt to image (n is up to 30 or more, and m is 3 or 4), so the number of imaging operations can range from 2 to 8 or even more.Other schemes can use the same broad principles, but the technology except DNA barcode technology is used to selectively engage dye and antibody.In scheme, usually only m signals are imaged at a time, so it is usually not too difficult to isolate dye signals from each other or from counterstain.However, the signal level measured can be medium or low, especially for weak species.Method and system described herein are particularly useful for isolating sample autofluorescence that can interfere with the detection of weaker signals in addition.As another benefit, the sample autofluorescence image obtained can be used for quasi-registration of sample images from continuous imaging sessions.

[0120] Other features and aspects of systems for obtaining sample images and methods for analyzing and classifying spectral images are described, for example, in the following references, the entire contents of each of which are incorporated herein by reference: U.S. Patent No. 8,634,607; U.S. Patent No. 7,555,155; U.S. Patent No. 8,103,331; U.S. Patent Application Publication No. US2014 / 0193061; U.S. Patent Application Publication No. US2014 / 0193050; and U.S. Patent Application No. 15 / 837,956.

[0121] Any of the method steps and other functions described herein may be performed by the controller 214 (e.g., via the processing device 216 of the controller 214) and / or one or more additional processing devices (such as a computer or preprogrammed integrated circuit) executing software programs or hardware-coded instructions. The software programs may be stored on a variety of tangible, processing device-readable storage media including (but not limited to) optical storage media such as CD-ROM or DVD media, magnetic storage media, and / or permanent solid-state storage media. The programs, when executed by the processing device 216 (or more generally, by the controller 214), cause the processing device (or controller) to perform any one or more of the control, calculation, and output functions described herein.

[0122] In addition to the processing device 216, the controller 214 may optionally include other components, including a display or output unit, an input unit (e.g., a pointing device, a voice recognition interface, a keyboard, and other such devices), a storage unit (e.g., a persistent or non-persistent storage unit that can store programs and data measured by the system, as well as calibration and control settings), and a sending and receiving unit for sending and receiving data and control signals to other electronic components including other computing devices.

[0123] Example

[0124] (1) Lung cancer samples

[0125] Lung cancer samples were prepared using the reagents listed in Table 1. TSA-biotin (Akoya Biosciences, Menlo Park, CA) was used to stain pan-cytokeratin (see Table 1) and was used with streptavidin, Alexa Fluor TM Final incubation with TUNAQ® 750 conjugate (ThermoFisher Scientific, Waltham MA) was performed at 1:200 dilution for 1 hour at room temperature.

[0126] use Figure 2 The system shown obtains a sample image. Four epitaxial filter cubes are used in the filter 206, each having Figure 5A-5D Optical response shown. Light source 202 generates light in six independently controlled wavelength bands with adjustable brightness and electronic shutter capability for each band.

[0127] Table 2 lists the bands.

[0128] LED Channel Central wavelength Bandwidth (FWHM) UV 385±5nm 11 purple 430±5nm 18 blue 475±5nm 22 yellow 550±5nm 82 red 638±5nm 18 NIR 735±5nm 32

[0129] Table 2

[0130] Figure 2 An embodiment of the system forms a compound epi-illumination fluorescence microscope operating at infinite conjugate. All components are selected to have high lateral resolution and high transmittance in the visible and near-infrared range (780nm). Lens 208 is a Nikon 10x plan-apochromat with a focal length of 20mm and a numerical aperture of 0.45 (Nikon USA, Melville, NY). The tube lens (part of the detector 212) is an apochromat with a focal length of 145.2mm, and the detector is a SCMOS Flash 2.8 sensor with a pixel size of 3.63 square microns (Hamamatsu US, Bridgewater NJ). The total magnification is 7.26, and each pixel corresponds to 0.5 microns of the sample.

[0131] The first filter cube is a three-band filter with 3 different excitation and emission bands. By selecting which band of the light source is activated, each excitation band is activated individually, while the other bands are not excited. Because the light source is electronically controlled, the system is able to quickly cycle through the excitation bands without mechanical movement.

[0132] Three sample images are acquired successively, corresponding to the sample response to each of the three excitation bands. Figure 5A Fluorescence emission is detected in the three emission peaks shown. Thus, sample fluorescence summed across all three filter emission bands is detected in each image, even though only one filter band is excited at a time.

[0133] Then, a second filter cube is placed in the light path. The second filter cube is another three-band filter that is used in the same manner as the first filter cube, with the light source activating the LEDs corresponding to the excitation bands in that filter cube. This provides three more images of the sample corresponding to the sample fluorescence in response to each of the three excitation bands, as recorded through its three emission bands.

[0134] The third and fourth filter cubes are single band filters, one excitation and one emission band each. Each of these filter cubes is recycled into the light path and one image is captured using each filter cube.

[0135] This provides a total of 8 images of the sample at 8 different combinations of excitation and emission wavelength bands.

[0136] The stage 210 is used to sweep out a raster pattern while taking images of the sample. For speed purposes, and to reduce the number of filter changing operations of the mechanism, the pattern is performed in sequence for each filter through a set of 4 lines of the raster pattern path, taking an image at each point in a given row, and stepping through a number of rows. The next filter is then engaged, the same pattern is performed, and its image is taken. This continues until the set of rows has been imaged for all filters. The raster pattern is then continued for the next set of 4 lines. In this way, the entire sample is imaged using a total of 24 lines in 6 groups. Overall, the acquisition time is 4 minutes and 35 seconds for a sample scan measuring 12 mm x 16 mm.

[0137] As part of the overall imaging operation, the controller 214 creates a map of the sample location and measures the focus at a grid of points within the sample area. This grid is used to set the focusing mechanism of the lens 208 during raster scanning. The normalized variance is used as a sharpness metric to select the best focus, and the focus is interpolated for each image in the raster using a Delaunay triangle mesh to fit the resulting surface.

[0138] Software running on the controller 214 (specifically, the processing device 216) is used to process each image and create a whole slide mosaic image containing 8 spectral channels corresponding to each filter and excitation combination. The individual images acquired during the raster imaging process were each corrected for shading using a different shading pattern for each filter and excitation setting. They were then assembled into a mosaic based on the known pixel size at the sample and the raster pattern grid. This mosaic was saved as a pyramid TIFF file using 512×512 pixel tiles and LZW lossless compression.

[0139] In spectral imaging, an image in which each pixel contains measurements at multiple spectral bands is traditionally called an image cube. Such a mosaic contains image cubes at several pyramid resolution levels.

[0140] Two additional steps were performed. First, the epitaxial filters were characterized by measuring the pixel shift at the sensor that was produced when the sample was imaged with each filter in turn. Unless the dichroic and emission filter elements were completely free of wedge, the epitaxial filter optics would introduce an image shift. In practice, image shifts of 2-10 seconds of arc are typical even for so-called "zero wedge" filter sets. This shift was systematic and repeatable. In the experiments described, one pixel corresponded to 5.5 seconds of arc.

[0141] Based on the pixel offset measured at the sensor and the known pixel size of 0.5 microns, the stage position is offset from nominal during the raster scan in order to oppose the wedge created by each filter. Therefore, the stage position used for a given field in the raster when imaging with the first filter is slightly different with each subsequent filter to offset the optical offset of the epi-optics. In this way, the pixels in the image correspond to exactly the same point in the specimen despite the optical offset introduced.

[0142] Second, a residual chromatic focus shift is measured for the lens based on measuring the best focus of the sample for all filter and excitation band combinations. During imaging, the focus setting is shifted by this amount in sync with the epi filter selection and LED band selection to compensate for this effect. If the focus dimension is expressed as the z-direction, the process attempts to record exactly the same layer of the sample along the z-direction in all spectral channels of the resulting multispectral image.

[0143] Once the images have been acquired as described above, an unmixing step is performed to separate the spectral channels of the mosaic into estimates of the contribution from each pure spectral contributor. As described above, each pure spectral contributor corresponds to an applied dye (or counterstain), or to endogenous autofluorescence of the sample.

[0144] The spectral response of each dye and counterstain was modeled taking into account the LED signal, camera response, excitation filter response and emission response and the measured excitation and emission responses of each dye and counterstain. This was done in the open source R programming environment (R Foundation for Statistical Computing; Vienna, Austria). The results for the first 7 data rows, corresponding to the spectral bands named B1-B7 in the table, are shown in Table 3.

[0145]

[0146]

[0147] Table 3

[0148] Fig. 6A The dye excitation responses of the above counterstain and six dyes used as immunofluorescence markers are shown, and Figure 6B The transmit response is shown. Figure 6C Shown are the autofluorescence emission responses of formalin-fixed, paraffin-embedded lung cancer specimens when excited at 387 nanometers and when excited at 425 nanometers.

[0149] Alternatively, pure spectra (or spectral library entries) can be derived from the measured information. For example, to obtain spectral library entries for autofluorescence, a sample without immunofluorescence labeling can be imaged in 8 separate spectral bands to obtain a measurement image, the intensity of the autofluorescence signal is extracted for each pixel, and the signal intensity is scaled by the exposure time to produce a time-normalized autofluorescence spectrum after a normalization step.

[0150] To obtain spectral library entries for individual dyes used in immunofluorescence labeling, samples can be prepared using primary antibodies, secondary antibodies, and / or dyes, using antigen retrieval solutions, blocking buffers, washing reagents, and other auxiliary reagents to produce single-stained samples. The sample is imaged to obtain measurement images in 8 separate spectral bands, the intensity of the combined dye and autofluorescence signal is extracted for individual pixels, the signal intensity is scaled by exposure time, and the signal is corrected for the autofluorescence contribution to obtain a time-normalized corrected dye spectrum.

[0151] To obtain a spectral library entry for a counterstain such as DAPI, where a sample is prepared with the counterstain to produce a counterstained sample, the sample is imaged to obtain a measurement image in 8 separate bands, the intensity of the combined dye or counterstain signal plus the autofluorescence signal is extracted for individual pixels, the signal intensity is scaled by exposure time, and the signal is corrected for the autofluorescence contribution to obtain a time-normalized corrected counterstain spectrum.

[0152] Two samples were prepared, one stained with Opal 620 alone, and one stained with Opal 690 alone. These samples were then imaged and the relative responses measured at bands B5, B6 and X. This was used to populate the X band row for both dyes in Table 3. Measurements of single dye samples of the other dyes showed no measurable response at the X band.

[0153] Table 3 shows the relative response of each dye of the image, wherein all spectral bands have equal exposure times normalized by the signal level in the brightest spectral band. It is usually useful to scale signal counts by exposure time and perform calculations in units of counts per unit time (such as counts per millisecond). The signal level in a scientific digital camera is proportional to the exposure time, and scaling by exposure time provides two practical benefits. First, it enables people to perform spectral calculations on images taken with unequal exposure times in different spectral bands with counts of exposure scale, so that the spectrum is not affected by acquisition conditions. Second, for the same reason, it enables people to compare spectra or perform spectral calculations involving multiple images, wherein the images have different exposure times from each other. This method was used in the present process.

[0154] Next, two FFPE samples without dye applied were imaged using the same apparatus and methods as above. This produced a pyramidal mosaic image cube with spectral bands B1-B7 and X. One sample was a lung cancer section and the other was a breast cancer section, each sample 4-5 microns thick.

[0155] Spectra are measured at multiple locations corresponding to identifiable biological structures such as stroma, erythrocytes, generalized extracellular matrix, collagen, etc. This measurement is done by pixel averaging the signal in a group of pixels selected to represent the structure of interest. This is done by deriving images for each individual spectral band plane from the pyramidal TIFF image and then using a 3D FT-IR software from Akoya Biosciences (Waltham, MA). The software does this by assembling these planes into an image cube.

[0156] In addition to measuring spectra of groups of pixels selected to correspond to specific biological structures, measurements are also taken along irregular paths extending through a variety of structures to obtain structure-averaged spectra.

[0157] The obtained spectra for four types of structures in lung cancer tissue samples and their averages are given in Table 4, and the obtained spectra for seven types of structures in breast cancer tissue samples and their averages are given in Table 5. All of them are listed in units of a time scale of counts per millisecond.

[0158]

[0159]

[0160] Table 4

[0161]

[0162] Table 5

[0163] Based on these, spectral unmixing is performed. In a linear system of independent spectral contributors, the following linear algebraic equation can be used to describe the measured spectrum at any image pixel:

[0164] M=S*A[1]

[0165] M is the measured spectrum at a given pixel, S is a matrix whose columns are the spectra of individual components (dyes, counterstains, or autofluorescence), and A is a column vector of the abundances of the components in the sample. In other words, equation (1) states that the measured signal M is the linear superposition of the spectrum S with the components in the sample according to their abundance A.

[0166] If S has a pseudo-inverse S + , which can be multiplied on the left by both sides of equation [1] to obtain

[0167] S + *M=S + *S*A=(S + *S)*A=I*A=A[2a]

[0168] therefore

[0169] A=S + *M[2b]

[0170] Equation (2b) is the central spectral unmixing equation, which enables one to solve the problem by multiplying the measured spectrum M at a given pixel by the pseudo-inverse spectral matrix S + to calculate the abundance vector A of the pure spectral contributors in the sample.

[0171] In this example, the goal is to decompose the measured signal into contributions from the 6 immunolabeling dyes, the DAPI counterstain, and tissue autofluorescence. Therefore, S has 8 columns, corresponding to the 7 dye (or counterstain) spectra, and the autofluorescence spectrum. The average spectrum of lung cancer normalized by the signal level of the brightest band is used for the autofluorescence spectrum.

[0172] Each column has 8 entries, corresponding to bands B1-B7 and X, meaning that S is an 8×8 matrix. Since S is square, we compute S by directly taking its inverse + More generally, S can be computed by well-known methods in linear algebra, such as the Moore-Penrose technique. + .

[0173] The obtained S + The matrix contains the coefficients that transform the raw spectral measurement M of a sample pixel into a vector of pure contributor abundances A in the time-scaled measurement space. It is used to transform the raw spectral mosaic image, where each pixel contains the measured signal M, into an unmixed mosaic image, where each pixel contains the abundance A of a separate pure spectral contributor (i.e., dye, counterstain, and endogenous autofluorescence).

[0174] The original sample images corresponding to bands B1-B7 and X are in Figures 7A-7H and the unmixed pure spectral contributor image is shown in Figures 8A-8H Shown in. Figures 7A-7H The images in 8A-8H show only selected areas of the sample. Figures 9A-9H An unmixed pure spectral contributor image of the entire sample is shown.

[0175] Comparison of paired images demonstrates the ability of the methods and systems described herein to provide accurate quantitative information about molecular targets in a sample. For example, Figure 7B The original image of B2 is shown, where the main spectral band of Dy430 emission is observed. This dye is a marker for CD8 in the sample. Figure 7B In the , one can see contributions associated with CD8, which is mainly located in the membrane of lymphocytes, especially cytotoxic T cells. Figure 7B It appears as a dense bright ring in the middle.

[0176] But in Figure 7B Many other structures can be seen in the image that are not related to the true CD8 localization in the sample, making this image an unreliable indicator of this type of cell. For example, there is a regular level of background signal in the extracellular matrix; large, bright, irregular features that appear to be collagen structures; and other cells in the tissue. Overall, most of the signal comes from sources other than the CD8 marker.

[0177] This interference makes it more difficult to accurately identify the presence, number, or location of cytotoxic T cells and tends to degrade analyses based on one or more of these measures. Based on the location and shape of the confounding features, the interference comes primarily from endogenous tissue autofluorescence rather than from other dyes used to label the sample.

[0178] Figure 8B is the unmixed pure spectral contributor image associated with the dye Dy430. Figure 7B In contrast, interference signals associated with endogenous tissues are either weak or nonexistent. Figure 7B The features associated with true CD8 seen in the images of were present without a significant decrease in intensity. Figure 7B Several faint CD8 ring features were difficult to locate due to background or interfering signals. This certainly illustrates the benefit of multispectral imaging and spectral unmixing as a technique to isolate signal and remove autofluorescence.

[0179] Comparison of other raw spectral images with the unmixed component abundance images of DAPI, Opal520, and Opal570 shows comparable effectiveness in isolating the desired component from interference from autofluorescence. Fig. 8A In the spectrally pure DAPI contributor abundance image, this nuclear counterstain was used to identify, segment, and count tumor and stromal cells. Fig. 8A The DAPI images in the provide information that is essentially free of unwanted additional content, while Fig. 7A The original spectrum in contains many small, bright structures that do not correspond to DAPI staining. These structures appear to arise from endogenous tissue autofluorescence. Fig. 7A The spectral image shown in corresponds to narrow-band excitation and emission filters that were selected to be optimal for DAPI fluorescence imaging. To the extent that the filter selection is indeed optimal, the signal-to-noise ratio shown in this image (its ability to discern the desired DAPI signal without other confounding signals) illustrates the best performance that can be achieved with conventional whole slide imaging.

[0180] Similarly, Opal570 is used to label FoxP3 protein in the sample, which tends to localize in the nucleus of regulatory T cells (eg, "Treg" cells). Fig.8D The unmixed pure spectral contributor image (corresponding to Opal570) shows clear labeling of the cell nucleus. This indicates that the immunofluorescence labeling of FoxP3 achieves high specificity, so Opal570 labeling appears to be localized to the expected structure. Brightly labeled Treg cells show 150-250 counts, and faint cells expressing 50 counts can be easily detected against a low background of 10 counts or less. In contrast, Fig.7DThe original sample image in has a more conventional background with many kinds of structures having a signal level of 25 counts that do not correspond to real FoxP3.

[0181] Next, consider the original sample image at band X Figure 7H . The excitation filter and emission filter of this band are selected to detect intrinsic autofluorescence from the FFPE sample, rather than any counterstain or dye. In addition, they are selected to be as insensitive as possible to the counterstain and (multiple) dyes applied to the sample, so that it is possible to obtain an image of the intrinsic autofluorescence in the sample despite the presence of 4 or more dyes. Typically, multispectral sample imaging protocols do not include selecting excitation and emission bands with these characteristics. Instead, traditional methods for obtaining images of intrinsic autofluorescence in such samples involve spectrally unmixing a multispectral image cube, where one of the spectra is autofluorescence, and observing the autofluorescence abundance image. In this case, the original spectral cube typically includes a large number of spectral channels to separate the autofluorescence signal from other fluorescence signals in the sample. This greatly increases computational time and data storage, making it unsuitable for many uses, such as whole slide scanning or digital pathology workflows.

[0182] In contrast, the X-band is only weakly affected by other signals (particularly due to non-endogenous dyes and counterstains). This makes it easier to obtain accurate estimates of endogenous sample autofluorescence without a large number of spectral channels. For example, in some cases only one spectral channel is used per dye and counterstain (i.e. single wavelength measurement), plus one channel for autofluorescence.

[0183] Figure 8H is the unmixed abundance image of the autofluorescence. This image looks roughly like Figure 7H This is not surprising since the X-ray bands are designed to respond only weakly to signals other than endogenous autofluorescence. Both images show features corresponding to collagen structures, generalized extracellular matrix, interstitial structures including nuclei, and erythrocytes.

[0184] However, there are important differences. Figure 7H The raw spectral image shown includes some signals associated with the CD68 target labeled with Opal620 and the PD1 target labeled with Opal690. Figure 8H The unmixed autofluorescence abundance images are largely or completely absent. This can be understood based on the dye spectra above. The X band shows a 1.4% response for the Opal620 dye and a 1.0% response for the Opal690 dye, compared to the time-normalized response in the band where each dye responds most strongly.

[0185] therefore, Figure 7HThe original spectral image in includes some signals associated with the Opal620 component and some signals associated with the Opal690 component. Spectral unmixing produces Figure 8H The autofluorescence image in more correctly reveals the true endogenous autofluorescence signal. In this specific example, the amount of difference is relatively modest.

[0186] In some embodiments, autofluorescence images can be analyzed to identify specific tissue structures, cells, or regions that are ignored or specially processed in downstream image analysis. Fig.8D In the discussion of Fig.7D The raw spectral image of shows a signal that is not entirely attributable to this dye component. Without wishing to be bound by theory, a possible explanation is that the autofluorescence of erythrocytes is different from the autofluorescence of the average sample structure. Although there is some signal in the FoxP3 abundance image, it is valuable to identify that these are erythrocytes and not Treg cells. Therefore, in some embodiments, erythrocytes can be identified based on the autofluorescence image and marked as erythrocytes for downstream analysis, ensuring that they are not mistaken for FoxP3 positive cell nuclei.

[0187] This step can also be used for other analysis processes. For example, in some embodiments, the exact results of red blood cell fluorescence emission may be different from the results here, although identifying and separating red blood cells may still be beneficial. In some embodiments, structures associated with collagen may confound or degrade specific measurements, and areas of the sample associated with collagen structures can be identified from the autofluorescence image and eliminated from the analysis process, or otherwise analyzed according to different criteria or algorithms.

[0188] It should be understood that various methods can be used to decompose the sample image. Specifically, the spectral unmixing calculation can be performed in a variety of ways, including by changing the order in which certain steps are performed, and omitting certain steps entirely. For example, it is not necessary to unmix all members of the abundance matrix to use any one unmixed component. Specific rows of the unmixing matrix can be selectively used in the unmixing process, depending on the pure spectral contributors being elucidated.

[0189] Furthermore, because the methods and systems described herein directly obtain a background image of the sample (which corresponds approximately - or even almost identically - to the sample autofluorescence image), the sample autofluorescence image can be obtained (e.g., from the background image) before unmixing the sample image to obtain a pure spectral contributor image. This two-step analysis process is particularly useful when adaptive spectral unmixing is performed depending on the characteristics of the autofluorescence spectrum at a specific sample location, and / or when pixels corresponding to certain sample structures (such as red blood cells and / or collagen) are identified from the autofluorescence image and marked (e.g., to exclude these pixels from further analysis).

[0190] In order to obtain an autofluorescence image before a pure spectral contributor image, the autofluorescence abundance can be simply unmixed first in a two-step unmixing process. Then, based on the sample image and the autofluorescence abundance A corresponding to the original spectral bands (e.g., bands B1-B7 in this example), x , instead of using the background image corresponding to band X, the contribution from non-endogenous spectral contributors (e.g., applied dye) is unmixed. The unmixing matrix coefficients are modified to take into account A as follows x Use:

[0191] A x =S + x *M[3]

[0192] Here, S + x Refers to the inverse spectral matrix S + The other rows of the unmixing matrix can also be modified to take into account the x rows of A. x The contribution introduced is reduced, and the result can be a more diagonal matrix. Because the matrix is ​​more diagonal, small-valued terms can be omitted, speeding up the calculation without a large impact on accuracy.

[0193] (2) Cell counting

[0194] To study cell counting based on sample autofluorescence images, two consecutive sections of lung cancer samples were prepared. One section was an unstained negative control without any dye deposition. The other section was stained with a multiple panel using the reagents listed in Table 6, followed by incubation with Opal Polaris780 Anti-Dig (Akoya Biosciences, Waltham, MA) at room temperature for 1 hour at a dilution of 1:25. All other incubation times were the same as in the previous examples. Sample images were obtained using the same system described in the previous examples.

[0195]

[0196] Table 6

[0197] After the measurement of the background image (assumed to correspond to the sample autofluorescence image), use The automatic tissue segmentation tool in the AUTOMATED® software (Akoya Biosciences, Menlo Park, CA) trained a pixel-based classification algorithm to identify red blood cells. Only autofluorescence images were used as input to the classifier. Fig. 10A shows a collection of images obtained from samples prepared with the dyes shown in Table 6, and Fig. 10B A collection of images derived from a negative control sample (no dye) is shown. Fig. 10A and 10B Image 901 in shows the respective autofluorescence images of the samples.

[0198] To train the machine learning algorithm, regions were manually drawn on the negative control autofluorescence images to identify pixels that were positive or negative for red blood cells. The results of the trained algorithm for both slices corresponded to Fig. 10A and 10B Binary classification mask for image 902 in .

[0199] In addition, a cell segmentation algorithm was developed using The adaptive cell segmentation tool in the software was used to identify FoxP3 positive cells. Only the unmixed Opal 570 / FoxP3 abundance image ( Fig. 10A and 10B The image 903 in FIG. 1 is used as input to the segmentation algorithm. A threshold for segmentation is set to identify objects with Opal570 signal and oval nuclei in the image ( Fig. 10A and 10B In the negative control sample ( Fig. 10B All cells identified in image 904) are false positive detections of autofluorescent species, such as red blood cells that have a morphology similar to that of a cell nucleus.

[0200] Finally, from both algorithms ( Fig. 10A and 10B The masks of images 902 and 904 in FIG. 1 and FIG. 2 are combined to retain only FoxP3 positive cells that do not overlap with any pixels identified as red blood cells in the panel. The resulting mask ( Fig. 10A and 10B Image 905) in FIG. 1 shows a reduction in the number of false positives relative to image 904.

Claims

1. A method for determining the abundance of multiple non-endogenous spectral contributors in a biological sample, include: exposing a plurality of non-endogenous spectral contributors in a biological sample to illumination light and measuring light emission from the sample to obtain N sample images, wherein each sample image corresponds to a different combination of a wavelength band of the illumination light and one or more wavelength bands of the light emission, The center wavelengths of the plurality of wavelength bands of the plurality of non-endogenous spectral contributors define a wavelength range, and N>1; exposing the sample to illumination light in a background excitation band and measuring light emission from the sample in a background spectral band to obtain a background image of the sample, wherein the background spectral band corresponds to wavelengths within the wavelength range; as well as Obtaining autofluorescence abundance information from the background image; decomposing at least some of the N sample images to obtain pure spectral contributor images, each of the pure spectral contributor images including only a contribution from one of the non-endogenous spectral contributors in the sample; and determining the abundance of the plurality of non-endogenous spectral contributors using the autofluorescence abundance information and the pure spectral contributor image, Wherein, for each non-endogenous spectral contributor among a plurality of non-endogenous spectral contributors in a sample exposed to illumination light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 10% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after the non-endogenous spectral contributor is excited in each wavelength band of the illumination light and the background excitation band. 2 . The method of claim 1 , further comprising obtaining an autofluorescence image of the sample from the background image.

3. The method according to claim 1, in, For each of the multiple non-endogenous spectral contributors in the sample exposed to illumination light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 4% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after the sample is excited in each wavelength band of the illumination light and the background excitation band.

4. The method according to claim 1, in, For each of the multiple non-endogenous spectral contributors in the sample exposed to illumination light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 2% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after the sample is excited in each wavelength band of the illumination light and the background excitation band.

5. The method according to claim 1, in, The background spectral band includes a spectral width Δλ having a full width at half maximum (FWHM) and a central wavelength λ c The wavelength distribution of the background spectrum band corresponds to λ c -Δλ / 2 to λ c Wavelengths within the range of +Δλ / 2.

6. The method according to claim 1, in, N>3。 7. The method according to claim 1, in, N>5。 8. The method according to claim 1, in, The sample includes M non-endogenous spectral contributors, and where M≤N.

9. The method according to claim 8, in, M>4。 10. The method according to claim 8, in, M>6。 11. The method of claim 2, further comprising displaying the autofluorescence image on a display device.

12. The method of claim 2, further comprising determining an amount of autofluorescence emission from the sample at each of a plurality of locations in the sample.

13. The method of claim 12, further comprising, at each of a plurality of locations in the sample, and for one or more sample images in the N sample images: Based on the amount of autofluorescence emission at each of the plurality of locations and at least one pure spectrum of autofluorescence emission from the sample, a value corresponding to the sample emission intensity is adjusted to correct for the autofluorescence emission from the sample.

14. The method according to claim 13, in, The at least one pure spectrum of autofluorescence emission comprises a plurality of pure spectra of autofluorescence emission, and wherein the plurality of pure spectra of autofluorescence emission each correspond to a different subset of the plurality of locations.

15. The method according to claim 14, further comprising: include: decomposing at least some of the N sample images to obtain M spectral contributor images based on an amount of autofluorescence emission from the sample at each of the plurality of locations, wherein each of the M spectral contributor images corresponds to light emission from only a different one of the non-endogenous spectral contributors; as well as At each of the plurality of locations, an amount of M non-endogenous spectral contributors in the sample is determined.

16. The method of claim 15, further comprising decomposing at least some of the N sample images based on at least one pure spectrum of autofluorescence emissions from the sample.

17. The method according to claim 16, in, The at least one pure spectrum of autofluorescence emissions comprises a plurality of pure spectra of autofluorescence emissions, and wherein the plurality of pure spectra of autofluorescence emissions each correspond to a different subset of the plurality of locations.

18. The method according to claim 9, in, The sum of the spectral emission intensity of each non-endogenous spectral contributor in the sample at each wavelength within the background spectral band is 10% or less of the total fluorescence emission intensity in the background spectral band.

19. The method of claim 2, further comprising classifying pixels of one or more of the sample images into different categories based on information derived from the autofluorescence image.

20. The method according to claim 19, in, The different categories correspond to different cell types in the sample.

21. The method according to claim 8, in, The M non-endogenous spectral contributors include one or more fluorescent species that selectively bind to different chemical moieties in the sample.

22. The method according to claim 21, in, The one or more fluorescent species include one or more immunofluorescent probes.

23. The method according to claim 21, in, The M non-endogenous spectral contributors include one or more counterstains.

24. A computer-readable storage medium comprising instructions which, when executed by a processing device, cause the processing device to: Decomposing N sample images of multiple non-endogenous spectral contributors in the sample using autofluorescence information of the biological sample to obtain multiple non-endogenous spectral contributor images of the sample; Obtain autofluorescence abundance information from background images; decomposing at least some of the N sample images to obtain pure spectral contributor images, each of the pure spectral contributor images comprising only a contribution from one of the non-endogenous spectral contributors in the sample, wherein N>1; using the autofluorescence abundance information and the pure spectral contributor image to determine an amount of a plurality of non-endogenous spectral contributors at a plurality of locations in a sample; as well as generating an output display including at least one of the non-endogenous spectral contributor images on a display device connected to the processing device, wherein each sample image corresponds to a different combination of a wavelength band of illumination light used to illuminate the sample and one or more wavelength bands of emission light from the sample, wherein center wavelengths of the plurality of wavelength bands of the plurality of non-endogenous spectral contributors define a wavelength range; wherein the background image corresponds to illumination of the sample with light in a background excitation band and measurement of light emission from the sample in a background spectral band; and Wherein, for each non-endogenous spectral contributor of a plurality of non-endogenous spectral contributors in a sample irradiated with light in a background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 10% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after the non-endogenous spectral contributor is excited in each wavelength band of the irradiation light and the background excitation band.

25. The storage medium of claim 24, further comprising instructions that, when executed by the processing device, cause the processing device to obtain an autofluorescence image of the sample from a background image of the sample.

26. The storage medium according to claim 25, in, Each non-endogenous spectral contributor image corresponds only to light emission from a different one of the non-endogenous spectral contributors in the sample.

27. The storage medium according to claim 25, in, For each of the multiple non-endogenous spectral contributors in the sample illuminated with light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 4% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after exciting the sample in each wavelength band of the illuminating light and the background excitation band.

28. The storage medium according to claim 25, in, For each of the multiple non-endogenous spectral contributors in the sample illuminated with light in the background excitation band, the spectral emission intensity at each wavelength within the background spectral band is 2% or less of the maximum measured spectral emission intensity of the non-endogenous spectral contributor after exciting the sample in each wavelength band of the illuminating light and the background excitation band.

29. The storage medium according to claim 24, in, The sample includes M non-endogenous spectral contributors, and where M≤N.

30. The storage medium according to claim 29, in, M>4。 31. The storage medium according to claim 29, in, M>6。 32. The storage medium according to claim 25, in, The autofluorescence information includes an amount of autofluorescence emission from the sample at each of a plurality of locations in the sample.

33. The storage medium of claim 32, further comprising instructions that, when executed by the processing device, cause the processing device to decompose a plurality of sample images using the autofluorescence information and at least one pure spectrum of autofluorescence emissions from the sample.

34. The storage medium of claim 33, further comprising instructions that, when executed by the processing device, cause the processing device to determine at least one pure spectrum of autofluorescence from the background image.

35. The storage medium according to claim 33, in, The at least one pure spectrum of autofluorescence emissions from the sample includes two or more different pure spectra of autofluorescence emissions from the sample, and wherein each of the different pure spectra corresponds to a different subset of the plurality of spatial locations.

36. The storage medium of claim 33, further comprising instructions which, when executed by the processing device, cause the processing device to adjust a value corresponding to the sample emission intensity in each sample image in the sample image to correct for the autofluorescence emission from the sample based on the amount of autofluorescence emission at each of the plurality of positions and at least one pure spectrum of the autofluorescence emission from the sample.

37. The storage medium according to claim 24, in, The sum of the spectral emission intensity of each non-endogenous spectral contributor in the sample at each wavelength within the background spectral band is 10% or less of the total fluorescence emission intensity in the background spectral band.

38. The storage medium of claim 24, further comprising instructions that, when executed by the processing device, cause the processing device to classify pixels of one or more of the sample images into different categories based on autofluorescence information.

39. The storage medium according to claim 38, in, The different categories correspond to different cell types in the sample.

40. The storage medium according to claim 29, in, The M non-endogenous spectral contributors include one or more fluorescent species that selectively bind to different chemical moieties in the sample.

41. The storage medium according to claim 40, in, The M non-endogenous spectral contributors include one or more counterstains.

Images