Method of analysing mixed fluorescence response of plurality of fluorophores, fluorescence analyser, fluorescence microscope and computer program

Abstract

To improve a process of spectral un-mixing.SOLUTION: A method of analysing a mixed fluorescence response of a plurality of fluorophores in a microscopic sample includes spectral un-mixing and a procedure for determining and validating reference emission spectra therefor. The procedure includes the steps of (a) providing a plurality of image acquisition settings for a sequence of images of the sample equal to, or greater than, the plurality of fluorophores and including an illumination setting, (b) acquiring the sequence of images of the sample using the plurality of image acquisition settings and storing each of the sequence of images of the sample together with the corresponding illumination setting, (c) determining candidate reference emission spectra for the fluorophores to be reconstructed from the sequence of images of the sample using one or more reference emission spectra determination algorithms, and (d) conditionally using said candidate reference emission spectra as the reference emission spectra in said spectral un-mixing.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a method for analyzing the mixed fluorescence response of a plurality of phosphors in a microscope sample, a fluorescence analyzer, a fluorescence microscope, and a computer program.

Background Art

[0002] WO 2021 / 185557 discloses a fluorescence microscope for imaging an object containing various phosphors having different spectral emission characteristics. The fluorescence microscope has an optical system configured to collect fluorescence emitted from different phosphors in the field of view in order to focus the fluorescence for detection. A spectral splitting device is provided that is configured to split the fluorescence collected in the field of view into at least two spectrally different fluorescence components. A multi-channel detection system having at least two image sensors is configured to detect at least two spatial light intensity distributions based on at least two spectrally different fluorescence components, and each spatial light intensity distribution represents an image of the object across the field of view. The fluorescence microscope further has a processor configured to identify the spatial distribution of different phosphor species based on spectral unmixing analysis of each spatial light intensity distribution.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In particular, the present invention aims to improve the process of spectral unmixing in such a system, as well as in any other fluorescence microscope and similar device in which the fluorescence response of a plurality of phosphors is analyzed, from the viewpoints of effectiveness and ease of use by the user.

Means for Solving the Problems

[0004] The present invention provides a method for analyzing the mixed fluorescence response of multiple phosphors in a microscope sample, wherein the mixed fluorescence response has contributions from the individual fluorescence responses of each of the multiple phosphors, and the method includes reconstructing the individual fluorescence responses from the mixed fluorescence response using spectral unmixing based on a reference emission spectrum or edge component spectrum of the phosphor to be reconstructed.

[0005] The above method includes a procedure for identifying and verifying a reference emission spectrum, which includes (a) supplying multiple image acquisition settings for image sequences of a sample equal to or larger than multiple phosphors, including illumination settings for each image sequence; (b) acquiring image sequences of the sample using the multiple image acquisition settings and storing each image sequence of the sample along with the corresponding illumination settings; (c) identifying candidate reference emission spectra for the phosphor to be reconstructed from the image sequences of the sample using one or more reference emission spectrum identification algorithms; and (d) conditionally using the above candidate reference emission spectra as reference emission spectra in the spectral unmixing described above.

[0006] The method proposed according to embodiments of the present invention makes it possible to reliably identify a reference emission spectrum used for spectral unmixing of fluorescence images. This is achieved with minimal user intervention by automatically obtaining the necessary images, performing spectral identification, selectively checking the validity of the spectral identification, and selectively providing the user with feedback on the results of the spectral identification. Since this method can be performed later directly on the sample being imaged, no additional reference sample is required.

[0007] In one embodiment of the present invention, the method further includes the following steps: (i) between steps (c) and (d) shown above, a step of performing a validation check on a reference emission spectrum candidate, wherein the result of the validation check indicates whether the reference emission spectrum candidate is valid for the phosphor to be reconstructed; and another step (ii) of using the reference emission spectrum candidate as the reference emission spectrum in the spectral unmixing described above, in accordance with step (d) shown above, if the result of the validation check is yes, or rejecting the reference emission spectrum candidate if the validation check is no. In one embodiment of the present invention, the validation check ensures that only valid reference emission spectra are used further, i.e., if a reference emission spectrum candidate is found to be invalid, it can be removed and, instead, for example, a default or previous end component spectrum can be used.

[0008] In one embodiment of the present invention, the lighting settings may be linearly independent. Linear independence is expected when the lighting settings are represented as a matrix having one lighting setting per column and the respective intensity per row. This is the case for a diagonal matrix when individually activated light sources are used, but it can also be the case when multiple light sources are activated. Thus, embodiments of the present invention are generally and flexibly applicable to these different cases.

[0009] According to one embodiment of the present invention, the illumination setup for each image sequence may include selecting a single light source from among several light sources, the single light source may include a light-emitting diode, a laser of a given central wavelength, or a wavelength selector, the wavelength selector may in particular be configured to select a (continuous) illumination spectrum from a broadband light source selected from, for example, a classical filter, an acousto-optic filter, or a selector and spectrometer. Thus, embodiments of the present invention are applicable to a number of different types of fluorescence microscopy techniques.

[0010] According to one embodiment of the present invention, the procedure for identifying and verifying a reference emission spectrum is generally executable in at least two temporal instances, including a preceding instance and a succeeding instance. This allows the previously obtained results to be reused in the subsequent instance of the procedure.

[0011] According to one embodiment of the present invention, performing a validation check in a subsequent instance may include performing an evaluation based on a reference emission spectrum obtained in a preceding instance. According to one embodiment of the present invention, if the result of performing the validation check is yes in the preceding instance, the reference emission spectrum candidate can be stored for use in the validation check in the subsequent instance. In particular, such an embodiment allows for incremental improvement for each instance.

[0012] According to one embodiment of the present invention, when a validation check in a preceding instance is successful, and the lighting settings used in a subsequent instance are based on those settings used in the preceding instance, such a gradual improvement of the results or advantageous (re)reuse of the results becomes particularly possible.

[0013] According to embodiments of the present invention, the image acquisition settings may include detector settings or camera settings selected based on sample characteristics, the sample characteristics including at least one of sample mobility or sensitivity, and the detector settings or camera settings including at least one of binning coefficient, scanning speed, scanning resolution, gain value, and exposure time. In other words, in such embodiments, the settings can be particularly adapted to samples that are more or less mobile or sensitive, thereby improving the results while protecting particularly sensitive samples.

[0014] In embodiments of the present invention, a fluorescence microscope can be used to identify a mixed fluorescence response, the fluorescence microscope being configured to operate in at least one microscope mode selected from wide-field mode, confocal mode, and light-sheet mode, and a procedure is performed to identify and verify a reference emission spectrum for at least one of these microscope modes. Thus, embodiments of the present invention allow for flexible operation of multiple modes in a single instrument.

[0015] According to one embodiment of the present invention, in response to a calibration request, a procedure can be performed to identify and verify a reference emission spectrum, and the calibration request may be at least one of a calibration request made by the user, a calibration request made based on the results of an image analysis step, a calibration request made based on a predetermined time, and a calibration request made when a change in the field of view is identified. In other words, embodiments of the present invention enable periodic (re)calibration or request-based (re)calibration, and thus improve analysis results can be obtained.

[0016] In particular, in one embodiment of the present invention, the above calibration request can be made when image analysis indicates that the number of adjacent pixels in a predetermined intensity range and / or the value corresponding to the correlation coefficient calculated between unmixed images exceeds a predetermined threshold. Therefore, reliable and automatic identification can be used to determine when (re)calibration is needed.

[0017] In embodiments of the present invention, user feedback can be provided to the user and / or the user can be given the option to use a default reference emission spectrum. This makes it possible to involve the know-how of a typically knowledgeable user.

[0018] In embodiments of the present invention, the one or more reference emission spectrum identification algorithms described above may include at least one of the following: the N-FINDR algorithm, PPI (pixel purity index), non-negative matrix factorization or tensor decomposition, VCA (vertex component analysis) or principal component analysis, a trained neural network, and an optimization algorithm. Such algorithms can be used individually or in combination and are particularly advantageous for obtaining reliable results.

[0019] According to one embodiment of the present invention, a reference excitation spectrum can be provided and used to identify a candidate reference emission spectrum.

[0020] In one embodiment of the present invention, validating a candidate reference emission spectrum may include identifying the Euclidean distance from a given reference emission spectrum, the spectral angle with the given reference emission spectrum, and at least one of the non-negativity of the candidate reference emission spectrum. The corresponding parameters are particularly suitable for identifying similarity to an expected or reference value and thus evaluating the validity of the spectrum.

[0021] In another embodiment of the present invention, the above-mentioned validity check may include comparing a validity value with a validity threshold, and the above-mentioned validity threshold is different for each of the above-mentioned phosphors. Thereby, each phosphor and its specific characteristics can be individually and appropriately considered.

[0022] In one embodiment of the present invention, the above-mentioned unmixing may include at least one of linear unmixing, phase unmixing, identification of a maximum likelihood estimator, or a combination thereof. Such a method is a particularly appropriate unmixing method, and the method used can preferably be selected to suit a specific analysis situation.

[0023] According to one embodiment of the present invention, in order to derive sample characteristics, a value representing the deviation of a reference emission spectrum candidate from an expected value can be used. Therefore, the embodiment of the present invention not only enables an advantageous (re)calibration for spectral unmixing, but also enables the identification of sample characteristics such as acidity by using the same method.

[0024] According to an embodiment of the present invention, the number of phosphors and their spectra can be derived from the mixed fluorescence response. Thereby, an unknown sample in which the number of phosphors and their spectra are unknown can be analyzed.

[0025] The present invention also provides a fluorescence analyzer configured to analyze the mixed fluorescence response of a plurality of phosphors in a microscope sample, where the mixed fluorescence response has contributions from the individual fluorescence responses of the plurality of phosphors. The above analysis includes reconstructing the individual fluorescence responses from the mixed fluorescence response using spectral unmixing based on a reference emission spectrum for the phosphor to be reconstructed. The fluorescence analyzer is configured to execute a procedure for specifying and verifying a reference emission spectrum as a response to calibration requirements, and this procedure includes: (a) supplying a plurality of image acquisition settings for an image sequence of a sample equal to or larger than the plurality of phosphors, including illumination settings for each image sequence; (b) acquiring an image sequence of the sample using the plurality of image acquisition settings and storing each image sequence of the sample together with the corresponding illumination settings; (c) specifying a reference emission spectrum candidate for the phosphor to be reconstructed from the image sequence of the sample using one or more reference emission spectrum specification algorithms; (d) conditionally using the reference emission spectrum candidate as the reference emission spectrum in the above spectral unmixing.

[0026] For such a fluorescence analyzer and other features and advantages according to one embodiment of the present invention, reference may be made to the description related to the method according to the embodiment of the present invention and its variants.

[0027] According to one embodiment of the present invention, the fluorescence analyzer may be configured to perform a procedure for identifying and verifying a reference emission spectrum, which further includes the following steps: (i) a validation check of a candidate reference emission spectrum between steps (c) and (d) above, wherein the result of the validation check indicates whether the candidate reference emission spectrum is valid for the phosphor to be reconstituted; and (ii) if the result of the validation check is yes, using the candidate reference emission spectrum as the reference emission spectrum in the spectral unmixing described above, in accordance with step (d) above, or if the validation check is no, rejecting the candidate reference emission spectrum. Again, refer to the above description for specific other features of embodiments of the present invention and their variations.

[0028] A fluorescence analyzer according to one embodiment of the present invention has one or more processors and one or more storage devices adapted to display a graphical user interface configured to provide control widgets for at least one of the triggers of steps (a) to (e), input of parameters for these, and evaluation of these results. User accessibility is particularly improved in the corresponding embodiments.

[0029] According to one embodiment of the present invention, the fluorescence analyzer can be configured to carry out the method described in the above embodiment, and therefore please refer in particular to the above description.

[0030] The present invention also provides a fluorescence microscope configured to identify the mixed fluorescence response of multiple phosphors in a microscopic sample. For a fluorescence microscope having the above-described fluorescence analyzer and, in one embodiment, multiple detectors and multiple light sources, please refer again to the above description.

[0031] The present invention also provides a computer program having program code for carrying out the methods described in various aspects above when the computer program is executed on a processor, and this computer program also enjoys the corresponding advantages. [Brief explanation of the drawing]

[0032] [Figure 1] This is a diagram showing a microscope system. [Figure 2] This figure shows multiple embodiments of a microscope system. [Figure 3] This figure shows multiple methods of fluorescence detection. [Figure 4] This figure shows a method according to one embodiment of the present invention. [Figure 5] This figure shows several other aspects of fluorescence detection. [Modes for carrying out the invention]

[0033] Conventional fluorescence microscopy uses filters to selectively capture a portion of the fluorescence emission spectrum of a given phosphor in order to increase specificity. However, when attempting to capture the fluorescence responses of multiple phosphors in this situation, classically, this requires switching filters, which is generally slow and therefore disadvantageous, for example, when observing moving samples.

[0034] More recently, microscopes with different detection channels for different wavelength ranges have also become known. For example, European Patent No. 3721279 discloses a microscope system having a detection unit adapted to detect the fluorescence response of a phosphor in different color channels using multiple detectors. In such systems, which will also be described below with reference to Figures 1 and 2, switching filters is no longer necessary to detect different color channels separately.

[0035] Fluorescence microscopes that use different detectors to provide detection of different color channels can also be provided as a system that allows the user to select between different microscope operating modes, including but not limited to wide-field, confocal, and light-sheet operating modes. Figure 1 shows a microscope system 1000 including such a microscope 100 in a more general way, and Figure 2 shows details of switching between wide-field and confocal operation, and the respective illumination and detection units in such a microscope 100. However, it should be noted that embodiments of the present invention are not limited to being used in connection with either a microscope having different detection channels or a microscope capable of operating in different microscope operating modes.

[0036] Regardless of whether filters or different detection channels are used, so-called “cross-excitation” and “cross-emission” can be observed. “Cross-excitation” is the phenomenon where a phosphor with a specific excitation wavelength may emit fluorescence to some extent when irradiated with excitation light of a different wavelength. “Cross-emission” is the effect of the fluorescence response, typically resulting in a spectrum that can be detected to some extent in an adjacent channel, rather than a sharp peak at a specific wavelength. In other words, using only filters or different detection channels can make clear separation between phosphors impossible, especially when different phosphors with relatively small differences between their excitation and emission wavelengths are used.

[0037] To address the problem of overlapping fluorescence emission, the aforementioned unmixing approaches have already been developed. These generally enable simultaneous imaging of multiple phosphors. An overview is provided, for example, in the paper “Spectral Imaging and Linear Unmixing in Light Microscopy” by T. Zimmermann, Adv. Biochem. Eng. Biotechnol. (2005) 95: 245-265. These unmixing approaches use the spectral distribution of the “pure” fluorescence emission of a single phosphor, such as one detected by a detector (a so-called “edge component spectrum” or “reference emission spectrum”), to reconstruct the spatial phosphor abundance from multiple detector images or contributions from different fluorescence emissions. These reference emission spectra can be measured using comparative reference data or calculated from literature values ​​and device sensitivity of the fluorescence emission spectrum.

[0038] However, with such unmixing methods, the quality of the unmixing result depends on how well the edge component spectra match the actual spectra of the phosphor in the sample. It is known that the emission spectra of phosphors can vary depending, in particular, on the chemical environment, target protein, illumination spectrum, or illumination history (i.e., especially "fading" or photoconversion effects). This variability can lead to suboptimal unmixing results, such as residual crosstalk between phosphor channels.

[0039] Before focusing specifically on solutions according to embodiments of the present invention, a microscope system usable in such embodiments will be described. As mentioned above, Figure 1 shows such a microscope system 1000. The microscope system 1000 may be configured to carry out the methods described herein. The microscope system 1000 includes a microscope 100 and a computer system 400. The microscope 100 is configured to take images and is connected to the computer system 400 by a wired or wireless communication path or interface unit 300. The microscope 100 can be configured to operate in different microscope operating modes, for example, a wide-field operating mode and a confocal operating mode, as further shown with reference to Figure 2. Although an upright microscope 100 is shown in Figure 1, embodiments of the present invention can be used in an inverted microscope, the details of which are shown in Figure 2, or in any other geometric arrangement of the microscope. For example, the components of a light-sheet microscope may be arranged horizontally.

[0040] The computer system 400 may be configured to carry out at least some of the methods described herein. The computer system 400 and the microscope 100, as well as the interface unit 300, which is entirely optional, may be separate entities but may be integrated within a single common housing. Although the computer system 400 is illustrated as a laptop computer, it may be part of the central processing system of the microscope 100, and / or the computer system 400 may be part of the dependent components of the microscope 100, such as sensors, actors, cameras, or illumination units of the microscope 100. Substantially the same applies to the interface unit 300.

[0041] The computer system 400 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors 440 and one or more storage devices 450, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed to various locations such as local clients and / or one or more remote server farms and / or data centers). The computer system 400 may include any circuit or combination of circuits.

[0042] In embodiments of the present invention, the computer system 400 may include, but is not limited to, one or more processors 440 incorporated into the housing of the computer system 400 shown in Figure 1. The processors 440 may be of any type, in any number and in any location, and may be provided in any part of the microscope system 1. As used herein, the term processor may refer to, but is not limited to, any type of computing circuit, such as a microprocessor, microcontroller, composite instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multicore processor, field-programmable gate array (FPGA) device, or any other type of processor or processing circuit in any part of the microscope 100 or microscope system 1000 (e.g., the camera). Other types of circuits that may be included in the computer system 400 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems.

[0043] The computer system 400 may include one or more storage devices 450, which may include one or more memory elements suitable for a specific purpose, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, and digital video discs (DVDs).

[0044] The computer system 400 may include a display device 410, one or more speakers, a keyboard 420, and / or one or more user interaction devices, which may be or include a mouse, or a trackpad 430 with buttons 432 and 434 as shown, a trackball, a touchscreen, a voice recognition device, or any other device that enables a user of the system to input information into and receive information from the computer system 400. As shown, the computer system 400 may be configured to display a graphical user interface 412 on the display device 410. The computer system 400 may further be configured to provide interactivity with the graphical user interface 412 for operating the microscope system 1000 via the keyboard and / or trackpad 430 and / or any other input device.

[0045] The microscope 100 is illustrated to have, among other things, a microscope housing 110, a focus adjustment knob 120, a transmitted light illumination unit 130, a stage 140 on which a sample 200 can be placed, at least one objective lens or lens 150, a dichroic unit 160, a tube 170 with an eyepiece or eyepiece set 172, an incident illumination unit 180 having, but not limited to, three different light sources 182-186 or illumination channels, and a camera or detection unit 190. The camera or detection unit 190 is illustrated to have, but not limited to, three different detectors, cameras or detection channels 192-196, as further shown in relation to Figure 2.

[0046] The light from the illumination unit 180 is coupled to the beam path shown by the dashed line by the dichroic 160, as shown by the dotted line. Another component of the embodiment of the microscope 100 is shown in Figure 2 as described above, and embodiments of the present invention are not limited to the specific configurations shown in Figures 1 and 2. The microscope may be capable of operating in different microscope operating modes, such as wide-field and confocal operating modes, via appropriate switching of light sources 182-186 or illumination unit 180 and detection modality in detection unit 190.

[0047] The computer system 400 and interface unit 300 may also be referred to as the microscope control device 500, but as used herein, this term is not limited to having the computer 400 and interface unit 300. As used herein, the term “microscope control device” should be understood in particular functionally and refers to one unit or group of units having one or more processors 440 and one or more storage devices 450 provided in the computer system 400, or otherwise configured to display a graphical user interface 412, for example, as described above and as further shown below in some embodiments.

[0048] Before moving on to a description of spectral unmixing according to embodiments of the present invention, Figure 2, described below, shows a portion of a fluorescence microscope 100 according to one embodiment of the present invention, which can be used, for example, as a fluorescence microscope 100 in the microscope system 1000 shown in Figure 1. The fluorescence microscope 100 shown in Figure 2 has two detection units, namely a first detection unit 190a and a second detection unit 190b. By a switchable or repositionable mirror 192 (indicated by double-headed arrows) or any other switching means, the observation light can be selectively coupled to either the first detection unit 190a or the second detection unit 190b. In the position shown in Figure 2, the observation light is coupled to the second detection unit 190b on the right via the mirror 192. When the mirror 192 is removed from the indicated position, as indicated by 192', the optical path further proceeds to the bottom (in the illustrated embodiment), and therefore the observation light is coupled to the first detection unit 190a, as indicated by the dotted border ray. Optical lenses in the illumination beam path and detection beam path are not specifically shown. Block filters or excitation / emission filters that can be provided in any conceivable manner are shown in 105.

[0049] In the embodiment shown in Figure 2, the first detection unit 190a is a wide-field detection unit, in which the image plane is indicated by 194a. By using a lens not specifically shown, the observation light is collimated and irradiated onto the detection device 196a in the first detection unit 190a. The detection device 196a may be configured to split the observation light into different detection channels. For a detection device 196a usable in the first detection unit 190a, see European Patent No. 3721279, in particular the detection unit 10 shown in Figure 3 of the same specification and the corresponding description. This disclosure is also incorporated herein by reference. The illumination unit for wide-field detection in the first detection unit 190a is simplified and shown by 180a. Its light can be coupled to the illumination beam path of the fluorescence microscope 100 in any manner known in the field of microscope illumination, for example, using a dichroic mirror 181a.

[0050] In the embodiment shown in Figure 2, the second detection unit 190b is a confocal detection unit, in which the image plane is indicated by 194b. A point light source 180b may be provided for confocal detection using the confocal detection unit 190b. The point light source 180b may be, in particular, a (single) pinhole aperture from which laser light can be focused, or the end of a light guide or optical fiber from which light is emitted as a point. The point light source 180b is conjugate to the intermediate image plane 194b and to the objective plane 210 in the sample 200, so that the illumination light can be scanned and focused to a point on the objective plane 210 using an X / Y scanner 185b, which can be positioned in the telecentric plane or a plane conjugate thereto, via a dichroic mirror 181b or any other spectral beam splitting device such as an acousto-optic beam splitter and an illumination optical system (not particularly shown). This allows the sample 200 to be scanned in the sample plane 21 as is generally known. The pinhole is shown in 187b. Again, for further details, see European Patent No. 3721279, in particular the detection unit 20 shown in Figure 3 of the same specification and its corresponding description. This disclosure is also incorporated herein by reference. The same applies to the detection device 196b, which can be configured to split the observation light into different detection channels. For the detection device 196b usable in the second detection unit 190b, see European Patent No. 3721279. This disclosure is also incorporated herein by reference.

[0051] Spectral unmixing techniques are applicable in conjunction with fluorescence microscopy using the fluorescence microscope 100 shown in Figures 1 and 2. Spectral unmixing is a technique that addresses the problem of overlapping emission spectra of phosphors as a result of “cross-excitation” or “bleeding” between different detection channels, such as detection channels 192–196 shown in Figure 1. These phenomena can lead to false positive results if not handled properly. The corresponding problem is particularly pronounced when the sample is labeled with three or more phosphors, even if detection is performed in different channels 192–196.

[0052] Spectral unmixing may include, in particular, linear unmixing, non-negative matrix factorization, deconvolution, and principal component analysis. The unmixing technique may be based on prior knowledge of the emission spectrum or can be used in connection with limiting the number of phosphors to less than or equal to the number of detection channels. At its core, and in the understanding used herein, spectral unmixing is a technique for decomposing a mixed multichannel image into spectral signatures and the abundance of each signature in each pixel.

[0053] Figure 3 illustrates the problem underlying embodiments of the present invention, in which detection in different detection channels, such as detection channels 192-196, is performed using a color beam splitter. Unlike Figure 1, the number of detection channels shown in Figure 3 is four, but substantially the same problem is observed when any other number of detection channels are used. Embodiments of the present invention are applicable to any number of detection channels. Figure 3 shows a quantum efficiency diagram in which quantum efficiency is plotted on the y-coordinate with respect to wavelength in nanometers on the x-coordinate. W1, W2, W3, and W4 represent wavelength components detectable by different spectral channels of the corresponding color beam splitter device, i.e., the "splitting" of colors between different channels is approximately performed at the wavelength where the graphs shown for adjacent wavelength components of wavelength components W1, W2, W3, and W4 intersect. W1 represents the blue channel, W2 represents the green channel, W3 represents the orange channel, and W4 represents the red channel.

[0054] In Figure 3, the spectra of various fluorescent dyes are shown by abbreviations known to those skilled in the art. As shown in the figure, the fluorescence response of DAPI has a maximum value in channel W1, while a considerable portion is still detectable in channel W2, where the maximum value for GFP is detectable. That is, if the sample is colored with these phosphors, these phosphors cannot be completely separated by using different detection channels, and the emission spectra overlap in channel W2. In other words, DAPI "bleeds" considerably into channel W2, requiring unmixing. The phosphor dsRed, for example, is detected in almost equal portions in channels W2 and W3, and therefore, a portion of its fluorescence response is detected in these channels.

[0055] Generally, to enable spectral unmixing, the reference emission spectrum can be measured directly in a reference sample containing only one of the phosphors of the sample of interest. However, this approach is extremely time-consuming because the user must prepare and then measure an additional sample. Furthermore, this approach is prone to errors because it requires user intervention (for reference sample preparation and definition of the region of interest) and does not account for inter-sample variability that can occur even with careful sample preparation. Using algorithms to identify the reference emission spectrum is not currently user-friendly because it typically requires acquiring and tracking multiple images (sometimes with different illumination and / or microscope settings) and using them as input to the algorithm. Moreover, the reconstructed spectrum is not immediately valid and must be manually entered into the unmixing workflow. For these reasons, such algorithms are often applied retrospectively and only by experts in conventional processes.

[0056] However, the methods according to several embodiments of the present invention, described below, enable more reliable and user-friendly identification of a reference emission spectrum used for spectral unmixing of fluorescence microscope images. This is achieved with minimal user intervention by automatically acquiring the necessary images, performing spectral identification, selectively checking the validity of the spectral identification, and selectively providing the user with feedback on the results of the spectral identification. Since this method can be performed directly on samples to be imaged later, no additional reference samples are required.

[0057] Figure 4 shows a method according to one embodiment of the present invention, which is shown in the form of a simplified flowchart and referred to as method 10. Method 10 is provided for analyzing the mixed fluorescence response of multiple phosphors in a microscope sample 200, where, as repeatedly stated, the mixed fluorescence response has contributions from the individual fluorescence responses of each of the multiple phosphors. The reasons that may lead to this situation have already been explained in great detail previously.

[0058] Method 10 involves reconstructing the individual fluorescence responses described above from the mixed fluorescence response using spectral unmixing, based on a reference emission spectrum for the phosphor to be reconstructed. As described above, the identification of the reference emission spectrum to be used for spectral unmixing is carried out in an advantageous manner according to embodiments of the present invention.

[0059] For this purpose, according to one embodiment of the present invention, a procedure is provided for identifying and verifying a reference emission spectrum, the steps of which are described below. According to one embodiment of the present invention, such a procedure may be initiated by a user request, for example, by pressing a button when the displayed image does not meet expectations. In one embodiment of the present invention, the corresponding procedure may also be requested depending on the results of the image analysis, for example, when the unmixed image has a large number of adjacent pixels with the same intensity, or when the calculation of the correlation coefficient between unmixed images gives a value that exceeds a certain threshold. The corresponding request, or step of the process, is shown as 1 in Figure 4.

[0060] In other words, according to one embodiment of the present invention, the above procedure for identifying and verifying a reference emission spectrum may be performed in response to a calibration request 1, which is at least one of a calibration request made by a user, a calibration request made based on the results of an image analysis step, a calibration request made based on a predetermined time, and a calibration request made when a change in the field of view is identified, and according to one embodiment of the present invention, this may be performed when the image analysis shows that the number of adjacent pixels in a predetermined intensity range and / or a value corresponding to a correlation coefficient calculated between unmixed images exceeds a predetermined threshold.

[0061] In the next step, step 2, according to one embodiment of the present invention, the required image sequence is identified. Generally, this is a sequence having a specific illumination setting for each image (e.g., an activated light-emitting diode, or an activated laser having a given center wavelength). The number of entries in the sequence is generally greater than or equal to the number of phosphors in the sample. In other words, according to one embodiment of the present invention, step 2 may include supplying multiple image acquisition settings for a sample 200, which may be equal to or greater than a plurality of phosphors, and including an illumination setting for each image sequence.

[0062] According to one embodiment of the present invention, step 3 may then involve acquiring an image sequence and storing it together with the corresponding lighting settings. In other words, according to one embodiment of the present invention, step 3 may include acquiring an image sequence of sample 200 using a plurality of image acquisition settings and storing each image sequence of sample 200 together with the corresponding lighting settings.

[0063] According to one embodiment of the present invention, preferably, the illumination intensity of the illumination source can be set to the previously identified optimal value, or more generally, the above procedure of identifying and verifying a reference emission spectrum can be performed in at least two temporal instances, including a preceding instance and a succeeding instance, and the illumination settings used in the succeeding instance may be based on the illumination settings used in the preceding instance, in particular, if the validation check described below was yes in the preceding instance.

[0064] According to embodiments of the present invention, an automatic illumination intensity control step may be included, if necessary, to provide an illumination setting optimized in terms of response and sample protection.

[0065] According to one embodiment of the present invention, step 4 may involve executing an algorithm for identifying edge component spectra from the acquired images. Typical algorithms include N-FINDR, PPI, non-negative matrix factorization or tensor decomposition, or VCA. Furthermore, this algorithm can be executed by a trained neural network or any optimization algorithm. Here again more generally, in one embodiment of the present invention, step 4 may involve identifying candidate reference emission spectra for the phosphor to be reconstructed from the image sequence of sample 200 using one or more reference emission spectrum identification algorithms.

[0066] Step 5, which may constitute part of a method according to one embodiment of the present invention, allows for a validity check of the identified edge component spectrum. The result of the validity check can indicate whether the reference emission spectrum candidate is valid for the phosphor to be reconstructed. As shown by steps 6 and 7, the above reference emission spectrum candidate is usable as a reference emission spectrum in spectral unmixing if the result of the validity check is yes, as indicated by Y, or the above reference emission spectrum candidate is rejected if the validity check is no, as indicated by N.

[0067] According to one embodiment of the present invention, a reasonable check is performed by identifying the edge component spectrum S det (λ k ) and the initial edge component spectrum S init (λ k ) and the degree to which they differ is sufficient. Here, λ k This gives the spectral position of the k-th spectral detector. This difference is, for example, the Euclidean distance.

number

number

[0068] Another check used according to embodiments of the present invention is, for example, a check for non-negativity (all λ k Regarding S det (λ k )≧0), or a check on the spectral separation of the identified reference emission spectrum candidates (including being unique for all such reference emission spectrum candidates).

[0069] If the identified edge component spectra are found to be valid, according to one embodiment of the present invention, they can be saved and used for future image acquisition for an unmixing step, including for live imaging. This unmixing step may be, for example, linear unmixing, phasor unmixing, a maximum likelihood estimator, or a hybrid approach. The saved edge component spectra can also be stored in the metadata of the captured image. In other words, in one embodiment of the present invention, the validation check performed in subsequent instances as already described above may include performing an evaluation based on a reference emission spectrum acquired in a preceding instance, and / or if the result of performing the above validation check is yes in the preceding instance, in one embodiment of the present invention, the above reference emission spectrum candidate can be stored for use in the above validation check in the subsequent instance.

[0070] If the identified edge component spectrum is not valid, in one embodiment of the present invention, feedback can be given to the user. For example, the user can be prompted to check whether the given phosphor is present in the imaged portion of the sample, whether the sample contains additional phosphors not specified, or whether the wrong phosphor has been specified. The user can also be asked whether the identified reference emission spectrum should be used. In the simplest case, if the validity of the identified edge component spectrum is not given, the initial reference emission spectrum is used.

[0071] In any case, and regardless of whether a validation check is performed, in embodiments of the present invention, step 6 may include conditionally using the above-mentioned candidate reference emission spectrum as the reference emission spectrum in the spectral unmixing described above.

[0072] Some embodiments of the present invention, partially described previously, may include the possibility of automatically requiring spectral calibration at a fixed temporal point (particularly during the acquisition of a series of images at different time points) to account for changes in the edge component spectrum due to photobleaching, photoconversion, changes in autofluorescence, changes in environmental conditions, etc. To account for changes in the edge component spectrum due to spatial variations, spectral calibration may be automatically required when the imaged field of view is changed. For validation, different thresholds can be used depending on the phosphor. In particular, if a given phosphor is an autofluorescent species, a higher threshold can be selected than for a non-autofluorescent species.

[0073] Many phosphors are particularly sensitive to changes in the acidity of their surroundings, such as those measured by pH values. If the changes in the spectral properties of a given phosphor are known (or measurable) as a function of pH, then knowledge of the changes in the edge component spectra of this phosphor can be used, for example, as a function of time, to measure the changes in pH. This itself can be used in other aspects of microscopy, e.g., to control environmental control (incubator), or to provide feedback to the user, as provided according to embodiments of the present invention. Since there are also physiological reasons that can cause pH changes, identifying these changes can be the subject of user research and embodiments of the present invention.

[0074] According to one embodiment of the present invention, the method can be extended to eliminate the need for knowledge of the initial spectrum (e.g., via user input). In this case, the number of spectra may be calculated from data in multiple images by an algorithm such as the Harsanyi-Farrand-Chang noisy whitening method and by knowledge-based inference, such as the initial edge component spectrum given by a Gaussian function. The identification of the edge component spectrum is then carried out according to the description above.

[0075] Embodiments of the present invention are particularly applicable to fluorescence microscopes equipped with multiple illumination sources and multiple detectors having known spectral characteristics, as already described in the embodiments relating to Figures 1 and 2. Multiple embodiments are shown relating to Figure 5, where the fluorescence microscope is indicated by 100 and the sample by 200, as described above. The phosphors numbered m=0,...,M-1 in sample 200 are generally referred to by 230, where c indicates their respective abundance. The illumination sources numbered k=0,...,K-1 are generally referred to by 180, as described above, where P indicates their respective illumination power. The detection channels numbered n=0,...,N-1 are generally referred to by 190, where I indicates the detection intensity. The detection path is referred to by 162 and the illumination path by 164. For further details, please refer to the descriptions already given relating to Figures 1 and 2 above.

[0076] As used herein, the term "and / or" includes all possible combinations of one or more of the items listed herein and may be abbreviated as " / ".

[0077] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus.

[0078] Some or all of the steps may be performed by a hardware device (or by using a hardware device), such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most critical steps may be performed by such a device.

[0079] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is feasible using a non-transient recording medium, which is a digital recording medium, etc., that stores electronically readable control signals and cooperates (or can cooperate) with a programmable computer system to carry out each method. Examples include floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memory. Thus, the digital recording medium may be computer-readable.

[0080] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so as to carry out any of the methods described herein.

[0081] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0082] Another embodiment includes a computer program stored in a machine-readable carrier for carrying out any of the methods described herein.

[0083] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.

[0084] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for carrying out any of the methods described herein when executed by a processor. The data carrier, digital recording medium, or recording medium is typically tangible and / or non-transient. Another embodiment of the present invention is an apparatus, such as those described herein, comprising a processor and a recording medium.

[0085] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0086] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to carry out any of the methods described herein.

[0087] Another embodiment includes a computer having an installed computer program for carrying out any of the methods described herein.

[0088] Another embodiment of the present invention includes an apparatus or system configured to transfer a computer program for carrying out any of the methods described herein (e.g., electronically or optically) to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0089] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to carry out any of the methods described herein. Generally, the methods are advantageously carried out by any hardware device. [Explanation of Symbols]

[0090] 10 ways 1. Calibration Requirements 2. Supply of multiple image acquisition settings 3. Acquisition of image sequence 4. Identification of candidate reference emission spectra 5. Validity Check 6. Conditional use of reference emission spectrum candidates 7. Rejection of the reference emission spectrum 500 Microscope Control Device 100 Microscopes 105 Block Filters 110 Microscope Housing 120 Focus adjustment knob 130 Transmitted light illumination section 140 Microscope Stages 150 Microscope Objective Lenses 160 Dichroic Elements 170 Microscope Tubes 172 Eyepiece Set 180 Incident Illumination Unit 182~186 light source 180a, 180b Wide-field illumination unit and confocal illumination unit 181a, 181b Dichroic mirror 185b X / Y Scanner 187b Pinhole 190 detection units 192-196 Detectors, detection channels 190a, 190b Wide-field detection unit and confocal detection unit 192 Adjustable Mirror 192' Relocated mirror position 194a, 194b Wide-field image plane and confocal image plane 196a, 196b Wide-field detection device and confocal detection device 200 samples 210 Objective surface 300 Interface Units 400 Computer Systems 410 displays 420 keyboards 430 Trackpad 432 First button 434 Second button 440 processors 450 storage devices 1000 Microscope Systems W1~W4 Detection Channel Spectra

Claims

1. A method (10) for analyzing a mixed fluorescence response of a plurality of fluorophores (230) in a microscopic sample (200), the mixed fluorescence response having contributions from individual fluorescence responses of each of the plurality of fluorophores (230), the method (10) comprising reconstructing the individual fluorescence responses from the mixed fluorescence response using spectral unmixing based on reference emission spectra for the fluorophores to be reconstructed, the method (10) comprising a step of identifying and validating the reference emission spectra, the step comprising the steps of: a) providing a plurality of image acquisition settings for image sequences of the sample (200) equal to or greater than a plurality of the phosphors (230), including an illumination setting for each of the image sequences; b) acquiring said image sequences of said sample (200) using a plurality of said image acquisition settings and storing each said image sequence of said sample (200) together with the corresponding said illumination setting; c) identifying (4) candidate reference emission spectra for the fluorophores to be reconstructed from the sequence of images of the sample (200) using one or more reference emission spectrum identification algorithms; d) conditionally using the candidate reference emission spectrum as the reference emission spectrum in the spectral unmixing; It contains steps including Method (10).

2. The method (10) comprises: i) between steps c) and d) a step (5) of performing a plausibility check on the candidate reference emission spectrum, the result of which indicates whether the candidate reference emission spectrum is valid for the phosphor to be reconstructed; ii) if the result of the plausibility check is yes, then according to step d), using the candidate reference emission spectrum as the reference emission spectrum in the spectral unmixing, or if the result of the plausibility check is no, then rejecting the candidate reference emission spectrum (6); further comprising: The method (10) of claim 1.

3. the illumination settings are linearly independent and / or the illumination settings for each of the image sequences include selecting a single light source from a plurality of light sources, the single source including a light emitting diode, a laser of a given center wavelength, or a wavelength selection device. The method (10) of claim 1.

4. performing the step of identifying and validating the reference emission spectrum at at least two temporal instances, including an earlier instance and a later instance; The method (10) of claim 2.

5. wherein the step (5) of performing the plausibility check in the subsequent instance includes performing an evaluation based on the reference emission spectrum obtained in the previous instance, and / or, if the result of the step (5) of performing the plausibility check in the previous instance is yes, storing the reference emission spectrum candidate for use in the plausibility check in the subsequent instance, and / or, if the plausibility check was yes in the previous instance, the lighting settings used in the subsequent instance are based on the lighting settings used in the previous instance. The method (10) of claim 4.

6. The image acquisition settings include detector settings or camera settings selected based on sample characteristics, the sample characteristics including at least one of the motility or sensitivity of the sample, and the detector settings or camera settings including at least one of a binning factor, a scanning speed, a scanning resolution, a gain value, and an exposure time. The method (10) of claim 1.

7. identifying the mixed fluorescent response using a fluorescence microscope (100), the fluorescence microscope (100) being configured to operate in at least one microscope mode selected from a wide-field mode, a confocal mode, and a light sheet mode, and performing the step of identifying and validating the reference emission spectrum for at least one of the microscope modes; The method (10) of claim 1.

8. performing the procedure of identifying and verifying the reference emission spectrum in response to a calibration request (1), the calibration request (1) being at least one of a calibration request made by a user, a calibration request made based on the results of an image analysis step, a calibration request made based on a predetermined time point, and a calibration request made when a change in field of view is identified; The method (10) of claim 1.

9. providing a reference excitation spectrum for use in step (4) identifying the candidate reference emission spectra; The method (10) of claim 1.

10. The validity check for the candidate reference emission spectrum includes determining at least one of a Euclidean distance from a given reference emission spectrum, a spectral angle with the given reference emission spectrum, and non-negativity for the candidate reference emission spectrum. The method (10) of claim 2.

11. the validity check includes comparing the validity value to a validity threshold, the validity threshold being different for each of the fluorophores; The method (10) of claim 2.

12. using values ​​representing the deviation of the candidate reference emission spectra from expected values ​​to derive sample characteristics and / or deriving the number of fluorophores and the spectra of the fluorophores from the composite fluorescence response; The method (10) of claim 1.

13. 1. A fluorescence analyzer (500) configured to analyze a mixed fluorescence response of a plurality of fluorophores (230) in a microscopic sample (200), the mixed fluorescence response comprising contributions of individual fluorescence responses of the plurality of fluorophores (230), the analysis comprising reconstructing the individual fluorescence responses from the mixed fluorescence response using spectral unmixing based on reference emission spectra for the fluorophores to be reconstructed, and configured to perform a procedure for identifying and validating the reference emission spectra, the procedure comprising the steps of: a) providing a plurality of image acquisition settings for image sequences of the sample (200) equal to or greater than a plurality of the phosphors (230), including an illumination setting for each of the image sequences; b) acquiring said image sequences of said sample (200) using a plurality of said image acquisition settings and storing each said image sequence of said sample (200) together with the corresponding said illumination setting; c) identifying (4) candidate reference emission spectra for the fluorophores to be reconstructed from the sequence of images of the sample (200) using one or more reference emission spectrum identification algorithms; d) conditionally using the candidate reference emission spectrum as the reference emission spectrum in the spectral unmixing; Contains Fluorescence analyzer (500).

14. The fluorescence analyzer (500) is configured to perform the procedure for identifying and validating the reference emission spectrum, which procedure further comprises the following steps: i) between steps c) and d) a step (5) of performing a plausibility check on the candidate reference emission spectrum, the result of which indicates whether the candidate reference emission spectrum is valid for the phosphor to be reconstructed; ii) if the result of the plausibility check is yes, then according to step (d), using the candidate reference emission spectrum as the reference emission spectrum in the spectral unmixing, or if the result of the plausibility check is no, then rejecting the candidate reference emission spectrum (6); Including, The fluorescence analyzer (500) of claim 13.

15. A fluorescence microscope (100), comprising: The fluorescence microscope (100) is configured to analyze the mixed fluorescence response of a plurality of fluorophores (230) in a microscope sample (200), and comprises a fluorescence analyzer (500) according to claim 13 or 14. Fluorescence microscope (100).

16. A computer program comprising instructions adapted to perform a method according to any one of claims 1 to 12.

Images