Multiple fluorescent dye quantification method and device

Abstract

Provided are a method and a device for rapidly and stably quantifying the spectral intensity of fluorescent emissions by increasing the number of fluorescent dyes that can be used simultaneously. The device comprises a filter control unit 2, a filter (spectral modulator) 3, a sensor array 4, a storage unit 5, and a regression model 6. The filter control unit 2 controls the spectral modulator 3. The spectral modulator 3 modulates spectral transmittance, and includes a first polarizer 32a, a liquid crystal variable retarder 31, and a second polarizer 32b. The sensor array 4 observes an observed object 8 including a plurality of fluorescent dyes through the spectral modulator 3. The storage unit 5 stores the luminance values of camera pixels captured under N types of modulation, as N-dimensional feature vectors. The regression model 6 is trained using the feature vectors under the modulation of each fluorescent dye, and quantifies the fluorescent dyes to reconstruct the intensity image 9 of each fluorescence.

Description

Method and apparatus for quantifying multiple fluorescent dyes

[0001] The present invention relates to a technology that can quantify multiple fluorescent dyes, and in particular to a technology that can simultaneously utilize and distinguish multiple fluorescent dyes in bioimaging such as spatial omics analysis, which quantitatively analyzes the gene expression of each cell using tissue sections.

[0002] In recent years, spatial transcriptome analysis, a type of spatial omics analysis, has become popular, enabling the analysis of gene expression profiles while preserving location information within tissues. This approach has been applied in fields such as developmental biology and drug discovery. In spatial transcriptome analysis, RNA molecules can be detected by hybridizing them with fluorescently labeled DNA probes. However, the simultaneous detection of multiple mRNAs presents limitations due to overlapping fluorescence spectra. Sequential RNA fluorescent molecular localization, a widely used method for spatial transcriptome analysis, observes RNA as a fluorescent molecule, revealing its molecular-level location. However, the use of absorption filters limits the number of dyes that can be obtained in a single staining procedure, necessitating multiple staining and imaging steps to identify multiple molecules.

[0003] On the other hand, the SeqFish+ method is known, which efficiently detects a large number of mRNAs by repeated molecular hybridization and deprobing. In the SeqFish+ method, since it is difficult to distinguish between 20 fluorescent dyes using a fluorescence microscope, a large number of mRNAs are identified by barcoding (reading sequences in which the glowing patterns are uniquely designed for each mRNA) using 20 types of pseudocolors obtained by 20 hybridizations. For example, 80 images can be taken to analyze the localization of approximately 10,000 types of mRNA. However, each hybridization takes just under 20 minutes, and each hybridization requires experimental manipulation, which results in the long and laborious process of repeated data acquisition (hybridization and reprobing).

[0004] Also known is a nucleic acid sequencing method including a spatial modulation module that uses a spatial random phase modulator to randomly modulate a light field to obtain a speckle image of a fluorescent signal, a liquid crystal spatial light modulator that constructs a specific two-dimensional code matrix, and an area array detector that detects a multicolor fluorescent two-dimensional intensity measurement matrix (see Patent Document 1). Patent Document 1 describes a method for preparing a nucleic acid chip having a spatial distribution using a fluorescent probe-labeled nucleic acid sequence, which emits multiple color fluorescent signals upon optical excitation. The emitted multiple color fluorescent signals are sequentially modulated, encoded, and collected using an imaging module and an area array probe to obtain a fluorescent two-dimensional intensity measurement matrix Y. A reconstruction algorithm then associates a scaling matrix A, which serves as prior information, with the fluorescent two-dimensional intensity measurement matrix Y to calculate y = A·X, thereby reconstructing a target signal X, i.e., the fluorescent molecular space, spectrum, and intensity distribution information of the nucleic acid sequence. In the invention disclosed in Patent Document 1, an area array probe detects fluorescence speckle signals (signals generated by coherent light diffusely reflected at each point on the surface where light is irradiated, and the light speckle signal image reconstruction algorithm is based mainly on a reconstruction algorithm related to the speckle field. Thus, the invention disclosed in Patent Document 1 does not have a technical idea of ​​using a modulated fluorescence spectrum as a feature quantity.

[0005] Furthermore, Non-Patent Document 1 discloses an imaging system for an advanced spectroscopic microscope equipped with a liquid crystal variable retarder and two polarizers sandwiching it. In this system, a spectral image cube is restored from a series of spectrally modulated images using compressed sensing based on the liquid crystal variable retarder and a sensor array, and a spatial spectral image cube is reconstructed from fewer spectral scans than required using conventional systems. Non-Patent Document 2 discloses a system that captures hyperspectral images with fewer measurements than required by conventional systems, and discloses a convolutional neural network (CNN) for reconstructing a hyperspectral cube captured by compressed sensing based on spectral modulation. Non-Patent Document 3 discloses a technology that applies a spatially random phase modulator to a wide-field microscope to enable random measurement of fluorescent signals.

[0006] Furthermore, Patent Document 2 discloses spectral analysis in which spectral information of a sample is input into a learning model for estimation. Spectral analysis involves detecting a response when a sample is given a stimulus, and obtaining information about the components that make up the sample (spectral information) based on the obtained signal. Spectral information includes the intensity of electromagnetic waves, including light, that characterize the stimulus and response, as well as temperature, mass, and the number of counts of fragments with a specific mass. However, spectral information cannot be obtained by observing the intensity of light passing through a filter whose spectral transmittance is changed (modulated) over time.

[0007] Furthermore, Patent Document 3 discloses an optical system for an optical filter and a solid-state image sensor. In an embodiment of Patent Document 3, a dichroic optical filter is shown, which is used to divide the fluorescence emission spectrum of a fluorescent agent into two spectral channels (e.g., first and second spectral bands). This dichroic optical filter, like a dichroic mirror, reflects light of a specific wavelength and transmits light of other wavelengths, and simply transmits or reflects light depending on the wavelength. Patent Document 3 does not disclose any description of quantifying a fluorescent dye using a trained regression model.

[0008] Patent Document 4 also discloses a process for calculating spectral data of a measurement object by acquiring spectral data from each of multiple unit areas multiple times and averaging the acquired spectral data from at least one unit area on the measurement object and a unit area adjacent to the unit area. The imaging means in Patent Document 4 uses a tunable filter provided in front of multiple pixels to change the transmission wavelength over time, thereby acquiring spectral data from each of multiple unit areas on the measurement object at multiple pixels. The tunable filter in Patent Document 4 changes the transmission wavelength over time, but does not change the spectral transmittance over time, which is clearly different from modulation. For example, a tunable filter with a 10 nm width has a transmittance of approximately 2% for the entire wavelength range from 400 to 1000 nm. Like Patent Document 3, Patent Document 4 does not disclose any mention of quantifying fluorescent dyes using a trained regression model.

[0009] International Publication Pamphlet WO2022 / 057584 JP 2020-101524 A JP 2017-534361 A JP 2016-80429 A

[0010] T. Yang et al., “Compressive hyperspectral microscopic imaging using spectral-coded illumination”, Optics & Laser Technology Vol.166, 109631 (2023). D. Gedalin et al., “DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks”, Optics Express Vol.27, Issue 24, pp.35811-35822 (2019).W.Li et al., “Single-frame wide-field nanoscopy based on ghost imaging via sparsity constraints”, Optica Vol.6, Issue 12, pp.1515-1523 (2019).

[0011] As mentioned above, in spatial transcriptome analysis, RNA molecules are detected by molecular hybridization with fluorescently labeled probes. However, if multiple fluorescent dyes can be used simultaneously for identification, various RNA molecules can be detected simultaneously, thereby improving the efficiency of analysis. In other words, even if fluorescent dyes with overlapping fluorescence spectra are used simultaneously, if multiple fluorescent dyes can be quantified, the analysis efficiency can be improved. Furthermore, in the sequential RNA fluorescent molecular localization method, the number of dyes that can be obtained in a single staining operation is limited due to the use of an absorption filter. However, if multiple fluorescent dyes can be quantified, the time required from sample acquisition (staining operation) to data analysis can be significantly reduced. Furthermore, in the SeqFish+ method, a large number of mRNAs are identified and analyzed by barcoding using 20 types of pseudocolors through 20 hybridizations. However, if the number of fluorescent dyes that can be used simultaneously can be increased without requiring time and effort for data acquisition, the analysis efficiency can be improved.

[0012] In view of the above circumstances, an object of the present invention is to provide a method and apparatus for quantifying multiple fluorescent dyes, which increases the number of fluorescent dyes that can be used simultaneously and quantifies the spectral intensity of fluorescent emission quickly and stably.

[0013] To solve the above problems, a first aspect of the present invention provides a method for quantifying multiple fluorescent dyes, in which an observation target containing multiple fluorescent dyes is observed through a filter with modulated spectral transmittance, and the intensity integrated in the wavelength direction is observed using a sensor array. Then, a group of observations obtained under N types of modulation (N is an integer of 2 or greater) is treated as an N-dimensional feature vector, and a regression model trained on the feature vectors for each of the fluorescent dyes under the same N types of modulation is used to quantify the fluorescent dyes and reconstruct an intensity image of each fluorescence.

[0014] Here, the fluorescence spectrum of each fluorescent dye to be observed is transmitted through a filter with modulated spectral transmittance according to the filter's spectral transmittance. The light passing through this filter is integrated in the wavelength direction by a sensor array, and the intensity is measured. Because the intensity varies depending on the spectral transmittance and the fluorescence spectrum, the observed values ​​obtained here change depending on the modulation of the spectral transmittance and the spectral distribution of the fluorescence spectrum, resulting in multiple observations. This set of observations is treated as a vector and used as a feature vector for quantifying the fluorescent dyes. The amount of each fluorescent dye is quantified to estimate the fluorescence intensity. To increase the information content of the observations, the spectral transmittance of the filter is modulated so that the spectral distribution of each fluorescent dye is a combination of different spectral transmittances. By observing under various spectral transmittances, the observed intensity vector is unique to each fluorescent dye and can be used as a feature vector for quantifying fluorescence. Regression models are also supervised learning models that make predictions about unknown cases (observations).

[0015] According to the present invention, fluorescent dyes can be quantified using feature vectors, thereby significantly reducing computer memory and computational costs. Furthermore, because fluorescent dyes can be quantified without measuring the sample's exact spectral distribution, a general camera (one that can observe intensity integrated in the wavelength direction using a sensor array) can be used, eliminating the need for a multispectral or hyperspectral camera. Furthermore, unlike the use of a hyperspectral camera, the spectral transmittance can be maintained at approximately 40%, enabling even weak fluorescence to be observed with low noise while maintaining its strong intensity. In the multiple fluorescent dye quantification method according to the first aspect of the present invention, the spectral transmittance is preferably 20% to 40%. Note that, in this specification, the wavelength of light used for fluorescent dyes is not limited to visible light, but can also be used in wavelengths ranging from ultraviolet light to infrared light (including near-infrared light).

[0016] The training in the multiple fluorescent dye quantification method according to the first aspect of the present invention is preferably sparse modeling. Sparse modeling has the advantage of enabling stable regression estimation even from a small number of features by assuming sparseness (i.e., that the observed target can be expressed by the superposition of a small number of factors) for the composition of the observed target. This reduces the time and cost required for observation and preparation to obtain feature vectors, and training is possible with a small amount of data to be processed. Furthermore, the training is preferably performed using the feature vectors of each fluorescent dye contained in the observed target as a dictionary matrix. For example, in the case of spatial transcriptome analysis, multiple fluorescent dyes can be quantified by using sparse modeling, in which feature vectors generated from the spectral distributions of fluorescent dyes used in molecular hybridization are used as a dictionary matrix. In other words, by assuming that the number of fluorescent dyes observed at the same pixel is small, stable quantification can be achieved. The training in the multiple fluorescent dye quantification method according to the present invention may also be performed by training a regression model using a neural network.

[0017] In the multiple fluorescent dye quantification method according to the first aspect of the present invention, the filter is preferably a spectral modulator, and the modulation of the spectral transmittance is preferably performed by controlling the voltage applied to the spectral modulator. Note that an electro-optical element may also be used as the spectral modulator. Furthermore, the above filter is preferably composed of, for example, a liquid crystal variable retarder and two polarizers sandwiching the liquid crystal variable retarder. A desired retardation (phase difference) can be obtained depending on the voltage applied to the nematic liquid crystal of the liquid crystal variable retarder. This is because there is no need to prepare a wave plate for each wavelength, and the retardation can be quickly switched, allowing for rapid polarization change and modulation of the spectral transmittance.

[0018] The voltage applied to the spectral modulator to observe the feature vector is preferably selected based on the spectral distribution of the fluorescent dye. It is also possible to obtain an optimal voltage waveform using techniques such as cosine similarity or deep learning. By controlling the spectral modulator using the obtained voltage waveform, a feature vector capable of quantifying the fluorescent dye can be obtained. For example, the voltage applied to observe the feature vector based on the spectral distribution of the fluorescent dye can be selected by selecting the minimum set of voltage candidates that maintains sufficient fluorescence quantification accuracy from a sufficiently large set of voltage candidates, or by randomly determining N types of voltages (N-dimensional voltage vectors) in advance and using optimization techniques such as machine learning or gradient descent to determine the best voltage vector that maximizes quantification accuracy.

[0019] In the multiple fluorescent dye quantification method according to the first aspect of the present invention, the number of dimensions of the feature vector may be such that the quantitative accuracy of the reconstructed fluorescence exceeds a predetermined value. The feature vector of the observed image is expressed in the form of a vector whose elements are pattern information of the luminance values ​​(pixel values) of each pixel of the image. When the fluorescence spectrum of the observed image is observed through a filter with modulated spectral transmittance and the intensity integrated in the wavelength direction using a sensor array, the feature vector changes with each change in spectral transmittance. The observation group obtained under N types of modulation is defined as an N-dimensional feature vector. The number of dimensions may be such that the quantitative accuracy of the reconstructed fluorescence exceeds a predetermined value. For example, when evaluating the quantitative accuracy of each fluorescence at each pixel, the smallest value among those objective quality evaluation values ​​such as the peak signal-to-noise ratio (PSNR) and the signal-to-noise ratio (SNR) exceed predetermined values ​​may be used.

[0020] In the method for quantifying multiple fluorescent dyes according to the first aspect of the present invention, a multi-order wave plate may be further provided in front of the filter whose spectral transmittance is changed over time. By providing the multi-order wave plate, it is possible to perform observations with a high frequency offset, thereby enabling more accurate quantification of the fluorescent dyes.

[0021] A multiple fluorescent dye quantification method according to a second aspect of the present invention performs modulation processing on an observation target containing multiple fluorescent dyes, observing the intensity integrated in the wavelength direction using a sensor array through a filter whose spectral transmittance is changed over time; the group of observations obtained under N types of modulation (N is an integer of 2 or greater) is treated as an N-dimensional feature vector; and a demodulation processing is performed using a regression model trained on the feature vectors under modulation for each fluorescent dye to quantify the fluorescent dyes and reconstruct an intensity image of each fluorescence.

[0022] In the multiple fluorescent dye quantification method according to the second aspect of the present invention, the learning is preferably sparse modeling. Furthermore, the learning preferably uses feature vectors for each of the fluorescent dyes included in the observation target as a dictionary matrix. Here, in the multiple fluorescent dye quantification method according to the second aspect of the present invention, the spectral transmittance is preferably 20% to 40%. In the multiple fluorescent dye quantification method according to the second aspect of the present invention, the filter is preferably a spectral modulator, and the modulation of the spectral transmittance is preferably performed by controlling the voltage applied to the spectral modulator. Furthermore, the filter is preferably composed of, for example, a liquid crystal variable retarder and two polarizers sandwiching the liquid crystal variable retarder. In the multiple fluorescent dye quantification method according to the second aspect of the present invention, the voltage applied to the spectral modulator for observing the feature vector is preferably selected in accordance with the spectral distribution group of the fluorescent dyes. In the multiple fluorescent dye quantification method according to the second aspect of the present invention, the number of dimensions of the feature vector may be such that the quantitative accuracy of the reconstructed fluorescence exceeds a predetermined value. In the multiple fluorescent dye quantification method according to the second aspect of the present invention, a multi-order wave plate may be further provided in front of the filter whose spectral transmittance is changed over time.

[0023] The multiple fluorescent dye quantification device of the present invention comprises a filter that modulates spectral transmittance, a filter control unit that controls the filter, a sensor array that observes the intensity of an object containing multiple fluorescent dyes through the filter, a memory unit that stores the observed values ​​of the sensor array observed under N types of modulation (N is an integer of 2 or more) as an N-dimensional feature vector, and a regression model that learns using the feature vector under modulation for each fluorescent dye, quantifies the fluorescent dyes, and reconstructs an intensity image of each fluorescence.

[0024] The filter control unit selects a group of voltages to be used for observation depending on the type of fluorescent dye, and modulates the spectral transmittance.

[0025] The present invention has the advantage of increasing the number of fluorescent dyes that can be used simultaneously, enabling rapid and stable quantification of the spectral intensity of fluorescent emission. In particular, in spatial omics analysis, the number of staining and imaging steps can be reduced, leading to shorter instrument operating times and reduced experimental costs. Furthermore, the present invention is applicable not only to spatial omics analysis but also to any bioimaging field that uses fluorescence technology, making it applicable to a variety of applications. Furthermore, while current techniques require multiple staining and imaging steps, which pose a risk of sample quality degradation during the procedure, the present invention enables the detection of multiple dyes with a single staining and imaging step, thereby maintaining sample quality for a longer period of time. The present invention is compatible with new fluorescent dyes, allowing for the incorporation of new fluorescent dyes as new dyes are developed, making it highly adaptable. Furthermore, the present invention allows fluorescence quantification using feature vectors without using a spectroscopic mechanism or three-dimensional spectral images acquired by compressed sensing, thereby significantly reducing memory and computational costs and enabling fluorescence quantification without the need to accurately measure the spectral distribution of the target.

[0026] Functional block diagram of a multiple fluorescent dye quantification device Schematic flow diagram of a multiple fluorescent dye quantification method Outline of a multiple fluorescent dye quantification device Graph comparing the fluorescence spectra of each dye Example of spectral transmittance using a spectral modulator Illustrative diagram of encoding Observation results of intensity integrated in the wavelength direction using a sensor array through a filter with modulated spectral transmittance Graph comparing spectral distributions when observed at various voltages for fluorescent dyes with different fluorescence spectra and feature vectors modulated by spectral transmittance Illustrative diagram of sparse modeling using feature vectors as a dictionary Illustrative diagram of a simulation using a multiple fluorescent dye quantification method Simulation results of a multiple fluorescent dye quantification method Graph showing the correlation between the number of observations and restoration quality Illustrative diagram of an optimization method for the number of observations Graph comparing sparse modeling and random selection regarding the correlation between the number of observations and error Functional block diagram of a multiple fluorescent dye quantification device of Example 4 Illustrative diagram of a multiple fluorescent dye quantification device of Example 4 Graph comparing spectral transmittance with and without a multi-order wave plate

[0027] An example of an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the scope of the present invention is not limited to the following examples and illustrated examples, and many modifications and variations are possible.

[0028] FIG. 1 shows a functional block diagram of the multiple fluorescent dye quantification apparatus of Example 1. As shown in FIG. 1, the multiple fluorescent dye quantification apparatus 1 includes a filter controller 2, a filter (also referred to as a spectral modulator) 3, a sensor array 4, a memory unit 5, and a regression model 6. The filter controller 2 controls the voltage applied to the spectral modulator 3. The spectral modulator 3 modulates the spectral transmittance and includes a first polarizer 32a, a liquid crystal variable retarder 31, and a second polarizer 32b. The sensor array 4 observes an observation target 8 containing multiple fluorescent dyes through the spectral modulator 3. The memory unit 5 stores the luminance values ​​of the camera pixels captured under N types of modulation (N is an integer equal to or greater than 2; in this example, N = 200) as an N-dimensional feature vector. The regression model 6 is trained using the feature vectors under modulation for each fluorescent dye, quantifies the fluorescent dyes, and reconstructs an intensity image 9 of each fluorescence.

[0029] Figure 2 shows a schematic flow diagram of the multiple fluorescent dye quantification method of Example 1. As shown in Figure 2, for an observation target containing multiple fluorescent dyes, the intensity integrated in the wavelength direction is observed using a sensor array through a filter with modulated spectral transmittance (Step S01). A group of observations obtained under N types of modulation (N = 200 in this example) is treated as an N-dimensional feature vector (Step S02). Using a regression model trained on the feature vectors under modulation for each fluorescent dye, the fluorescent dyes are quantified and an intensity image of each fluorescence is reconstructed (Step S03).

[0030] Figure 3 shows a schematic diagram of a multiple fluorescent dye quantification device. The liquid crystal variable retarder 31 shown in Figure 3 is positioned to function similarly to a color filter placed in front of an image sensor or the like. The liquid crystal variable retarder 31 is an element whose spectral transmittance (spectral transmission characteristics) can be changed by applying a voltage, thereby modulating the spectral transmittance. As shown in Figure 3, in spectral encoding, the target 8 is imaged by the sensor array 4 via the spectral modulator 3 while varying the voltage applied to the liquid crystal variable retarder 31 in the filter control unit 2. The resulting observation group is converted into a 200-dimensional feature vector. Seven types of fluorescent dyes (fluorescent pigments) are used in the target 8. In fluorescence decoding, based on the resulting observation group converted into a 200-dimensional feature vector, a regression model 6 trained on the feature vectors under modulation for the seven types of fluorescent dyes is used to quantify the fluorescent dyes (pigments) and reconstruct an intensity image of each fluorescence. Combining encoding and decoding enables accurate identification and efficient imaging, even when multiple fluorescent dyes are used.

[0031] Figure 4 shows a comparison graph of the fluorescence spectra of each dye (source: https: / / www.thermofisher.com / jp / ja / home / brands / molecular-probes / key-molecular-probes-products / alexa-fluor / alexa-fluor-dyes-across-the-spectrum.html, see Figure 1). Observation of the dyes with fluorescence spectra numbered 1 to 20 shown in Figure 4 reveals that they cover a wide range of the spectrum, from near-ultraviolet to visible to near-infrared, and each dye has a different fluorescence spectrum. However, for example, dye number 13 and dye number 14 have similar fluorescence spectra, making it difficult to quantify the fluorescent pigments (dyes) when observed without using the spectral modulator 3.

[0032] Figure 5 shows an example of the spectral transmittance obtained by the spectral modulator. In Figure 5, the horizontal axis represents wavelength and the vertical axis represents transmittance, showing how the spectral transmittance characteristics change from (1) to (4). The spectral transmittance can be modulated by changing the voltage applied to the liquid crystal variable retarder 31 in the filter control unit 2. As mentioned above, in this example, fluorescent dyes with seven different fluorescent spectra were quantified using 200 types of modulation.

[0033] Fig. 6 is an explanatory diagram of encoding, where (1) shows the spectral distribution of two fluorescent dyes with different fluorescence spectra observed at various voltages, and (2) shows a comparison graph when the spectral transmittance is changed using a spectral modulator. Fig. 7 shows the results of observing the intensity integrated in the wavelength direction using a sensor array through a filter with modulated spectral transmittance, where (1) shows the result when an image is taken with a voltage v1 applied, and (2) shows the result when an image is taken with a voltage v2 applied.

[0034] As shown in Figure 6(1), the fluorescence spectra of dye d2 and dye d3 appear similar. However, as shown in Figure 6(2), when the spectral transmittance of the spectral modulator 3 is changed between v1 and v2, and images are taken, as shown in Figure 7(1), when image capture is performed with voltage v1 applied, the intensity at the peak position of dye d2 is 0.9 or more, while the intensity at the peak position of dye d3 is approximately 0.4. Also, as shown in Figure 7(1), when image capture is performed with voltage v2 applied, the intensity at the peak position of dye d2 is approximately 0.5, while the intensity at the peak position of dye d3 is approximately 1.0. Thus, even when dyes have different fluorescence spectra, the differences in the resulting spectral distributions become clear by observing while changing the spectral transmittance using the spectral modulator 3, making it easy to distinguish and quantify the fluorescent dyes.

[0035] FIG. 8 shows a comparison graph of the spectral distributions and feature vectors obtained by modulating the spectral transmittance of fluorescent dyes with different fluorescence spectra observed at various voltages. (1) shows the spectral distribution of a fluorescent dye with closely spaced peaks, (2) shows the spectral distribution of a fluorescent dye with nearly identical peaks but different spreads, (3) shows the feature vector of a fluorescent dye with closely spaced peaks, and (4) shows the feature vector of a fluorescent dye with nearly identical peaks but different spreads. When the peak positions are offset, as in the case of dyes d2 and d3 shown in FIG. 8(1), the light intensity observed at each voltage differs, resulting in a difference in the feature vector obtained by modulating the spectral transmittance, as shown in FIG. 8(3). Furthermore, even when the peaks are nearly identical, as in the case of dyes d4 and d7 shown in FIG. 8(2), the difference is clearly evident in the feature vector obtained by modulating the spectral transmittance, as shown in FIG. 8(4). Thus, by observing while varying the voltage applied to the liquid crystal variable retarder 31, it is possible to distinguish and quantify even fluorescent dyes with very similar fluorescence spectra.

[0036] FIG. 9 is an explanatory diagram of sparse modeling using a dictionary matrix for feature vectors. In this embodiment, a group of 200 observation images observed under 200 types of modulation is used as the feature vector on the horizontal axis, and matrix Y can be created based on the pixel count on the vertical axis and the feature vector. The dictionary matrix D is the feature vector of the fluorescent dye actually used. By identifying the dye used, the fluorescence spectrum is identified, and the intensity obtained when observed under each modulation can be simulated. Furthermore, a matrix X is created that indicates the amount of each fluorescent dye observed at each pixel. By calculating the product of X and D, the result can be expressed as a linear equation of Y = X · D. Here, matrix X represents the fluorescence intensity (brightness value) of each pixel, with the number of rows representing the number of pixels and the number of columns representing the number of fluorescent dyes. For example, if the observed image contains seven types of fluorescent dyes, and the row value of a certain pixel is [7, 0, 3, 0, 0, 0, 0], the seven types of fluorescent light (numbered 0 to 6) can be quantified and expressed as follows: fluorescent light number 0 has a brightness value of 7, fluorescent light number 2 has a brightness value of 3, and so on. Matrix Y is obtained through observation, and dictionary matrix D can be created by specifying the fluorescent dyes used. Matrix X can then be solved from these two. By solving the linear equation in this way, it is possible to analyze how much of each fluorescent dye is present in each pixel.

[0037] FIG. 10 is an explanatory diagram of a simulation using the multiple fluorescent dye quantification method of Example 1. In the simulation, seven types of fluorescent dyes were used, and a group of observation images was created using the liquid crystal variable retarder 31. The fluorescent images (7 types) are actual observation images of seven types of RNA. Each of the seven types of RNA was labeled with a different fluorescent dye, and the images obtained when each was observed under various voltages could be synthesized by multiplying the intensity of each pixel in the RNA observation image by the feature vector of the fluorescent dye used for labeling. When seven types of fluorescent dyes were simultaneously observed under a given voltage, the result was the sum of the observation images of each fluorescence obtained under that voltage. Since actual observations also contain noise, Gaussian noise with an intensity of approximately 3% was added depending on the brightness of the observed image.

[0038] Figure 11 shows the simulation results of the multiple fluorescent dye quantification method of Example 1. A group of observation images was created using the method shown in Figure 10, and fluorescence decoding was performed, resulting in the intensity image shown in Figure 11. When comparing the actual images using seven types of fluorescent dyes (d1 to d7) with the obtained intensity images, the peak signal-to-noise ratio (PSNR), which is the ratio of maximum power to noise, was 50 dB or higher in all cases. It can also be seen that the more indistinguishable the observed images were with the naked eye, the more successfully the inverse problem was solved.

[0039] In the above-described Example 1, the number of observations was set to 200. In this Example, however, the intensity images obtained by varying the number of observations were compared with actual images using the same seven fluorescent dyes as in Example 1, and the results were used to clarify the correlation between the number of observations and restoration quality by showing the change in the peak S / N ratio. Figure 12 is a graph showing the correlation between the number of observations and restoration quality. As shown in Figure 12, for each of the dyes (d1 to d7), the peak S / N ratio increases as the number of observations increases. However, even with an observation number of around 25, the peak S / N ratio is generally 50 dB or higher, indicating that sufficient restoration quality can be obtained.

[0040] Next, a method for optimizing the number of observations when the number of observations is reduced, that is, what voltage should be set at each observation point, will be described. FIG. 13 shows an explanatory diagram of the method for optimizing the number of observations. Ｔ " is the transposed matrix of the matrix X in Y=X·D explained in the first embodiment with reference to FIG. 9, and "Y Ｔ" represents the transposed matrix of matrix Y. Matrix β is like the dictionary matrix in Figure 9 with the rows and columns swapped, and is a matrix with the number of rows representing the number of fluorescent dyes and the number of columns representing the optimal number of observations. Here, we assume that the fluorescence intensity image (X) corresponding to each dye can be expressed by taking a linear sum using a small number of images (Y) out of the many images observed under various voltages. The coefficient matrix in this linear sum is β. When solving matrix β, regularization is performed so that many column elements are all 0. Observations corresponding to columns where all elements are 0 can be considered unnecessary for estimating the fluorescence intensity image. By adjusting the strength of regularization, the number of observations can be increased or decreased, and the X Ｔ and β・Y Ｔ The error can be estimated by evaluating the difference between

[0041] Figure 14 shows a graph comparing the correlation between the number of observations and the error when optimized using sparse modeling and when randomly selected. Specifically, the pixel where each dye appears is represented by a linear combination of the observed images, and sparse modeling is used to reconstruct the location and amount of each dye using as few observation images as possible. By solving the problem of which observations can adequately reconstruct the actual location and amount of dye using as few observations as possible, the amount of dye present at each pixel is estimated using as few observations as possible. As shown in Figure 14, unlike random selection, the multiple fluorescent dye quantification method of the present invention significantly reduced error with as few observations as possible (7-8 observations), demonstrating superior performance compared to random selection even with the same number of observations.

[0042] FIG. 15 shows a functional block diagram of the multiple fluorescent dye quantification apparatus of Example 4. As shown in FIG. 15, the multiple fluorescent dye quantification apparatus 1a includes a filter control unit 2, a filter 3, a sensor array 4, a memory unit 5, a regression model 6, and a multi-order wave plate 7. The filter control unit 2, the filter 3, the sensor array 4, the memory unit 5, and the regression model 6 are the same as those of the multiple fluorescent dye quantification apparatus 1 of Example 1. Unlike the multiple fluorescent dye quantification apparatus 1 of Example 1, the multiple fluorescent dye quantification apparatus 1a includes a multi-order wave plate 7 between the filter 3 and the observation target 8. The multi-order wave plate 7 is a wave plate in which the retardance of the optical path shifts by an integer multiple of the wavelength in addition to the design retardance, and has the advantage of being less expensive to implement than a zero-order wave plate. The sensor array 4 observes the observation target 8 containing multiple fluorescent dyes through the spectral modulator 3 and the multi-order wave plate 7. By providing the multi-order wave plate 7 between the filter 3 and the observation target 8, observations can be performed with a high frequency offset, enabling more accurate quantification of the fluorescent dyes.

[0043] Fig. 16 shows a schematic diagram of the multiple fluorescent dye quantification apparatus of Example 4. As shown in Fig. 16, in spectral encoding, the voltage applied to the liquid crystal variable retarder 31 in the filter control unit 2 is changed while the object 8 to be observed is imaged by the sensor array 4 via the multi-order wave plate 7 and the spectral modulator 3, and the resulting observation group is used as a 200-dimensional feature vector.

[0044] FIG. 17 is a graph comparing the spectral transmittance of a device with and without a multi-order wave plate 7 in front of the variable retarder 31. FIG. 17(1) shows the case where an applied voltage is 1 V. Compared to the spectral transmittance when using only the liquid crystal variable retarder 31, the spectral transmittance changes more dramatically with wavelength when the multi-order wave plate 7 is provided in front of the liquid crystal variable retarder 31. The more dramatic this change, the more effective it is for discriminating and quantifying fluorescent dyes with small spectral differences. FIG. 17(2) shows the case where an applied voltage is 10 V. In particular, the spectral transmittance when using only the liquid crystal variable retarder 31 changes very slowly with wavelength, whereas the spectral transmittance when the multi-order wave plate 7 is provided in front of the liquid crystal variable retarder 31 changes more dramatically with wavelength. The spectral transmittance pattern, which is important for discriminating between fluorescent dyes, depends on the type of fluorescent dye to be discriminated or quantified. However, the use of the multi-order wave plate 7 increases the number of selectable spectral transmittances, thereby improving discrimination and quantification performance.

[0045] Other Examples In the regression model, deep learning may be used to learn the tendency of each pixel from the proximity relationship between pixels.

[0046] The present invention is useful as a technique for simultaneously using and distinguishing multiple fluorescent dyes in bioimaging, including spatial omics analysis.

[0047] REFERENCE SIGNS LIST 1, 1a Multiple fluorescent dye quantification device 2 Filter control unit 3 Filter (spectral modulator) 4 Sensor array 5 Memory unit 6 Regression model 7 Multi-order wave plate 8 Observation target 9 Intensity image 31 Liquid crystal variable retarder 32a First polarizer 32b Second polarizer d1 to d7 Dyes v1, v2 Voltage

Claims

1. For an observation target containing a plurality of fluorescent dyes, the intensity integrated in the wavelength direction is observed using a sensor array through a filter whose spectral transmittance is modulated, and an observation group obtained under N types (N is an integer of 2 or more) of modulations is used as an N-dimensional feature vector. A method for quantifying a plurality of fluorescent dyes, characterized in that the fluorescent dyes are quantified and intensity images of each fluorescence are reconstructed using a regression model learned by the feature vector under the modulation for each fluorescent dye.

2. The method for quantifying a plurality of fluorescent dyes according to claim 1, wherein the learning uses the feature vector of each fluorescent dye contained in the observation target as a dictionary matrix.

3. The method for quantifying a plurality of fluorescent dyes according to claim 1 or 2, wherein the learning is sparse modeling.

4. The filter is a spectral modulator, and the modulation modulates the spectral transmittance by controlling the applied voltage of the spectral modulator. The method for quantifying a plurality of fluorescent dyes according to claim 1.

5. The filter is composed of a liquid crystal variable retarder and two polarizers sandwiching it. The method for quantifying a plurality of fluorescent dyes according to claim 1.

6. The spectral modulator selects the applied voltage used for observing the feature vector according to the spectral distribution group of the fluorescent dyes. The method for quantifying a plurality of fluorescent dyes according to claim 4.

7. The method for quantifying a plurality of fluorescent dyes according to claim 3, wherein the number of dimensions of the feature vector is such that the quantification accuracy of the reconstructed fluorescence exceeds a predetermined value.

8. The spectral transmittance is 20% to 40%. The method for quantifying a plurality of fluorescent dyes according to claim 1.

9. The filter further includes a multi-order wave plate. The method for quantifying a plurality of fluorescent dyes according to claim 1.

10. For an observation target containing a plurality of fluorescent dyes, a modulation process is performed to observe the intensity integrated in the wavelength direction using a sensor array through a filter whose spectral transmittance is changed over time. An observation group obtained under N types (N is an integer of 2 or more) of modulations is used as an N-dimensional feature vector. A method for quantifying a plurality of fluorescent dyes, characterized in that the fluorescent dyes are quantified and intensity images of each fluorescence are reconstructed by a demodulation process using a regression model learned by the feature vector under the modulation for each fluorescent dye.

11. A filter that modulates the spectral transmittance, a filter control unit that controls the filter, a sensor array that observes the intensity of an observation target containing a plurality of fluorescent dyes through the filter, a storage unit that stores the observation values of the sensor array observed under N types (N is an integer of 2 or more) of modulations as an N-dimensional feature vector, and a regression model that is trained by the feature vector under the modulation for each fluorescent dye to quantify the fluorescent dye and reconstruct an intensity image of each fluorescence. A multi-fluorescent dye quantification device characterized by comprising:

12. The multi-fluorescent dye quantification device according to claim 11, wherein the filter control unit selects a voltage group used for observation according to the type of fluorescent dye and modulates the spectral transmittance.

Images