FLUORESCENCE IMAGE ACQUISITION METHOD, FLUORESCENCE IMAGE ACQUISITION DEVICE, AND FLUORESCENCE IMAGE ACQUISITION PROGRAM

Abstract

The fluorescence image acquisition system 1 includes an irradiation device 2 that irradiates a sample with each of a plurality of excitation light beams having a plurality of wavelength distributions, an image acquisition device 3 that acquires a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, for each of a plurality of fluorescence beams corresponding to the respective plurality of excitation light beams, and an image processing device 4 that processes the plurality of first fluorescence images and the plurality of second fluorescence images. The image processing device 4 calculates, for the first fluorescence image and the second fluorescence image, an intensity ratio that is the ratio between the intensity value of a pixel of the first fluorescence image and the intensity value of a pixel of the second fluorescence image corresponding to the pixel, calculates the intensity ratio for each of the plurality of excitation light beams, and determines whether the pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation light beams.

Description

[Technical field]

[0001] One aspect of the embodiment relates to a fluorescence image acquiring method, a fluorescence image acquiring device, and a fluorescence image acquiring program. [Background technology]

[0002] Conventionally, a multiple staining method has been used in which a sample such as a biological tissue is simultaneously stained with a plurality of substances in the sample. In order to observe the substances in the sample that have been multiple stained, the sample is irradiated with excitation light to obtain a fluorescent image. For example, Non-Patent Document 1 below discloses the application of a nonnegative matrix factorization (NMF) method to blind unmix a fluorescent image in which fluorescence in a plurality of wavelength ranges is observed to obtain a separated image for each substance in the sample. In addition, Non-Patent Document 2 below discloses a method of unmixing a fluorescent image by clustering the fluorescent image, extracting the maximum value of the fluorescent intensity for each clustered pixel group, and generating a separated image based on the maximum value. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Binjie Qin et al., “Target / Background Classification Regularized Nonnegative Matrix Factorization for Fluorescence Unmixing”, IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL.65, NO.4, APRIL 2016 [Non-Patent Document 2] Tristan D. McRae et al., “Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning”, PLOS ONE, December 2,2019 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional techniques, it tends to be difficult to distinguish single-color pixels in a fluorescent image when observing the fluorescent image. Therefore, it is desirable to be able to distinguish single-color pixels in a fluorescent image in order to observe the fluorescent image efficiently.

[0005] Therefore, one aspect of the embodiments has been made in consideration of such problems, and has an object to provide a fluorescence image acquisition method, a fluorescence image acquisition device, and a fluorescence image acquisition program that are capable of distinguishing monochromatic pixels among fluorescent pixels. [Means for solving the problem]

[0006] A fluorescence image acquisition method according to a first aspect of an embodiment includes an irradiation step of irradiating a sample with each of excitation light having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescence corresponding to each of the plurality of excitation light, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state; a calculation step of calculating, for the first fluorescence image and the second fluorescence image, an intensity ratio that is a ratio between an intensity value of a pixel of the first fluorescence image and an intensity value of a pixel of the second fluorescence image corresponding to the pixel, and calculating the intensity ratio for each of the plurality of excitation light beams; and a discrimination step of discriminating whether or not the pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation light beams.

[0007] Alternatively, a fluorescence image acquisition apparatus according to a second aspect of the embodiment includes an irradiation device that irradiates a sample with each of a plurality of excitation light beams having a plurality of wavelength distributions, an image acquisition device that acquires, for each of a plurality of fluorescence beams corresponding to the respective plurality of excitation light beams, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from that of the first optical state, and an image processing device that processes the plurality of first fluorescence images and the plurality of second fluorescence images, wherein the image processing device calculates, for the first fluorescence image and the second fluorescence image, an intensity ratio that is a ratio between an intensity value of a pixel of the first fluorescence image and an intensity value of a pixel of the second fluorescence image corresponding to the pixel, calculates an intensity ratio for each of the plurality of excitation light beams, and determines whether or not the pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation light beams.

[0008] Alternatively, a fluorescence image acquisition program according to a third aspect of the embodiment is a fluorescence image acquisition program for determining whether a pixel is a monochromatic pixel based on a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, the first fluorescence image and the second fluorescence image being acquired through a fluorescence filter unit having multiple reflection wavelength bands and multiple transmission wavelength bands for each of a plurality of fluorescence beams corresponding to the respective excitation beams by irradiating a sample with excitation light beams having multiple wavelength distributions, and causing a computer to function as an intensity ratio calculation unit that calculates an intensity ratio for each of the plurality of excitation beams, the intensity ratio being the ratio between the intensity value of the pixel in the first fluorescence image and the intensity value of the pixel in the second fluorescence image corresponding to the pixel, and calculates the intensity ratio for each of the plurality of excitation beams, and a monochromatic pixel determination unit that determines whether a pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation beams.

[0009] According to the first, second, or third aspect, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from the first optical state are acquired for each of a plurality of fluorescences corresponding to a plurality of excitation lights. Furthermore, an intensity ratio, which is a ratio between an intensity value of a pixel in the first fluorescence image and an intensity value of a pixel in the second fluorescence image, is calculated, and an intensity ratio for each of a plurality of excitation lights is calculated. In addition, based on the intensity ratio for each of a plurality of excitation lights, it is determined whether the pixel is a monochromatic pixel. As a result, even in a fluorescence image including a pixel reflecting a mixture of fluorescence from a plurality of dyes and a monochromatic pixel reflecting monochromatic fluorescence from a single dye, a monochromatic pixel can be determined from a plurality of pixels in the fluorescence image. That is, a monochromatic pixel in the fluorescence image can be determined. Effect of the Invention

[0010] According to one aspect of the present disclosure, single-color pixels in a fluorescent image can be identified. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a fluorescence image acquisition system 1 according to a first embodiment. [Diagram 2] FIG. 2 is a perspective view showing the configuration of the fluorescent image acquisition system 1 of FIG. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of the image processing device 4 in FIG. 1. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of the image processing device 4 in FIG. [Diagram 5] 5A and 5B are diagrams for explaining the fluorescence intensities of a first fluorescence image and a second fluorescence image. [Figure 6] 4A and 4B are diagrams for explaining characteristics of an intensity ratio in a first pixel for a first excitation light and an intensity ratio in the first pixel for a second excitation light. [Figure 7] FIG. 13 is a diagram illustrating a characteristic of the intensity ratio. [Figure 8]FIG. 2 is a diagram showing an image of a group of pixels clustered by the clustering unit 204. [Figure 9] 13 shows an image of matrix data Y' regenerated by the statistical value calculation unit 205 and its corresponding dye matrix data X'. [Figure 10] 4 is a flowchart showing the steps of a fluorescence image acquiring method according to the first embodiment. [Figure 11] FIG. 11 is a schematic diagram of a fluorescence image acquiring system 1A according to a second embodiment. [Figure 12] 11 is a diagram for explaining the intensity ratio of each pixel when a color camera is used. FIG. [Figure 13] FIG. 13 is a diagram for explaining the intensity ratio of each pixel when a hyperspectral camera is used. [Figure 14] FIG. 11 is a schematic diagram of a fluorescence image acquiring system 1B according to a third embodiment. [Figure 15] FIG. 13 is a schematic diagram of a fluorescence image acquiring system 1C according to a fourth embodiment. [Figure 16] FIG. 13 is a diagram showing the results of clustering in an example of generating a fluorescent image. [Figure 17] FIG. 13 is a diagram showing the results of generating fluorescent images from each dye in an example of generating a fluorescent image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description, the same elements or elements having the same functions will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0013] [First embodiment] FIG. 1 is a schematic diagram of a fluorescence image acquisition system 1, which is a fluorescence image acquisition device according to a first embodiment. The fluorescence image acquisition system 1 is a device for generating a fluorescence image for identifying the distribution of a pigment in a sample such as a biological tissue to be observed. The image generated by the fluorescence image acquisition system 1 is used for the purpose of developing a drug, examining a treatment method, and the like through the analysis of the image. The fluorescence image acquisition system 1 includes an irradiation device 2 configured to irradiate an excitation light onto a sample S, an image acquisition device 3 that irradiates the sample S with the excitation light and acquires an image of the fluorescence generated in response to the irradiation, and an image processing device 4 that processes the image acquired by the image acquisition device 3. The image acquisition device 3 and the image processing device 4 may be configured to be capable of transmitting and receiving image data between them using wired communication or wireless communication, or may be configured to be capable of inputting and outputting image data via a recording medium.

[0014] Fig. 2 is a perspective view showing the configuration of the fluorescence image acquisition system 1 of Fig. 1. In Fig. 2, the optical path of the excitation light is indicated by a dotted line with an arrow, and the optical path of the fluorescence is indicated by a solid line with an arrow. The irradiation device 2 is configured to include an excitation light source 2a and an excitation light filter section 2b. The image acquisition device 3 is configured to include a dichroic mirror 11, a fluorescence filter section 3a, a first optical filter 13, and a first camera 15.

[0015] The excitation light source 2a is a light source capable of switching and irradiating excitation light of a plurality of wavelength bands (wavelength distributions), and is, for example, an LED (Light Emitting Diode) light source, a light source consisting of a plurality of monochromatic laser light sources, or a light source combining a white light source and a wavelength selection optical element. The excitation light filter unit 2b is a multi-bandpass filter provided on the optical path of the excitation light of the excitation light source 2a and having a property of transmitting light of a plurality of predetermined wavelength bands. The transmission wavelength band of this excitation light filter unit 2b is set according to the plurality of wavelength bands of the excitation light that can be used. The dichroic mirror 11 is an optical member provided between the excitation light filter unit 2b and the sample S, and has a property of reflecting the excitation light toward the sample S and transmitting the fluorescence emitted from the sample S in response. In the first embodiment, the irradiation device 2 irradiates excitation light of M (M is an integer of 2 or more) wavelength bands. Hereinafter, the excitation light of the M wavelength bands is referred to as the first excitation light, the second excitation light, ..., and the M excitation light.

[0016] The fluorescence filter section 3a is provided on the optical path of the fluorescence transmitted by the dichroic mirror 11, and is a multi-bandpass filter having the property of transmitting light in a predetermined number of wavelength ranges. The fluorescence filter section 3a has a number of reflection wavelength ranges and a number of transmission wavelength ranges. The reflection wavelength ranges and the transmission wavelength ranges are arranged alternately. The reflection wavelength ranges are, for example, wavelength ranges between the transmission wavelength ranges. The transmission wavelength ranges of this fluorescence filter section 3a are set according to the wavelength ranges of fluorescence generated in a pigment that may be contained in the sample S to be observed.

[0017] The first optical filter 13 is an optical system provided on the optical path of the fluorescence transmitted by the fluorescence filter unit 3a, and for acquiring wavelength information of the fluorescence. The first optical filter 13 includes a switching mechanism (not shown) that can switch the position of the first optical filter 13 so as to slide between a position on the optical path of the fluorescence from the fluorescence filter unit 3a and a position off the optical path. In the first embodiment, a state in which the first optical filter 13 is provided on the optical path of the fluorescence and thereby wavelength information of the fluorescence can be acquired via the first optical filter 13 is referred to as a first optical state, and a state in which the first optical filter 13 is not provided on the optical path of the fluorescence and thus wavelength information of the fluorescence can be acquired without the first optical filter 13 is referred to as a second optical state. The image acquisition device 3 can measure the fluorescence in the first optical state and in the second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state.

[0018] The first optical filter 13 is a filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescent filter unit 3a. In the first embodiment, the first optical filter 13 is a gradient filter having a wavelength characteristic of transmittance such that the transmittance increases linearly as the wavelength increases. That is, the transmittance of the first optical filter 13 increases monotonically across each transmission wavelength range of the fluorescent filter unit 3a. In the following description of the first embodiment, the first optical filter 13, which is such a gradient filter, is also referred to as the first gradient filter 13. The first optical filter 13 may be a gradient filter having a wavelength characteristic of transmittance such that the transmittance decreases linearly as the wavelength increases. The first optical filter 13 may be a gradient filter having a transmittance that changes monotonically across each reflection wavelength range of the fluorescent filter unit 3a.

[0019] The first camera 15 is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer equal to or greater than 2, for example, 2048×2048), and is a camera that captures the fluorescence transmitted through the first gradient filter 13 when the first gradient filter 13 is provided on the optical path of the fluorescence to obtain a first fluorescence image. That is, the first fluorescence image is a fluorescence image acquired in a first optical state. Moreover, the first camera 15 captures the fluorescence and obtains a second fluorescence image when the first gradient filter 13 is removed from the optical path of the fluorescence. That is, the second fluorescence image is a fluorescence image acquired in a second optical state. The first camera 15 outputs the acquired first and second fluorescence images to the image processing device 4 by using communication or via a recording medium. In the first embodiment, the N pixels are respectively referred to as a first pixel, a second pixel, ..., and an Nth pixel.

[0020] Next, the configuration of the image processing device 4 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 4, and Fig. 4 is a block diagram showing the functional configuration of the image processing device 4.

[0021] 3, the image processing device 4 is physically a computer or the like including a processor such as a CPU (Central Processing Unit) 101, a recording medium such as a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, each of which is electrically connected. Note that the image processing device 4 may include, as input / output devices, a display, a keyboard, a mouse, a touch panel display, etc., or may include a data recording device such as a hard disk drive or a semiconductor memory. The image processing device 4 may also be composed of multiple computers.

[0022] As shown in FIG. 4, the image processing device 4 includes, as functional components, an image acquisition unit 201, an intensity ratio calculation unit 202, a single-color pixel discrimination unit 203, a clustering unit 204, a statistical value calculation unit 205, and an image generation unit 206. Each functional unit of the image processing device 4 shown in FIG. 4 is realized by reading a program (a fluorescent image acquisition program according to the embodiment) onto hardware such as the CPU 101 and the RAM 102, and operating the communication module 104 and the input / output module 106 under the control of the CPU 101, and reading and writing data in the RAM 102. The CPU 101 of the image processing device 4 executes this computer program to cause each functional unit of FIG. 4 to function, and sequentially executes processes corresponding to a fluorescent image acquisition method described later. The CPU 101 may be a standalone hardware unit, or may be implemented in a programmable logic such as an FPGA, like a software processor. The RAM and ROM may also be standalone hardware units, or may be built into a programmable logic such as an FPGA. Various data required for executing this computer program and various data generated by executing this computer program are all stored in an internal memory such as ROM 103, RAM 102, or a storage medium such as a hard disk drive. The functions of the functional components of the image processing device 4 will be described in detail below.

[0023] The image acquiring section 201 acquires a first fluorescence image and a second fluorescence image for each of the first to Mth excitation light beams from the image acquiring device 3. The first fluorescence image is a fluorescence image acquired in a first optical state for a plurality of fluorescence beams corresponding to each of the first to Mth excitation light beams irradiated from the irradiation device 2. The second fluorescence image is a fluorescence image acquired in a second optical state for a plurality of fluorescence beams corresponding to each of the first to Mth excitation light beams irradiated from the irradiation device 2.

[0024] An example of the fluorescence intensity of the first and second fluorescence images will be described with reference to FIG. 5. In FIG. 5, part (a) shows the wavelength characteristic of the fluorescence intensity of the first fluorescence image acquired in the first optical state, and part (b) shows the wavelength characteristic of the fluorescence intensity of the second fluorescence image acquired in the second optical state. Here, it is assumed that three types of fluorescence, that is, the first fluorescence, the second fluorescence, and the third fluorescence, are imaged. In part (a) of FIG. 5, FI1, FI2, and FI3 respectively indicate the fluorescence intensity of the first fluorescence, the second fluorescence, and the third fluorescence transmitted through the first gradient filter 13. In part (b) of FIG. 5, FI4, FI5, and FI6 respectively indicate the fluorescence intensity of the first fluorescence, the second fluorescence, and the third fluorescence not transmitted through the first gradient filter. Moreover, T1 typically indicates the transmittance in the transmission wavelength range of the fluorescence filter section 3a. As shown in (b) of Fig. 5, in the second optical state, the magnitudes of the fluorescence intensities are approximately the same, but as shown in (a) of Fig. 5, in the first optical state, the fluorescence intensity FI1 of the first fluorescence increases, followed by the fluorescence intensity FI2 of the second fluorescence and the fluorescence intensity FI3 of the third fluorescence. This is because the transmittance of the first gradient filter 13 increases linearly as the wavelength increases.

[0025] The intensity ratio calculation unit 202 calculates an intensity ratio, which is a ratio between the intensity value of a pixel in the first fluorescence image and the intensity value of a pixel in the second fluorescence image corresponding to that pixel, for the first and second fluorescence images acquired for each of the first to Mth excitation lights, and calculates the intensity ratio for each of the first to Mth excitation lights. Here, for the mth excitation light (m is an integer between 1 and M), the intensity ratio, which is a ratio between the intensity value at the nth pixel (n is an integer between 1 and N) of the first fluorescence image and the intensity value at the nth pixel of the second fluorescence image, is defined as R mn Then, the intensity ratio calculation unit 202 calculates the intensity ratio R mn In the first embodiment, the intensity ratio is a ratio of the intensity value of a pixel of the first fluorescent image to the intensity value of a pixel of the second fluorescent image. The intensity ratio may be a ratio of the intensity value of a pixel of the second fluorescent image to the intensity value of the pixel of the first fluorescent image.

[0026] With reference to FIG. 6, as an example, the intensity ratio R 11 , and the intensity ratio R at the first pixel for the second excitation light 21 6, it is assumed, as an example, that one type of first dye is irradiated with the first excitation light and the second excitation light.

[0027] In part (a) of Fig. 6, excitation light spectra ES1 and ES2 of the first and second excitation lights are shown in schematic form with respect to the absorption spectrum AS of the first dye. In part (b) of Fig. 6, transmittance T1 in the transmission wavelength range of the fluorescence filter section 3a and transmittance T2 of the first gradient filter 13 taking the transmittance T1 into account are shown in schematic form with respect to the fluorescence spectrum FS of the first dye. Here, the wavelength is λ, the absorption spectrum AS is f AS (λ), the excitation light spectrum ES1 is f ES1 (λ), and the excitation light spectrum ES2 is f ES2 (λ), fluorescence spectrum FS FS (λ), transmittance T1 is S T1 (λ), transmittance T2 is S T2 (λ), the intensity value I of the first pixel acquired in the first optical state when irradiated with the first excitation light is 11 can be calculated as in the following formula (1). On the other hand, the intensity value I of the first pixel acquired in the second optical state when irradiated with the first excitation light is 12 can be calculated as shown in the following formula (2).

[0028]

number

[0029] When irradiated with the second excitation light, the intensity value I of the first pixel acquired in the first optical state 21 can be calculated as in the following formula (3). On the other hand, the intensity value I of the first pixel acquired in the second optical state when irradiated with the second excitation light is 22 can be calculated as shown in the following formula (4).

[0030]

number

[0031] From the above formulas (1) and (2), the intensity ratio R 11 From the above equations (3) and (4), the intensity ratio R 21 is calculated as shown in the following formula (6).

[0032]

number

[0033] In the above formula (5), ∫f AS (λ)f ES1 Since (λ)dλ is cancelled, the intensity ratio R 11 does not depend on the first excitation light. Similarly, the intensity ratio R 21 does not depend on the second excitation light. Therefore, as shown in the above formulas (5) and (6), the intensity ratio R 11 and intensity ratio R 21 However, the intensity ratio R 11 and intensity ratio R 21 is actually calculated based on the measured intensity values, so the intensity ratio R 11 and intensity ratio R 21 Note that do not necessarily have the same value.

[0034] Here, when the first pixel is a pixel that reflects a mixture of fluorescence from the first dye and the second dye, the intensity ratio R' at the first pixel corresponding to the first excitation light is 11 , and the intensity ratio R′ at the first pixel corresponding to the second excitation light 21 The absorption spectrum of the second dye is f' AS (λ), and the fluorescence spectrum of the second dye is f' FS (λ). Intensity ratio R' 11 and R' 21is calculated using the following equations (7) and (8).

[0035]

number

[0036] In the above formula (7), the values ​​in the numerator and denominator that depend on the first excitation light are not canceled out, so the intensity ratio R' 11 depends on the first excitation light. Similarly, the intensity ratio R' 21 depends on the second excitation light. That is, the intensity ratio R' 11 and intensity ratio R' 21 will not have the same value.

[0037] The characteristics of the intensity ratios described above will be explained. FIG. 7 is a graph that shows a schematic diagram of an intensity ratio RX1 when the first pixel reflects the fluorescence from the first dye, an intensity ratio RX2 when the first pixel reflects the fluorescence from the second dye, and an intensity ratio RX3 when the first pixel reflects a mixture of the fluorescence from the first dye and the second dye. FIG. 7 shows the intensity ratios for the first to third excitation lights. As shown in FIG. 7, the intensity ratios RX1 and RX2 have the same value for each excitation light, whereas the intensity ratio RX3 has a different value for each excitation light. Furthermore, as shown in FIG. 7, the intensity ratios RX1 and RX2 have different values ​​from each other. This is because, as shown in the above formulas (5) and (6), the transmittance T2 (i.e., S T2 ) is a value that depends on the wavelength λ, and the transmittance T1 (i.e., S T1 ) is a value that depends on the wavelength λ. T1 and S T2 is a constant independent of the wavelength λ, the intensity ratio is a constant, and the intensity ratios RX1 and RX2 have the same value. The monochromatic pixel discrimination unit 203 described later can discriminate whether a pixel is a monochromatic pixel based on such characteristics of the intensity ratio.

[0038] The monochromatic pixel discrimination unit 203 discriminates whether each pixel is a monochromatic pixel based on the intensity ratios of the first to Mth excitation lights. The monochromatic pixel discrimination unit 203 calculates V1 to V2 which are indices of the variation in the intensity ratios of the first to Mth excitation lights for the first to Nth pixels, respectively. N The monochrome pixel discrimination unit 203 calculates each of the indices V1 and determines whether each pixel is monochrome based on the indices. The method of calculating the index V1 for the first pixel by the monochrome pixel discrimination unit 203 will be described below. The monochrome pixel discrimination unit 203 calculates the indices V2 to V N Calculate.

[0039] For example, the single-color pixel discrimination unit 203 calculates the index V1 by the following formula (9): ave1 is the intensity ratio R of the first to Mth excitation lights in the first pixel. 11 ~R M1 In this case, the single-color pixel discriminating unit 203 can easily calculate the index V1. ave1 is calculated, for example, by the following formula (10) or (11). 1i is the intensity of the first pixel in the first optical state when irradiated with the i-th excitation light, y 2i is the intensity of the first pixel in the second optical state when irradiated with the i-th excitation light. ave1 is not limited to the following formulas (10) and (11), R ave1 is the weighted average or R 11 ~R M1 It may be an intermediate value between the above.

[0040]

number

[0041] Alternatively, the single-color pixel discriminating unit 203 calculates the index V1 by the following formula (12): i1 is R i1 For example, noise σ i1 When the first camera 15 is a CMOS (Complementary Metal-Oxide Semiconductor) camera, y' is calculated by the following formula (13). 1i is y1i is the intensity when converted to the number of electrons, y' 2i is y 2i is converted to the number of electrons, and ε is the readout noise of the CMOS camera. If shot noise is dominant, the readout noise ε can be ignored. In the case of the following formula (12), the monochromatic pixel discrimination unit 203 follows a chi-square distribution with M-1 degrees of freedom, so that it is possible to set a threshold value described later regardless of the sample S.

[0042]

number

[0043] Alternatively, the monochrome pixel discriminating unit 203 calculates the index V1 by the following formula (14): where p is a constant equal to or greater than 1, and w i is the weight corresponding to the ith excitation light, and RF i For example, R ave1 , the intensity ratio calculated from the dye reference information, and the statistical value of the intensity ratio calculated from multiple pixels. RF i may be changed for each excitation light, taking into consideration the measurement conditions for each excitation light, noise, the influence of background light, etc. Alternatively, it may be an arbitrary value. In the following formula (14), R i1 R i1 The function G(R i1 ) may be substituted.

[0044]

number

[0045] Indicators V1~V N is not limited to the indices of the above formulas (9) to (14), and may be any indices other than the above formulas (9) to (14) as long as they can evaluate the variation in the intensity ratio.

[0046] The monochrome pixel discrimination unit 203 uses the indices V1 to V NBased on this, it is determined whether each of the first to Nth pixels is a monochromatic pixel. As an example, the monochromatic pixel determination unit 203 determines a pixel whose variation index is equal to or smaller than a threshold as a monochromatic pixel, and determines a pixel whose variation index is larger than the threshold as not being a monochromatic pixel. The threshold may be a value preset by a user, or may be determined by the monochromatic pixel determination unit 203. Note that the case where the threshold is determined by the monochromatic pixel determination unit 203 is not limited to the case where it is automatically determined by the monochromatic pixel determination unit 203, and the monochromatic pixel determination unit 203 may determine the threshold in accordance with a parameter input by the user.

[0047] The clustering unit 204 performs clustering on a plurality of pixels (hereinafter, referred to as "a plurality of monochromatic pixels") that are determined to be monochromatic pixels by the monochromatic pixel determination unit 203. Prior to the clustering process, the clustering unit 204 generates matrix data Y in which values ​​of fluorescence intensity of N pixels constituting each of the C fluorescent images, each of which is composed of N pixels, are arranged in parallel in a one-dimensional manner, the C fluorescent images being C fluorescent images obtained by irradiating a sample with each of excitation lights having C wavelength distributions (C is an integer of 2 or more). The C fluorescent images may be a compilation of M first fluorescent images and M second fluorescent images obtained by the image acquisition device 3 by irradiating the sample with the first to M excitation lights, or may be M second fluorescent images. Alternatively, the C fluorescent images may be C fluorescent images captured in a second optical state by irradiating the sample with C excitation lights separately from the first to M excitation lights.

[0048] As an example, the clustering unit 204 clusters the multiple single-color pixels into L pixel groups (L is an integer between 2 and N-1) based on the average value of the intensity ratios of the multiple single-color pixels. Here, the number L of pixel groups to be clustered is set in advance as a parameter stored in the image processing device 4, for example, corresponding to the number of types of dyes that may be present in the sample S. The number L of pixel groups may be determined according to the type of excitation light or the number C of wavelength distributions of the excitation light, or may be determined independently of the type of excitation light or the number C of wavelength distributions of the excitation light. The average values ​​of the intensity ratios of single-color pixels of the same color are the same or close to each other. The clustering unit 204 clusters the multiple single-color pixels into L pixel groups by determining the distance (closeness of value) between the average values ​​of the intensity ratios of the multiple single-color pixels. The clustering unit 204 divides and regenerates matrix data Y in which the fluorescence intensity values ​​of the pixels of the C fluorescent images are arranged in parallel in one dimension into cluster matrices for each of the L pixel groups. The average value of the intensity ratios of the multiple single-color pixels is, for example, a simple average or a weighted average. The average value may be calculated using other calculation methods. The clustering unit 204 may perform clustering based on the median value of the intensity ratios of the multiple single-color pixels. The clustering unit 204 may perform clustering based on the intensities of the C excitation lights of the multiple single-color pixels.

[0049] Fig. 8 shows an image of pixel groups clustered by the clustering unit 204. As shown in Fig. 8, the sample S contains three types of dyes, dye C1, dye C2, and dye C3, and the clustering unit 204 clusters a plurality of single-color pixels into three pixel groups PGr1 to PGr3.

[0050] The statistical value calculation unit 205 obtains a mixing matrix A for generating K dye images showing the distribution of each of K dyes (K is an integer between 2 and C) from C fluorescent images based on L cluster matrices obtained for the sample S. Generally, according to a non-negative matrix factorization (NMF) calculation method, the relationship between matrix data Y, which is an observation matrix, and dye matrix data X, in which K dye images are arranged in parallel in a one-dimensional manner for each pixel, is expressed by the following formula using the mixing matrix A: Y=AX Here, Y is matrix data with C rows and N columns, A is matrix data with C rows and K columns, and X is matrix data with K rows and N columns. Conversely, once the value of the mixing matrix A is found, the pigment matrix data X is expressed as the inverse matrix A of the mixing matrix A. -1 and matrix data Y are used to calculate the following equation: X=A -1 Y This process is called unmixing.

[0051] Here, the statistical value calculation unit 205 regenerates the matrix data Y' by compressing the matrix data Y generated by the clustering unit 204 in units of pixel groups clustered by the clustering unit 204. In detail, the statistical value calculation unit 205 calculates a statistical value for each pixel group of the clustered cluster matrix for the fluorescence intensity of each row of the matrix data Y, and compresses the pixel group of each row into one pixel having the calculated statistical value. In this way, the statistical value calculation unit 205 regenerates the matrix data Y' which is matrix data of C rows and L columns. The statistical value calculation unit 205 may calculate an average value based on the integrated value of the fluorescence intensity, may calculate a mode value of the fluorescence intensity, or may calculate a median value of the fluorescence intensity as the statistical value.

[0052] Furthermore, the statistical value calculation unit 205 also calculates the following formula including the mixing matrix A for the reproduced matrix data Y' and the dye matrix data X' compressed in the same manner from the dye matrix data X: Y'=AX' holds, a mixing matrix A is derived based on the matrix data Y'. Fig. 9 shows an image of the matrix data Y' regenerated by the statistical value calculation unit 205, and the corresponding dye matrix data X'. One square shown in Fig. 9 represents one element of the matrix data. In this way, the dye matrix data X and matrix data Y divided into three pixel groups PGr1 to PGr3 are compressed into three columns of dye matrix data X' and matrix data Y', with the statistical values ​​for each of the pixel groups PGr1 to PGr3 being used as representative values.

[0053] The statistical value calculation unit 205 derives the mixing matrix A based on the matrix data Y' as follows. That is, the statistical value calculation unit 205 sets an initial value to the mixing matrix A, calculates the loss function (loss value) Loss of the following formula (15) while sequentially changing the value of the mixing matrix A, and derives the mixing matrix A that reduces the value of the loss function Loss. Note that a regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree to which the regularization term is emphasized) may be added to this loss function.

[0054]

number

[0055] As described above, the statistical value calculation unit 205 calculates a loss function for each of the L cluster matrices divided by the clustering unit 204 by referring to the statistical values ​​of the C matrix data Y', calculates the loss function Loss based on the sum of the L loss functions, and obtains the mixing matrix A based on the loss function Loss. At this time, the statistical value calculation unit 205 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values ​​a, b, and c of the statistical values ​​of the C matrix data Y', and then obtains the loss function Loss by calculating the sum of the corrected loss functions. Note that the statistical value calculation unit 205 may calculate the loss function for each of the L cluster matrices by correcting the row components of the difference value Y'-AX' for each wavelength band of the excitation light by dividing them by the C statistical values ​​corresponding to each wavelength band of the excitation light.

[0056] Alternatively, the statistical value calculation unit 205 may generate the mixing matrix A as follows. The statistical value calculation unit 205 may calculate the above-mentioned statistical value for each of the L clustered pixel groups for each of the C fluorescent images, and then arrange the calculated statistical values ​​for each of the C fluorescent images to generate a mixing matrix A with C rows and L columns. That is, the mixing matrix A may be the same as the matrix data Y'. In this case, the number of clusters and the number of dyes are equal, and the mixing matrix A can be generated more easily.

[0057] The image generating unit 206 obtains K dye images by unmixing C fluorescent images obtained from the sample S to be observed using the mixing matrix A derived by the statistical value calculating unit 205. Specifically, the image generating unit 206 obtains K dye images by applying the inverse matrix A of the mixing matrix A to the matrix data Y generated by the clustering unit 204 based on the C fluorescent images. -1 By applying the above formula, the dye matrix data X is calculated. Then, the image generation unit 206 reconstructs K dye images from the dye matrix data X, and outputs the reconstructed K dye images. The output destination at this time may be an output device of the image processing device 4, such as a display or a touch panel display, or may be an external device connected to the image processing device so as to be able to communicate data with the image processing device.

[0058] Next, a procedure for an observation process of the sample S using the fluorescence image acquisition system 1 according to the first embodiment, that is, the flow of the fluorescence image acquisition method according to the first embodiment, will be described. Fig. 10 is a flowchart showing the procedure for the observation process by the fluorescence image acquisition system 1.

[0059] First, the irradiation device 2 irradiates the sample S with each of the excitation lights having a plurality of wavelength distributions (step S1; irradiation step). Next, the image acquisition unit 201 of the image processing device 4 acquires a first fluorescence image in the first optical state and a second fluorescence image in the second optical state in which the fluorescence is measured with wavelength characteristics different from the first optical state through the fluorescence filter unit 3a for each of the plurality of fluorescence corresponding to each of the plurality of excitation lights (step S2; acquisition step). In the first embodiment, step S2 includes a first acquisition step of imaging the fluorescence in the first optical state and acquiring the first fluorescence image, and a second acquisition step of imaging the fluorescence in the second optical state and acquiring the second fluorescence image. Note that steps S1 and S2 may be repeated alternately. For example, after irradiating the sample with the first excitation light in the irradiation step and acquiring the first fluorescence image in the first acquisition step, the process may return to the irradiation step to irradiate the sample with the first excitation light and acquire the second fluorescence image in the second acquisition step, or the process may return to the irradiation step to irradiate the sample with the second excitation light.

[0060] Furthermore, the intensity ratio calculation section 202 of the image processing device 4 calculates an intensity ratio, which is a ratio between the intensity value of a pixel in the first fluorescent image and the intensity value of a pixel in the second fluorescent image corresponding to that pixel, for the first and second fluorescent images, and calculates the intensity ratio for each of the multiple excitation lights (step S3; calculation step). Next, the monochromatic pixel discrimination section 203 of the image processing device 4 discriminates whether the pixel is a monochromatic pixel based on the intensity ratio for each of the multiple excitation lights (step S4; discrimination step).

[0061] Furthermore, the clustering unit 204 of the image processing device 4 clusters a plurality of pixels determined to be monochromatic pixels in the discrimination step into L pixel groups for C fluorescence images each composed of N pixels, the C fluorescence images being obtained by irradiating the sample with excitation light of C wavelength distributions (step S5; clustering step). Next, the clustering unit 204 of the image processing device 4 generates matrix data Y in which the N pixels of the C fluorescence images are arranged in parallel (step S6; clustering step).

[0062] Furthermore, the statistical value calculation unit 205 of the image processing device 4 calculates the statistical value of the L pixel groups, thereby regenerating matrix data Y' based on the matrix data Y generated by the clustering unit 204 (step S7; calculation step). Next, the statistical value calculation unit 205 of the image processing device 4 derives a mixing matrix A based on the matrix data Y' (step S8; image generation step). Next, the image generation unit 206 of the image processing device 4 unmixes the matrix data Y generated based on the C fluorescent images of the sample S using the mixing matrix A, thereby regenerating K dye images (step S9; image generation step). Finally, the image generation unit 206 of the image processing device 4 outputs the regenerated K dye images (step S10). With the above, the observation process of the sample S is completed.

[0063] According to the fluorescence image acquisition system 1 of the first embodiment described above, for each of a plurality of fluorescences corresponding to a plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from the first optical state are acquired. Furthermore, an intensity ratio, which is a ratio between the intensity value of the pixel of the first fluorescence image and the intensity value of the pixel of the second fluorescence image, is calculated, and an intensity ratio for each of a plurality of excitation lights is calculated. In addition, based on the intensity ratio for each of a plurality of excitation lights, it is determined whether the pixel is a monochromatic pixel. As a result, even in a fluorescence image including a pixel reflecting a mixture of fluorescence from a plurality of dyes and a monochromatic pixel reflecting monochromatic fluorescence from a single dye, a monochromatic pixel can be determined from a plurality of pixels in the fluorescence image. That is, a monochromatic pixel in the fluorescence image can be determined.

[0064] The above-mentioned effects will be specifically described. According to the fluorescence image acquisition system 1 according to the first embodiment, the intensity ratio calculation unit 202 calculates the intensity ratio R mn If the n-th pixel is a monochrome pixel, the intensity ratio R 1n ~R Mn is the same value, or compared with the case where the nth pixel is not a monochrome pixel, the intensity ratio R 1n ~R Mn Based on such a feature, the single-color pixel discrimination unit 203 determines the intensity ratio R 1n ~R Mn By calculating an index of variation for the n-th pixel and judging whether the index is equal to or smaller than a threshold value, it is possible to determine whether the n-th pixel is a monochrome pixel.

[0065] In the first embodiment, the acquisition step (step S2) includes a first acquisition step of capturing an image of the fluorescence in a first optical state using the first optical filter 13 having different transmittances in each reflection wavelength range or each transmission wavelength range of the fluorescence filter unit 3a for each of the multiple fluorescences to acquire a first fluorescence image, and a second acquisition step of capturing an image of the fluorescence in a second optical state for each of the multiple fluorescences to acquire a second fluorescence image. This can improve the accuracy of discrimination of monochromatic pixels. Furthermore, by using the first optical filter 13, the transmission characteristics or reflection characteristics of the multiple fluorescences can be changed between multiple wavelength ranges, and monochromatic images of multiple colors can be further discriminated. In addition, by using the first optical filter 13, the first optical state can be easily established, and the first fluorescence image can be easily acquired.

[0066] In the first embodiment, the first optical filter 13 is a first gradient filter 13 whose transmittance changes monotonically across each transmission wavelength range. By using the first gradient filter 13, the transmittance can be changed monotonically across each transmission wavelength range, and monochromatic images of multiple colors can be distinguished with greater accuracy. Furthermore, by using the first gradient filter 13, the first optical state can be easily achieved, and the first fluorescent image can be easily acquired.

[0067] Furthermore, in the first embodiment, in the second acquisition step, the fluorescence is imaged in the second optical state in which the first optical filter 13 is removed. Since the second optical state can be achieved by removing the first optical filter 13, the second optical state can be achieved more easily. Therefore, the second fluorescence image can be acquired more easily.

[0068] In the first embodiment, the image processing device 4 targets C fluorescence images each composed of N pixels (N is an integer of 2 or more) obtained by irradiating a sample with each of excitation light having a wavelength distribution of C (C is an integer of 2 or more), clusters a plurality of pixels determined to be monochromatic pixels into L pixel groups (L is an integer of 2 or more and N-1 or less), generates L cluster matrices in which the C fluorescence images are arranged for each of the clustered pixel groups, calculates statistics of intensity values ​​of pixel groups constituting the C fluorescence images for each of the L cluster matrices, and performs unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices, thereby generating K fluorescence images showing distributions for each of K dyes (K is an integer of 2 or more and C or less). In this case, a plurality of pixels determined to be monochromatic pixels are clustered into L pixel groups, and L cluster matrices are generated in which the C fluorescence images obtained by irradiating a sample with each of excitation light having a wavelength distribution of C are arranged for each of the clustered pixel groups. In addition, the statistical values ​​of the intensity values ​​of the pixel groups constituting the C fluorescent images are calculated for each of the L cluster matrices, and the C fluorescent images are unmixed using the statistical values ​​of each of the C fluorescent images to generate K fluorescent images, thereby enabling accurate separation of images for each dye.

[0069] The above-mentioned effects will be specifically described. In the first embodiment, since the first inclined filter 13 has a transmittance that depends on the wavelength λ, the intensity ratios of the monochromatic pixels that show the same monochromatic color are the same or close to each other. Based on such a feature, the clustering unit 204 can cluster the monochromatic pixels into L pixel groups by, for example, determining the distance (closeness of value) between the average values ​​of the intensity ratios of the monochromatic pixels. The average value of the intensity ratios of the monochromatic pixels is, for example, a simple average or a weighted average. The average value may be calculated using other calculation methods. The clustering unit 204 may perform clustering by determining the distance between the median values ​​of the intensity ratios of the monochromatic pixels. The clustering unit 204 may perform clustering based on the intensities of the C excitation lights of the monochromatic pixels.

[0070] [Second embodiment] A second embodiment of the present disclosure will be described. Fig. 11 is a schematic configuration diagram of a fluorescence image acquisition system 1A which is a fluorescence image acquisition device according to the second embodiment. In the fluorescence image acquisition system 1A, the configuration of an image acquisition device 3A is different from the configuration of the image acquisition device 3 of the fluorescence image acquisition system 1. The image acquisition device 3A does not include a first optical filter 13. The image acquisition device 3A includes a first camera 15A having a different configuration from the first camera 15.

[0071] The first camera 15A is a camera that acquires at least two or more fluorescence images obtained by measuring the fluorescence from the sample S with different wavelength characteristics. The first camera 15A can capture each of the multiple fluorescences and acquire multiple fluorescence images including at least a first fluorescence image and a second fluorescence image. In the second embodiment, for example, a color camera or a hyperspectral camera can be used as the first camera 15A.

[0072] When a color camera is used as the first camera 15A, a plurality of images separated for each RGB component can be obtained by capturing each of the plurality of fluorescent lights once. As an example, the color camera captures a plurality of images separated for each RGB component as follows. The color camera is provided with specific color filters, for example, red, green, and blue color filters, corresponding to each pixel on the solid-state imaging element. An element provided with a red filter is referred to as an R element, an element provided with a green filter is referred to as a G element, and an element provided with a blue filter is referred to as a B element. In the color camera, the R element, the G element, and the B element are arranged in a mosaic pattern. The B element captures only the B component (blue component) of the fluorescent light, and the R component (red component) and the G component (green component) of the fluorescent light in the B element are interpolated by Bayer conversion processing. By performing the same processing on the R element and the G element, the color camera can capture a plurality of images separated for each RGB component. The Bayer conversion processing refers to a processing that supplements the missing color from the surrounding elements using a certain algorithm. The image acquisition section 201 can acquire a first fluorescence image and a second fluorescence image from a plurality of images separated for each RGB component. In the following description, the image of the G component is referred to as the first fluorescence image, and the image of the B component is referred to as the second fluorescence image.

[0073] The intensity ratio of each pixel when the above color camera is used will be described with reference to Fig. 12. Fig. 12 is a diagram showing a schematic of the spectral characteristics of each filter with respect to the fluorescence spectrum. SC1 is the spectral characteristic of a red filter, SC2 is the spectral characteristic of a green filter, and SC3 is the spectral characteristic of a blue filter. When the first excitation light is irradiated onto the sample S, the intensity IA of a pixel of the first fluorescence image, which is an image of the G component, is 11 and the intensity IA of the pixel in the second fluorescent image, which is the image of the B component. 12 and the intensity ratio RA 11 The spectral characteristics of the green filter are calculated, for example, by the following formulas (16) to (18). G (λ), the spectral characteristic of the blue filter is S B(λ). As in the first embodiment, in the case of a monochromatic pixel, the intensity ratio does not depend on the first excitation light. Therefore, in the second embodiment as well, it is possible to determine whether each pixel is a monochromatic pixel by calculating the intensity ratio for each of the multiple excitation lights. Note that the intensity of each pixel may be multiplied by a coefficient generated during Bayer conversion processing. Since the coefficient does not depend on the excitation light, the intensity ratio of a monochromatic pixel does not depend on the excitation light even when the coefficient is multiplied.

[0074]

number

[0075] When a hyperspectral camera is used as the first camera 15A, a plurality of images separated for different wavelength characteristics can be obtained by capturing an image of each of a plurality of fluorescent lights once. As an example, the hyperspectral camera obtains a plurality of images separated for different wavelength characteristics as follows. In the hyperspectral camera, filters of various wavelength bands are configured in a mosaic pattern in each pixel on the solid-state imaging element. For example, filters of 4×4=16 wavelength bands are configured in each pixel, and each pixel has information of 16 wavelength bands. This allows the hyperspectral camera to obtain a plurality of images separated for example for 16 different wavelength characteristics. The image obtaining unit 201 obtains a first fluorescent image and a second fluorescent image from the plurality of images separated for different wavelength characteristics. In the following description, the 16 wavelength bands of the hyperspectral camera correspond to, by way of example, wavelengths of 465 nm, 474 nm, 485 nm, 496 nm, 510 nm, 522 nm, 534 nm, 546 nm, 548 nm, 562 nm, 578 nm, 586 nm, 600 nm, 608 nm, 624 nm, and 630 nm, respectively, and the image acquired using a filter in the wavelength band corresponding to 474 nm is referred to as the first fluorescence image, and the image acquired using a filter in the wavelength band corresponding to 630 nm is referred to as the second fluorescence image.

[0076] The intensity ratio of each pixel when the hyperspectral camera is used will be described with reference to Fig. 13. Fig. 13 is a diagram showing a schematic of the spectral characteristics of each filter with respect to the fluorescence spectrum. SC4 is the spectral characteristic of a filter with a wavelength band corresponding to 474 nm, and SC5 is the spectral characteristic of a filter with a wavelength band corresponding to 630 nm. When the first excitation light is irradiated onto the sample S, the intensity IB of the pixel of the first fluorescence image is 11 and the intensity IB of the pixel in the second fluorescent image. 12 and the intensity ratio RB 11 Here, the spectral characteristics of the filter for the wavelength band corresponding to 474 nm are calculated as S 474 (λ), the spectral characteristics of the filter in the wavelength band corresponding to 630 nm are S 630 (λ). As in the first embodiment, in the case of a monochromatic pixel, the intensity ratio does not depend on the first excitation light. Therefore, in the second embodiment, it is possible to determine whether each pixel is a monochromatic pixel by calculating the intensity ratio for each of the multiple excitation lights.

[0077]

number

[0078] Alternatively, the hyperspectral camera may disperse the fluorescence using a spectroscope rather than using a filter on a solid-state imaging element. In such a hyperspectral camera, for example, the fluorescence is dispersed into a plurality of wavelength bands by being incident on a spectroscope via a lens, and the fluorescence for each wavelength band is received by an imaging element, so that each pixel has information on the plurality of wavelength bands. Even with such a hyperspectral camera, it is possible to acquire a plurality of images separated according to different wavelength characteristics. Thus, the image acquisition unit 201 can acquire the first fluorescence image and the second fluorescence image from the plurality of images separated according to different wavelength characteristics.

[0079] In the fluorescence image acquisition system 1A according to the second embodiment described above, the image acquisition device 3A also captures each of the multiple fluorescent lights, thereby acquiring a first fluorescence image and a second fluorescence image. Therefore, the fluorescence image acquisition system 1A also achieves the same effects as the fluorescence image acquisition system 1 according to the first embodiment. Furthermore, both the first fluorescence image and the second fluorescence image can be acquired by capturing each of the multiple fluorescent lights once. Therefore, the first fluorescence image and the second fluorescence image can be acquired efficiently.

[0080] Moreover, in the second embodiment, the image acquisition device 3A captures each of the multiple fluorescent lights using, for example, a color camera. By capturing an image of each of the multiple fluorescent lights once using the color camera, it is possible to acquire multiple images separated for each RGB component, for example. From these multiple images, a first fluorescent light image and a second fluorescent light image can be acquired.

[0081] In the second embodiment, the image acquisition device 3A captures each of the multiple fluorescences using, for example, a hyperspectral camera. By capturing an image of each of the multiple fluorescences once using the hyperspectral camera, multiple images separated according to different wavelength characteristics can be acquired. From these multiple images, a first fluorescence image and a second fluorescence image can be acquired.

[0082] [Third embodiment] A third embodiment of the present disclosure will now be described. Fig. 14 is a schematic configuration diagram of a fluorescence image acquiring system 1B which is a fluorescence image acquiring device according to the third embodiment. An image acquiring device 3B of the fluorescence image acquiring system 1B further includes a second camera 17 in addition to the configuration of the image acquiring device 3 of the fluorescence image acquiring system 1.

[0083] The second camera 17 is a camera having, for example, the same configuration as the first camera 15. The second camera 17 is a camera that captures the fluorescence reflected by the first gradient filter 13 to obtain a second fluorescence image when the first gradient filter 13 is provided on the optical path of the fluorescence. That is, in the fluorescence image acquisition system 1B, the fluorescence passing through the first gradient filter 13 is captured by the first camera 15 to obtain a first fluorescence image, and the fluorescence reflected by the first gradient filter 13 is captured by the second camera 17 to obtain a second fluorescence image.

[0084] In the fluorescence image acquisition system 1B according to the third embodiment described above, the fluorescence passing through the first gradient filter 13 is captured by the first camera 15, and the fluorescence reflected by the first gradient filter 13 is captured by the second camera 17, thereby acquiring a first fluorescence image and a second fluorescence image. Therefore, the fluorescence image acquisition system 1B also achieves the same effects as the fluorescence image acquisition system 1 according to the first embodiment.

[0085] [Fourth embodiment] A fourth embodiment of the present disclosure will be described. Fig. 15 is a schematic configuration diagram of a fluorescence image acquisition system 1C that is a fluorescence image acquisition device according to the fourth embodiment. An image acquisition device 3C of the fluorescence image acquisition system 1C further includes a second optical filter 19 in addition to the configuration of the image acquisition device 3 of the fluorescence image acquisition system 1. The image acquisition device 3C is provided with a switching mechanism (not shown) that can switch between the first optical filter 13 and the second optical filter 19 so as to switch from a state in which the first optical filter 13 is provided on the optical path of the fluorescence from the fluorescence filter section 3a to a state in which the second optical filter 19 is provided on the optical path of the fluorescence from the fluorescence filter section 3a.

[0086] The second optical filter 19 has a transmittance different from that of the first optical filter 13, and is a filter whose transmittance differs in each reflection wavelength range or each transmission wavelength range of the fluorescent filter unit 3a. In the fourth embodiment, the second optical filter 19 is a gradient filter having a wavelength characteristic of transmittance such that the transmittance decreases linearly as the wavelength increases. That is, the transmittance of the second optical filter 19 decreases monotonically across each transmission wavelength range of the fluorescent filter unit 3a. In the following description of the fourth embodiment, the second optical filter 19, which is such a gradient filter, is also referred to as the second gradient filter 19. The second gradient filter 19 exhibits a change in transmittance different from that of the first gradient filter 13. The second optical filter 19 may be a gradient filter having a wavelength characteristic of transmittance such that the transmittance increases linearly as the wavelength increases. The second optical filter 19 may be a gradient filter whose transmittance changes monotonically across each reflection wavelength range of the fluorescent filter unit 3a.

[0087] When the first gradient filter 13 is provided on the optical path of the fluorescence, the first camera 15 captures the fluorescence and obtains a first fluorescence image. Also, when the second gradient filter 19 is provided on the optical path of the fluorescence, the first camera 15 captures the fluorescence and obtains a second fluorescence image.

[0088] In the fluorescence image acquisition system 1C according to the fourth embodiment described above, the first fluorescence image and the second fluorescence image are acquired by capturing an image of the fluorescence when the first gradient filter 13 is provided on the optical path of the fluorescence, and capturing an image of the fluorescence when the second gradient filter 19 is provided on the optical path of the fluorescence. Thus, the fluorescence image acquisition system 1C also achieves the same effects as the fluorescence image acquisition system 1 according to the first embodiment.

[0089] Moreover, in the fourth embodiment, the image acquisition device 3C captures fluorescence in a second optical state using a second gradient filter 19 that exhibits a different change in transmittance from the first gradient filter 13. By using the second gradient filter 19, the transmittance can be changed monotonically across each transmission wavelength range, and monochromatic images of multiple colors can be distinguished with even greater accuracy. Furthermore, by using the second gradient filter 19, the second optical state can be easily established, and the second fluorescence image can be easily acquired.

[0090] Moreover, in the fourth embodiment, the image acquisition device 3C captures an image of fluorescence in a second optical state using a second optical filter 19 having a transmittance different from that of the first optical filter 13. By using the second optical filter 19, the second optical state can be easily achieved, and the second fluorescence image can be easily acquired.

[0091] [Example of generating a fluorescent image] An example of generating a fluorescent image by the fluorescent image acquisition system 1 according to the first embodiment will be described. In this example, samples S11 and S21, which are red fluorescent tapes, samples S12 and S22, which are orange fluorescent tapes, samples S13 and S23, which are green fluorescent tapes, and samples S14 and S24, which are blue fluorescent tapes, are prepared, and these fluorescent tapes are arranged in a cross shape on a slide. Part (a) of FIG. 16 is a fluorescent image obtained by capturing images of these fluorescent tapes with a monochrome camera. Part (b) of FIG. 16 shows a pixel group composed of blue monochrome pixels among the pixel groups clustered by the clustering unit 204. As shown in part (b) of FIG. 16, in this example, monochrome pixels are clustered with high accuracy.

[0092] Part (a) of Fig. 17 shows a fluorescence image from the blue dye (samples S14, S24) among the dye images generated by the image generation unit 206, part (b) of Fig. 17 shows a fluorescence image from the green dye (samples S13, S23), part (c) of Fig. 17 shows a fluorescence image from the orange dye (samples S12, S22), and part (d) of Fig. 17 shows a fluorescence image from the red dye (samples S11, S21). As shown in parts (a) to (d) of Fig. 17, in this generation example, fluorescence images from each dye are generated with high accuracy.

[0093] Various embodiments of the present disclosure have been described above, but the present disclosure is not limited to the above-described embodiments, and may be modified or applied to other embodiments without changing the gist of the present disclosure described in each claim.

[0094] In the second embodiment, the first camera may have a filter on the solid-state imaging element that can arbitrarily switch the transmission wavelength range or reflection wavelength range. Examples of such filters include a variable bandpass filter that has the functions of a variable filter and a bandpass filter, a liquid crystal tunable filter, and an AOTF (Acousto-Optic Tunable Filter). Alternatively, the first camera may separate the fluorescence into different wavelength characteristics using a spectroscope such as a prism.

[0095] When generating matrix data Y before clustering, the image processing device 4 according to each embodiment may generate the data in which N pixels constituting an image are arranged in the row direction according to a specified rule, or may generate the data in which N pixels constituting the image are arranged in the row direction according to a random rule. However, the data of a fluorescence image of C rows constituting one piece of matrix data Y is set as data in which N pixels are arranged according to the same rule. Even if matrix data Y generated by random arrangement is used, the same matrix data Y' can be regenerated by clustering.

[0096] Furthermore, when generating the matrix data Y before clustering, the image processing device 4 according to each embodiment may generate the data by excluding background pixels (pixels in which no dye is present) contained in the fluorescence image.

[0097] As a method for clustering a group of pixels in the image processing device 4, a method using machine learning such as the K-means method, a method using deep learning, or the like may be adopted.

[0098] The image acquisition device according to each embodiment may acquire, in addition to the first and second fluorescence images, a third fluorescence image, ..., a Qth fluorescence image in a third optical state, ..., a Qth optical state (Q is an integer equal to or greater than 3) in which fluorescence is measured with wavelength characteristics different from those of the first and second optical states. Fluorescence is measured with wavelength characteristics different from each other in the third to Qth optical states. In this case, an intensity ratio and a variation index for each excitation light may be calculated from a combination of two fluorescence images from among the first to Qth fluorescence images, and an average of the variation indexes for all combinations may be calculated to determine whether a pixel is a monochromatic pixel.

[0099] In the first, third and fourth embodiments, it is preferable that the acquisition step includes a first acquisition step of imaging the fluorescence in a first optical state using a first optical filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescence filter unit for each of the multiple fluorescences to acquire a first fluorescence image, and a second acquisition step of imaging the fluorescence in a second optical state for each of the multiple fluorescences to acquire a second fluorescence image. In the first, third and fourth embodiments, it is preferable that the image acquisition device images the fluorescence in a first optical state using a first optical filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescence filter unit for each of the multiple fluorescences to acquire a first fluorescence image, and images the fluorescence in the second optical state for each of the multiple fluorescences to acquire a second fluorescence image. In this case, the accuracy of discrimination of monochromatic pixels can be improved. Furthermore, by using the first optical filter, the transmission characteristics or reflection characteristics of the multiple fluorescences can be changed between multiple wavelength ranges, and monochromatic pixels of multiple colors can be further discriminated. In addition, the first optical state can be easily achieved by using the first optical filter, and the first fluorescent image can be easily acquired.

[0100] In the first, third and fourth embodiments, it is preferable that the first optical filter is a first gradient filter whose transmittance changes monotonically across each reflection wavelength range, or whose transmittance changes monotonically across each transmission wavelength range. In this case, by using the first gradient filter, the transmittance can be changed monotonically across each reflection wavelength range, or the transmittance can be changed monotonically across each transmission wavelength range, and monochrome pixels of multiple colors can be distinguished with even greater accuracy. Furthermore, by using the first gradient filter, the first optical state can be easily achieved, and the first fluorescence image can be easily acquired.

[0101] In the fourth embodiment, it is preferable that in the second acquisition step, the fluorescence is imaged in the second optical state using a second gradient filter that exhibits a different change in transmittance from the first gradient filter. In the fourth embodiment, it is preferable that the image acquisition device images the fluorescence in the second optical state using a second gradient filter that exhibits a different change in transmittance from the first gradient filter. In this case, too, by using the second gradient filter, the transmittance can be monotonically changed across each reflection wavelength range or across each transmission wavelength range, and monochromatic images of multiple colors can be more accurately distinguished. Furthermore, by using the second gradient filter, the second optical state can be easily established, and the second fluorescence image can be easily acquired.

[0102] In the fourth embodiment, it is preferable that in the second acquisition step, the fluorescence is imaged in a second optical state using a second optical filter having a transmittance different from that of the first optical filter. In the fourth embodiment, it is preferable that the image acquisition device images the fluorescence in a second optical state using a second optical filter having a transmittance different from that of the first optical filter. In this case, the second optical state can be easily achieved by using the second optical filter, and the second fluorescence image can be easily acquired.

[0103] Also, in the first embodiment, it is preferable that in the second acquisition step, the fluorescence is imaged in the second optical state in which the first optical filter is removed. Also, in the first embodiment, it is preferable that the image acquisition device images the fluorescence in the second optical state in which the first optical filter is removed. In this case, since the second optical state can be achieved by removing the first optical filter, the second optical state can be achieved more easily. Therefore, the second fluorescence image can be acquired more easily.

[0104] In the second embodiment, it is preferable that in the acquisition step, each of the multiple fluorescent lights is imaged and a first fluorescent image and a second fluorescent image are acquired. In the second embodiment, it is preferable that the image acquisition device images each of the multiple fluorescent lights and acquires a first fluorescent image and a second fluorescent image. In this case, both the first fluorescent image and the second fluorescent image can be acquired by imaging each of the multiple fluorescent lights once. Therefore, the first fluorescent image and the second fluorescent image can be acquired efficiently.

[0105] In the second embodiment, it is preferable that in the acquiring step, each of the multiple fluorescence is imaged using a color camera. In the second embodiment, it is preferable that the image acquiring device is configured to image each of the multiple fluorescence using a color camera. By imaging each of the multiple fluorescence once using the color camera, it is possible to acquire multiple images separated for each RGB component, for example. From these multiple images, a first fluorescence image and a second fluorescence image can be acquired.

[0106] In the second embodiment, it is preferable that the acquisition step uses a hyperspectral camera to capture each of the multiple fluorescences. In the second embodiment, it is preferable that the image acquisition device uses a hyperspectral camera to capture each of the multiple fluorescences. By capturing an image of each of the multiple fluorescences once using the hyperspectral camera, it is possible to acquire multiple images separated for different wavelength characteristics. From these multiple images, a first fluorescence image and a second fluorescence image can be acquired.

[0107] In addition, in the first to fourth embodiments, it is preferable that the method further comprises the following steps: a clustering step of clustering a plurality of pixels determined to be monochromatic pixels in the discrimination step into L pixel groups (L is an integer greater than or equal to 2 and less than N-1) and generating L cluster matrices in which the C fluorescence images are arranged for each of the clustered pixel groups, the C fluorescence images being obtained by irradiating a sample with excitation light having each of C wavelength distributions (C is an integer greater than or equal to 2), the C fluorescence images each consisting of N pixels (N is an integer greater than or equal to 2); a calculation step of calculating statistics of intensity values ​​of the pixel groups constituting the C fluorescence images for each of the L cluster matrices; and an image generation step of performing unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices, and generating K fluorescence images showing the distribution for each of the K dyes (K is an integer greater than or equal to 2 and less than C). In the first to fourth embodiments, the image processing device also preferably clusters a plurality of pixels determined to be monochromatic pixels into L pixel groups (L is an integer between 2 and N-1) for C fluorescence images each composed of N pixels (N is an integer between 2 and 1), which are obtained by irradiating a sample with excitation light having a wavelength distribution of C (C is an integer between 2 and 1), generates L cluster matrices in which the C fluorescence images are arranged for each of the clustered pixel groups, calculates statistics of the intensity values ​​of the pixel groups constituting the C fluorescence images for each of the L cluster matrices, and performs unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices to generate K fluorescence images showing the distribution for each of K dyes (K is an integer between 2 and 1, and C). In this case, a plurality of pixels determined to be monochromatic pixels are clustered into L pixel groups, and L cluster matrices are generated in which the C fluorescence images obtained by irradiating a sample with excitation light having a wavelength distribution of C are arranged for each of the clustered pixel groups. In addition, the statistical values ​​of the intensity values ​​of the pixel groups constituting the C fluorescent images are calculated for each of the L cluster matrices, and the C fluorescent images are unmixed using the statistical values ​​of each of the C fluorescent images to generate K fluorescent images, thereby enabling accurate separation of images for each dye.

[0108] A fluorescence image acquisition method according to an embodiment is a fluorescence image acquisition method comprising: [1] "an irradiation step of irradiating a sample with each of excitation light having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescence corresponding to each of the plurality of excitation light, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state; a calculation step of calculating, for the first fluorescence image and the second fluorescence image, an intensity ratio that is a ratio between an intensity value of a pixel of the first fluorescence image and an intensity value of a pixel of the second fluorescence image corresponding to the pixel, and calculating the intensity ratio for each of the plurality of excitation light; and a discrimination step of discriminating whether or not the pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation light."

[0109] The fluorescence image acquisition method of the embodiment may be the fluorescence image acquisition method described in [1] above, [2] "wherein the acquisition step includes a first acquisition step of capturing an image of the fluorescence in a first optical state using a first optical filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescence filter section for each of the multiple fluorescences to acquire a first fluorescence image, and a second acquisition step of capturing an image of the fluorescence in a second optical state for each of the multiple fluorescences to acquire a second fluorescence image."

[0110] The fluorescence image acquisition method of the embodiment may be [3] "the fluorescence image acquisition method described in the above [2], wherein the first optical filter is a first gradient filter whose transmittance changes monotonically across each reflection wavelength range, or whose transmittance changes monotonically across each transmission wavelength range."

[0111] The fluorescence image acquisition method of the embodiment may be [4] "the fluorescence image acquisition method described in [3] above, in which in the second acquisition step, fluorescence is imaged in a second optical state using a second gradient filter that exhibits a different change in transmittance from the first gradient filter."

[0112] The fluorescence image acquisition method of the embodiment may be [5] "the fluorescence image acquisition method described in the above [2] or [3], in which in the second acquisition step, fluorescence is imaged in a second optical state using a second optical filter having a transmittance different from that of the first optical filter."

[0113] The fluorescence image acquisition method of the embodiment may be [6] "the fluorescence image acquisition method described in [2] or [3] above, in which, in the second acquisition step, fluorescence is imaged in a second optical state in which the first optical filter is removed."

[0114] The fluorescence image acquisition method of the embodiment may be [7] "the fluorescence image acquisition method described in the above [1], in which, in the acquisition step, each of the multiple fluorescences is imaged to acquire a first fluorescence image and a second fluorescence image."

[0115] The fluorescence image acquisition method of the embodiment may be [8] "the fluorescence image acquisition method according to the above [7], in which, in the acquisition step, each of the multiple fluorescence is imaged using a color camera."

[0116] The fluorescence image acquisition method of the embodiment may be [9] "the fluorescence image acquisition method according to the above [7], in which, in the acquisition step, each of the multiple fluorescences is imaged using a hyperspectral camera."

[0117] The fluorescence image acquisition method of the embodiment may be

[10] "the fluorescence image acquisition method according to any one of [2] to [9] above, further comprising: a clustering step of clustering a plurality of pixels determined to be monochromatic pixels in the discrimination step into L pixel groups (L is an integer between 2 and N-1) and generating L cluster matrices in which the C fluorescence images are obtained by irradiating a sample with excitation light having a wavelength distribution of C (C is an integer between 2 and C), each of the C fluorescence images being composed of N pixels (N is an integer between 2 and C), a calculation step of calculating statistics of intensity values ​​of the pixel groups constituting the C fluorescence images for each of the L cluster matrices, and an image generation step of performing unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices, and generating K fluorescence images showing a distribution for each of K dyes (K is an integer between 2 and C)."

[0118] The fluorescence image acquisition device of the embodiment is

[11] "a fluorescence image acquisition device comprising: an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions; an image acquisition device that acquires, for each of a plurality of fluorescence corresponding to each of the plurality of excitation light, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state; and an image processing device that processes the plurality of first fluorescence images and the plurality of second fluorescence images, wherein the image processing device calculates, for the first fluorescence image and the second fluorescence image, an intensity ratio that is the ratio between the intensity value of a pixel of the first fluorescence image and the intensity value of a pixel of the second fluorescence image corresponding to the pixel, calculates the intensity ratio for each of the plurality of excitation light, and determines whether the pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation light."

[0119] The fluorescence image acquisition device of the embodiment may be

[12] "the fluorescence image acquisition device described in the above

[11] , which images, for each of the multiple fluorescence, the fluorescence in a first optical state using a first optical filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescence filter section, and acquires a first fluorescence image, and images, for each of the multiple fluorescence, the fluorescence in a second optical state, and acquires a second fluorescence image."

[0120] The fluorescence image acquisition device of the embodiment may be

[13] "the fluorescence image acquisition device described in

[12] above, wherein the first optical filter is a first gradient filter whose transmittance changes monotonically across each reflection wavelength range, or whose transmittance changes monotonically across each transmission wavelength range."

[0121] The fluorescence image acquisition device of the embodiment may be

[14] "the fluorescence image acquisition device described in

[13] above, which images fluorescence in a second optical state using a second gradient filter that exhibits a transmittance change different from that of the first gradient filter."

[0122] The fluorescence image acquisition device of the embodiment may be

[15] "the fluorescence image acquisition device described in

[12] or

[13] above, which captures fluorescence in a second optical state using a second optical filter having a transmittance different from that of the first optical filter."

[0123] The fluorescence image acquisition device of the embodiment may be

[16] "the fluorescence image acquisition device described in

[12] or

[13] above, which captures fluorescence in a second optical state in which the first optical filter is removed."

[0124] The fluorescence image acquisition device of the embodiment may be

[17] "the fluorescence image acquisition device described in

[11] above, which captures each of the multiple fluorescences and acquires a first fluorescence image and a second fluorescence image."

[0125] The fluorescence image acquisition device of the embodiment may be

[18] "the fluorescence image acquisition device according to the above

[17] , which uses a color camera to capture each of the multiple fluorescence images."

[0126] The fluorescence image acquisition device of the embodiment may be

[19] "the fluorescence image acquisition device according to the above

[17] , which captures each of the multiple fluorescence images using a hyperspectral camera."

[0127] The fluorescence image acquisition device of the embodiment may be

[20] "the image processing device is a fluorescence image acquisition device described in any of

[12] to

[19] above, which targets C fluorescence images obtained by irradiating a sample with each of excitation light having a wavelength distribution of C (C is an integer of 2 or more), each of which is composed of N (N is an integer of 2 or more) pixels, clusters a plurality of pixels determined to be monochromatic pixels into L (L is an integer of 2 or more and N-1 or less) pixel groups, generates L cluster matrices in which the C fluorescence images are arranged for each clustered pixel group, calculates statistics of intensity values ​​of the pixel groups constituting the C fluorescence images for each of the L cluster matrices, performs unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices, and generates K fluorescence images showing distributions for each of K dyes (K is an integer of 2 or more and C or less)."

[0128] A fluorescence image acquisition program of an embodiment is a fluorescence image acquisition program for determining whether a pixel is a monochromatic pixel based on a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, the first fluorescence image being acquired through a fluorescence filter unit having multiple reflection wavelength bands and multiple transmission wavelength bands for each of a plurality of fluorescence lights corresponding to each of the plurality of excitation lights by irradiating a sample with each of excitation lights having a plurality of wavelength distributions, the fluorescence image acquisition program causing a computer to function as an intensity ratio calculation unit that calculates an intensity ratio, which is a ratio between the intensity value of the pixel of the first fluorescence image and the intensity value of the pixel of the second fluorescence image corresponding to the pixel, and calculates the intensity ratio for each of the plurality of excitation lights, and a monochromatic pixel discrimination unit that discriminates whether a pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation lights. [Explanation of symbols]

[0129] 1, 1A, 1B, 1C...fluorescence image acquisition system, 2...illumination device, 2a...excitation light source, 2b...excitation light filter section, 3, 3A, 3B, 3C...image acquisition device, 3a...fluorescence filter section, 4...image processing device, 11...dichroic mirror, 13...first optical filter (first gradient filter), 15, 15A...first camera, 17...second camera, 19...second optical filter (second gradient filter), 201...image acquisition section, 202...intensity ratio calculation section, 203...monochrome pixel discrimination section, 204...clustering section, 205...statistical value calculation section, 206...image generation section, C1 to C3...dyes, PGr1 to PGr3...pixel groups, S, S11, S12, S13, S14, S21, S22, S23, S24...sample.

Claims

1. an irradiation step of irradiating a sample with each of the excitation lights having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescence beams corresponding to the plurality of excitation light beams, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those of the first optical state; a calculation step of calculating an intensity ratio, which is a ratio between an intensity value of a pixel of the first fluorescent image and an intensity value of a pixel of the second fluorescent image corresponding to the pixel, for the first fluorescent image and the second fluorescent image, and calculating the intensity ratio for each of the plurality of excitation lights; a determining step of determining whether the pixel is a monochromatic pixel based on an intensity ratio for each of the plurality of excitation lights; A fluorescence image acquisition method comprising:

2. the acquisition step includes a first acquisition step of capturing an image of the fluorescence in the first optical state using a first optical filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescence filter unit for each of the plurality of fluorescence, and acquiring the first fluorescence image; a second acquisition step of capturing an image of the fluorescence in the second optical state for each of the plurality of fluorescences to acquire the second fluorescence image; The method for acquiring a fluorescent image according to claim 1 .

3. The first optical filter is a first gradient filter whose transmittance changes monotonically across the respective reflection wavelength ranges, or whose transmittance changes monotonically across the respective transmission wavelength ranges. The method for acquiring a fluorescent image according to claim 2 .

4. In the second acquisition step, the fluorescence is captured in the second optical state using a second gradient filter that exhibits a change in transmittance different from that of the first gradient filter. The method for acquiring a fluorescent image according to claim 3 .

5. In the second acquisition step, the fluorescence is captured in the second optical state using a second optical filter having a transmittance different from that of the first optical filter. The method for acquiring a fluorescent image according to claim 2 .

6. In the second acquisition step, the fluorescence is captured in the second optical state in which the first optical filter is removed. The method for acquiring a fluorescent image according to claim 2 .

7. In the acquiring step, each of the plurality of fluorescent lights is imaged to acquire the first fluorescent image and the second fluorescent image. The method for acquiring a fluorescent image according to claim 1 .

8. In the acquiring step, an image of each of the plurality of fluorescent lights is captured using a color camera. The method for acquiring a fluorescent image according to claim 7 .

9. In the acquisition step, each of the plurality of fluorescent lights is imaged using a hyperspectral camera. The method for acquiring a fluorescent image according to claim 7 .

10. a clustering step of clustering a plurality of pixels determined to be monochromatic pixels in the discrimination step into L pixel groups (L is an integer equal to or greater than 2 and equal to N-1) for C fluorescence images obtained by irradiating a sample with excitation light having each of C wavelength distributions (C is an integer equal to or greater than 2), the C fluorescence images each being composed of N pixels (N is an integer equal to or greater than 2), and generating L cluster matrices in which the C fluorescence images are arranged for each of the clustered pixel groups; a calculation step of calculating a statistic of the intensity values ​​of the pixel groups constituting the C fluorescent images for each of the L cluster matrices; and an image generating step of performing unmixing on the C fluorescent images using the statistics of the C fluorescent images for each of the L cluster matrices to generate K fluorescent images showing distributions for each of K dyes (K is an integer equal to or greater than 2 and equal to or less than C). The method for acquiring a fluorescent image according to any one of claims 2 to 9.

11. an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions; an image acquisition device that acquires, for each of a plurality of fluorescent lights corresponding to the plurality of excitation light beams, a first fluorescent image in a first optical state via a fluorescent filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescent image in a second optical state in which the fluorescent light is measured with wavelength characteristics different from those of the first optical state; an image processing device that processes a plurality of the first fluorescent light images and a plurality of the second fluorescent light images, The image processing device includes: calculating an intensity ratio for each of the first and second fluorescent images, the intensity ratio being a ratio between an intensity value of a pixel of the first fluorescent image and an intensity value of a pixel of the second fluorescent image corresponding to the pixel, and calculating an intensity ratio for each of the plurality of excitation lights; determining whether the pixel is a monochromatic pixel based on the intensity ratio for each of the plurality of excitation lights; Fluorescence image acquisition device.

12. the image acquisition device images the fluorescence in the first optical state using a first optical filter having a different transmittance in each reflection wavelength range or each transmission wavelength range of the fluorescence filter unit for each of the plurality of fluorescence rays, and acquires the first fluorescence image; capturing an image of each of the plurality of fluorescent lights in the second optical state to obtain a second fluorescent light image; The fluorescence image acquisition device according to claim 11.

13. The first optical filter is a first gradient filter whose transmittance changes monotonically across the respective reflection wavelength ranges, or whose transmittance changes monotonically across the respective transmission wavelength ranges. The fluorescence image acquisition device according to claim 12.

14. the image acquisition device captures the fluorescence in the second optical state using a second gradient filter that exhibits a change in transmittance different from that of the first gradient filter; The fluorescence image acquisition device according to claim 13.

15. the image acquisition device captures the fluorescent light in the second optical state using a second optical filter having a transmittance different from that of the first optical filter; The fluorescence image acquisition device according to claim 12.

16. the image acquisition device captures the fluorescent light in the second optical state in which the first optical filter is removed. The fluorescence image acquisition device according to claim 12.

17. the image acquisition device captures images of the plurality of fluorescent lights and acquires the first fluorescent light image and the second fluorescent light image. The fluorescence image acquisition device according to claim 11.

18. The image acquisition device captures images of each of the plurality of fluorescent lights using a color camera. The fluorescence image acquisition device according to claim 17.

19. The image acquisition device captures each of the plurality of fluorescent lights using a hyperspectral camera. The fluorescence image acquisition device according to claim 17.

20. The image processing device includes: C fluorescence images are obtained by irradiating a sample with excitation light having C wavelength distributions (C is an integer of 2 or more), each of the C fluorescence images being composed of N pixels (N is an integer of 2 or more), clustering a plurality of pixels determined to be monochromatic pixels into L pixel groups (L is an integer of 2 or more and N-1 or less), and generating L cluster matrices in which the C fluorescence images are arranged for each of the clustered pixel groups; calculating a statistical value of the intensity values ​​of the pixel groups constituting the C fluorescent images for each of the L cluster matrices; performing unmixing on the C fluorescence images using the statistical values ​​of the C fluorescence images for each of the L cluster matrices to generate K fluorescence images (K is an integer between 2 and C) showing distributions for each of the K dyes; The fluorescence image acquisition device according to any one of claims 12 to 19.

21. a fluorescence image acquisition program for determining whether a pixel is a monochromatic pixel based on a first fluorescence image in a first optical state acquired through a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges for each of a plurality of fluorescence beams corresponding to each of the plurality of excitation light beams by irradiating a sample with each of the plurality of excitation light beams, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those of the first optical state, the program comprising: Computer, an intensity ratio calculation unit that calculates an intensity ratio, which is a ratio between an intensity value of a pixel of the first fluorescent image and an intensity value of a pixel of the second fluorescent image corresponding to the pixel, for the first fluorescent image and the second fluorescent image, and calculates the intensity ratio for each of the plurality of excitation lights; and a fluorescence image acquisition program that causes the pixel to function as a monochrome pixel discrimination unit that discriminates whether the pixel is a monochrome pixel based on the intensity ratio for each of the plurality of excitation lights.

Images