Fluorescence image acquisition method, fluorescence image acquisition device, and fluorescence image acquisition program

Abstract

A fluorescence image acquisition system (1) is provided with: an irradiation device (2) that irradiates a sample with excitation light having a plurality of wavelength distributions; an image acquisition device (3) that acquires a plurality of fluorescent lights corresponding to each of the plurality of excitation lights via a fluorescent light filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; acquiring a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with a wavelength characteristic different from that of the first optical state; and an image processing device (4) for processing the plurality of first fluorescence images and the plurality of second fluorescence images, the image processing device (4) calculating, for the first fluorescence images and the second fluorescence images, an intensity ratio, which is a ratio between an intensity value of a pixel of the first fluorescence image and an intensity value of a pixel of the second fluorescence image corresponding to the pixel; an intensity ratio of each of the plurality of excitation lights is calculated, and whether the pixel is a monochromatic pixel is determined based on the intensity ratio of each of the plurality of excitation lights.

Description

Technical Field

[0001] One aspect of the embodiment relates to a fluorescence image acquisition method, a fluorescence image acquisition apparatus, and a fluorescence image acquisition program. Background Art

[0002] Currently, multiple staining methods are used to simultaneously stain multiple substances within a sample, such as biological tissue. Furthermore, in order to observe the substances within the multi-stained sample, the sample is irradiated with excitation light to obtain a fluorescence image. For example, the following non-patent document 1 discloses a method for applying non-negative matrix factorization (NMF) to blindly unmix a fluorescence image in which fluorescence in multiple wavelength ranges is observed to obtain a separated image of each substance within the sample. Furthermore, the following non-patent document 2 discloses a method for unmixing a fluorescence image, clustering the fluorescence image, extracting the maximum value of the fluorescence intensity in each clustered pixel group, and generating a separated image based on this maximum value.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-patent literature 1: Binjie Qin et al., "Target / Background ClassificationRegularized Nonnegative Matrix Factorization for Fluorescence Unmixing", IEEETRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT,VOL.65,NO.4,APRIL 2016

[0006] Non-patent literature 2: Tristan D.McRae et al., "Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning", PLOS ONE, December 2, 2019 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] However, in conventional techniques, it is difficult to distinguish single-color pixels in a fluorescence image when observing the fluorescence image. Therefore, in order to observe the fluorescence image efficiently, it is desirable to distinguish single-color pixels in the fluorescence image.

[0009] Therefore, one aspect of the embodiments has been developed in view of the above-mentioned problems, and an object of the invention is to provide a fluorescence image acquisition method, a fluorescence image acquisition device, and a fluorescence image acquisition program that can distinguish monochromatic pixels among fluorescent pixels.

[0010] Technical solutions to solve problems

[0011] A first aspect of the embodiment provides a fluorescence image acquisition method, comprising: an irradiation step of irradiating a sample with each of excitation lights having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescences corresponding to the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, via a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; a calculation step of calculating, for the first fluorescence image and the second fluorescence image, an intensity ratio, which is a ratio of an intensity value of a pixel of the first fluorescence image to an intensity value of a pixel of the second fluorescence image corresponding to the pixel, and calculating the intensity ratio for each of the plurality of excitation lights; and a determination step of determining whether a pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights.

[0012] Alternatively, a second aspect of the embodiment provides a fluorescence image acquisition device comprising: an irradiation device for irradiating a sample with each of a plurality of excitation lights having a wavelength distribution; an image acquisition device for acquiring, for each of a plurality of fluorescence lights corresponding to the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, via a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and an image processing device for processing the plurality of first fluorescence images and the plurality of second fluorescence images, the image processing device calculating, for the first fluorescence images and the second fluorescence images, an intensity ratio, which is a ratio of an intensity value of a pixel in the first fluorescence image to an intensity value of a pixel in the second fluorescence image corresponding to the pixel, and calculating the intensity ratio for each of the plurality of excitation lights, and determining whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights.

[0013] Alternatively, a third aspect of the embodiment provides a fluorescence image acquisition program for determining whether a pixel is a monochrome pixel based on a first fluorescence image in a first optical state acquired through a fluorescence filter section having multiple reflection wavelength ranges and multiple transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, by irradiating a sample with each of excitation lights having a plurality of wavelength distributions. In the fluorescence image acquisition program, a computer is caused to function as an intensity ratio calculation section and a monochrome pixel determination section. The intensity ratio calculation section calculates, for the first and second fluorescence images, an intensity ratio, i.e., a ratio of an intensity value of a pixel in the first fluorescence image to an intensity value of a pixel in the second fluorescence image corresponding to the pixel, and calculates the intensity ratio for each of the plurality of excitation lights. The monochrome pixel determination section determines whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights.

[0014] According to the first, second, or third aspects, for each of the multiple fluorescences corresponding to the multiple excitation lights, a first fluorescence image is acquired under a first optical state and a second fluorescence image is acquired under a second optical state in which the fluorescence is measured using wavelength characteristics different from those of the first optical state. Then, the intensity ratio, i.e., the ratio of the intensity value of a pixel in the first fluorescence image to the intensity value of a pixel in the second fluorescence image, is calculated to calculate the intensity ratio for each of the multiple excitation lights. Furthermore, based on the intensity ratio of each of the multiple excitation lights, it is determined whether the pixel is a monochrome pixel. Thus, even in a fluorescence image containing a mixture of pixels reflecting fluorescence from multiple pigments and monochrome pixels reflecting monochromatic fluorescence from a single pigment, it is possible to identify the monochrome pixels based on the multiple pixels of the fluorescence image. In other words, it is possible to identify the monochrome pixels in the fluorescence image.

[0015] Effects of the Invention

[0016] According to an aspect of the present disclosure, it is possible to discriminate monochrome pixels in a fluorescent image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic configuration diagram of the fluorescence image acquisition system 1 according to the first embodiment.

[0018] Figure 2 Yes Figure 1 A perspective view of the structure of the fluorescence image acquisition system 1.

[0019] Figure 3 Yes Figure 1 FIG. 1 is a block diagram showing an example of a hardware configuration of the image processing device 4.

[0020] Figure 4 Yes Figure 1A block diagram of the functional structure of the image processing device 4 is shown.

[0021] Figure 5 3 is a diagram for explaining the fluorescence intensities of the first fluorescent image and the second fluorescent image.

[0022] Figure 6 1 and 2 are diagrams for explaining characteristics of an intensity ratio in a first pixel with respect to first excitation light and an intensity ratio in a first pixel with respect to second excitation light.

[0023] Figure 7 It is a diagram schematically showing the characteristics of the intensity ratio.

[0024] Figure 8 3 is a diagram showing an image of a pixel group clustered by the clustering unit 204 .

[0025] Figure 9 An image showing the matrix data Y′ regenerated by the statistical value calculation unit 205 and the corresponding pigment matrix data X′.

[0026] Figure 10 This is a flowchart showing the procedure of the fluorescence image acquisition method according to the first embodiment.

[0027] Figure 11 It is a schematic configuration diagram of a fluorescence image acquisition system 1A according to the second embodiment.

[0028] Figure 12 This is a diagram for explaining the intensity ratio of each pixel when a color camera is used.

[0029] Figure 13 This is a diagram for explaining the intensity ratio of each pixel when a hyperspectral camera is used.

[0030] Figure 14 It is a schematic configuration diagram of a fluorescence image acquisition system 1B according to the third embodiment.

[0031] Figure 15 It is a schematic configuration diagram of a fluorescence image acquisition system 1C according to a fourth embodiment.

[0032] Figure 16 This is a diagram showing the clustering results in the example of generating a fluorescence image.

[0033] Figure 17 This is a diagram showing the results of generating a fluorescence image from each dye in an example of generating a fluorescence image. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same elements or elements having the same functions will be denoted by the same reference numerals, and repeated descriptions will be omitted.

[0035] [First embodiment]

[0036] Figure 1 This is a schematic diagram of a fluorescence image acquisition system 1, which serves as a fluorescence image acquisition device according to a first embodiment. Fluorescence image acquisition system 1 is a device for generating fluorescence images used to determine the distribution of pigments within a sample, such as biological tissue, serving as an observation target. The images generated by fluorescence image acquisition system 1 are used for purposes such as developing pharmaceuticals and studying treatment methods through analysis of these images. Fluorescence image acquisition system 1 includes an irradiation device 2 configured to irradiate sample S with excitation light, an image acquisition device 3 configured to acquire an image of fluorescence generated in response to irradiation of sample S with the excitation light, and an image processing device 4 that processes the image acquired by image acquisition device 3. Image acquisition device 3 and image processing device 4 can be configured to transmit and receive image data between them using wired or wireless communication, or they can be configured to input and output image data via a recording medium.

[0037] Figure 2 Yes Figure 1 A perspective view of the structure of the fluorescence image acquisition system 1. Figure 2 In the figure, the dotted line with an arrow represents the optical path of the excitation light, and the solid line with an arrow represents the optical path of the fluorescence. The irradiation device 2 comprises an excitation light source 2a and an excitation optical filter 2b. The image acquisition device 3 comprises a dichroic mirror 11, a fluorescence filter 3a, a first optical filter 13, and a first camera 15.

[0038] The excitation light source 2a is a light source that can switch between multiple bands (wavelength distributions) of excitation light for irradiation, and is, for example, an LED (Light Emitting Diode) light source, a light source composed of multiple monochromatic laser light sources, or a light source that combines a white light source and a wavelength selection optical element. The excitation optical filter section 2b is a multi-bandpass filter, which is arranged on the optical path of the excitation light of the excitation light source 2a and has the property of allowing light in a specified multiple wavelength range to pass through. The transmission band of the excitation optical filter section 2b is set according to the multiple bands of excitation light that can be used. The color separation mirror 11 is an optical component, which is arranged between the excitation optical filter section 2b and the sample S and has the property of reflecting the excitation light toward the sample S and allowing the corresponding fluorescence emitted from the sample S to pass through. In the first embodiment, the irradiation device 2 irradiates M (M is an integer greater than 2) bands of excitation light. Hereinafter, the excitation light in the M bands will be referred to as the first excitation light, the second excitation light, ..., and the Mth excitation light, respectively.

[0039] The fluorescence filter section 3a is a multi-bandpass filter, positioned in the optical path of the fluorescence transmitted by the dichroic mirror 11, and has the property of transmitting light within a plurality of predetermined wavelength ranges. The fluorescence filter section 3a has multiple reflection wavelength ranges and multiple transmission wavelength ranges. The reflection wavelength ranges and transmission wavelength ranges are arranged alternately. The reflection wavelength range is, for example, a wavelength range between the transmission wavelength ranges. The transmission wavelength range of the fluorescence filter section 3a is set based on the wavelength band of fluorescence generated by the pigment contained in the sample S to be observed.

[0040] The first optical filter 13 is an optical system positioned on the optical path of the fluorescence transmitted by the fluorescence filter section 3a and used to obtain wavelength information of the fluorescence. The first optical filter 13 includes a switching mechanism (not shown) that can switch the position of the first optical filter 13 between a position on the optical path of the fluorescence from the fluorescence filter section 3a and a position offset from the optical path. In the first embodiment, the state in which the wavelength information of the fluorescence can be obtained through the first optical filter 13 by placing the first optical filter 13 on the optical path of the fluorescence is referred to as the first optical state, while the state in which the wavelength information of the fluorescence can be obtained without passing through the first optical filter 13 by not placing the first optical filter 13 on the optical path of the fluorescence is referred to as the second optical state. The image acquisition device 3 can measure fluorescence in both the first optical state and the second optical state in which fluorescence is measured with wavelength characteristics different from those in the first optical state.

[0041] The first optical filter 13 is a filter whose transmittance varies in each reflection wavelength range or each transmission wavelength range of the fluorescence filter section 3a. In the first embodiment, the first optical filter 13 is a tilted filter having a wavelength characteristic in which the transmittance increases linearly with increasing wavelength. That is, the transmittance of the first optical filter 13 increases monotonically within each transmission wavelength range of the fluorescence filter section 3a. In the following description of the first embodiment, this tilted filter, i.e., the first optical filter 13, will be referred to as the first tilted filter 13. The first optical filter 13 may also be a tilted filter having a wavelength characteristic in which the transmittance decreases linearly with increasing wavelength. Alternatively, the first optical filter 13 may be a tilted filter whose transmittance varies monotonically within each reflection wavelength range of the fluorescence filter section 3a.

[0042] The first camera 15 is a photographing device that photographs a two-dimensional image composed of N (N is an integer greater than or equal to 2, for example, 2048×2048) pixels. It is a camera that photographs the fluorescence transmitted through the first tilted filter 13 to obtain a first fluorescence image when the first tilted filter 13 is set on the optical path of the fluorescence. That is, the first fluorescence image is a fluorescence image obtained in a first optical state. In addition, the first camera 15 photographs the fluorescence when the first tilted filter 13 deviates from the optical path of the fluorescence to obtain a second fluorescence image. That is, the second fluorescence image is a fluorescence image obtained in a second optical state. The first camera 15 outputs the obtained first and second fluorescence images to the image processing device 4 using communication or via a recording medium. In the first embodiment, the N pixels are respectively set as the first pixel, the second pixel, ..., and the Nth pixel.

[0043] Next, refer to Figure 3 and Figure 4 The configuration of the image processing device 4 will be described. Figure 3 is a block diagram showing an example of the hardware configuration of the image processing device 4. Figure 4 3 is a block diagram showing the functional structure of the image processing device 4 .

[0044] like Figure 3 As shown, the image processing device 4 is a computer that physically includes a CPU (Central Processing Unit) 101 as a processor, a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103 as a recording medium, a communication module 104, and an input / output module 106, etc., which are electrically connected. Furthermore, the image processing device 4 may include a display, keyboard, mouse, touch panel, monitor, etc. as input / output devices, and may also include a data recording device such as a hard disk drive or semiconductor memory. Furthermore, the image processing device 4 may be composed of multiple computers.

[0045] like Figure 4 As shown, the image processing device 4 includes, as functional components, an image acquisition unit 201 , an intensity ratio calculation unit 202 , a monochrome pixel determination unit 203 , a clustering unit 204 , a statistic calculation unit 205 , and an image generation unit 206 . Figure 4 Each functional unit of the image processing device 4 shown is realized as follows: by reading a program (fluorescence image acquisition program of the embodiment) into the hardware such as the CPU 101 and the RAM 102, the communication module 104 and the input / output module 106 are operated under the control of the CPU 101, and data in the RAM 102 is read and written. The CPU 101 of the image processing device 4 executes the computer program to Figure 4 Each functional unit functions to sequentially execute the processing corresponding to the fluorescence image acquisition method described below. Furthermore, CPU 101 may be a single piece of hardware or a component such as a soft processor installed in a programmable logic device such as an FPGA. RAM and ROM may also be single pieces of hardware or components built into a programmable logic device such as an FPGA. The various data required to execute the computer program and the various data generated by executing the computer program are all stored in a built-in memory such as ROM 103 or RAM 102, or a storage medium such as a hard disk drive. The functions of the functional components of image processing device 4 are described in detail below.

[0046] The image acquisition unit 201 acquires a first fluorescent image and a second fluorescent image from the image acquisition device 3 for each of the first to M-th excitation lights. The first fluorescent image is a fluorescent image acquired in a first optical state for a plurality of fluorescent lights corresponding to the first to M-th excitation lights emitted from the irradiation device 2. The second fluorescent image is a fluorescent image acquired in a second optical state for a plurality of fluorescent lights corresponding to the first to M-th excitation lights emitted from the irradiation device 2.

[0047] Reference Figure 5 An example of the fluorescence intensity of the first fluorescent image and the second fluorescent image is described. Figure 5 In FIG, part (a) shows the wavelength characteristics of the fluorescence intensity of the first fluorescence image acquired in the first optical state, and part (b) shows the wavelength characteristics of the fluorescence intensity of the second fluorescence image acquired in the second optical state. Here, it is assumed that the first fluorescence, the second fluorescence, and the third fluorescence are captured as three types of fluorescence. Figure 5 In part (a), FI1, FI2, and FI3 respectively represent the fluorescence intensities of the first fluorescence, the second fluorescence, and the third fluorescence after passing through the first tilt filter 13. Figure 5 In part (b), FI4, FI5, and FI6 represent the fluorescence intensities of the first fluorescence, the second fluorescence, and the third fluorescence that have not passed through the first tilt filter, respectively. In addition, T1 schematically represents the transmittance within the transmission wavelength range of the fluorescence filter section 3a. Figure 5 As shown in part (b) of FIG. 1 , in the second optical state, the magnitude of each fluorescence intensity is about the same, but as Figure 5 As shown in part (a) of FIG, in the first optical state, the fluorescence intensity FI1 of the first fluorescence, the fluorescence intensity FI2 of the second fluorescence, and the fluorescence intensity FI3 of the third fluorescence increase in this order. This is because the transmittance of the first tilt filter 13 increases linearly with increasing wavelength.

[0048] The intensity ratio calculation unit 202 calculates the intensity ratio, i.e., the ratio of the intensity value of a pixel in the first fluorescent image to the intensity value of a pixel in the second fluorescent image corresponding to the pixel, for each of the first to M-th excitation lights. Here, when the ratio of the intensity value of the n-th pixel (n is an integer from 1 to N) in the first fluorescent image to the intensity value of the n-th pixel in the second fluorescent image with respect to the m-th excitation light (m is an integer from 1 to M), i.e., the intensity ratio, is R. mn When the intensity ratio calculation unit 202 calculates the intensity ratio R based on the measured intensity value mn In the first embodiment, the intensity ratio is the ratio of the intensity value of a pixel in the first fluorescent image to the intensity value of a pixel in the second fluorescent image. Alternatively, the intensity ratio may be the ratio of the intensity value of a pixel in the second fluorescent image to the intensity value of a pixel in the first fluorescent image.

[0049] As an example, refer to Figure 6 The intensity ratio R in the first pixel relative to the first excitation light is shown as 11 , and the intensity ratio R in the first pixel relative to the second excitation light 21 Features. Figure 6 In FIG. 1 , a case where a first pigment is irradiated with the first excitation light and the second excitation light is assumed as an example.

[0050] exist Figure 6 Part (a) schematically shows the excitation light spectra ES1 and ES2 of the first excitation light and the second excitation light with respect to the absorption spectrum AS of the first pigment. Figure 6 Part (b) schematically shows the transmittance T1 of the fluorescence filter section 3a within the transmission wavelength range, and the transmittance T2 of the first tilt filter 13 after adding the transmittance T1, relative to the fluorescence spectrum FS of the first dye. AS (λ), the excitation light spectrum ES1 is set to f ES1 (λ), set the excitation light spectrum ES2 to f ES2 (λ), the fluorescence spectrum FS is set to f FS (λ), let the transmittance T1 be S T1 (λ), let the transmittance T2 be S T2 (λ), then when the first excitation light is irradiated, the intensity value I of the first pixel obtained in the first optical state is 11 It can be calculated as shown in the following formula (1). On the other hand, when the first excitation light is irradiated, the intensity value I of the first pixel acquired in the second optical state is 12 It can be calculated as shown in the following formula (2).

[0051] [Mathematical formula 1]

[0052] I 11 =∫f AS (λ)f ES1 (λ)dλ×∫f FS (λ)S T2 (λ)dλ…(1)

[0053] I 12 =∫f AS (λ)f ES1 (λ)dλ×∫f FS (λ)S T1 (λ)dλ…(2)

[0054] When the second excitation light is irradiated, the intensity value I of the first pixel obtained in the first optical state is 21 It can be calculated as shown in the following formula (3). On the other hand, when the second excitation light is irradiated, the intensity value I of the first pixel obtained in the second optical state is 22 It can be calculated as shown in the following formula (4).

[0055] [Mathematical formula 2]

[0056] I 21 =∫f AS (λ)f ES2 (λ)dλ×∫f FS (λ)S T2 (λ)dλ…(3)

[0057] I 22 =∫f AS (λ)f ES2 (λ)dλ×∫f FS (λ)S T1 (λ)dλ…(4)

[0058] According to the above equations (1) and (2), the intensity ratio R in the first pixel corresponding to the first excitation light is 11 It is calculated as shown in the following formula (5). According to the above formulas (3) and (4), the intensity ratio R in the first pixel corresponding to the second excitation light is 21 It is calculated as shown in the following formula (6).

[0059] [Mathematical formula 3]

[0060]

[0061] In the above formula (5), the common value in the denominator and the numerator that depends on the first excitation light, ie, ∫f AS (λ)f ES1 (λ)dλ, therefore, the intensity ratio R 11Independent of the first excitation light. Similarly, the intensity ratio R 21 Therefore, as shown in the above formulas (5) and (6), the intensity ratio R 11 and intensity ratio R 21 However, the intensity is greater than R 11 and intensity ratio R 21 The calculation is based on the actual measured intensity value. Therefore, please note that the intensity ratio R 11 and intensity ratio R 21 They do not necessarily have the same value.

[0062] Here, when the first pixel reflects the fluorescence from the first pigment and the fluorescence from the second pigment in addition to the fluorescence from the first pigment, the intensity ratio R′ in the first pixel corresponding to the first excitation light is 11 , and the intensity ratio R' in the first pixel corresponding to the second excitation light 21 Let the absorption spectrum of the second pigment be f' AS (λ), the fluorescence spectrum of the second dye is f' FS (λ). Intensity ratio R' 11 and R' 21 It is calculated as shown in the following formulas (7) and (8).

[0063] [Formula 4]

[0064]

[0065] In the above formula (7), the values ​​in the denominator and numerator that depend on the first excitation light are eliminated, so the intensity ratio R' 11 Depends on the first excitation light. Similarly, the intensity ratio R' 21 Depends on the second excitation light. That is, the intensity ratio R' 11 and intensity ratio R' 21 They do not become the same value.

[0066] The characteristics of the above-mentioned intensity ratio will be described. Figure 7 This is a graph schematically showing the intensity ratio RX1 when the first pixel reflects the fluorescence from the first pigment, the intensity ratio RX2 when the first pixel reflects the fluorescence from the second pigment, and the intensity ratio RX3 when the first pixel reflects the fluorescence from the first pigment and the second pigment in combination. Figure 7 In the figure, the intensity ratio of each of the first to third excitation lights is shown. Figure 7 As shown in FIG. 1 , the intensity ratio RX1 and the intensity ratio RX2 are the same for each excitation light, whereas the intensity ratio RX3 is different for each excitation light. Figure 7As shown in the above equations (5) and (6), the intensity ratio RX1 and the intensity ratio RX2 are different values. This is because the transmittance T2 (i.e., S T2 ) is a value that depends on the wavelength λ. In addition, the transmittance T1 (ie, S T1 ) is a value that depends on the wavelength λ. Assume that T1 and S T2 When the intensity ratio is constant and independent of the wavelength λ, the intensity ratio becomes constant, and the intensity ratio RX1 and the intensity ratio RX2 become the same value. The monochrome pixel determination unit 203 described later can determine whether a pixel is a monochrome pixel based on the characteristics of the intensity ratio.

[0067] The monochrome pixel determination unit 203 determines whether each pixel is a monochrome pixel based on the intensity ratio of each of the first to M-th excitation lights. The monochrome pixel determination unit 203 calculates the deviation indexes V1 to V2 of the intensity ratio of each of the first to M-th excitation lights for each of the first to N-th pixels. N In each of the above, it is determined whether each pixel is monochrome based on the index. The following describes a method for calculating the index V1 for the first pixel by the monochrome pixel determination unit 203. The monochrome pixel determination unit 203 calculates the indexes V2 to V1 in the same manner as the index V1. N .

[0068] As an example, the monochrome pixel determination unit 203 calculates the index V1 using the following equation (9). ave1 is the intensity ratio R of the first to M-th excitation light in the first pixel 11 ~R M1 In this case, the monochrome pixel determination unit 203 can easily calculate the index V1. ave1 For example, it can be calculated by the following formula (10) or (11). 1i is the intensity of the first pixel in the first optical state when irradiated with the i-th excitation light, y 2i is the intensity of the first pixel in the second optical state when irradiated with the i-th excitation light. ave1 Not limited to the following formulas (10) and (11), R ave1 Can be weighted average or R 11 ~R M1 The middle value of .

[0069] [Formula 5]

[0070]

[0071] Alternatively, the monochrome pixel determination unit 203 calculates the index V1 using the following equation (12). i1 Ri1 For example, when the first camera 15 is a CMOS (Complementary Metal-Oxide Semiconductor) camera, the noise σ is calculated by the following formula (13): i1 Here, y' 1i To y 1i Intensity when converted to electron number, y' 2i To y 2i The intensity converted to electron counts is represented by ε, which represents the readout noise of the CMOS camera. Furthermore, if shot noise dominates, the readout noise ε can be ignored. In the case of the following equation (12), the monochrome pixel discrimination unit 203 follows a chi-square distribution with M-1 degrees of freedom, so the threshold value described below can be set regardless of the sample S.

[0072] [Formula 6]

[0073]

[0074] Alternatively, the monochrome pixel determination unit 203 calculates the index V1 using the following equation (14). Here, p is a constant greater than 1, and w i is the weight corresponding to the i-th excitation light, RF i For example, according to R ave1 , intensity ratio calculated based on pigment reference information, and statistical value of intensity ratio obtained from multiple pixels. Taking into account the measurement conditions of each excitation light, noise, background light, etc., RF i It can be changed for each excitation light. Alternatively, it can be any value. In the following formula (14), R i1 Replace with R i1 The related function G(R i1 ).

[0075] [Formula 7]

[0076]

[0077] Indicators V1~V N The index is not limited to the above-mentioned formulas (9) to (14), and any index other than the above-mentioned formulas (9) to (14) may be used as long as it can evaluate the deviation of the intensity ratio.

[0078] The monochrome pixel determination unit 203 determines the pixel values ​​based on the indices V1 to V N, and determines whether each of the first to Nth pixels is a monochrome pixel. As an example, the monochrome pixel determination unit 203 determines a pixel whose deviation index is below a threshold as a monochrome pixel, and determines a pixel whose deviation index is greater than the threshold as a non-monochrome pixel. The threshold may be a value preset by the user or determined by the monochrome pixel determination unit 203. In addition, when the threshold is determined by the monochrome pixel determination unit 203, it is not limited to the case where the monochrome pixel determination unit 203 automatically determines the threshold. The monochrome pixel determination unit 203 may determine the threshold based on a parameter input by the user.

[0079] The clustering unit 204 performs clustering on a plurality of pixels (hereinafter referred to as "a plurality of monochrome pixels") identified as monochrome pixels by the monochrome pixel discrimination unit 203. Prior to the clustering process, the clustering unit 204 generates matrix data Y in which the fluorescence intensity values ​​of N pixels are arranged in a one-dimensional array in parallel. The fluorescence intensity values ​​of the N pixels constitute each of C fluorescence images, each consisting of N pixels, obtained by irradiating the sample with each of C excitation lights having a wavelength distribution (C is an integer greater than or equal to 2). The C fluorescence images can be an image formed by summarizing the M first fluorescence images and M second fluorescence images acquired by the image acquisition device 3 by irradiating the sample with the first to Mth excitation lights, or they can be M second fluorescence images. Alternatively, they can be C fluorescence images captured under the second optical state by irradiating the sample with C excitation lights separately from the first to Mth excitation lights.

[0080] For example, the clustering unit 204 clusters the multiple monochrome pixels into L pixel groups (L is an integer greater than or equal to 2 and less than or equal to N-1) based on the average value of their respective intensity ratios. The number of clustered pixel groups, L, corresponds to, for example, the number of pigment types that can be present in the sample S and is pre-set as a parameter stored in the image processing device 4. The number of pixel groups, L, can be determined based on the type of excitation light and the number of wavelength distributions, C, or independently of the type of excitation light and the number of wavelength distributions, C. The average values ​​of the intensity ratios of monochromatic pixels of the same color are the same or close to each other. The clustering unit 204 clusters the multiple monochrome pixels into L pixel groups by determining the distance (the degree of similarity) between the average values ​​of the intensity ratios of each of the multiple monochrome pixels. The clustering unit 204 divides the matrix data Y, which is a one-dimensional array of fluorescence intensity values ​​for C pixels of the fluorescence image, into cluster matrices for each of the L pixel groups and regenerates the cluster matrix. The average value of the intensity ratios of the multiple monochrome pixels can be, for example, a simple average or a weighted average. The average value can be calculated using other calculation methods. The clustering unit 204 can perform clustering based on the median value of the intensity ratio of each of the multiple monochrome pixels. The clustering unit 204 can also perform clustering based on the intensity of each of the C excitation lights of each of the multiple monochrome pixels.

[0081] Figure 8 , an image of a pixel group clustered by the clustering unit 204 is shown in FIG. Figure 8 As shown, the sample S contains three types of pigments, namely, pigment C1, pigment C2, and pigment C3. The clustering unit 204 clusters a plurality of single-color pixels into three pixel groups PGr1 to PGr3.

[0082] The statistical value calculation unit 205 obtains a mixing matrix A for generating K pigment images representing the distribution of K pigments (K is an integer greater than or equal to 2 and less than or equal to C) from the C fluorescence images based on the L cluster matrices obtained for the sample S. Typically, the mixing matrix A is used to express the relationship between matrix data Y, which is an observation matrix, and pigment matrix data X, which is a one-dimensional array of K pigment images arranged in parallel for each pixel, using the following equation:

[0083] Y=AX

[0084] Here, Y is the matrix data of C rows and N columns, A is the matrix data of C rows and K columns, and X is the matrix data of K rows and N columns. Conversely, if the value of the mixing matrix A is obtained, the pigment matrix data X can be obtained using the inverse matrix A of the mixing matrix A. －1 And matrix data Y, through the following formula;

[0085] X=A －1 Y

[0086] This process is called demixing.

[0087] Here, the statistical value calculation unit 205 compresses the matrix data Y generated by the clustering unit 204, based on the pixel groups clustered by the clustering unit 204, to regenerate matrix data Y'. Specifically, the statistical value calculation unit 205 calculates a statistical value for each pixel group in the clustered matrix, using the fluorescence intensity of each row of the matrix data Y as the target. The statistical value calculation unit 205 compresses the pixel groups in each row into a single pixel having the calculated statistical value. Thus, the statistical value calculation unit 205 regenerates matrix data Y', which consists of C rows and L columns. The statistical value calculation unit 205 can calculate the average value based on the cumulative value of the fluorescence intensity, the mode of the fluorescence intensity, or the median of the fluorescence intensity.

[0088] In addition, the statistical value calculation unit 205 uses the following equation which also includes the mixing matrix A in the regenerated matrix data Y' and the pigment matrix data X' compressed in the same manner as the pigment matrix data X:

[0089] Y'=AX'

[0090] The property holds true, and the mixing matrix A is derived based on the matrix data Y'. Figure 9Matrix data Y' regenerated by the statistical value calculation unit 205 and the corresponding pigment matrix data X' are shown in FIG. Figure 9 Each square in the figure represents one element of the matrix data. Thus, the pigment matrix data X and matrix data Y divided into three pixel groups PGr1 to PGr3 are compressed into three columns of pigment matrix data X' and matrix data Y', using the statistical values ​​of each pixel group PGr1 to PGr3 as representative values.

[0091] The statistical value calculation unit 205 derives the mixing matrix A based on the matrix data Y' as follows. Specifically, the statistical value calculation unit 205 sets an initial value for the mixing matrix A, sequentially changes the value of the mixing matrix A, and simultaneously calculates the loss function (loss value) Los represented by the following equation (15), thereby deriving the mixing matrix A so as to reduce the value of the loss function Los. Furthermore, a regularization term such as the L1 norm λ|A| (λ represents a coefficient indicating the degree of emphasis on the regularization term) may be added to this loss function.

[0092] [Formula 8]

[0093]

[0094] In equation (15), j is a parameter representing the position of the matrix data row (corresponding to the wavelength band of the excitation light). The matrix subscript 1j represents the matrix data of the j-th row of the first cluster matrix, the matrix subscript 2j represents the matrix data of the j-th row of the second cluster matrix, and the matrix subscript 3j represents the matrix data of the j-th row of the third cluster matrix. Furthermore, the parameters a, b, and c represent the average of the statistical values ​​of each column of the matrix data Y'.

[0095] As described above, the statistical value calculation unit 205 calculates a loss function for each of the L cluster matrices divided by the clustering unit 204, referring to the statistical values ​​of the C matrix data Y'. Based on the sum of the L loss functions, the loss function Los is calculated, and the mixing matrix A is obtained based on the loss function Los. At this time, the statistical value calculation unit 205 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values ​​a, b, and c of the statistical values ​​of the C matrix data Y', and then calculates the sum of the corrected loss functions to obtain the loss function Los. Alternatively, the statistical value calculation unit 205 may correct the loss function for each of the L cluster matrices by dividing the row component of each wavelength band of the excitation light of the differential value Y'-AX' by the C statistical values ​​corresponding to each wavelength band of the excitation light.

[0096] Alternatively, the statistics calculation unit 205 can generate the mixing matrix A as follows. For each of the C fluorescence images, the statistics calculation unit 205 can calculate the aforementioned statistics for each of the L clustered pixel groups. Then, the calculated statistics are arranged for each of the C fluorescence images to generate a mixing matrix A with C rows and L columns. In other words, the mixing matrix A can be identical to the matrix data Y'. In this case, the number of clusters and the number of pigments are equal, making it easier to generate the mixing matrix A.

[0097] The image generation unit 206 unmixes the C fluorescence images obtained with the sample S as the observation target using the mixing matrix A derived by the statistic calculation unit 205, thereby obtaining K pigment images. Specifically, the image generation unit 206 applies the inverse matrix A of the mixing matrix A to the matrix data Y generated by the clustering unit 204 based on the C fluorescence images. －1 , calculates the pigment matrix data X. The image generation unit 206 then regenerates K pigment images based on the pigment matrix data X and outputs the regenerated K pigment images. The output destination may be an output device of the image processing device 4, such as a display or touch panel display, or an external device connected to the image processing device for data communication.

[0098] Next, the procedure of observation processing for the sample S using the fluorescence image acquisition system 1 according to the first embodiment, that is, the flow of the fluorescence image acquisition method according to the first embodiment, will be described. Figure 10 1 is a flowchart showing the procedure of observation processing performed by the fluorescence image acquisition system 1 .

[0099] First, the sample S is irradiated with each of the excitation lights of a plurality of wavelength distributions using the irradiation device 2 (step S1; irradiation step). Next, the image acquisition unit 201 of the image processing device 4 acquires, for each of the plurality of fluorescences corresponding to the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, via the fluorescence filter unit 3a (step S2; acquisition step). In the first embodiment, step S2 includes a first acquisition step of acquiring the first fluorescence image by photographing the fluorescence in the first optical state, and a second acquisition step of acquiring the second fluorescence image by photographing the fluorescence in the second optical state. In addition, steps S1 and S2 can be repeated alternately. For example, the first excitation light can be irradiated in the irradiation step, and after acquiring the first fluorescence image in the first acquisition step, the process returns to the irradiation step, irradiates the first excitation light, and acquires the second fluorescence image in the second acquisition step. Alternatively, the process can return to the irradiation step and irradiate the second excitation light again.

[0100] Furthermore, the intensity ratio calculation unit 202 of the image processing device 4 calculates the intensity ratio, which is the ratio of the intensity value of a pixel in the first fluorescent image to the intensity value of the corresponding pixel in the second fluorescent image, for the first and second fluorescent images, and calculates the intensity ratio for each of the plurality of excitation lights (step S3; calculation step). Next, the monochrome pixel determination unit 203 of the image processing device 4 determines whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights (step S4; determination step).

[0101] Furthermore, the clustering unit 204 of the image processing device 4 clusters the plurality of pixels identified as monochromatic pixels in the discrimination step into L pixel groups, using the C fluorescence images, each consisting of N pixels, obtained by irradiating the sample with excitation light having C wavelength distributions (step S5; clustering step). Next, the clustering unit 204 of the image processing device 4 generates matrix data Y in which the N pixels of the C fluorescence images are arranged in parallel (step S6; clustering step).

[0102] Furthermore, the statistical value calculation unit 205 of the image processing device 4 calculates the statistical values ​​of the L pixel groups, and matrix data Y' is regenerated based on the matrix data Y generated by the clustering unit 204 (step S7; calculation step). Next, the statistical value calculation unit 205 of the image processing device 4 derives a mixing matrix A based on the matrix data Y' (step S8; image generation step). Next, the image generation unit 206 of the image processing device 4 uses the mixing matrix A to unmix the matrix data Y generated based on the C fluorescence images of the sample S, thereby regenerating K pigment images (step S9; image generation step). Finally, the image generation unit 206 of the image processing device 4 outputs the regenerated K pigment images (step S10). The above-described observation process for the sample S is completed.

[0103] According to the fluorescence image acquisition system 1 of the first embodiment described above, for each of the multiple fluorescences corresponding to the multiple excitation lights, a first fluorescence image is acquired under a first optical state, and a second fluorescence image is acquired under a second optical state, in which the fluorescence is measured using wavelength characteristics different from those of the first optical state. Then, the intensity ratio is calculated, which is the ratio of the intensity value of a pixel in the first fluorescence image to the intensity value of a pixel in the second fluorescence image. Furthermore, based on the intensity ratio of each of the multiple excitation lights, whether a pixel is a single color pixel is determined. Thus, even in a fluorescence image containing a mixture of pixels reflecting fluorescence from multiple pigments and single color pixels reflecting fluorescence from a single pigment, single color pixels can be identified based on the multiple pixels in the fluorescence image. In other words, single color pixels in the fluorescence image can be identified.

[0104] The above-mentioned effects will be described in detail. According to the fluorescence image acquisition system 1 of the first embodiment, the intensity ratio calculation unit 202 calculates the intensity ratio R which is the ratio of the intensity values ​​in the n-th pixel. mn When the nth pixel is a monochrome pixel, the intensity ratio R 1n ~R Mn is the same value, or the nth pixel is not a monochrome pixel, the intensity ratio R 1n ~R Mn Based on this feature, the monochrome pixel determination unit 203 calculates the intensity ratio R 1n ~R Mn By calculating a deviation index and determining whether the index is below a threshold, it is possible to determine whether the n-th pixel is a monochrome pixel.

[0105] In the first embodiment, the acquisition step (step S2) includes: a first acquisition step for capturing each of the multiple fluorescent lights in a first optical state using a first optical filter 13 having a different transmittance within each reflection wavelength range or each transmission wavelength range of the fluorescence filter section 3a to obtain a first fluorescent light image; and a second acquisition step for capturing each of the multiple fluorescent lights in a second optical state to obtain a second fluorescent light image. This improves the accuracy of distinguishing single-color pixels. Furthermore, by using the first optical filter 13, the transmittance or reflectance characteristics of the multiple fluorescent lights can be varied across multiple wavelength ranges, further enabling the distinction of multi-color single-color images. Furthermore, by using the first optical filter 13, the first optical state can be easily established, making it easier to acquire the first fluorescent light image.

[0106] Furthermore, in the first embodiment, the first optical filter 13 is a first tilted filter 13 whose transmittance varies monotonically within each transmission wavelength range. The use of the first tilted filter 13 enables the transmittance to vary monotonically within each transmission wavelength range, enabling higher-precision discrimination of multi-color monochromatic images. Furthermore, the use of the first tilted filter 13 facilitates the first optical state, making it easier to acquire a first fluorescent image.

[0107] Furthermore, in the first embodiment, in the second acquisition step, fluorescence is captured in the second optical state, i.e., after the first optical filter 13 is removed. Removing the first optical filter 13 allows the second optical state to be set, making it easier to set the second optical state. Consequently, the second fluorescence image can be acquired more easily.

[0108] In the first embodiment, the image processing device 4 clusters a plurality of pixels identified as monochromatic pixels into L pixel groups (L is an integer greater than or equal to 2 and less than or equal to N-1) based on C fluorescence images, each consisting of N pixels (N is an integer greater than or equal to 2), obtained by irradiating a sample with excitation light of C wavelength distributions. The image processing device 4 then generates L cluster matrices arranged for each pixel group resulting from the clustering of the C fluorescence images. For each of the L cluster matrices, a statistical value of the intensity values ​​of the pixel groups constituting the C fluorescence images is calculated. Using the statistical value of the C fluorescence images in each of the L cluster matrices, the image processing device 4 performs demixing on the C fluorescence images to generate K fluorescence images representing the distribution of each of K pigments (K is an integer greater than or equal to 2 and less than or equal to C). In this case, the plurality of pixels identified as monochromatic pixels are clustered into L pixel groups, and L cluster matrices are generated, arranged for each pixel group comprising the C fluorescence images acquired by irradiating the sample with excitation light of C wavelength distributions. Furthermore, for each of the L cluster matrices, the statistical value of the intensity values ​​of the pixel groups that make up the C fluorescence images is calculated. Using the statistical values ​​of each of the C fluorescence images, the C fluorescence images are demixed to generate K fluorescence images. This allows for highly accurate separation images of each pigment.

[0109] The above-mentioned effects are described in detail. In the first embodiment, because the first tilt filter 13 has a transmittance that depends on the wavelength λ, the intensity ratios of the monochrome pixels representing the same monochrome become the same value or values ​​close to each other. Based on this feature, the clustering unit 204 can cluster the multiple monochrome pixels into L pixel groups, for example, by judging the distance between the average values ​​of the intensity ratios of each of the multiple monochrome pixels (the approximation of the values). The average value of the intensity ratio of each of the multiple monochrome pixels is, for example, a simple average or a weighted average. The average value can be calculated using other calculation methods. The clustering unit 204 can also perform clustering by judging the distance between the median values ​​of the intensity ratios of each of the multiple monochrome pixels. The clustering unit 204 can also perform clustering based on the intensity of each of the C excitation lights of each of the multiple monochrome pixels.

[0110] [Second embodiment]

[0111] A second embodiment of the present disclosure will be described. Figure 11 This is a schematic diagram of the structure of a fluorescence image acquisition system 1A, which serves as a fluorescence image acquisition device according to a second embodiment. In fluorescence image acquisition system 1A, the structure of image acquisition device 3A differs from that of image acquisition device 3 in fluorescence image acquisition system 1. Image acquisition device 3A does not include first optical filter 13. Instead, image acquisition device 3A includes first camera 15A, which has a different structure from first camera 15.

[0112] The first camera 15A is a camera that captures at least two fluorescence images obtained by measuring fluorescence from the sample S with different wavelength characteristics. The first camera 15A can capture each of the multiple fluorescence images and capture multiple fluorescence images including at least a first fluorescence image and a second fluorescence image. In the second embodiment, a color camera or a hyperspectral camera, for example, can be used as the first camera 15A.

[0113] When a color camera is used as the first camera 15A, by capturing each of the multiple fluorescent lights once, for example, multiple images separated by each RGB component can be acquired. As an example, the color camera acquires multiple images separated by each RGB component as follows. In a color camera, specific color filters, such as red, green, and blue color filters, are provided corresponding to each pixel on a solid-state imaging element. The element provided with a red filter is recorded as an R element, the element provided with a green filter is recorded as a G element, and the element provided with a blue filter is recorded as a B element. In a color camera, the R element, the G element, and the B element are arranged in a mosaic shape. The B element only acquires the B component (blue component) of the fluorescent light, and the R component (red component) and the G component (green component) of the fluorescent light in the B element are interpolated through the Bayer transform process. By performing the same process on the R element and the G element, the color camera can acquire multiple images separated by each RGB component. The Bayer transform process refers to a process of supplementing the insufficient colors from the surrounding elements through a certain algorithm. The image acquisition unit 201 can acquire a first fluorescent light image and a second fluorescent light image from a plurality of images separated for each RGB component. In the following description, the image of the G component is referred to as the first fluorescent light image, and the image of the B component is referred to as the second fluorescent light image.

[0114] Reference Figure 12 The intensity ratio of each pixel when the above-mentioned color camera is used will be described. Figure 12 This diagram schematically shows the spectral characteristics of each filter relative to the fluorescence spectrum. SC1 is the spectral characteristic of the red filter, SC2 is the spectral characteristic of the green filter, and SC3 is the spectral characteristic of the blue filter. When the sample S is irradiated with the first excitation light, the pixel intensity IA of the first fluorescence image, which is the image of the G component, is 11 , the intensity IA of the pixel in the second fluorescent image as the B component image 12 , and intensity ratio RA 11 For example, it can be calculated as shown in the following equations (16) to (18). Here, the spectral characteristics of the green filter are represented by S G (λ), the spectral characteristics of the blue filter are set to S B(λ). Similar to the first embodiment, in the case of monochrome pixels, the intensity ratio is independent of the first excitation light. Therefore, in the second embodiment, by calculating the intensity ratio of each of the multiple excitation lights, it is also possible to determine whether each pixel is a monochrome pixel. Furthermore, the intensity of each pixel can be multiplied by a coefficient generated during the Bayer transform process. Because this coefficient is independent of the excitation light, when multiplied by this coefficient, the intensity ratio of the monochrome pixel is also independent of the excitation light.

[0115] [Formula 9]

[0116] IA 11 =f∫f AS (λ)f ES1 (λ)dλ×∫f FS (λ)S T1 (λ)S G (λ)dλ…(16)

[0117] IA 12 =∫f AS (λ)f ES1 (λ)dλ×∫f FS (λ)S T1 (λ)S B (λ)dλ…(17)

[0118]

[0119] When a hyperspectral camera is used as the first camera 15A, by capturing each of the multiple fluorescences once, it is possible to acquire multiple images separated by different wavelength characteristics. As an example, the hyperspectral camera acquires multiple images separated by different wavelength characteristics as follows. In the hyperspectral camera, filters of various wavelength bandwidths are arranged in a mosaic pattern within each pixel on the solid-state imaging element. For example, filters of 4×4=16 wavelength bandwidths are arranged within each pixel, and each pixel has information about 16 wavelength bandwidths. Thus, the hyperspectral camera can acquire multiple images separated by each of the 16 different wavelength characteristics. The image acquisition unit 201 acquires the first and second fluorescence images from the multiple images separated by different wavelength characteristics. In the following description, as an example, each of the 16 wavelength bandwidths of the hyperspectral camera corresponds to a wavelength of 465nm, 474nm, 485nm, 496nm, 510nm, 522nm, 534nm, 546nm, 548nm, 562nm, 578nm, 586nm, 600nm, 608nm, 624nm, and 630nm. The image obtained by the filter with the wavelength bandwidth corresponding to 474nm is set as the first fluorescent image, and the image obtained by the filter with the wavelength bandwidth corresponding to 630nm is set as the second fluorescent image.

[0120] Reference Figure 13 The intensity ratio of each pixel when the hyperspectral camera is used will be described. Figure 13 This diagram schematically shows the spectral characteristics of each filter relative to the fluorescence spectrum. SC4 is the spectral characteristic of the filter with a wavelength bandwidth corresponding to 474 nm, and SC5 is the spectral characteristic of the filter with a wavelength bandwidth corresponding to 630 nm. When the sample S is irradiated with the first excitation light, the intensity IB of the pixel of the first fluorescence image is 11 , the intensity IB of the pixel in the second fluorescent image 12 , and intensity ratio RB 11 For example, it can be calculated as shown in the following equations (19) to (21). Here, the spectral characteristics of the filter with a wavelength bandwidth corresponding to 474 nm are represented by S 474 (λ), the spectral characteristics of the filter with a wavelength bandwidth corresponding to 630 nm are set as S 630 (λ). As in the first embodiment, in the case of a monochrome pixel, the intensity ratio does not depend on the first excitation light. Therefore, in the second embodiment, by calculating the intensity ratio of each of the multiple excitation lights, it is also possible to determine whether each pixel is a monochrome pixel.

[0121] [Formula 10]

[0122] IB 11 =∫f AS(λ)f ES1 (λ)dλ×∫f FS (λ)S T1 (λ)S 474 (λ)dλ…(19)

[0123] IB 12 =∫f AS (λ)f ES1 (λ)dλ×∫f FS (λ)S T1 (λ)S 630 (λ)dλ…(20)

[0124]

[0125] Alternatively, a hyperspectral camera can use a spectrometer to separate fluorescence, rather than using filters on a solid-state imaging element. In such a hyperspectral camera, for example, fluorescence is incident on a spectrometer via a lens, splitting the fluorescence into multiple wavelength bands. The imaging element then receives fluorescence corresponding to each wavelength band. This allows each pixel to capture information from multiple wavelength bands. Even such a hyperspectral camera can capture multiple images separated by different wavelength characteristics. Therefore, the image acquisition unit 201 can acquire the first and second fluorescence images from these multiple images separated by different wavelength characteristics.

[0126] In the fluorescence image acquisition system 1A of the second embodiment described above, the image acquisition device 3A captures each of the multiple fluorescence images to acquire a first fluorescence image and a second fluorescence image. Therefore, the fluorescence image acquisition system 1A also achieves the same operational effects as the fluorescence image acquisition system 1 of the first embodiment. Furthermore, by capturing each of the multiple fluorescence images once, both the first and second fluorescence images can be acquired. Therefore, the first and second fluorescence images can be acquired efficiently.

[0127] In the second embodiment, the image acquisition device 3A uses, for example, a color camera to capture each of the multiple fluorescent images. By capturing each of the multiple fluorescent images once using the color camera, multiple images separated by, for example, RGB components can be acquired. A first fluorescent image and a second fluorescent image can be acquired from these multiple images.

[0128] In the second embodiment, the image acquisition device 3A uses, for example, a hyperspectral camera to capture each of the multiple fluorescent lights. By capturing each of the multiple fluorescent lights once using the hyperspectral camera, multiple images separated by different wavelength characteristics can be acquired. A first fluorescent light image and a second fluorescent light image can be acquired from these multiple images.

[0129] [Third embodiment]

[0130] A third embodiment of the present disclosure will be described. Figure 14 FIG2 is a schematic diagram of a fluorescence image acquisition system 1B as a fluorescence image acquisition device according to a third embodiment. An image acquisition device 3B of the fluorescence image acquisition system 1B includes a second camera 17 in addition to the configuration of the image acquisition device 3 of the fluorescence image acquisition system 1 .

[0131] The second camera 17, for example, has the same structure as the first camera 15. When the first tilted filter 13 is positioned in the optical path of the fluorescence, the second camera 17 captures the fluorescence reflected by the first tilted filter 13 to obtain a second fluorescence image. Specifically, in the fluorescence image acquisition system 1B, the first camera 15 captures the fluorescence transmitted through the first tilted filter 13 to obtain a first fluorescence image, while the second camera 17 captures the fluorescence reflected by the first tilted filter 13 to obtain a second fluorescence image.

[0132] In the aforementioned third embodiment of the fluorescence image acquisition system 1B, the first camera 15 captures the fluorescence transmitted through the first tilted filter 13, and the second camera 17 captures the fluorescence reflected by the first tilted filter 13, thereby acquiring a first fluorescence image and a second fluorescence image. Therefore, the fluorescence image acquisition system 1B also achieves the same operational effects as the fluorescence image acquisition system 1 of the first embodiment.

[0133] [Fourth embodiment]

[0134] A fourth embodiment of the present disclosure will be described. Figure 15 This is a schematic diagram of the structure of a fluorescence image acquisition system 1C, which serves as a fluorescence image acquisition device according to a fourth embodiment. Image acquisition device 3C of fluorescence image acquisition system 1C, in addition to having the components of image acquisition device 3 of fluorescence image acquisition system 1, further includes a second optical filter 19. Image acquisition device 3C is provided with a switching mechanism (not shown) capable of switching between first optical filter 13 and second optical filter 19. This mechanism allows the first optical filter 13 and second optical filter 19 to be switched from a state in which the first optical filter 13 is positioned on the optical path of fluorescence from fluorescence filter section 3a to a state in which the second optical filter 19 is positioned on the optical path of fluorescence from fluorescence filter section 3a.

[0135] The second optical filter 19 has a different transmittance than the first optical filter 13, and the transmittance varies within each reflection wavelength range or each transmission wavelength range of the fluorescence filter section 3a. In the fourth embodiment, the second optical filter 19 is a tilted filter having a wavelength characteristic in which the transmittance decreases linearly with increasing wavelength. In other words, the transmittance of the second optical filter 19 decreases monotonically within each transmission wavelength range of the fluorescence filter section 3a. In the subsequent description of the fourth embodiment, the second optical filter 19, which is a tilted filter, will be referred to as the second tilted filter 19. The second tilted filter 19 represents a different variation in transmittance than the first tilted filter 13. The second optical filter 19 may also have a wavelength characteristic in which the transmittance increases linearly with increasing wavelength. Alternatively, the second optical filter 19 may be a tilted filter whose transmittance changes monotonically within each reflection wavelength range of the fluorescence filter section 3a.

[0136] The first camera 15 captures the fluorescence when the first tilted filter 13 is placed on the optical path of the fluorescence and acquires a first fluorescence image. In addition, the first camera 15 captures the fluorescence when the second tilted filter 19 is placed on the optical path of the fluorescence and acquires a second fluorescence image.

[0137] In the fluorescence image acquisition system 1C of the fourth embodiment described above, fluorescence is captured when the first tilted filter 13 is positioned on the optical path of the fluorescence, and when the second tilted filter 19 is positioned on the optical path of the fluorescence, thereby acquiring a first fluorescence image and a second fluorescence image. Therefore, the fluorescence image acquisition system 1C also achieves the same operational advantages as the fluorescence image acquisition system 1 of the first embodiment.

[0138] Furthermore, in the fourth embodiment, the image acquisition device 3C captures fluorescence in the second optical state using a second tilted filter 19 that exhibits a different transmittance variation than the first tilted filter 13. The use of the second tilted filter 19 enables the transmittance to vary monotonically within each transmission wavelength range, enabling higher-precision discrimination of multi-color monochromatic images. Furthermore, the use of the second tilted filter 19 facilitates the second optical state and facilitates the acquisition of a second fluorescence image.

[0139] In the fourth embodiment, the image acquisition device 3C captures fluorescence in the second optical state using a second optical filter 19 having a transmittance different from that of the first optical filter 13. By using the second optical filter 19, the second optical state can be easily established, and the second fluorescence image can be easily acquired.

[0140] [Example of generating a fluorescence image]

[0141] An example of generating a fluorescence image using the fluorescence image acquisition system 1 of the first embodiment will be described. In this example, samples S11 and S21 representing red fluorescent bands, samples S12 and S22 representing orange fluorescent bands, samples S13 and S23 representing green fluorescent bands, and samples S14 and S24 representing blue fluorescent bands are prepared, and these fluorescent bands are arranged in a cross shape on a slide. Figure 16 Part (a) is a fluorescent image obtained by photographing these fluorescent bands using a black and white camera. Figure 16 Part (b) of FIG. 2 shows a pixel group composed of blue single-color pixels among the pixel groups clustered by the clustering unit 204. Figure 16 As shown in part (b) of , in this generation example, monochrome pixels are clustered with high precision.

[0142] Figure 17 Part (a) shows the fluorescence image of the blue pigment (samples S14 and S24) among the pigment images generated by the image generating unit 206. Figure 17 Part (b) shows the fluorescence image from the green pigment (samples S13 and S23). Figure 17 Part (c) shows the fluorescence image from the orange pigment (samples S12 and S22). Figure 17 Part (d) shows the fluorescence image from the red pigment (samples S11 and S21). Figure 17 As shown in parts (a) to (d) of FIG. , in this generation example, a fluorescence image from each pigment is generated with high precision.

[0143] Although various embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments, and can be modified or applied to other embodiments within the scope of the spirit described in the claims.

[0144] In a second embodiment, the first camera can incorporate a filter on the solid-state imaging element that can arbitrarily switch between a transmission wavelength range and a reflection wavelength range. Examples of such filters include variable bandpass filters (which function as both variable and bandpass filters), liquid crystal tunable filters, and AOTFs (acousto-optic tunable filters). Alternatively, the first camera can utilize a spectrometer such as a prism to separate fluorescence according to its wavelength characteristics.

[0145] When generating matrix data Y before clustering, the image processing device 4 of each embodiment may generate the N pixels constituting the image as data arranged in the row direction according to a predetermined rule, or as data arranged in the row direction according to a random rule. However, the data of the fluorescence image constituting C rows of matrix data Y is set to data consisting of N pixels arranged in the same rule. Even when using matrix data Y generated by random arrangement, the same matrix data Y' can be regenerated through clustering.

[0146] Furthermore, the image processing device 4 of each embodiment may generate the matrix data Y before clustering by excluding background pixels (pixels in which no pigment exists) included in the fluorescent image.

[0147] As a method for clustering pixel groups in the image processing device 4 , a method using machine learning such as the K-means method, a method using deep learning, or the like can be adopted.

[0148] In addition to acquiring the first and second fluorescence images, the image acquisition device of each embodiment can also acquire third, ..., and Qth fluorescence images in third, ..., and Qth optical states (Q is an integer greater than or equal to 3), in which fluorescence is measured using wavelength characteristics different from those of the first and second optical states. In the third through Qth optical states, fluorescence is measured using wavelength characteristics that differ from each other. In this case, the intensity ratio and deviation index of each excitation light can be calculated from a combination of two fluorescence images from the first through Qth fluorescence images. The deviation indexes for all combinations can be averaged to determine whether a pixel is a single color pixel.

[0149] In the first, third, and fourth embodiments, the acquisition step preferably includes: a first acquisition step of capturing the fluorescence light for each of the plurality of fluorescent lights in a first optical state using a first optical filter having different transmittances in each reflection wavelength range or each transmission wavelength range of the fluorescence filter portion to acquire a first fluorescence image; and a second acquisition step of capturing the fluorescence light for each of the plurality of fluorescent lights in a second optical state to acquire a second fluorescence image. In the first, third, and fourth embodiments, the image acquisition device preferably captures the fluorescence light for each of the plurality of fluorescent lights in the first optical state using a first optical filter having different transmittances in each reflection wavelength range or each transmission wavelength range of the fluorescence filter portion to acquire a first fluorescence image, and captures the fluorescence light for each of the plurality of fluorescent lights in the second optical state to acquire a second fluorescence image. In this case, the accuracy of distinguishing single-color pixels can be improved. Furthermore, by using the first optical filter, the transmittance or reflectance characteristics of the plurality of fluorescent lights can be varied across multiple wavelength ranges, enabling further discrimination of multiple-color single-color pixels. Furthermore, by using the first optical filter, the first optical state can be easily established, making it easier to acquire the first fluorescence image.

[0150] Furthermore, in the first, third, and fourth embodiments described above, the first optical filter is preferably a first tilted filter whose transmittance varies monotonically within each reflection wavelength range, or whose transmittance varies monotonically within each transmission wavelength range. In this case, the use of the first tilted filter enables the transmittance to vary monotonically within each reflection wavelength range, or the transmittance to vary monotonically within each transmission wavelength range, enabling higher-precision discrimination of multi-color monochromatic pixels. Furthermore, the use of the first tilted filter facilitates the first optical state, making it easier to acquire the first fluorescent image.

[0151] Furthermore, in the fourth embodiment, preferably, in the second acquisition step, fluorescence is captured in a second optical state using a second tilted filter that exhibits a different transmittance variation than the first tilted filter. Furthermore, in the fourth embodiment, preferably, the image acquisition device captures fluorescence in the second optical state using a second tilted filter that exhibits a different transmittance variation than the first tilted filter. In this case, the use of the second tilted filter enables monotonically varying transmittance within each reflected wavelength range or monotonically varying transmittance within each transmitted wavelength range, enabling higher precision discrimination of multi-color monochromatic images. Furthermore, the use of the second tilted filter facilitates the second optical state and facilitates acquisition of the second fluorescence image.

[0152] Furthermore, in the fourth embodiment, preferably, in the second acquisition step, fluorescence is captured in a second optical state using a second optical filter having a different transmittance than the first optical filter. Furthermore, in the fourth embodiment, preferably, the image acquisition device captures fluorescence in the second optical state using a second optical filter having a different transmittance than the first optical filter. In this case, the use of the second optical filter facilitates the second optical state and facilitates the acquisition of a second fluorescence image.

[0153] Furthermore, in the first embodiment described above, it is preferred that, in the second acquisition step, fluorescence is captured in a second optical state, which is a state in which the first optical filter is removed. Furthermore, in the first embodiment described above, it is preferred that the image acquisition device captures fluorescence in the second optical state, which is a state in which the first optical filter is removed. In this case, the second optical state can be set by removing the first optical filter, making it easier to set the second optical state. Consequently, it is easier to acquire the second fluorescence image.

[0154] Furthermore, in the second embodiment, preferably, in the acquisition step, each of the plurality of fluorescent lights is captured to acquire a first fluorescent image and a second fluorescent image. Furthermore, in the second embodiment, preferably, the image acquisition device captures each of the plurality of fluorescent lights to acquire a first fluorescent image and a second fluorescent image. In this case, by capturing each of the plurality of fluorescent lights once, both the first fluorescent image and the second fluorescent image can be acquired. Therefore, the first fluorescent image and the second fluorescent image can be acquired efficiently.

[0155] In the second embodiment, preferably, in the acquisition step, each of the multiple fluorescent lights is imaged using a color camera. In the second embodiment, preferably, the image acquisition device uses a color camera to image each of the multiple fluorescent lights. By capturing each of the multiple fluorescent lights once using a color camera, multiple images separated by, for example, RGB components can be acquired. A first fluorescent light image and a second fluorescent light image can be acquired from these multiple images.

[0156] Furthermore, in the second embodiment, preferably, in the acquisition step, each of the multiple fluorescent lights is imaged using a hyperspectral camera. Furthermore, in the second embodiment, preferably, the image acquisition device uses a hyperspectral camera to image each of the multiple fluorescent lights. By capturing each of the multiple fluorescent lights once using the hyperspectral camera, multiple images separated by different wavelength characteristics can be acquired. A first fluorescent light image and a second fluorescent light image can be acquired from these multiple images.

[0157] In addition, in the first to fourth embodiments described above, it is also preferred that the method further comprises: a clustering step of clustering a plurality of pixels discriminated as monochrome pixels by the discrimination step into L pixel groups (L is an integer not less than 2 and not more than N-1) based on C fluorescence images each of which is obtained by irradiating the sample with excitation light having C wavelength distributions (C is an integer not less than 2) and each of which is composed of N pixels (N is an integer not less than 2), and generating L cluster matrices arranged for each pixel group after clustering the C fluorescence images; a calculation step of calculating, for each of the L cluster matrices, a statistical value of the intensity values ​​of the pixel groups constituting the C fluorescence images; and an image generation step of performing demixing on the C fluorescence images using the statistical value of the C fluorescence images of each of the L cluster matrices to generate K fluorescence images representing the distribution of each of K pigments (K is an integer not less than 2 and not more than C). In the first to fourth embodiments described above, the image processing device preferably clusters a plurality of pixels identified as monochromatic pixels into L pixel groups (L is an integer from 2 to N-1) based on C fluorescence images obtained by irradiating a sample with excitation light having C wavelength distributions (C is an integer from 2 to 4), each consisting of N pixels (N is an integer from 2 to 4). The device then generates L cluster matrices arranged for each pixel group resulting from clustering the C fluorescence images. For each of the L cluster matrices, the device calculates a statistical value of the intensity values ​​of the pixel groups comprising the C fluorescence images. Using the statistical value of the C fluorescence images in each of the L cluster matrices, the device performs demixing on the C fluorescence images to generate K fluorescence images representing the distribution of each of K pigments (K is an integer from 2 to C). In this case, the plurality of pixels identified as monochromatic pixels are clustered into L pixel groups, and the L cluster matrices are generated, arranged for each pixel group resulting from clustering the C fluorescence images obtained by irradiating the sample with excitation light having C wavelength distributions. In addition, for each of the L cluster matrices, a statistical value of the intensity values ​​of the pixel groups that make up the C fluorescence images is calculated. Using the statistical values ​​of each of the C fluorescence images, the C fluorescence images are demixed to generate K fluorescence images. This allows for highly accurate separation images of each pigment.

[0158] The fluorescence image acquisition method of the embodiment is [1] "a fluorescence image acquisition method comprising: an irradiation step of irradiating a sample with each of a plurality of excitation lights having a wavelength distribution; an acquisition step of acquiring, for each of a plurality of fluorescences corresponding to the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with a wavelength characteristic different from that of the first optical state via a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; a calculation step of calculating, for the first fluorescence image and the second fluorescence image, a ratio of an intensity value of a pixel of the first fluorescence image to an intensity value of a pixel of the second fluorescence image corresponding to the pixel, i.e., an intensity ratio, and calculating the intensity ratio for each of the plurality of excitation lights; and a determination step of determining whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights."

[0159] The fluorescence image acquisition method of the embodiment may also be, [2] "the fluorescence image acquisition method according to the above [1], wherein the acquisition step includes: for each of the plurality of fluorescences, photographing the fluorescence in a first optical state using a first optical filter having a transmittance different in each reflection wavelength range or each transmission wavelength range of the fluorescence filter portion to obtain a first fluorescence image; and for each of the plurality of fluorescences, photographing the fluorescence in a second optical state to obtain a second fluorescence image."

[0160] The fluorescence image acquisition method of the embodiment may also be, [3] "according to the fluorescence image acquisition method described in the above [2], wherein the first optical filter is a first tilted filter whose transmittance changes monotonically within each reflection wavelength range, or whose transmittance changes monotonically within each transmission wavelength range."

[0161] The fluorescence image acquisition method of the embodiment may also be, [4] "according to the fluorescence image acquisition method described in the above [3], wherein, in the second acquisition step, fluorescence is captured in a second optical state using a second tilted filter that represents a change in transmittance different from that of the first tilted filter."

[0162] The fluorescence image acquisition method of the embodiment may also be, [5] "a fluorescence image acquisition method according to the above-mentioned [2] or [3], wherein, in the second acquisition step, fluorescence is captured in a second optical state using a second optical filter having a transmittance different from that of the first optical filter."

[0163] The fluorescence image acquisition method of the embodiment may also be, [6] "the fluorescence image acquisition method according to the above [2] or [3], wherein, in the second acquisition step, fluorescence is captured in a second optical state, which is a state in which the first optical filter is removed."

[0164] The fluorescence image acquisition method of the embodiment may also be, [7] "according to the fluorescence image acquisition method described in the above [1], wherein, in the acquisition step, each of the multiple fluorescences is photographed to acquire a first fluorescence image and a second fluorescence image."

[0165] The fluorescence image acquisition method of the embodiment may also be, [8] "the fluorescence image acquisition method according to the above [7], wherein, in the acquisition step, each of the plurality of fluorescences is photographed using a color camera."

[0166] The fluorescence image acquisition method of the embodiment may also be, [9] "the fluorescence image acquisition method according to the above [7], wherein, in the acquisition step, each of the multiple fluorescences is photographed using a hyperspectral camera."

[0167] The fluorescence image acquisition method of the embodiment may also be,

[10] "the fluorescence image acquisition method according to any one of the above [2] to [9], further comprising: a clustering step of clustering a plurality of pixels identified as monochrome pixels by the identification step into L (L is an integer from 2 to N-1) pixel groups, each of which is obtained by irradiating the sample with each of C (C is an integer from 2 to 9) wavelength distribution excitation light, and each of which is composed of N (N is an integer from 2 to 2) pixels, to generate L cluster matrices arranged according to each pixel group clustered with the C fluorescence images; a calculation step of calculating, for each of the L cluster matrices, a statistical value of the intensity value of the pixel group constituting the C fluorescence images; and an image generation step of using the statistical value of the C fluorescence images of each of the L cluster matrices to perform demixing on the C fluorescence images to generate K fluorescence images representing the distribution of each of K (K is an integer from 2 to C) pigments."

[0168] The fluorescence image acquisition device of the embodiment is

[11] "a fluorescence image acquisition device comprising: an irradiation device for irradiating a sample with each of a plurality of excitation lights having a wavelength distribution; an image acquisition device for acquiring, for each of a plurality of fluorescence lights corresponding to the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with a wavelength characteristic different from that of the first optical state via a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and an image processing device for processing the plurality of first fluorescence images and the plurality of second fluorescence images, the image processing device calculating, for the first fluorescence image and the second fluorescence image, a ratio of an intensity value of a pixel of the first fluorescence image to an intensity value of a pixel of the second fluorescence image corresponding to the pixel, i.e., an intensity ratio, calculating the intensity ratio of each of the plurality of excitation lights, and judging whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights".

[0169] The fluorescence image acquisition device of the embodiment may also be,

[12] "a fluorescence image acquisition device according to the above-mentioned

[11] , wherein the image acquisition device captures the fluorescence for each of the plurality of fluorescences in a first optical state using a first optical filter having a transmittance different in each reflection wavelength range or each transmission wavelength range of the fluorescence filter portion to obtain a first fluorescence image, and captures the fluorescence for each of the plurality of fluorescences in a second optical state to obtain a second fluorescence image."

[0170] The fluorescence image acquisition device of the embodiment may also be,

[13] "a fluorescence image acquisition device according to the above-mentioned

[12] , wherein the first optical filter is a first tilted filter whose transmittance changes monotonically within each reflection wavelength range or whose transmittance changes monotonically within each transmission wavelength range."

[0171] The fluorescence image acquisition device of the embodiment may also be,

[14] "a fluorescence image acquisition device according to the above-mentioned

[13] , wherein the image acquisition device captures fluorescence in a second optical state using a second tilted filter representing a change in transmittance different from that of the first tilted filter."

[0172] The fluorescence image acquisition device of the embodiment may also be,

[15] "a fluorescence image acquisition device according to the above-mentioned

[12] or

[13] , wherein the image acquisition device captures fluorescence in a second optical state using a second optical filter having a transmittance different from that of the first optical filter."

[0173] The fluorescence image acquisition device of the embodiment may also be,

[16] "a fluorescence image acquisition device according to the above-mentioned

[12] or

[13] , wherein the image acquisition device captures fluorescence in a second optical state in which the first optical filter is removed."

[0174] The fluorescence image acquisition device of the embodiment may also be,

[17] "the fluorescence image acquisition device according to the above-mentioned

[11] , wherein the image acquisition device captures each of the plurality of fluorescences to acquire a first fluorescence image and a second fluorescence image."

[0175] The fluorescence image acquisition device of the embodiment may also be,

[18] "the fluorescence image acquisition device according to the above

[17] , wherein the image acquisition device uses a color camera to capture each of the multiple fluorescences."

[0176] The fluorescence image acquisition device of the embodiment may also be,

[19] "the fluorescence image acquisition device according to the above

[17] , wherein the image acquisition device uses a hyperspectral camera to capture each of the multiple fluorescences."

[0177] The fluorescence image acquisition device of the embodiment may also be,

[20] "a fluorescence image acquisition device according to any one of the above

[12] to

[19] , wherein the image processing device takes C fluorescence images obtained by irradiating the sample with each of C (C is an integer greater than 2) wavelength distribution excitation light and each consisting of N (N is an integer greater than 2) pixels as the object, clusters a plurality of pixels judged as monochrome pixels into L (L is an integer greater than 2 and less than N-1) pixel groups, generates L cluster matrices arranged according to each pixel group clustered with the C fluorescence images, calculates the statistical value of the intensity value of the pixel group constituting the C fluorescence images for each of the L cluster matrices, uses the statistical value of the C fluorescence images of each of the L cluster matrices, performs demixing on the C fluorescence images as the object, and generates K fluorescence images representing the distribution of each of K (K is an integer greater than 2 and less than C) pigments".

[0178] A fluorescence image acquisition program according to an embodiment is a "fluorescence image acquisition program for determining whether a pixel is a monochrome pixel, for each of a plurality of fluorescences corresponding to the plurality of excitation lights, by irradiating a specimen with each of excitation lights having a plurality of wavelength distributions, based on a first fluorescence image in a first optical state acquired through a fluorescence filter having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state. In the fluorescence image acquisition program, a computer is caused to function as an intensity ratio calculation unit and a monochrome pixel determination unit. The intensity ratio calculation unit calculates, for each of the plurality of excitation lights, an intensity ratio, which is a ratio of an intensity value of a pixel in the first fluorescence image to an intensity value of a pixel in the corresponding second fluorescence image. The monochrome pixel determination unit determines whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights."

[0179] Description of Reference Numerals

[0180] 1, 1A, 1B, 1C…fluorescence image acquisition system, 2…irradiation device, 2a…excitation light source, 2b…excitation optical filter unit, 3, 3A, 3B, 3C…image acquisition device, 3a…fluorescence filter unit, 4…image processing device, 11…dichroic mirror, 13…first optical filter (first tilted filter), 15, 15A…first camera, 17…second camera, 19…second optical filter (second tilted filter), 201…image acquisition unit, 202…intensity ratio calculation unit, 203…monochrome pixel discrimination unit, 204…clustering unit, 205…statistics calculation unit, 206…image generation unit, C1–C3…pigment, PGr1–PGr3…pixel group, S, S11, S12, S13, S14, S21, S22, S23, S24…sample

Claims

1. A method for acquiring a fluorescence image, comprising: an irradiation step of irradiating the sample with each of a plurality of excitation lights having a wavelength distribution; For each of a plurality of fluorescences corresponding to the plurality of excitation lights, a first fluorescence image is acquired in a first optical state, and a second fluorescence image is acquired in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, via a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; a step of calculating, for the first fluorescent image and the second fluorescent image, an intensity ratio between an intensity value of a pixel of the first fluorescent image and an intensity value of a pixel of the second fluorescent image corresponding to the pixel, and calculating the intensity ratio for each of the plurality of excitation lights; as well as A determination step of determining whether the pixel is a monochrome pixel based on the intensity ratio of each of the plurality of excitation lights.

2. The fluorescence image acquisition method according to claim 1, wherein: The acquisition step includes: for each of the plurality of fluorescences, photographing the fluorescence in the first optical state using a first optical filter having different transmittances in each reflection wavelength range or each transmission wavelength range of the fluorescence filter section to acquire a first fluorescence image; and For each of the plurality of fluorescent lights, the fluorescent light is photographed in the second optical state to obtain a second fluorescent light image.

3. The fluorescence image acquisition method according to claim 2, wherein: The first optical filter is a first tilt filter whose transmittance changes monotonically within the respective reflection wavelength ranges, or whose transmittance changes monotonically within the respective transmission wavelength ranges.

4. The fluorescence image acquisition method according to claim 3, wherein: In the second acquisition step, the fluorescence is captured in the second optical state using a second tilted filter that shows a different change in transmittance from that of the first tilted filter.

5. The fluorescence image acquisition method according to claim 2 or 3, wherein: In the second acquisition step, the fluorescence is photographed in the second optical state using a second optical filter having a transmittance different from that of the first optical filter.

6. The fluorescence image acquisition method according to claim 2 or 3, wherein: In the second acquisition step, the fluorescence is photographed in the second optical state in which the first optical filter is removed.

7. The fluorescence image acquisition method according to claim 1, wherein: In the acquisition step, each of the plurality of fluorescence images is photographed to acquire the first fluorescence image and the second fluorescence image.

8. The fluorescence image acquisition method according to claim 7, wherein: In the acquisition step, each of the plurality of fluorescent lights is photographed using a color camera.

9. The fluorescent image acquisition method according to claim 7, wherein: In the acquisition step, each of the plurality of fluorescent lights is photographed using a hyperspectral camera.

10. The fluorescent image acquisition method according to any one of claims 2 to 9, wherein: Also features: a clustering step of clustering a plurality of pixels discriminated as monochrome pixels by the discrimination step into L pixel groups, with the C fluorescence images each consisting of N pixels, obtained by irradiating the sample with excitation light having C wavelength distributions, and generating L cluster matrices arranged for each pixel group in which the C fluorescence images are clustered, wherein C is an integer greater than or equal to 2, N is an integer greater than or equal to 2, and L is an integer greater than or equal to 2 and less than or equal to N-1; a calculation step of calculating the statistical value of the intensity values ​​of the pixel groups constituting the C fluorescence images for each of the L cluster matrices; and An image generation step is performed using the statistical values ​​of the C fluorescence images of each of the L cluster matrices to perform demixing on the C fluorescence images to generate the K fluorescence images representing the distribution of each of the K pigments, wherein K is an integer greater than 2 and less than C.

11. A fluorescence image acquisition device comprising: an irradiation device for irradiating the sample with each of a plurality of excitation lights of wavelength distribution; an image acquisition device for acquiring, for each of a plurality of fluorescences corresponding to the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, via a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and an image processing device for processing the plurality of first fluorescent images and the plurality of second fluorescent images, The image processing device, For the first fluorescent image and the second fluorescent image, the intensity ratio is calculated, which is a ratio of the intensity value of a pixel in the first fluorescent image to the intensity value of a pixel in the second fluorescent image corresponding to the pixel, and the intensity ratio of each of the plurality of excitation lights is calculated. Based on the intensity ratio of each of the plurality of excitation lights, it is determined whether the pixel is a monochrome pixel.

12. The fluorescence image acquisition device according to claim 11, wherein: The image acquisition device captures each of the plurality of fluorescences in the first optical state using a first optical filter having different transmittances in each reflection wavelength range or each transmission wavelength range of the fluorescence filter portion, thereby acquiring the first fluorescence image. For each of the plurality of fluorescent lights, the fluorescent light is photographed in the second optical state to acquire a second fluorescent light image.

13. The fluorescence image acquisition device according to claim 12, wherein: The first optical filter is a first tilt filter whose transmittance changes monotonically within the respective reflection wavelength ranges or whose transmittance changes monotonically within the respective transmission wavelength ranges.

14. The fluorescence image acquisition device according to claim 13, wherein: The image acquisition device captures the fluorescence in the second optical state using a second tilted filter that exhibits a different transmittance change from that of the first tilted filter.

15. The fluorescence image acquisition device according to claim 12 or 13, wherein: The image acquisition device captures the fluorescence in the second optical state using a second optical filter having a transmittance different from that of the first optical filter.

16. The fluorescence image acquisition device according to claim 12 or 13, wherein: The image acquisition device captures the fluorescence in the second optical state, which is a state in which the first optical filter is removed.

17. The fluorescence image acquisition device according to claim 11, wherein: The image acquisition device captures each of the plurality of fluorescent lights to acquire the first fluorescent image and the second fluorescent image.

18. The fluorescence image acquisition device according to claim 17, wherein: The image acquisition device captures each of the plurality of fluorescent lights using a color camera.

19. The fluorescence image acquisition device according to claim 17, wherein: The image acquisition device captures each of the plurality of fluorescent lights using a hyperspectral camera.

20. The fluorescence image acquisition device according to any one of claims 12 to 19, wherein: The image processing device, Taking C fluorescence images each consisting of N pixels, obtained by irradiating a sample with excitation light having C wavelength distributions, as the target, a plurality of pixels determined to be monochrome pixels are clustered into L pixel groups, and L cluster matrices are generated, each arranged for each pixel group in which the C fluorescence images are clustered, wherein C is an integer greater than or equal to 2, N is an integer greater than or equal to 2, and L is an integer greater than or equal to N-1. calculating, for each of the L cluster matrices, a statistical value of the intensity values ​​of the pixel groups constituting the C fluorescence images, Using the statistical values ​​of the C fluorescence images in each of the L cluster matrices, unmixing is performed on the C fluorescence images to generate the K fluorescence images representing the distribution of each of the K pigments, where K is an integer greater than 2 and less than C.

21. A fluorescence image acquisition program, wherein: The fluorescence image acquisition program is a program for determining whether a pixel is a monochrome pixel based on a first fluorescence image in a first optical state obtained through a fluorescence filter portion having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those in the first optical state, by irradiating a sample with each of excitation lights having a plurality of wavelength distributions, and for each of a plurality of fluorescences corresponding to the plurality of excitation lights. In the fluorescent image acquisition program, a computer is made to function as an intensity ratio calculation unit and a monochrome pixel determination unit. The intensity ratio calculation unit calculates, for the first fluorescent image and the second fluorescent image, an intensity ratio, which is a ratio of an intensity value of a pixel in the first fluorescent image to an intensity value of a pixel in the second fluorescent image corresponding to the pixel, and calculates the intensity ratio of each of the plurality of excitation lights. The monochrome pixel determination unit determines whether the pixel is a monochrome pixel based on an intensity ratio of each of the plurality of excitation lights.

Images