Image acquisition method, device, and program

Abstract

Distinguishing between monochromatic pixels in fluorescent pixels. [Solution] The image acquisition method includes an irradiation step of irradiating a sample S with each of a plurality of excitation lights having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescence corresponding to each of the plurality of excitation lights, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, via a fluorescence filter unit 3a having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; a calculation step of calculating, for each pixel, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light, and calculating vector parameters of each pixel based on the first and second vectors; and a discrimination step of discriminating whether each pixel is a monochromatic pixel based on the vector parameters.

Description

[Technical Field]

[0001] One aspect of the present disclosure relates to an image acquisition method, an apparatus, and a program. [Background technology]

[0002] Conventionally, a multi-staining technique has been used to simultaneously stain multiple substances in a sample, such as biological tissue. To observe the substances in a multi-stained sample, excitation light is irradiated onto the sample to acquire a fluorescence image. For example, Non-Patent Document 1 below discloses a method for unmixing fluorescence images, in which the fluorescence image is clustered, the maximum fluorescence intensity value is extracted for each clustered pixel group, and a separated image is generated based on the maximum value. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tristan D. McRae et al., “Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning”, PLOSONE, December 2,2019 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, it tends to be difficult to distinguish between single-color pixels in a fluorescent image when observing the fluorescent image. Therefore, it is desirable to be able to distinguish between single-color pixels in a fluorescent image in order to observe the fluorescent image efficiently.

[0005] One aspect of the present disclosure has been made in view of the above-described circumstances, and an object of the present disclosure is to provide an image acquisition method, device, and program that are capable of distinguishing between monochromatic pixels in fluorescent pixels. [Means for solving the problem]

[0006] (1) An image acquisition method according to one aspect of the present disclosure includes an irradiation step of irradiating a sample with excitation light of a plurality of wavelength distributions, an acquisition step of acquiring, for each of a plurality of fluorescence corresponding to each of the plurality of excitation light, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, a calculation step of calculating, for each pixel, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light, and calculating vector parameters of each pixel based on the first vector and the second vector, and a determination step of determining whether each pixel is a monochromatic pixel based on the vector parameters.

[0007] In an image acquisition method according to one aspect of the present disclosure, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from those of the first optical state are acquired for each of a plurality of fluorescence light beams corresponding to a plurality of excitation light beams. Furthermore, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light beam and a second vector corresponding to the intensity value of the pixel in the second fluorescence image for each excitation light beam are calculated for each pixel, and vector parameters for each pixel are calculated based on the first and second vectors. Then, based on the vector parameters, it is determined whether each pixel is a monochromatic pixel. Here, for a monochromatic pixel, the intensity ratio, which is the ratio between the intensity value of the pixel in the first fluorescence image and the intensity value of the pixel in the second fluorescence image corresponding to that pixel, is generally constant regardless of the excitation light beam. Therefore, for a monochromatic pixel, the first vector corresponding to the intensity value of the pixel in the first fluorescence image for each excitation light beam and the second vector corresponding to the intensity value of the pixel in the second fluorescence image for each excitation light beam have a predetermined relationship, and for example, are positioned on generally the same straight line. From the above, by determining whether a pixel is a monochromatic pixel from the vector parameters calculated based on the first and second vectors, it is possible to accurately identify monochromatic pixels from multiple pixels in a fluorescence image, even in a fluorescence image that includes pixels that reflect mixed fluorescence from multiple dyes and monochromatic pixels that reflect monochromatic fluorescence from a single dye. In other words, it is possible to identify monochromatic pixels in a fluorescence image.

[0008] (2) In the image acquisition method of (1) above, the calculation step may calculate a vector parameter indicating the similarity between the first vector and the second vector. This allows pixels associated with similar (substantially aligned) first and second vectors to be appropriately determined as monochromatic pixels, enabling highly accurate determination of monochromatic pixels.

[0009] (3) In the image acquisition method of (1) above, the calculation step may calculate vector parameters from an inner product or a cross product of a first vector and a second vector. For monochromatic pixels, there is a certain tendency in the value of the inner product or the cross product of the first vector and the second vector. Therefore, by calculating the vector parameters from the inner product or the cross product of the first vector and the second vector, monochromatic pixels can be identified with high accuracy based on the vector parameters.

[0010] (4) In the image acquisition methods (1) to (3) above, the irradiating step may use a light source capable of switching between excitation light beams having a plurality of wavelength distributions, a multi-bandpass filter, and a filter wheel having a plurality of bandpass filters to irradiate the sample with each of the excitation light beams having a plurality of wavelength distributions. With this configuration, it is possible to distinguish single-color pixels with high accuracy while employing the Sedat optical system.

[0011] (5) The image acquisition methods (1) to (4) above may further include a clustering step of classifying a plurality of pixels determined to be monochromatic pixels in the determination step into clusters, thereby making it possible to estimate regions for each dye.

[0012] (6) The image acquisition method of (5) above may further include a calculation step of calculating a statistical value of the intensity values ​​for each cluster, thereby allowing the intensity values ​​to be appropriately characterized on a cluster-by-cluster basis, for example, by calculating an average value as the statistical value.

[0013] (7) In the image acquisition method of (6) above, in the calculation step, weighting may be performed for each cluster according to vector parameters, and statistical values ​​may be calculated. For example, weighting according to the degree of monochromaticity can improve the accuracy of unmixing, which will be described later.

[0014] (8) The image acquisition method of (6) or (7) above may further include an unmixing step of unmixing the fluorescence image using statistics for each cluster, in which the fluorescence image is unmixed using a mixing matrix generated from the statistics to acquire a dye image. This allows efficient and highly accurate acquisition of dye images.

[0015] (9) An apparatus according to one aspect of the present disclosure includes an irradiation device that irradiates a sample with excitation light of a plurality of wavelength distributions, an image acquisition device that acquires, for each of a plurality of fluorescence corresponding to the plurality of excitation light, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and an image processing device that processes the plurality of first fluorescence images and second fluorescence images, wherein the image processing device is configured to calculate, for each pixel, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light, calculate vector parameters for each pixel based on the first vector and the second vector, and determine whether each pixel is a monochromatic pixel based on the vector parameters.

[0016] A program according to one aspect of the present disclosure is for determining whether or not a pixel is a monochromatic pixel based on a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from that of the first optical state, the first fluorescence image being acquired by irradiating a sample with excitation light beams having a plurality of wavelength distributions, and the second fluorescence image being acquired through a fluorescence filter unit having a plurality of reflection wavelength bands and a plurality of transmission wavelength bands, for each of a plurality of fluorescence beams corresponding to the plurality of excitation light beams. The program causes a computer to function as a vector parameter calculation unit that calculates, for each pixel, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light beam and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light beam, and calculates vector parameters of each pixel based on the first and second vectors, and a monochromatic pixel determination unit that determines whether or not each pixel is a monochromatic pixel based on the vector parameters. [Effects of the Invention]

[0017] According to one aspect of the present disclosure, it is possible to provide an image acquisition method, device, and program capable of distinguishing between monochromatic pixels in fluorescent pixels. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a schematic diagram of a fluorescence image acquisition system 1 according to the present embodiment. [Figure 2] FIG. 2 is a perspective view showing the configuration of the fluorescence image acquisition system 1 of FIG. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of the image processing device 4 in FIG. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the image processing device 4 of FIG. [Figure 5] 4A and 4B are diagrams for explaining the fluorescence intensities of the first and second fluorescent images. [Figure 6] 4A and 4B are diagrams for explaining characteristics of the intensity ratio in the first pixel to the first excitation light and the intensity ratio in the first pixel to the second excitation light. [Figure 7] FIG. 10 is a diagram schematically illustrating characteristics of the intensity ratio. [Figure 8] FIG. 10 is a diagram illustrating a problem with discrimination based on intensity ratio. [Figure 9] FIG. 10 is a diagram illustrating a problem with discrimination based on intensity ratio. [Figure 10] FIG. 10 is a diagram illustrating a problem with discrimination based on intensity ratio. [Figure 11] FIG. 10 is a diagram illustrating discrimination based on vector parameters. [Figure 12] FIG. 10 is a diagram illustrating discrimination based on vector parameters. [Figure 13] FIG. 10 is a diagram illustrating discrimination based on vector parameters. [Figure 14] FIG. 10 is a diagram illustrating an example of threshold determination. [Figure 15] FIG. 10 is a diagram illustrating an example of threshold determination. [Figure 16] FIG. 10 is a diagram illustrating an example of threshold determination. [Figure 17] FIG. 10 is a diagram showing an image of a group of pixels clustered by the clustering unit 204. [Figure 18] 10 shows an image of matrix data Y' regenerated by the statistical value calculation unit 205 and the corresponding dye matrix data X'. [Figure 19] 1 is a flowchart showing the procedure of a fluorescence image acquisition method according to the present embodiment. [Figure 20] This explains the effect of discrimination based on vector parameters. [Figure 21] This explains the effect of discrimination based on vector parameters. [Figure 22] 10A and 10B are diagrams illustrating statistical value calculations using monochromaticity as weights. [Figure 23] 10A and 10B are diagrams illustrating statistical value calculations using monochromaticity as weights. [Figure 24] FIG. 10 is a diagram illustrating the unmixing result (true solution). [Figure 25] FIG. 10 is a diagram illustrating an unmixing result (without weighting). [Figure 26]FIG. 10 is a diagram illustrating an unmixing result (weighted). [Figure 27] FIG. 10 is a diagram schematically illustrating an image acquisition device 3A according to a modified example. [Figure 28] FIG. 10 is a diagram illustrating an intensity ratio. [Figure 29] FIG. 10 is a diagram illustrating an intensity ratio. [Figure 30] The effects of the configuration according to the modified example will be described. [Figure 31] 10A and 10B are diagrams illustrating the reason why determination is possible even in the configuration according to the modified example. [Figure 32] 10A and 10B are diagrams illustrating intensity ratios when switching between multi-bandpass filters. [Figure 33] 10 is a graph showing a change in intensity ratio when switching the multi-bandpass filter. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description, the same elements or elements having the same functions will be denoted by the same reference numerals, and redundant description will be omitted.

[0020] FIG. 1 is a schematic diagram of a fluorescence image acquisition system 1, which is a fluorescence image acquisition device according to a first embodiment. The fluorescence image acquisition system 1 is a device for generating fluorescence images for identifying the distribution of dyes in a sample, such as biological tissue, that is the object of observation. The images generated by the fluorescence image acquisition system 1 are used for purposes such as drug development and the study of treatment methods through analysis of the images. The fluorescence image acquisition system 1 includes an irradiation device 2 configured to irradiate the sample S with excitation light, an image acquisition device 3 that irradiates the sample S with the excitation light and acquires an image of the fluorescence generated in response to the irradiation, and an image processing device 4 that processes the image acquired by the image acquisition device 3. The image acquisition device 3 and the image processing device 4 may be configured to be able to send and receive image data between them using wired or wireless communication, or may be configured to be able to input and output image data via a recording medium.

[0021] FIG. 2 is a perspective view showing the configuration of the fluorescence image acquisition system 1 of FIG. 1. In FIG. 2, the optical path of the excitation light is indicated by a dotted line with an arrow, and the optical path of the fluorescence is indicated by a solid line with an arrow. The irradiation device 2 includes an excitation light source 2a and an excitation light filter unit 2b. The image acquisition device 3 includes a dichroic mirror 11, a fluorescence filter unit 3a, a first optical filter 13, and a first camera 15. The image acquisition device 3 executes an acquisition step of acquiring a first fluorescence image (described below) in a first optical state and a second fluorescence image (described below) in a second optical state in which fluorescence is measured with wavelength characteristics different from those of the first optical state, for each of the multiple fluorescence beams corresponding to the multiple excitation light beams output from the irradiation device 2, via the fluorescence filter unit 3a having multiple reflection wavelength bands and multiple transmission wavelength bands. Note that the excitation light is, for example, a light beam output in a beam form.

[0022] The excitation light source 2a is a light source capable of switching between and irradiating excitation light in multiple wavelength bands (wavelength distributions), such as an LED (Light Emitting Diode) light source, a light source consisting of multiple monochromatic laser light sources, or a light source combining a white light source and a wavelength-selective optical element. The excitation light filter unit 2b is a multi-bandpass filter disposed on the optical path of the excitation light from the excitation light source 2a and capable of transmitting light in multiple predetermined wavelength bands. The transmission wavelength bands of the excitation light filter unit 2b are set according to the multiple wavelength bands of the excitation light that can be used. The dichroic mirror 11 is disposed between the excitation light filter unit 2b and the sample S and is an optical element that reflects the excitation light toward the sample S and transmits the fluorescence emitted from the sample S in response. In this embodiment, the irradiation device 2 performs an irradiation step of irradiating the sample S with excitation light in each of M wavelength bands (M is an integer greater than or equal to 2). Hereinafter, the excitation light in each of the M wavelength bands will be referred to as the first excitation light, the second excitation light, ..., and the M excitation light.

[0023] The fluorescence filter unit 3a is a multi-bandpass filter that is provided on the optical path of the fluorescence transmitted by the dichroic mirror 11 and has the property of transmitting light in a plurality of predetermined wavelength ranges. The fluorescence filter unit 3a has a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges. The reflection wavelength ranges and transmission wavelength ranges are arranged alternately. The reflection wavelength ranges are, for example, wavelength ranges between the transmission wavelength ranges. The transmission wavelength ranges of this fluorescence filter unit 3a are set according to the wavelength ranges of fluorescence generated by pigments that may be contained in the sample S to be observed.

[0024] The first optical filter 13 is an optical system provided on the optical path of the fluorescence transmitted by the fluorescence filter unit 3a for acquiring wavelength information of the fluorescence. The first optical filter 13 includes a switching mechanism (not shown) that switches the position of the first optical filter 13 so that the first optical filter 13 slides between a position on the optical path of the fluorescence from the fluorescence filter unit 3a and a position off the optical path. In this embodiment, a state in which the first optical filter 13 is provided on the optical path of the fluorescence and thereby wavelength information of the fluorescence can be acquired via the first optical filter 13 is referred to as a first optical state, and a state in which the first optical filter 13 is not provided on the optical path of the fluorescence and thus wavelength information of the fluorescence can be acquired without the first optical filter 13 is referred to as a second optical state. The image acquisition device 3 can measure fluorescence in the first optical state and in a second optical state in which fluorescence is measured with wavelength characteristics different from those of the first optical state.

[0025] The first optical filter 13 is a filter whose transmittance varies across each reflection wavelength range or each transmission wavelength range of the fluorescent filter unit 3a. In the first embodiment, the first optical filter 13 is a gradient filter having wavelength characteristics of transmittance such that the transmittance increases linearly as the wavelength increases. That is, the transmittance of the first optical filter 13 increases monotonically across each transmission wavelength range of the fluorescent filter unit 3a. In the following description of the first embodiment, the first optical filter 13, which is such a gradient filter, will also be referred to as the first gradient filter 13. The first optical filter 13 may also be a gradient filter having wavelength characteristics of transmittance such that the transmittance decreases linearly as the wavelength increases. The first optical filter 13 may also be a gradient filter whose transmittance changes monotonically across each reflection wavelength range of the fluorescent filter unit 3a.

[0026] First camera 15 is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer greater than or equal to 2, for example, 2048 × 2048). When first gradient filter 13 is provided on the optical path of the fluorescence, first camera 15 captures the fluorescence passing through first gradient filter 13 to acquire a first fluorescence image. That is, the first fluorescence image is a fluorescence image acquired in a first optical state. Furthermore, when first gradient filter 13 is removed from the optical path of the fluorescence, first camera 15 captures the fluorescence to acquire a second fluorescence image. That is, the second fluorescence image is a fluorescence image acquired in a second optical state. First camera 15 outputs the acquired first and second fluorescence images to image processing device 4 via communication or via a recording medium. In the first embodiment, the N pixels are referred to as a first pixel, a second pixel, ..., and an Nth pixel, respectively.

[0027] Next, the configuration of the image processing device 4 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 4, and Fig. 4 is a block diagram showing the functional configuration of the image processing device 4. The image processing device 4 is a device that processes a plurality of first fluorescent light images and second fluorescent light images.

[0028] 3, the image processing device 4 is physically a computer or the like including a processor such as a CPU (Central Processing Unit) 101, a recording medium such as a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, all of which are electrically connected to one another. Note that the image processing device 4 may include input / output devices such as a display, a keyboard, a mouse, a touch panel display, or a data recording device such as a hard disk drive or semiconductor memory. The image processing device 4 may also be composed of multiple computers.

[0029] As shown in FIG. 4 , the image processing device 4 includes, as functional components, an image acquisition unit 201, a vector parameter calculation unit 202, a single-color pixel discrimination unit 203, a clustering unit 204, a statistical value calculation unit 205, and an image generation unit 206. The functional units of the image processing device 4 shown in FIG. 4 are implemented by loading a program onto hardware such as the CPU 101 and RAM 102, which, under the control of the CPU 101, operates the communication module 104, the input / output module 106, and reads and writes data from and to the RAM 102. The CPU 101 of the image processing device 4 executes the computer program to cause the functional units of FIG. 4 to function and sequentially execute processes corresponding to the fluorescence image acquisition method described below. The CPU 101 may be a standalone piece of hardware or may be implemented in a programmable logic device such as an FPGA, like a software processor. The RAM and ROM may also be standalone pieces of hardware or may be incorporated into a programmable logic device such as an FPGA. All of the various data required to execute this computer program and the various data generated by executing this computer program are stored in built-in memories such as ROM 103 and RAM 102, or in storage media such as a hard disk drive. The functions of the functional components of the image processing device 4 will be described in detail below.

[0030] The image acquisition unit 201 acquires a first fluorescence image and a second fluorescence image for each of the first to Mth excitation lights from the image acquisition device 3. The first fluorescence image is a fluorescence image acquired in a first optical state for a plurality of fluorescence lights corresponding to the first to Mth excitation lights irradiated from the irradiation device 2. The second fluorescence image is a fluorescence image acquired in a second optical state for a plurality of fluorescence lights corresponding to the first to Mth excitation lights irradiated from the irradiation device 2.

[0031] An example of the fluorescence intensity of the first and second fluorescence images will be described with reference to FIG. 5. FIG. 5(a) shows the wavelength characteristics of the fluorescence intensity of the first fluorescence image acquired in the first optical state, and FIG. 5(b) shows the wavelength characteristics of the fluorescence intensity of the second fluorescence image acquired in the second optical state. Here, it is assumed that three types of fluorescence, namely, first fluorescence, second fluorescence, and third fluorescence, are captured. In FIG. 5(a), FI1, FI2, and FI3 respectively represent the fluorescence intensities of the first fluorescence, second fluorescence, and third fluorescence that have passed through the first gradient filter 13. In FIG. 5(b), FI4, FI5, and FI6 respectively represent the fluorescence intensities of the first fluorescence, second fluorescence, and third fluorescence that have not passed through the first gradient filter. Furthermore, T1 schematically represents the transmittance in the transmission wavelength range of the fluorescence filter unit 3a. As shown in Fig. 5(b), the magnitudes of the fluorescence intensities are approximately the same in the second optical state, but as shown in Fig. 5(a), in the first optical state, the fluorescence intensity FI1 of the first fluorescence increases, followed by the fluorescence intensity FI2 of the second fluorescence and the fluorescence intensity FI3 of the third fluorescence, in that order. This is because the transmittance of the first gradient filter 13 increases linearly as the wavelength increases.

[0032] The vector parameter calculation unit 202 performs a calculation step in which, for each pixel of the first fluorescence image and the second fluorescence image acquired for each of the first to Mth excitation lights, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light are calculated, and vector parameters of each pixel are calculated based on the first and second vectors.

[0033] In this embodiment, the process of "determining whether each pixel is a monochromatic pixel based on the vector parameters of each pixel calculated from the first vector and the second vector" (details will be described later) is performed. As a comparative example, the process of "determining whether each pixel is a monochromatic pixel based on the intensity ratio, which is the ratio between the intensity value of the pixel in the first fluorescent image and the intensity value of the pixel in the second fluorescent image corresponding to that pixel," will be described, and the advantages of the determination method according to this embodiment compared to the comparative example will be described.

[0034] In the configuration according to the comparative example, an intensity ratio is calculated, which is the ratio between the intensity value of a pixel in the first fluorescence image and the intensity value of a pixel in the second fluorescence image corresponding to that pixel, and the intensity ratios for each of the first to Mth excitation light beams are calculated. Here, for the mth excitation light beam (m is an integer of 1 to M), the intensity ratio, which is the ratio between the intensity value at the nth pixel (n is an integer of 1 to N) in the first fluorescence image and the intensity value at the nth pixel in the second fluorescence image, is defined as R. mn Then, based on the measured intensity values, the intensity ratio R mn is calculated. The intensity ratio is the ratio of the intensity value of the pixel of the first fluorescent image to the intensity value of the pixel of the second fluorescent image. The intensity ratio may be the ratio of the intensity value of the pixel of the second fluorescent image to the intensity value of the pixel of the first fluorescent image.

[0035] Referring to FIG. 6, for example, the intensity ratio R 11 , and the intensity ratio R at the first pixel relative to the second excitation light 21 The following describes the characteristics of the first gradient filter 13. In FIG. 6, as an example, it is assumed that one type of first dye is irradiated with the first excitation light and the second excitation light. In FIG. 6(a), the excitation light spectra ES1 and ES2 of the first excitation light and the second excitation light are shown in relation to the absorption spectrum AS of the first dye. In FIG. 6(b), the transmittance T1 in the transmission wavelength range of the fluorescence filter section 3a and the transmittance T2 of the first gradient filter 13 taking the transmittance T1 into account are shown in relation to the fluorescence spectrum FS of the first dye. Here, the wavelength is λ, the absorption spectrum AS is f AS (λ), the excitation light spectrum ES1 is f ES1(λ), the excitation light spectrum ES2 is f ES2 (λ), fluorescence spectrum FS FS (λ), transmittance T1 is S T1 (λ), transmittance T2 is S T2 (λ), the intensity value I of the first pixel acquired in the first optical state when irradiated with the first excitation light is 11 can be calculated as in the following formula (1): On the other hand, the intensity value I of the first pixel acquired in the second optical state when irradiated with the first excitation light is 12 can be calculated as in the following formula (2).

[0036]

number

[0037] The intensity value I of the first pixel acquired in the first optical state when irradiated with the second excitation light 21 can be calculated as in the following formula (3): On the other hand, the intensity value I of the first pixel acquired in the second optical state when irradiated with the second excitation light is 22 can be calculated as in the following formula (4).

[0038]

number

[0039] From the above formulas (1) and (2), the intensity ratio R at the first pixel corresponding to the first excitation light is 11 is calculated by the following formula (5): From the above formulas (3) and (4), the intensity ratio R 21 is calculated as shown in the following formula (6).

[0040]

number

[0041] In the above formula (5), ∫f AS (λ)fES1 Since (λ)dλ is cancelled, the intensity ratio R 11 does not depend on the first excitation light. Similarly, the intensity ratio R 21 does not depend on the second excitation light. Therefore, as shown in the above formulas (5) and (6), the intensity ratio R 11 and intensity ratio R 21 However, the intensity ratio R 11 and intensity ratio R 21 is actually calculated based on the measured intensity value, so the intensity ratio R 11 and intensity ratio R 21 Note that these do not necessarily have the same value.

[0042] Here, when the first pixel is a pixel that reflects a mixture of fluorescence from the first dye and the second dye, the intensity ratio R' at the first pixel corresponding to the first excitation light is 11 , and the intensity ratio R′ at the first pixel corresponding to the second excitation light 21 The absorption spectrum of the second dye is f' AS (λ), and the fluorescence spectrum of the second dye is f' FS (λ). Intensity ratio R' 11 and R' 21 is calculated using the following equations (7) and (8).

[0043]

number

[0044] In the above formula (7), the values ​​in the numerator and denominator that depend on the first excitation light are not canceled out, so the intensity ratio R' 11 is dependent on the first excitation light. Similarly, the intensity ratio R' 21 depends on the second excitation light. That is, the intensity ratio R' 11 and intensity ratio R' 21 will not be the same value.

[0045] The characteristics of the intensity ratios described above will be explained. FIG. 7 is a graph schematically showing the intensity ratio RX1 when the first pixel reflects fluorescence from the first dye, the intensity ratio RX2 when the first pixel reflects fluorescence from the second dye, and the intensity ratio RX3 when the first pixel reflects a mixture of fluorescence from the first dye and the second dye. FIG. 7 shows the intensity ratios for each of the first to third excitation lights. As shown in FIG. 7, the intensity ratios RX1 and RX2 have the same value for each excitation light, whereas the intensity ratio RX3 has a different value for each excitation light. Furthermore, as shown in FIG. 7, the intensity ratios RX1 and RX2 have different values ​​from each other. This is because, as shown in the above formulas (5) and (6), the transmittance T2 of the first gradient filter 13 taking the transmittance T1 into account (i.e., S T2 ) is a value that depends on the wavelength λ, and the transmittance T1 (i.e., S T1 ) is a value that depends on the wavelength λ. T1 and S T2 is a constant independent of the wavelength λ, the intensity ratio is a constant, and the intensity ratio RX1 and the intensity ratio RX2 have the same value. In the comparative example, it is possible to determine whether a pixel is a monochromatic pixel based on such characteristics of the intensity ratio.

[0046] Here, the method of determining whether a pixel is a monochromatic pixel based on the intensity ratio (the comparative example described above) has a problem in that the accuracy of determining whether a pixel is a monochromatic pixel decreases when the intensity of the excitation light is low. Figures 8 to 10 are diagrams explaining the problem with the comparative example (determination based on the intensity ratio).

[0047] As shown in Figure 8(a), four fluorescent tapes (red, orange, green, and blue) were arranged in a vertical arrangement of four strips and a horizontal arrangement of four strips. In the example shown in Figure 8(a), the vertical arrangement of four strips was on top and the horizontal arrangement of four strips was on the bottom. More specifically, red fluorescent tape samples S11 and S21, orange fluorescent tape samples S12 and S22, green fluorescent tape samples S13 and S23, and blue fluorescent tape samples S14 and S24 were prepared and arranged in a cross shape on a slide. Figure 8(b) shows a fluorescent image (observed image) captured by a monochrome camera of such fluorescent tape. Figure 8(c) shows an image in which monochromatic and mixed-color pixels are distinguished based on the intensity ratio. In Figure 8(c), the areas marked with "circles" indicate overlapping areas of fluorescent tape. Ideally, areas where fluorescent tape does not overlap and areas SC1 where fluorescent tape of the same color overlap among overlapping fluorescent tape areas should be determined as monochromatic pixels, while areas where fluorescent tape of different colors overlap should be determined as mixed pixels. First, for example, in the area SC2 where red and orange overlap, their spectra are similar, so even though it is actually a mixed pixel area, it may not be properly determined. Second, in the example shown in Figure 8(c), the excitation light intensity of sample S24, which is horizontally oriented blue fluorescent tape, is weak, so area SC3 (the area overlapping sample S13, which is vertically oriented green fluorescent tape), which should actually be determined as a mixed pixel, is determined as a monochromatic pixel.

[0048] The second problem will be described with reference to FIG. 9. FIG. 9(a) is an example of a second fluorescence image (observation image), and FIG. 9(b) is an image displaying the intensity ratio between the second fluorescence image of FIG. 9(a) and the corresponding first fluorescence image. A region E500 in FIG. 9(a) where the intensity of excitation light is weak becomes an unstable region E550, as shown in FIG. 9(b), where it is not possible to accurately determine whether the pixel is a single-color pixel or a mixed-color pixel. This is because, when calculating the intensity ratio for each pixel, pixels with low excitation light intensity are affected by noise, resulting in a large error in the intensity ratio.

[0049] In such a case, it is not possible to accurately distinguish between monochromatic pixels. As shown in Fig. 10(a), the region SC11 where it is not possible to clearly distinguish between monochromatic pixels and mixed pixels due to the low intensity of the excitation light described above results in a region SC110 where it is not possible to distinguish between monochromatic pixels in the image shown in Fig. 10(b), in which monochromatic pixels are distinguished based on a predetermined threshold value, for example. As such, in the discrimination method based on the intensity ratio, it may not be possible to accurately discriminate between monochromatic pixels and mixed pixels for pixels with low excitation light intensity.

[0050] To solve this problem, the image processing device 4 according to this embodiment determines whether each pixel is a monochromatic pixel based on a vector parameter determined by a first vector corresponding to the intensity value of the first fluorescence image for each excitation light and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light, rather than on the intensity ratio described above.

[0051] 11 to 13 are diagrams illustrating discrimination based on vector parameters. FIG. 11 illustrates the case of a monochromatic pixel, and FIG. 12 illustrates the case of a mixed pixel. In FIG. 11(a), the upper graph shows the intensity of a pixel in the second fluorescent image for each excitation light ("excitation wavelength 1," "excitation wavelength 2," and "excitation wavelength 3"), and the lower graph shows the intensity of the pixel in the first fluorescent image for each excitation light ("excitation wavelength 1," "excitation wavelength 2," and "excitation wavelength 3"). As shown in FIG. 11(a), for a monochromatic pixel, the intensity ratio α is a constant value regardless of the excitation light, as described above. Now, as shown in FIG. 11(b), considering vectors corresponding to the intensity values ​​for each excitation light ("excitation wavelength 1," "excitation wavelength 2," and "excitation wavelength 3"), the intensity ratio α is constant, so the first vector corresponding to the intensity values ​​of the first fluorescent image and the second vector corresponding to the intensity values ​​of the second fluorescent image are positioned approximately on the same line.

[0052] 12(a), in a mixed pixel, the "constant intensity ratio" does not hold true as described above, and the intensity ratio γ1 for "excitation wavelength 1," the intensity ratio γ2 for "excitation wavelength 2," and the intensity ratio γ3 for "excitation wavelength 3" each have different values. Therefore, as shown in FIG. 12(b), when considering vectors corresponding to the intensity values ​​of each excitation light ("excitation wavelength 1," "excitation wavelength 2," and "excitation wavelength 3"), the intensity ratio is not constant, and therefore the first vector corresponding to the intensity values ​​of the first fluorescence image and the second vector corresponding to the intensity values ​​of the second fluorescence image do not lie on the same straight line.

[0053] In this way, the relationship between the first vector and the second vector differs depending on whether the pixel is a monochrome pixel or a mixed pixel, so it is possible to appropriately determine whether each pixel is a monochrome pixel or not based on the vector parameters calculated based on the first vector and the second vector.

[0054] The vector parameter may be, for example, the angle θ between the first vector and the second vector. The vector parameter may be an index indicating the similarity between the first vector and the second vector. That is, in the calculation step, the vector parameter calculation unit 202 may calculate a vector parameter indicating the similarity between the first vector and the second vector. The vector parameter may be calculated from the dot product or cross product of the first vector and the second vector. That is, in the calculation step, the vector parameter calculation unit 202 may calculate the vector parameter from the dot product or cross product of the first vector and the second vector.

[0055] An example of vector parameters (indicators of monochromaticity) is shown below in equations (9) to (14). In each of the following equations, θ is the angle between the first vector and the second vector, a → is the first vector, b → indicates the second vector, "·" indicates the dot product, and "×" indicates the cross product.

[0056]

number

[0057] Furthermore, discrimination based on vector parameters is less likely to encounter the problem of decreased discrimination accuracy when the excitation light intensity is low, which was an issue in the comparative example described above. Consider the case where the intensity of the excitation light of "excitation wavelength 3" among the excitation lights is extremely low, as shown in FIG. 13 . When the intensity of one of the excitation lights is low, as shown in FIG. 13 , a vector appears only on the plane of the intensity of the other excitation lights (here, "excitation wavelength 1" and "excitation wavelength 2"). In this case, calculation of vector parameters (such as the angle θ) is simply performed on the plane on which the vector appears. The inclusion of excitation light with low intensity does not deteriorate the calculation accuracy of the vector parameters (and thus the discrimination accuracy of single-color pixels). For example, the angle θ between vectors is mainly determined by vectors with high intensity, so the inclusion of excitation light with low intensity is unlikely to lead to a deterioration in discrimination accuracy.

[0058] In addition, if there is a bias in intensity between the dyes, the influence of the dye with the greater intensity may become greater, and for example, a mixed pixel may be determined to be a monochromatic pixel. To address this, the intensity may be corrected (normalized) by multiplying by a coefficient so that the average intensity of each excitation light is equal. However, multiplying by a coefficient increases the risk of misidentifying a monochromatic pixel as a mixed pixel, so the coefficient may be determined appropriately taking into account the balance.

[0059] The monochromatic pixel determination unit 203 executes a determination step of determining whether or not each pixel is a monochromatic pixel based on the vector parameters calculated by the vector parameter calculation unit 202. As an example, the monochromatic pixel determination unit 203 may determine that a pixel whose vector parameter (index) is equal to or smaller than a threshold is a monochromatic pixel, and that a pixel whose vector parameter (index) is greater than the threshold is not a monochromatic pixel (is a mixed pixel).

[0060] 14 to 16 are diagrams illustrating an example of threshold determination. In FIGS. 14 and 15, (a) shows the use of a sample S containing three dyes ("Dye 1," "Dye 2," and "Dye 3"), and (b) shows an image in which each pixel is color-coded according to a value set based on calculated vector parameters so that the value decreases as the degree of monochromaticity increases and increases as the degree of monochromaticity decreases (presumably due to a mixture). In the example shown in FIG. 14, a user who views the image shown in FIG. 14(b) manually determines a threshold and inputs the threshold determined by the user, thereby generating an image (monochromatic pixel image) in which monochromatic pixel regions are segmented and displayed, as shown in FIG. 14(c). In the example shown in Fig. 15, a histogram is generated in which the horizontal axis represents an index (vector parameter) and the vertical axis represents the number of pixels, as shown in Fig. 15(c), and a user who views the histogram manually determines a threshold value and inputs the determined threshold value to generate an image (monochromatic pixel image) in which monochromatic pixel regions are segmented and displayed, as shown in Fig. 15(d). The user who views the histogram may set the threshold value to, for example, an inflection point where the number of pixels suddenly increases (suddenly changes).

[0061] The threshold may be determined automatically without relying on user settings. For example, the threshold may be determined automatically using a binarization technique (such as Otsu's binarization or the modal method), which is a well-known technique. Furthermore, assuming that clustering (described later) is performed, it is not necessary to strictly identify monochromatic pixels at this stage; only pixels with a sufficiently high degree of monochromaticity need be detected. Therefore, as shown in FIG. 16 , for example, a process may be performed automatically in which the top 1% of pixels with small index (vector parameter) values ​​are determined as the threshold. The percentage, such as 1%, does not need to be set strictly; it is sufficient that a sufficiently small percentage is set so that only pixels with a clearly high degree of monochromaticity are detected. In this case, the percentage setting must be performed in advance by the user, so the overall process can be considered to be semi-automatic.

[0062] The clustering unit 204 executes a clustering step of classifying a plurality of pixels (hereinafter referred to as "a plurality of monochromatic pixels") determined by the monochromatic pixel determination unit 203 into clusters. Prior to the clustering process, the clustering unit 204 generates matrix data Y in which the fluorescence intensity values ​​of N pixels constituting each of the C fluorescence images, each of which is composed of N pixels, are arranged in parallel in a one-dimensional manner and are obtained by irradiating a sample with excitation light beams having C wavelength distributions (C is an integer equal to or greater than 2). The C fluorescence images may be a compilation of M first fluorescence images and M second fluorescence images acquired by the image acquisition device 3 by irradiating the sample with the first to M excitation light beams, or may be M second fluorescence images. Alternatively, the C fluorescence images may be C fluorescence images captured in a second optical state by irradiating the sample with C excitation light beams separate from the first to M excitation light beams.

[0063] As an example, the clustering unit 204 clusters the multiple single-color pixels into L pixel groups (L is an integer between 2 and N-1) based on the average value of the intensity ratios of each of the multiple single-color pixels. Here, the number L of pixel groups to be clustered is preset as a parameter stored in the image processing device 4, corresponding to, for example, the number of types of dyes that can be present in the sample S. The number L of pixel groups may be determined according to the type of excitation light or the number C of wavelength distributions of the excitation light, or may be determined independently of the type of excitation light or the number C of wavelength distributions of the excitation light. The average values ​​of the intensity ratios of single-color pixels of the same color are the same or close to each other. The clustering unit 204 clusters the multiple single-color pixels into L pixel groups by determining the distance (closeness of values) between the average values ​​of the intensity ratios for each of the multiple single-color pixels. The clustering unit 204 divides matrix data Y, in which the fluorescence intensity values ​​of the pixels of C fluorescence images are arranged in parallel in one dimension, into cluster matrices for each of the L pixel groups to regenerate the data. The average value of the intensity ratios of the plurality of single-color pixels may be, for example, a simple average or a weighted average. The average value may also be calculated using other calculation methods. The clustering unit 204 may perform clustering based on the median value of the intensity ratios of the plurality of single-color pixels. The clustering unit 204 may perform clustering based on the intensities of the plurality of single-color pixels for each of the C excitation lights.

[0064] Fig. 17 shows an image of pixel groups clustered by the clustering unit 204. As shown in Fig. 17, the sample S contains three types of dyes: dye C1, dye C2, and dye C3, and the clustering unit 204 clusters a plurality of single-color pixels into three pixel groups PGr1 to PGr3.

[0065] The statistical value calculation unit 205 executes a calculation step of calculating the statistical values ​​of intensity values ​​based on L cluster matrices obtained for the sample S. The statistical value calculation unit 205 obtains a mixing matrix A for generating K dye images showing the distribution of each of K dyes (K is an integer between 2 and C) from C fluorescent images based on the L cluster matrices obtained for the sample S. Generally, according to the non-negative matrix factorization (NMF) calculation method, the relationship between matrix data Y, which is an observation matrix, and dye matrix data X, in which K dye images are arranged in parallel in one dimension for each pixel, is expressed by the following equation (15) using the mixing matrix A: Y = AX (15) Here, Y is matrix data with C rows and N columns, A is matrix data with C rows and K columns, and X is matrix data with K rows and N columns. Conversely, once the value of mixing matrix A is determined, the dye matrix data X can be calculated by the inverse matrix A of mixing matrix A. -1 and matrix data Y, the following equation (16) is used: X=A -1 Y···(16) This process is called unmixing.

[0066] Here, the statistical value calculation unit 205 regenerates matrix data Y' by compressing the matrix data Y generated by the clustering unit 204 in units of pixel groups clustered by the clustering unit 204. In detail, the statistical value calculation unit 205 calculates a statistical value for each pixel group of the clustered cluster matrix for the fluorescence intensity of each row of the matrix data Y, and compresses the pixel group of each row into a single pixel having the calculated statistical value. In this way, the statistical value calculation unit 205 regenerates matrix data Y', which is matrix data with C rows and L columns. As the statistical value, the statistical value calculation unit 205 may calculate an average value based on the integrated value of the fluorescence intensity, may calculate the mode of the fluorescence intensity, or may calculate the median value of the fluorescence intensity.

[0067] Furthermore, the statistical value calculation unit 205 also calculates the following equation (17) including the mixing matrix A in the reproduced matrix data Y' and the dye matrix data X' compressed in the same manner from the dye matrix data X: Y'=AX' (17) holds, the mixing matrix A is derived based on the matrix data Y'. Fig. 18 shows an image of the matrix data Y' regenerated by the statistical value calculation unit 205 and the corresponding dye matrix data X'. One square shown in Fig. 18 represents one element of the matrix data. In this way, the dye matrix data X and matrix data Y divided into three pixel groups PGr1 to PGr3 are data compressed into three columns of dye matrix data X' and matrix data Y', with the statistical values ​​for each of the pixel groups PGr1 to PGr3 being used as representative values.

[0068] The statistical value calculation unit 205 derives the mixing matrix A based on the matrix data Y' as follows. That is, the statistical value calculation unit 205 sets an initial value to the mixing matrix A, calculates the loss function (loss value) Loss of the following equation (18) while sequentially changing the value of the mixing matrix A, and derives the mixing matrix A that reduces the value of the loss function Loss. Note that a regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree to which importance is attached to the regularization term) may be added to this loss function.

[0069]

number

[0070] As described above, the statistical value calculation unit 205 calculates a loss function for each of the L cluster matrices divided by the clustering unit 204 by referring to the statistical values ​​of the C pieces of matrix data Y', calculates the loss function Los based on the sum of the L loss functions, and obtains the mixing matrix A based on the loss function Los. In this case, the statistical value calculation unit 205 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values ​​a, b, and c of the statistical values ​​of the C pieces of matrix data Y', and then obtains the loss function Los by calculating the sum of the corrected loss functions. Note that the statistical value calculation unit 205 may calculate the loss function for each of the L cluster matrices by correcting the row components of the difference value Y'-AX' for each wavelength band of the excitation light by dividing them by the C statistical values ​​corresponding to each wavelength band of the excitation light.

[0071] Alternatively, the statistical value calculation unit 205 may generate the mixing matrix A as follows: For each of the C fluorescent images, the statistical value calculation unit 205 may calculate the above statistical value for each of the L clustered pixel groups, and then arrange the calculated statistical values ​​for each of the C fluorescent images to generate a mixing matrix A with C rows and L columns. That is, the mixing matrix A may be the same as the matrix data Y'. In this case, the number of clusters and the number of dyes are equal, making it easier to generate the mixing matrix A.

[0072] The image generation unit 206 executes an unmixing step of acquiring K dye images by unmixing C fluorescent images obtained from the sample S to be observed using a mixing matrix A, which is a statistical value derived by the statistical value calculation unit 205. Specifically, the image generation unit 206 adds an inverse matrix A of the mixing matrix A to matrix data Y generated by the clustering unit 204 based on the C fluorescent images. -1The dye matrix data X is calculated by applying the above formula. Then, the image generation unit 206 reconstructs K dye images from the dye matrix data X and outputs the reconstructed K dye images. The output destination at this time may be an output device of the image processing device 4, such as a display or a touch panel display, or may be an external device connected to the image processing device so as to be able to communicate data with it.

[0073] Next, the procedure for the observation process of the sample S using the fluorescence image acquisition system 1 according to the first embodiment, that is, the flow of the fluorescence image acquisition method according to the first embodiment, will be described. Fig. 19 is a flowchart showing the procedure for the observation process by the fluorescence image acquisition system 1.

[0074] First, the irradiation device 2 irradiates the sample S with excitation light of each of a plurality of wavelength distributions (step S1; irradiation step). Next, the image acquisition unit 201 of the image processing device 4 acquires, via the fluorescence filter unit 3a, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from those of the first optical state, for each of a plurality of fluorescence corresponding to each of the plurality of excitation light (step S2; acquisition step). Steps S1 and S2 may be repeated alternately. For example, after irradiating the sample with the first excitation light in the irradiation step and acquiring the first fluorescence image in the first acquisition step, the process may return to the irradiation step to irradiate the sample with the first excitation light and acquire the second fluorescence image in the second acquisition step, or the process may then return to the irradiation step to irradiate the sample with the second excitation light.

[0075] Furthermore, the vector parameter calculation unit 202 of the image processing device 4 calculates vectors corresponding to the intensity values ​​for the first fluorescent image and the second fluorescent image, and calculates vector parameters from the vectors (step S3; calculation step). Next, the monochromatic pixel determination unit 203 of the image processing device 4 determines whether each pixel is a monochromatic pixel based on the vector parameters (step S4; determination step).

[0076] Furthermore, the clustering unit 204 of the image processing device 4 clusters a plurality of pixels determined to be monochromatic pixels in the discrimination step into L pixel groups for C fluorescence images each consisting of N pixels, the C fluorescence images being obtained by irradiating the sample with excitation light having C wavelength distributions (step S5; clustering step). Next, the clustering unit 204 of the image processing device 4 generates matrix data Y in which the N pixels of the C fluorescence images are arranged in parallel (step S6; clustering step).

[0077] Furthermore, the statistical value calculation unit 205 of the image processing device 4 calculates statistical values ​​of the L pixel groups, thereby regenerating matrix data Y' based on the matrix data Y generated by the clustering unit 204 (step S7; calculation step). Next, the statistical value calculation unit 205 of the image processing device 4 derives a mixing matrix A based on the matrix data Y' (step S8; image generation step). Next, the image generation unit 206 of the image processing device 4 unmixes the matrix data Y generated based on C fluorescence images of the sample S using the mixing matrix A, thereby regenerating K dye images (step S9; image generation step). Finally, the image generation unit 206 of the image processing device 4 outputs the regenerated K dye images (step S10). This completes the observation process of the sample S.

[0078] Next, the effects of the image acquisition method according to this embodiment will be described.

[0079] The image acquisition method according to this embodiment includes an irradiation step of irradiating a sample S with excitation light beams each having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescence beams corresponding to the plurality of excitation light beams, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those of the first optical state, via a fluorescence filter unit 3a having a plurality of reflection wavelength bands and a plurality of transmission wavelength bands; a calculation step of calculating, for each pixel, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light beam and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light beam, and calculating vector parameters of each pixel based on the first and second vectors; and a determination step of determining whether each pixel is a monochromatic pixel based on the vector parameters.

[0080] In the image acquisition method according to this embodiment, a first fluorescence image in a first optical state and a second fluorescence image in a second optical state in which fluorescence is measured with wavelength characteristics different from those of the first optical state are acquired for each of a plurality of fluorescence light beams corresponding to a plurality of excitation light beams. Furthermore, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light beam and a second vector corresponding to the intensity value of the pixel in the second fluorescence image for each excitation light beam are calculated for each pixel, and vector parameters for each pixel are calculated based on the first and second vectors. Then, based on the vector parameters, it is determined whether each pixel is a monochromatic pixel. For monochromatic pixels, the intensity ratio, which is the ratio between the intensity value of the pixel in the first fluorescence image and the intensity value of the pixel in the second fluorescence image corresponding to that pixel, is generally constant regardless of the excitation light beam. Therefore, for monochromatic pixels, the first vector corresponding to the intensity value of the pixel in the first fluorescence image for each excitation light beam and the second vector corresponding to the intensity value of the pixel in the second fluorescence image for each excitation light beam have a predetermined relationship, and for example, they are positioned on roughly the same straight line. From the above, by determining whether a pixel is a monochromatic pixel from the vector parameters calculated based on the first and second vectors, it is possible to accurately identify monochromatic pixels from multiple pixels in a fluorescence image, even in a fluorescence image that includes pixels that reflect mixed fluorescence from multiple dyes and monochromatic pixels that reflect monochromatic fluorescence from a single dye. In other words, it is possible to identify monochromatic pixels in a fluorescence image.

[0081] FIG. 20 illustrates the effect of discrimination based on vector parameters. FIG. 20 shows pixel discrimination results when three fluorescent tapes are arranged crosswise. Ideally, areas where fluorescent tapes do not overlap and areas SC101 where fluorescent tapes of the same color overlap should be discriminated as monochromatic pixels, while areas where fluorescent tapes of different colors overlap should be discriminated as mixed pixels. FIG. 20(a) shows the results of discrimination between monochromatic and mixed pixels using the intensity ratio variation as an index in a comparative example. FIG. 20(b) shows the results of discrimination between monochromatic and mixed pixels using vector parameters (here, the angle θ between the first and second vectors) as an index. As shown in FIG. 20(a), in the configuration where discrimination is based on intensity ratio, in the vertical fluorescent tape on the far left, a location that should be discriminated as a monochromatic pixel is discriminated as a mixed pixel (or the result is difficult to discriminate between). Therefore, discrimination accuracy is not sufficiently ensured. In this regard, as shown in Figure 20(b), the configuration that discriminates based on vector parameters can generally properly distinguish between single-color pixels and mixed pixels, even for the vertical fluorescent tape on the far left. Furthermore, as shown in Figure 20(c), by setting an appropriate threshold for the image in Figure 20(b) and displaying the single-color pixels and mixed pixels separately, single-color pixels and mixed pixels can be properly detected. In the example of Figure 20(c), the mixed pixel area indicated by the circle can be properly detected.

[0082] As shown in FIG. 21(a), autofluorescence cannot be ignored in an actual sample. In this regard, autofluorescence regions are not mixed with the dye, resulting in monochromatic pixels. Therefore, as shown in FIG. 21(b), regions of strong autofluorescence and regions of strong dye occur in the sample. In this regard, as shown in FIG. 21(c), by detecting regions of strong autofluorescence based on vector parameters (e.g., the angle θ between the vectors), it is possible to separately detect regions of strong dye (regions where mixed pixels also exist) and regions of strong autofluorescence.

[0083] In the calculation step, a vector parameter indicating the similarity between the first vector and the second vector may be calculated, whereby pixels associated with similar (similar) first and second vectors (located on roughly the same straight line) can be appropriately determined as monochromatic pixels, thereby enabling highly accurate determination of monochromatic pixels.

[0084] In the calculation step, the vector parameters may be calculated from the inner product or cross product of the first vector and the second vector. For monochromatic pixels, there is a certain tendency in the value of the inner product or cross product of the first vector and the second vector. Therefore, by calculating the vector parameters from the inner product or cross product of the first vector and the second vector, monochromatic pixels can be identified with high accuracy based on the vector parameters.

[0085] The image acquisition method may further include a clustering step of classifying the pixels determined to be monochromatic pixels in the determining step into clusters, thereby making it possible to estimate the regions for each pigment.

[0086] The image acquisition method may further comprise a calculation step of calculating a statistical value of the intensity values ​​for each cluster, whereby the intensity values ​​can be appropriately characterized on a cluster-by-cluster basis, for example by calculating an average value as the statistical value.

[0087] The image acquisition method may further include an unmixing step of unmixing the fluorescence image using statistical values ​​for each cluster, in which the fluorescence image is unmixed using a mixing matrix generated from the statistical values ​​to acquire a dye image, thereby enabling efficient and highly accurate acquisition of the dye image.

[0088] Various embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments, and may be modified or applied to other things within the scope that does not change the gist of the claims.

[0089] For example, in the calculation step, weighting may be performed for each cluster according to the vector parameters, and the statistical values ​​may be calculated. For example, weighting according to the degree of monochromaticity can improve the accuracy of unmixing.

[0090] 22 and 23 are diagrams illustrating statistical value calculations using the degree of monochromaticity as a weight. Fig. 22(a) shows an image obtained when a statistical value (e.g., an average value) is simply calculated for each cluster, and Fig. 22(b) shows an image obtained when, for example, only the top N% of the degree of monochromaticity is used and weighting is performed according to the degree of monochromaticity in accordance with a vector parameter. For example, as shown in Fig. 23, it is assumed that cosθ (the above-mentioned formula (9)) is used as the vector parameter. In this case, for example, the pixels may be sorted in descending order of the value of cosθ (highest degree of monochromaticity), and only the above 80% of the pixels may be used and weighting may be performed so that the weight increases according to the degree of monochromaticity.

[0091] FIG. 24 is a diagram illustrating the unmixing result (true solution), FIG. 25 is a diagram illustrating the unmixing result (without weighting), and FIG. 26 is a diagram illustrating the unmixing result (with weighting). In each diagram, (a) shows the unmixed image, and (b) shows the mixing matrix (intensity for each wavelength). In addition, in FIGS. 25(b) and 26(b), the mixing matrix of the true solution shown in FIG. 24(b) is superimposed. As is clear from FIGS. 25(b) and 26(b), the mixing matrix when weighting is performed (FIG. 26(b)) is closer to the mixing matrix of the true solution, and unmixing can be performed with high precision.

[0092] 27, the irradiation step may be performed using an image acquisition device 3A employing a Sedat optical system instead of the image acquisition device 3. The image acquisition device 3 includes a filter wheel 300 having multiple monochrome bandpass filters instead of the fluorescence filter unit 3a, which is a multi-bandpass filter. In this case, the irradiation step may irradiate the sample with excitation light of multiple wavelength bands, using an excitation light source 2a capable of switching between and irradiating excitation light of multiple wavelength bands, an excitation light filter unit 2b, and the filter wheel 300 having multiple monochrome bandpass filters. With this configuration, it is possible to distinguish monochromatic pixels with high accuracy while employing the Sedat optical system.

[0093] FIG. 28(a) shows the fluorescence spectrum f(λ) of a certain dye when a monochrome bandpass filter with a transmission spectrum S1(λ) is used, and the transmission spectrum S2(λ) taking the transmission spectrum S1(λ) into consideration. FIG. 28(b) shows the excitation light spectrum ES1 of a certain excitation light relative to the absorption spectrum AS of a certain dye. FIG. 29(a) shows the fluorescence spectrum f(λ) of a certain dye when a monochrome bandpass filter with a transmission spectrum S'1(λ) is used, and the transmission spectrum S'2(λ) taking the transmission spectrum S'1(λ) into consideration. FIG. 29(b) shows the excitation light spectrum ES2 of a different excitation light relative to the absorption spectrum AS of a certain dye. When the filter wheel 300 is used, the monochrome bandpass filter is switched each time the excitation light is changed, and the intensity ratios with and without the filter (first optical state and second optical state) are calculated. The intensity ratio corresponding to FIG. 28 is expressed by the following equation (19), and the intensity ratio corresponding to FIG. 29 is expressed by the following equation (20). In this way, in the image acquisition device 3A employing the Sedat optical system, the intensity ratio differs for each excitation light.

[0094]

number

[0095] The image acquisition device 3A acquired the measurement image shown in Fig. 30(a), and as shown in Fig. 30(b), single-color pixels and mixed pixels were distinguished using a vector parameter (here, the angle θ between the vectors) as an index, and a threshold was set and the single-color pixels and mixed pixels were displayed separately, as shown in Fig. 30(c). As shown in Fig. 30(b) and Fig. 30(c), even when the image acquisition device 3A employing the Sedat optical system was used, single-color pixels and mixed pixels could be appropriately distinguished.

[0096] The reason why monochromatic pixels and mixed pixels can be appropriately distinguished even when using an image acquisition device 3A employing the Sedat optical system will be explained with reference to FIG. 31. FIG. 31(a) shows an example of vector parameters for monochromatic pixels, and FIG. 31(b) shows an example of vector parameters for mixed pixels. As shown in FIG. 31(a), when the intensity of each dye increases with a specific excitation light, the deviation in vector parameters (e.g., the angle θ between the vectors) decreases. This is because the angle is primarily determined by the excitation light with the higher intensity. On the other hand, as shown in FIG. 31(b), in mixed pixels, the angle θ between the vectors is clearly larger than that of monochromatic pixels. Therefore, even when using an image acquisition device 3A employing the Sedat optical system, monochromatic pixels and mixed pixels can be appropriately distinguished.

[0097] Furthermore, the multi-bandpass filter may be changed (switched) during measurement to match the excitation light. For example, as shown in Figures 32(a) and 32(b), a multi-bandpass filter with a transmission spectrum S'1(λ) may be switched to match the excitation light so that the excitation light spectrum ES3 of the excitation light ("excitation light 3") is not transmitted to the detection side. In this case, as shown in Figure 33, switching the multi-bandpass filter changes the intensity ratio, but the amount of change is not large, so that approximately monochromatic pixel detection is possible.

[0098] Furthermore, although the above description has been given assuming that vector parameters are calculated from vectors (first vector, second vector) corresponding to the intensity values ​​of the fluorescence images in two optical states (first fluorescence image, second fluorescence image), vector parameters may be calculated from vectors corresponding to the intensity values ​​of the fluorescence images in three or more optical states. In other words, three or more filters and excitation wavelengths may be used to calculate the vector parameters.

[0099] For example, the vector a1 → ,a2 → ,…a N → Then, examples of vector parameters are expressed by the following equations (21) to (23). i,j indicates the sum of all combinations. Instead of the sum, the average value may be used.

[0100]

number

[0101] Also, a → =a1 → +a2 → +…+a N → As a → and a i → The indexes (i=1, 2, ..., N) may be summed together. In this case, the number of summations can be reduced compared to the above example. Examples of vector parameters are shown in the following equations (24) to (26). An average value may be used instead of summation.

[0102]

number

[0103] 2...illumination device, 3, 3A...image acquisition device, 3a...fluorescence filter section, 4...image processing device, 202...vector parameter calculation section, 203...single-color pixel discrimination section, 300...filter wheel, A...mixing matrix.

Claims

1. an irradiation step of irradiating a sample with each of excitation lights having a plurality of wavelength distributions; an acquisition step of acquiring, for each of a plurality of fluorescence beams corresponding to the plurality of excitation beams, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those of the first optical state; a calculation step of calculating, for each pixel, a first vector corresponding to the intensity value of the first fluorescence image for each excitation light and a second vector corresponding to the intensity value of the second fluorescence image for each excitation light, and calculating vector parameters of each pixel based on the first vector and the second vector; a determining step of determining whether each of the pixels is a monochromatic pixel based on the vector parameters; An image acquisition method comprising:

2. The image acquisition method according to claim 1 , wherein the calculation step calculates the vector parameter indicating a similarity between the first vector and the second vector.

3. The image acquisition method according to claim 1 , wherein the calculation step calculates the vector parameters from an inner product or a cross product of the first vector and the second vector.

4. 2. The image acquisition method according to claim 1, wherein in the irradiating step, the excitation light having each of the plurality of wavelength distributions is irradiated onto the sample using a light source capable of switching between and irradiating excitation light having each of the plurality of wavelength distributions, a multi-bandpass filter, and a filter wheel having a plurality of bandpass filters.

5. The image acquisition method according to claim 1 , further comprising a clustering step of classifying a plurality of pixels determined to be monochromatic pixels in said determining step into clusters.

6. The image acquisition method of claim 5 , further comprising a calculating step of calculating statistics of intensity values ​​for each of the clusters.

7. 7. The image acquisition method according to claim 6, wherein in said calculating step, weighting is performed for each cluster according to the vector parameters, and said statistical value is calculated.

8. an unmixing step of unmixing a fluorescence image using the statistical value for each cluster; 8. The image acquisition method according to claim 6, wherein the unmixing step acquires a dye image by unmixing the fluorescence image using a mixing matrix generated from the statistical values.

9. an irradiation device that irradiates a sample with each of excitation lights having a plurality of wavelength distributions; an image acquisition device that acquires, for each of a plurality of fluorescence beams corresponding to a plurality of excitation beams, a first fluorescence image in a first optical state via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and a second fluorescence image in a second optical state in which the fluorescence is measured with wavelength characteristics different from those of the first optical state; an image processing device that processes a plurality of the first fluorescent light images and the second fluorescent light images, The image processing device includes: calculating, for each pixel, a first vector corresponding to the intensity value of the first fluorescent image for each excitation light and a second vector corresponding to the intensity value of the second fluorescent image for each excitation light, and calculating vector parameters for each pixel based on the first vector and the second vector; determining whether each of the pixels is a monochromatic pixel based on the vector parameters; 20. An apparatus configured to:

10. a program for determining whether a pixel is a monochromatic pixel based on a first fluorescence image in a first optical state, the first fluorescence image being acquired through a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges for each of a plurality of fluorescence beams corresponding to each of the plurality of excitation light beams by irradiating a sample with each of the plurality of excitation light beams having a plurality of wavelength distributions, and a second fluorescence image being acquired in a second optical state in which the fluorescence is measured with wavelength characteristics different from those of the first optical state, the program comprising: Computer, a vector parameter calculation unit that calculates, for each pixel, a first vector corresponding to the intensity value of the first fluorescent image for each excitation light and a second vector corresponding to the intensity value of the second fluorescent image for each excitation light, and calculates vector parameters for each pixel based on the first vector and the second vector; and a program that functions as a monochrome pixel determination unit that determines whether each of the pixels is a monochrome pixel based on the vector parameters;

Images