Dye image acquisition method, dye image acquisition device, and dye image acquisition program

Abstract

To improve throughput while increasing the accuracy of a separation image.SOLUTION: A dye image acquisition system 1 comprises: an image acquisition device 3 that irradiates a sample S with rays of excitation light in C (C is an integer of 2 or more) wavelength distributions to acquire C fluorescence images each constituted of N (N is an integer of 2 or more) pixels; and an image processing device 5 that creates a dye image. The image processing device 5 clusters the N pixels into L (L is an integer of 2 or more and N-1 or less) pixel groups on the basis of the fluorescence intensity of each pixel in the C fluorescence images, to generate L cluster matrices in which the C fluorescence images are arranged for each of the clustered pixel groups, calculates a statistical value of the fluorescence intensities of the pixel groups constituting the C fluorescence images, for each of the L cluster matrices, and subjects the C fluorescence images to unmixing by using the statistical value of the C fluorescence images for each of the L cluster matrices, to generate K dye images indicating the distribution for each of K (K is an integer of 2 or more and C or less) dyes.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the embodiment relates to a dye image acquiring method, a dye image acquiring device, and a dye image acquiring program. [Background technology]

[0002] Conventionally, a method of multiple identification has been used in which a sample such as a biological tissue is simultaneously stained with a plurality of substances in the sample. In order to observe the substances in the sample that have been subjected to multiple identification, a fluorescent image is obtained by irradiating the sample with excitation light. For example, Non-Patent Document 1 below discloses the application of a nonnegative matrix factorization (NMF) method to blind unmix a fluorescent image in which fluorescence in a plurality of wavelength ranges is observed to obtain a separated image for each substance in the sample. In addition, Non-Patent Document 2 below discloses a method of unmixing a fluorescent image by clustering the fluorescent image, extracting the maximum value of the fluorescent intensity for each clustered pixel group, and generating a separated image based on the maximum value. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Binjie Qin et al., “Target / Background ClassificationRegularized Nonnegative Matrix Factorization for Fluorescence Unmixing”, IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL.65, NO.4, APRIL2016 [Non-Patent Document 2] Tristan D. McRae et al., “Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning”, PLOSONE, December 2,2019 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional methods described above, in order to improve the accuracy of the separated images obtained by unmixing, it is necessary to increase the types of excitation light, the number of fluorescence bands to be observed, or the number of images, which tends to increase the calculation time for unmixing and reduce throughput.

[0005] Therefore, one aspect of the embodiments has been made in consideration of such problems, and has an objective of providing a dye image acquisition method, a dye image acquisition device, and a dye image acquisition program that are capable of improving throughput while increasing the accuracy of the separated images obtained by unmixing. [Means for solving the problem]

[0006] A dye image acquisition method according to a first aspect of an embodiment includes an image acquisition step of irradiating a sample with excitation light having C (C is an integer of 2 or more) wavelength distributions and acquiring C fluorescence images, each of which is composed of N (N is an integer of 2 or more) pixels; a clustering step of clustering the N pixels into L (L is an integer of 2 or more and N-1 or less) pixel groups based on the intensity values ​​of each pixel of the C fluorescence images, and generating L cluster matrices in which the C fluorescence images are arranged for each clustered pixel group; a calculation step of calculating statistics of the intensity values ​​of the pixel groups constituting the C fluorescence images for each of the L cluster matrices; and an image generation step of unmixing the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices, and generating K dye images showing the distribution of each of the K dyes (K is an integer of 2 or more and C or less).

[0007] Alternatively, a dye image acquisition device according to a second aspect of the embodiment includes an image acquisition device that irradiates a sample with excitation light having a wavelength distribution of C (C is an integer of 2 or more) and acquires C fluorescence images, each of which is composed of N (N is an integer of 2 or more) pixels, and an image processing device that generates a dye image that shows the distribution of dyes in the sample, in which the image processing device clusters the N pixels into L (L is an integer of 2 or more and N-1 or less) pixel groups based on the intensity values ​​of each pixel of the C fluorescence images, generates L cluster matrices in which the C fluorescence images are arranged for each clustered pixel group, calculates statistics of the intensity values ​​of the pixel groups that make up the C fluorescence images for each of the L cluster matrices, and performs unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices, thereby generating K dye images that show the distribution of each of the K dyes (K is an integer of 2 or more and C or less).

[0008] Alternatively, a dye image acquisition program according to a third aspect of the embodiment is a dye image acquisition program for generating a dye image showing the distribution of dyes in a sample based on C fluorescent images, each of which is composed of N pixels (N is an integer of 2 or more), obtained by irradiating a sample with excitation light having a wavelength distribution of C (C is an integer of 2 or more), and causes a computer to function as a clustering unit that clusters the N pixels into L (L is an integer of 2 or more and N-1 or less) pixel groups based on the intensity values ​​of each pixel of the C fluorescent images, and generates L cluster matrices in which the C fluorescent images are arranged for each clustered pixel group, a calculation unit that calculates statistics of the intensity values ​​of the pixel groups constituting the C fluorescent images for each of the L cluster matrices, and an image generation unit that performs unmixing on the C fluorescent images using the statistics of the C fluorescent images for each of the L cluster matrices, and generates K dye images showing the distribution of each of the K (K is an integer of 2 or more and C or less) dyes.

[0009] According to the first, second, or third aspect, C fluorescent images are obtained by capturing fluorescent images of a sample using excitation light with different wavelength distributions, N pixels of these C fluorescent images are clustered into L pixel groups based on the intensity value of each pixel, and L cluster matrices are generated in which the C fluorescent images are arranged for each of the L pixel groups. In addition, statistics of the intensity values ​​of the pixel groups constituting the C fluorescent images are calculated for each of the L cluster matrices, and the C fluorescent images are unmixed using the statistics of each of the C fluorescent images to generate K dye images. This makes it possible to reduce the amount of calculation required for unmixing even if the type of excitation light used for observation or the number of fluorescent bands observed increases. In addition, by performing unmixing using the statistics of the fluorescent images for each cluster matrix, the accuracy of separation of dye images can be improved. As a result, it is possible to improve the throughput when obtaining separated images while increasing the accuracy of the separated images.

[0010] The dye image acquisition method of the embodiment includes the steps of: [1] "irradiating a sample with excitation light having a wavelength distribution of C (C is an integer of 2 or more) and acquiring the C fluorescent images, each of which is composed of N (N is an integer of 2 or more) pixels; and a clustering step of clustering the N pixels into L pixel groups (L is an integer equal to or greater than 2 and equal to or less than N-1) based on the intensity values ​​of each pixel of the C fluorescent images, and generating L cluster matrices in which the C fluorescent images are arranged for each clustered pixel group; a calculation step of calculating a statistic of the intensity values ​​of the pixel groups constituting the C fluorescent images for each of the L cluster matrices; an image generating step of performing unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices to generate K dye images (K is an integer between 2 and C) showing distributions for each of the K dyes; A dye image acquisition method comprising the steps of:

[0011] In the dye image acquisition method of the embodiment, [2] "in the clustering step, the N pixels are clustered based on distribution information of the intensity values ​​for each of the C excitation lights; The dye image obtaining method according to the above item [1] may be used.

[0012] The dye image acquisition method of the embodiment further includes a wavelength information acquisition step of acquiring wavelength information related to a fluorescence wavelength corresponding to each pixel of the fluorescence image, In the clustering step, the N pixels are clustered based on the wavelength information corresponding to each of the pixels. The dye image obtaining method according to the above item [1] or [2] may be provided.

[0013] In the dye image acquisition method of the embodiment, [4] "in the wavelength information acquisition step, the sample is irradiated with excitation light having any one of the C wavelength distributions, and fluorescence from the sample is separated via a wavelength information acquisition optical system that separates the fluorescence with different wavelength characteristics, and a plurality of separated fluorescence images are obtained by capturing images of the separated fluorescence, and the wavelength information is acquired based on the plurality of separated fluorescence images; The dye image obtaining method according to the above item [3] may also be used.

[0014] In the dye image acquisition method of the embodiment, [5] "in the wavelength information acquisition step, the sample is irradiated with excitation light having any one of the C wavelength distributions, and fluorescence from the sample is captured with a camera capable of detecting at least two or more fluorescent wavelengths to obtain a fluorescence image, and the wavelength information is obtained based on the fluorescence image. The dye image obtaining method according to the above item [3] may also be used.

[0015] In the dye image acquisition method of the embodiment, [6] "in the calculation step, the statistical value is calculated based on an integrated value, a mode value, or a median value of the intensity values ​​of the pixel group; The dye image obtaining method according to any one of the above items [1] to [5] may be used.

[0016] In the dye image acquisition method of the embodiment, [7] "in the image generation step, a mixing matrix is ​​obtained using the statistics of the C fluorescent images for each of the L cluster matrices, and unmixing is performed using the mixing matrix; The dye image obtaining method according to any one of the above items [1] to [6] may be used.

[0017] In the dye image acquisition method of the embodiment, [8] "in the image generation step, a loss value is calculated based on the statistics of the C fluorescent images for each of the L cluster matrices using non-negative matrix factorization, and the mixing matrix is ​​obtained based on the sum of the loss values; The dye image obtaining method according to the above item [7] may also be used.

[0018] In the dye image acquisition method of the embodiment, [9] "in the image generation step, the loss value for each of the L cluster matrices is corrected based on the statistical value, and the mixing matrix is ​​obtained based on the sum of the corrected loss values; The dye image obtaining method according to the above item [8] may also be used.

[0019] The dye image acquisition device of the embodiment includes:

[10] "an image acquisition device that irradiates a sample with excitation light having a wavelength distribution of C (C is an integer of 2 or more) and acquires the C fluorescent images, each of which is composed of N (N is an integer of 2 or more) pixels; and an image processing device for generating a dye image showing a distribution of the dye in the sample, The image processing device includes: clustering the N pixels into L pixel groups (L is an integer between 2 and N-1) based on the intensity values ​​of each pixel of the C fluorescent images, and generating L cluster matrices in which the C fluorescent images are arranged for each clustered pixel group; calculating a statistic of the intensity values ​​of the pixel groups constituting the C fluorescent images for each of the L cluster matrices; performing unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices to generate K dye images (K is an integer between 2 and C) showing distributions for each of the K dyes; Dye image acquisition device."

[0020] In the dye image acquisition device of the embodiment,

[11] "the image processing device clusters the N pixels based on distribution information of the intensity values ​​for each of the C excitation lights; The dye image acquiring device according to the above item

[10] may be used.

[0021] In the dye image acquisition device of the embodiment,

[12] "the image processing device is Further acquiring wavelength information regarding a fluorescence wavelength corresponding to each pixel of the fluorescence image; clustering the N pixels based on the wavelength information corresponding to each pixel; The dye image acquiring device according to the above item

[10] or

[11] may be used.

[0022] The dye image acquisition device according to the embodiment is, irradiating the sample with excitation light having any one of the C wavelength distributions, separating the fluorescence from the sample through a wavelength information acquisition optical system that separates the fluorescence with different wavelength characteristics, and capturing an image of each of the separated fluorescence to obtain a plurality of separated fluorescence images; the image processing device acquires the wavelength information based on the plurality of separated fluorescent images. The dye image acquiring device according to the above item

[12] may be used.

[0023] In an embodiment, the dye image acquisition device is,

[14] "illuminating the sample with excitation light having any one of the C wavelength distributions, capturing an image of the fluorescence from the sample with a camera capable of detecting at least two or more fluorescence wavelengths to acquire a fluorescence image, and acquiring the wavelength information based on the fluorescence image. The dye image acquiring device according to the above item

[12] may be used.

[0024] In the dye image acquisition device of the embodiment,

[15] "the image processing device calculates the statistical value based on an integrated value, a mode value, or a median value of the intensity values ​​of the pixel group; The dye image obtaining device according to any one of the above items

[10] to

[14] may be used.

[0025] In the dye image acquisition device of the embodiment,

[16] "the image processing device obtains a mixing matrix using the statistics of the C fluorescent images for each of the L cluster matrices, and performs unmixing using the mixing matrix. The dye image obtaining device according to any one of the above items

[10] to

[15] may be used.

[0026] In the dye image acquisition device of the embodiment,

[17] "the image processing device calculates a loss value based on the statistics of the C fluorescent images for each of the L cluster matrices using non-negative matrix factorization, and obtains the mixing matrix based on the sum of the loss values; The dye image acquiring device according to the above item

[16] may be used.

[0027] In the dye image acquisition device of the embodiment,

[18] "the image processing device corrects the loss value for each of the L cluster matrices based on the statistical value, and obtains the mixing matrix based on the sum of the corrected loss values; The dye image acquiring device according to the above item

[17] may be used.

[0028] A dye image acquisition program according to an embodiment of the present invention is a dye image acquisition program for generating a dye image showing a distribution of dyes in a sample, based on C fluorescence images each of which is composed of N pixels (N is an integer of 2 or more) and which are acquired by irradiating a sample with excitation light having a wavelength distribution of C (C is an integer of 2 or more), Computer, a clustering unit that clusters the N pixels into L pixel groups (L is an integer equal to or greater than 2 and equal to or less than N-1) based on the intensity values ​​of each pixel of the C fluorescent images, and generates L cluster matrices in which the C fluorescent images are arranged for each clustered pixel group; a calculation unit for calculating a statistical value of the intensity values ​​of the pixel groups constituting the C fluorescent images for each of the L cluster matrices; and an image generation unit that performs unmixing on the C fluorescence images using the statistics of the C fluorescence images for each of the L cluster matrices to generate K dye images (K is an integer between 2 and C) showing distributions for each of the K dyes; "This is a dye image acquisition program that functions as a Effect of the Invention

[0029] According to one aspect of the embodiment, it is possible to improve throughput while improving the accuracy of separated images obtained by unmixing. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a schematic configuration diagram of a dye image acquisition system 1 according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing a configuration of an image acquisition device 3 in FIG. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of the image processing device 5 in FIG. 1. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of the image processing device 5 of FIG. [Diagram 5] FIG. 5 is a diagram showing an image of a group of pixels clustered by a first clustering function of clustering section 203 in FIG. 4. [Figure 6] 1 is a graph showing wavelength characteristics of the absorptance of excitation light for multiple dyes contained in sample S. [Figure 7] 5 is a graph showing the distribution of centroid fluorescence wavelengths identified by the clustering unit 203 of FIG. 4. [Figure 8]FIG. 5 is a diagram showing an image of a group of pixels clustered by the second clustering function of clustering section 203 in FIG. 4. [Figure 9] 5 is a diagram showing an image of matrix data Y' and dye matrix data X' regenerated by the statistical value calculation unit 204 in FIG. 4. FIG. [Figure 10] 1 is a flowchart showing the procedure of a dye image acquisition method according to an embodiment. [Figure 11] FIG. 2 is a diagram showing an example of a dye image generated by the dye image acquisition system 1 according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description, the same elements or elements having the same functions will be denoted by the same reference numerals, and duplicated description will be omitted.

[0032] FIG. 1 is a schematic diagram of a dye image acquisition system 1, which is a dye image acquisition device according to an embodiment. The dye image acquisition system 1 is a device for generating dye images for identifying the distribution of dyes in a sample such as a biological tissue to be observed. The images generated by the dye image acquisition system 1 are used for the purpose of developing medicines and examining treatment methods through the analysis of the images. For this reason, the dye image acquisition system 1 is required to generate images capable of quantitatively identifying the distribution of many substances (dyes) contained in a sample with high throughput. The dye image acquisition system 1 includes an image acquisition device 3 that irradiates a sample S with excitation light and acquires an image of the fluorescence generated in response to the irradiation, and an image processing device 5 that processes the image acquired by the image acquisition device 3. The image acquisition device 3 and the image processing device 5 may be configured to be capable of transmitting and receiving image data between them using wired or wireless communication, or may be configured to be capable of inputting and outputting image data via a recording medium.

[0033] Fig. 2 is a perspective view showing the configuration of the image acquisition device 3 in Fig. 1. In Fig. 2, the optical path of the excitation light is indicated by a dotted line with an arrow, and the optical path of the fluorescence is indicated by a solid line with an arrow. The image acquisition device 3 includes an excitation light source 7, a light source side filter set 9a, a dichroic mirror 11, a camera side filter set 9b, a wavelength information acquisition optical system 13, a first camera 15a, and a second camera 15b.

[0034] The excitation light source 7 is a light source capable of switching between and irradiating excitation light of a plurality of wavelength bands (wavelength distributions), and is, for example, an LED (Light Emitting Diode) light source, a light source consisting of a plurality of monochromatic laser light sources, or a light source combining a white light source and a wavelength selection optical element. The light source side filter set 9a is a multi-bandpass filter provided on the optical path of the excitation light from the excitation light source 7 and having a property of transmitting light of a plurality of predetermined wavelength bands. The transmission wavelength band of this light source side filter set 9a is set according to the plurality of wavelength bands of the excitation light that can be used. The dichroic mirror 11 is an optical member provided between the light source side filter set 9a and the sample S and having a property of reflecting the excitation light toward the sample S and transmitting the fluorescence emitted from the sample S in response to the reflection. The camera side filter set 9b is a multi-bandpass filter provided on the optical path of the fluorescence transmitted by the dichroic mirror 11 and having a property of transmitting light of a plurality of predetermined wavelength bands. The transmission wavelength band of this camera side filter set 9b is set according to the wavelength band of the fluorescence generated in the pigment that can be contained in the sample S to be observed.

[0035] The wavelength information acquisition optical system 13 is an optical system provided on the optical path of the fluorescence transmitted by the camera-side filter set 9b, and is used to acquire wavelength information of the fluorescence. That is, the wavelength information acquisition optical system 13 separates the fluorescence from the sample S into two optical paths with different wavelength characteristics. For example, a dichroic mirror having a wavelength characteristic of transmittance in which the transmittance increases linearly as the wavelength increases is used as the wavelength information acquisition optical system 13. The wavelength information acquisition optical system 13 using such a dichroic mirror separates the fluorescence with different wavelength characteristics, reflects a part of the fluorescence with a wavelength characteristic in which the reflectance decreases as the wavelength increases, and transmits a part of the fluorescence with a wavelength characteristic in which the transmittance increases as the wavelength increases. The wavelength information acquisition optical system 13 is provided with a support mechanism (not shown) that detachably supports the wavelength information acquisition optical system 13 on the optical path of the fluorescence from the camera-side filter set 9b.

[0036] The first camera 15a is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer equal to or greater than 2, for example, 2048×2048), and is a camera that captures one component of the fluorescence separated by the wavelength information acquisition optical system 13 to acquire one separated fluorescence image when the wavelength information acquisition optical system 13 is supported on the optical path of the fluorescence. Moreover, the first camera 15a captures the fluorescence transmitted through the camera-side filter set 9b to acquire a fluorescence image when the wavelength information acquisition optical system 13 is removed from the optical path of the fluorescence. The first camera 15a outputs the acquired separated fluorescence image or fluorescence image to the image processing device 5 by communication or via a recording medium. The second camera 15b is an imaging device that captures a two-dimensional image composed of the same N pixels as the first camera 15a, and is a camera that captures the other component of the fluorescence separated by the wavelength information acquisition optical system 13 to acquire the other separated fluorescence image when the wavelength information acquisition optical system 13 is supported on the optical path of the fluorescence. The second camera 15b outputs the acquired separated fluorescent image to the image processing device 5 using communication or via a recording medium.

[0037] The fluorescence image may be acquired by adding together one separated fluorescence image acquired by the first camera 15a and the other separated fluorescence image acquired by the second camera 15b using the image processing device 5. In this case, the support mechanism in the wavelength information acquisition optical system 13 may be omitted.

[0038] Next, the configuration of the image processing device 5 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 5, and Fig. 4 is a block diagram showing the functional configuration of the image processing device 5.

[0039] 3, the image processing device 5 is physically a computer or the like including a processor such as a CPU (Central Processing Unit) 101, a recording medium such as a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, each of which is electrically connected. Note that the image processing device 5 may include, as input / output devices, a display, a keyboard, a mouse, a touch panel display, etc., or may include a data recording device such as a hard disk drive or a semiconductor memory. The image processing device 5 may also be composed of multiple computers.

[0040] As shown in FIG. 4, the image processing device 5 includes an image acquisition unit 201, a wavelength information acquisition unit 202, a clustering unit 203, a statistical value calculation unit 204, and an image generation unit 205 as functional components. Each functional unit of the image processing device 5 shown in FIG. 4 is realized by reading a program (a dye image acquisition program according to the embodiment) onto hardware such as the CPU 101 and the RAM 102, and operating the communication module 104 and the input / output module 106 under the control of the CPU 101, and reading and writing data in the RAM 102. The CPU 101 of the image processing device 5 executes this computer program to operate each functional unit of FIG. 4, and sequentially executes processing corresponding to a dye image acquisition method described later. The CPU 101 may be a standalone hardware, or may be implemented in a programmable logic such as an FPGA like a software processor. The RAM and ROM may also be standalone hardware, or may be built into a programmable logic such as an FPGA. Various data required for executing this computer program and various data generated by executing this computer program are all stored in an internal memory such as ROM 103, RAM 102, or a storage medium such as a hard disk drive. The functions of the functional components of the image processing device 5 will be described in detail below.

[0041] The image acquisition unit 201 acquires C (C is an integer equal to or greater than 2) pre-specified fluorescence images of the sample S from the image acquisition device 3. These C fluorescence images are fluorescence images composed of N pixels that are generated by irradiating the sample S with excitation light of C wavelength bands and capturing the fluorescence generated from the sample S in response to the excitation light with the wavelength information acquisition optical system 13 removed from the optical path of the fluorescence. At this time, the number C of acquired fluorescence images (the number C of wavelength bands of excitation light irradiated to the sample S) is pre-specified to be equal to or greater than the maximum number of dyes that can be contained in the sample S. Note that the ratio of the excitation light intensities when the image acquisition device 3 acquires the C fluorescence images is set to be the same, or the image acquisition unit 201 relatively corrects the luminance values ​​of the C fluorescence images so that the fluorescence images are regarded as having the same excitation light intensities.

[0042] Furthermore, the image acquiring unit 201 acquires pre-specified C sets of separated fluorescence images of the sample S from the image acquiring device 3. These C sets of separated fluorescence images are sets of separated fluorescence images composed of N pixels that are generated by irradiating the sample S with excitation light of C wavelength bands, respectively, with the wavelength information acquisition optical system 13 supported on the optical path of the fluorescence, and then separating and imaging the fluorescence generated from the sample S in response to the irradiation into two components.

[0043] The wavelength information acquisition unit 202 estimates a centroid fluorescence wavelength indicating the center of gravity of the wavelength distribution of the fluorescence by calculating the ratio of the fluorescence intensity (brightness value) of one separated fluorescent image to the fluorescence intensity of the other separated fluorescent image for each of the C sets of separated fluorescent images. At this time, the wavelength information acquisition unit 202 calculates the average value of the fluorescence intensity of one separated fluorescent image to the average value of the fluorescence intensity of the other separated fluorescent image for a group of pixels clustered by the clustering unit 203 described below, and calculates the ratio of these average values. The wavelength information acquisition unit 202 acquires the estimated centroid fluorescence wavelength as wavelength information related to the fluorescence wavelength.

[0044] The clustering unit 203 performs clustering on the N pixels constituting the C fluorescent images, based on the C fluorescent images acquired by the image acquiring unit 201 and the wavelength information acquired by the wavelength information acquiring unit 202. Prior to the clustering process, the clustering unit 203 generates matrix data Y in which the fluorescent intensity values ​​of the N pixels constituting each of the C fluorescent images are arranged in parallel in a one-dimensional manner.

[0045] Next, the clustering unit 203 has a function (first clustering function) of clustering N pixels into C pixel groups based on distribution information of the fluorescence intensity for each excitation light in C wavelength bands. In detail, the clustering unit 203 clusters pixels having the same wavelength band of excitation light with the highest fluorescence intensity into the same pixel group. FIG. 5 shows an image of pixel groups clustered by the first clustering function of the clustering unit 203, and FIG. 6 shows the wavelength characteristics of the excitation light absorption rate of multiple dyes contained in the sample S. As shown in FIG. 5, assuming that the dyes contained in the sample S are three types, dye C1, dye C2, and dye C3, and six fluorescence images are obtained using excitation light in six types of wavelength bands, the clustering unit 203 clusters N pixels contained in the six fluorescence images GC1 to GC6 into six pixel groups PGr1 to PGr6. As shown in FIG. 6, generally, different kinds of dyes have wavelength characteristics of different absorptances, and the three kinds of dyes C1, C2, and C3 have wavelength characteristics CW1, CW2, and CW3 with different peak wavelengths. Therefore, in six kinds of wavelength bands EW1, EW2, EW3, EW4, EW5, and EW6 of excitation light, the dye with the highest absorptance is determined to be one of the three kinds of dyes C1, C2, and C3. For example, the dye C1 has the highest absorptance of excitation light in the wavelength band EW1, the dye C1 has the highest absorptance of excitation light in the wavelength band EW2, and the dye C2 has the highest absorptance of excitation light in the wavelength band EW3. Using this property, the clustering unit 203 can cluster N pixels into pixel groups in a range where the same dye is distributed by the first clustering function. However, the six pixel groups PGr1 to PGr6 clustered by the first clustering function do not correspond one-to-one to the three kinds of dyes C1, C2, and C3.

[0046] In addition, the clustering unit 203 has a function (second clustering function) of further clustering the C pixel groups clustered by the first clustering function into L pixel groups (L is an integer between 2 and N-1) based on the wavelength information. Here, the number L of pixel groups to be clustered is set in advance as a parameter stored in the image processing device 5 in correspondence with the number of types of dyes that may exist in the sample S. That is, for each of the C pixel groups clustered by the first clustering function, the clustering unit 203 specifies a centroid fluorescence wavelength estimated for the wavelength band of excitation light corresponding to the pixel group. More specifically, the clustering unit 203 acquires wavelength information from the wavelength information acquisition unit 202 for a pixel group clustered as having the highest absorptance in a certain wavelength band, and specifies the centroid fluorescence wavelength based on the acquired wavelength information. At this time, the wavelength information acquisition unit 202 acquires the wavelength information using the average value of the fluorescence intensity in the pixel groups of a set of separated fluorescent images obtained corresponding to that wavelength band. Furthermore, the clustering unit 203 determines the distance (closeness of values) between the centroid fluorescence wavelengths identified for each of the C pixel groups, thereby clustering the C pixel groups into L pixel groups. Then, the clustering unit 203 divides and regenerates matrix data Y, in which the fluorescence intensity values ​​of the pixels of the C fluorescent images are arranged in parallel in a one-dimensional manner, into cluster matrices for each of the L pixel groups.

[0047] Fig. 7 shows the distribution of the centroid fluorescence wavelengths identified by the clustering unit 203, and Fig. 8 shows an image of pixel groups clustered by the second clustering function of the clustering unit 203. In the example shown in Figs. 7 and 8, centroid fluorescence wavelengths FW1 to FW6 are identified for each of the six pixel groups PGr1 to PGr6 clustered by the first clustering function, and pixel group PGr1 and pixel group PGr2, which are close to each other in distance of the centroid fluorescence wavelengths, are grouped into a new pixel group PGr 01 Similarly, pixel group PGr3 and pixel group PGr4 are clustered into pixel group PGr 02 The pixel groups PGr5 and PGr6 are clustered into the pixel group PGr 03As a result, the pixels of the C fluorescent images can be divided into L pixel groups corresponding to the distribution of pixels assumed to be contained in the sample S. However, the number of divisions L by the second clustering function is set to be equal to or less than the number C of fluorescent images (the number C of wavelength bands of excitation light).

[0048] The statistical value calculation unit 204 obtains a mixing matrix A for generating K dye images showing the distribution of each of K dyes (K is an integer between 2 and C) from C fluorescent images based on L cluster matrices obtained for the sample S. Generally, according to a non-negative matrix factorization (NMF) calculation method, the relationship between matrix data Y, which is an observation matrix, and dye matrix data X, in which K dye images are arranged in parallel in a one-dimensional manner for each pixel, is expressed by the following formula using the mixing matrix A: Y=AX Here, Y is matrix data with C rows and N columns, A is matrix data with C rows and K columns, and X is matrix data with K rows and N columns. Conversely, once the value of the mixing matrix A is found, the pigment matrix data X is expressed as the inverse matrix A of the mixing matrix A. -1 and matrix data Y are used to obtain the following equation: X=A -1 Y (This process is called unmixing.)

[0049] Here, the statistical value calculation unit 204 regenerates the matrix data Y' by compressing the matrix data Y generated by the clustering unit 203 in units of pixel groups clustered by the clustering unit 203. In detail, the statistical value calculation unit 204 calculates a statistical value for each pixel group of the clustered cluster matrix for the fluorescence intensity of each row of the matrix data Y, and compresses the pixel group of each row into one pixel having the calculated statistical value. In this way, the statistical value calculation unit 204 regenerates the matrix data Y' which is matrix data of C rows and L columns. The statistical value calculation unit 204 may calculate an average value based on the integrated value of the fluorescence intensity, may calculate a mode value of the fluorescence intensity, or may calculate a median value of the fluorescence intensity as the statistical value.

[0050] Furthermore, the statistical value calculation unit 204 also calculates the following formula including the mixing matrix A for the reproduced matrix data Y' and the dye matrix data X' compressed in the same manner from the dye matrix data X: Y'=AX' Using the property that the above holds, the mixing matrix A is derived based on the matrix data Y'. FIG. 9 shows an image of the matrix data Y' regenerated by the statistical value calculation unit 204 and the corresponding dye matrix data X'. One square shown in FIG. 9 represents one element of the matrix data. In this way, the three pixel groups PGr 01 ~PGr 03 The pigment matrix data X and the matrix data Y are divided into the pixel group PGr 01 ~PGr 03 The data is compressed into three columns of dye matrix data X' and matrix data Y', with the statistical values ​​for each pixel being used as representative values.

[0051] The statistical value calculation unit 204 derives the mixing matrix A based on the matrix data Y' as follows. That is, the statistical value calculation unit 204 sets an initial value to the mixing matrix A, calculates the following loss function (loss value) Loss while sequentially changing the value of the mixing matrix A, and derives the mixing matrix A that reduces the value of the loss function Loss. Note that a regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree to which the regularization term is emphasized) may be added to this loss function.

number

[0052] As described above, the statistical value calculation unit 204 calculates a loss function for each of the L cluster matrices divided by the clustering unit 203 by referring to the statistical values ​​of the C matrix data Y', calculates a loss function Loss based on the sum of the L loss functions, and obtains a mixing matrix A based on the loss function Loss. At this time, the statistical value calculation unit 204 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values ​​a, b, and c of the statistical values ​​of the C matrix data Y', and then obtains the loss function Loss by calculating the sum of the corrected loss functions. Note that the statistical value calculation unit 204 may calculate the loss function for each of the L cluster matrices by correcting the row components of the difference value Y'-AX' for each wavelength band of the excitation light by dividing them by the C statistical values ​​corresponding to each wavelength band of the excitation light.

[0053] The above formula can also be generalized as follows. That is, the statistical value calculation unit 204 derives the mixing matrix A and the dye matrix data X' based on the matrix data Y' as follows. That is, the statistical value calculation unit 204 sets initial values ​​to the mixing matrix A and the dye matrix data X', calculates a loss function (loss value) Loss using the following formula while sequentially changing the values ​​of the mixing matrix A and the dye matrix data X', and derives the mixing matrix A and the dye matrix data X' that reduces the value of the loss function Loss. Note that a regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree to which the regularization term is emphasized) may be added to this loss function. Also, the calculation may be performed with a constraint that the mixing matrix A and the dye matrix data X' are non-negative values.

number

[0054] As described above, the statistical value calculation unit 204 calculates a loss function by referring to the statistical values ​​of the C matrix data Y′ for each of the L cluster matrices divided by the clustering unit 203, and calculates the L loss functions Los i The loss function Los is calculated based on the sum of the L cluster matrices, and the mixing matrix A is obtained based on the loss function Los. i may be calculated by correcting by dividing the row components of the difference values ​​Y'-AX' for each wavelength band of the excitation light by the C statistical values ​​corresponding to each wavelength band of the excitation light.

[0055] The image generating unit 205 obtains K dye images by unmixing C fluorescent images obtained from the sample S to be observed using the mixing matrix A derived by the statistical value calculating unit 204. Specifically, the image generating unit 205 obtains K dye images by applying the inverse matrix A of the mixing matrix A to the matrix data Y generated by the clustering unit 203 based on the C fluorescent images. -1 By applying the above formula, the dye matrix data X is calculated. Then, the image generation unit 205 regenerates K dye images from the dye matrix data X, and outputs the regenerated K dye images. The output destination at this time may be an output device of the image processing device 5, such as a display or a touch panel display, or may be an external device connected to the image processing device so as to be able to communicate data with the image processing device.

[0056] Next, a procedure of an observation process for the sample S using the dye image acquisition system 1 according to the present embodiment, that is, a flow of the dye image acquisition method according to the present embodiment, will be described. Fig. 10 is a flowchart showing the procedure of the observation process by the dye image acquisition system 1.

[0057] First, the image acquiring unit 201 of the image processing device 5 acquires C fluorescent images and C sets of separated fluorescent images of the sample S (step S1; image acquiring step). Next, the clustering unit 203 of the image processing device 5 generates matrix data Y in which N pixels of the C fluorescent images are arranged in parallel (step S2).

[0058] Furthermore, the clustering unit 203 of the image processing device 5 executes a first clustering function, and N pixels of the fluorescence image are clustered into C pixel groups using distribution information of the fluorescence intensity for each excitation wavelength band (step S3; clustering step). Next, the wavelength information acquisition unit 202 of the image processing device 5 acquires wavelength information indicating the centroid fluorescence wavelength for each of the C pixel groups by referring to the set of separated fluorescent images (step S4; wavelength information acquisition step). Thereafter, the clustering unit 203 of the image processing device 5 executes a second clustering function, and the C pixel groups are clustered into L pixel groups by determining the distance between the centroid fluorescence wavelengths identified for each of the C pixel groups based on the wavelength information (step S5; clustering step).

[0059] Next, the statistical value calculation unit 204 of the image processing device 5 calculates the statistical value of the L pixel groups, thereby regenerating matrix data Y' based on the matrix data Y generated by the clustering unit 203 (step S6; calculation step). Then, the statistical value calculation unit 204 of the image processing device 5 derives a mixing matrix A based on the matrix data Y' (step S7; image generation step). Furthermore, the image generation unit 205 of the image processing device 5 unmixes the matrix data Y generated based on the C fluorescent images of the sample S using the mixing matrix A, thereby regenerating K dye images (step S8; image generation step). Finally, the image generation unit 205 of the image processing device 5 outputs the regenerated K dye images (step S9). With the above, the observation process of the sample S is completed.

[0060] According to the dye image acquisition system 1 described above, C fluorescent images are acquired by capturing fluorescent images of the sample S using excitation light of different wavelength bands, N pixels of these C fluorescent images are clustered into L pixel groups based on the fluorescent intensity of each pixel, and L cluster matrices are generated in which the C fluorescent images are arranged for each of the L pixel groups. In addition, statistical values ​​of the fluorescent intensity of the pixel groups constituting the C fluorescent images are calculated for each of the L cluster matrices, and the C fluorescent images are unmixed using the statistical values ​​of each of the C fluorescent images to generate K dye images. This makes it possible to reduce the amount of calculation required for unmixing even if the type of excitation light used for observation or the number of fluorescent bands observed increases. In addition, the accuracy of separation of dye images can be improved by performing unmixing using the statistical values ​​of the fluorescent images for each cluster matrix. As a result, it is possible to improve the throughput when acquiring dye images while increasing the accuracy of the separated dye images.

[0061] In the present embodiment, in the clustering step, N pixels are clustered based on the distribution information of the fluorescence intensity for each of the C excitation lights. This makes it possible to cluster the pixels according to the fluorescence intensity for each excitation light, and by performing unmixing based on the clustering results, it is possible to improve the accuracy of separation of the dye images.

[0062] Furthermore, in this embodiment, a wavelength information acquisition step is further executed to acquire wavelength information related to the fluorescence wavelength for each pixel group clustered by the first clustering function, and in the clustering step, the C pixel groups are further clustered into L pixel groups based on the wavelength information corresponding to each pixel group. This makes it possible to cluster the pixels using the fluorescence wavelength corresponding to each pixel group of the fluorescence image, and by unmixing the fluorescence image using the clustering result, it is possible to improve the accuracy of separation of dye images for each dye that may be present in the sample S.

[0063] Furthermore, in this embodiment, the fluorescence from the sample S generated using excitation light in any one of the C wavelength bands is separated into two components via the wavelength information acquisition optical system 13, a set of separated fluorescence images capturing the fluorescence of each of the two separated components is acquired, and wavelength information is acquired based on the set of separated fluorescence images. In this case, the fluorescence from the sample S generated by irradiation with excitation light is separated with different wavelength characteristics, and wavelength information is acquired based on the set of separated fluorescence images capturing the separated fluorescence, so that the wavelength information can be analyzed with high accuracy. As a result, the accuracy of separation of the dye images can be improved.

[0064] In a conventional fluorescence imaging method for a sample containing multiple dyes, multiple excitation light filters (bandpass filters) each having multiple transmission wavelength bands are switched between, and the fluorescence generated from the sample is imaged through the fluorescence filter while switching between multiple fluorescence filters (bandpass filters) each having multiple transmission wavelength bands, thereby acquiring multiple fluorescence images. Therefore, switching of the excitation light filter and switching of the corresponding fluorescence filter are required. As a result, the time required for switching the filter and the time required for adjusting the position of the stage of the sample S to match the irradiation range of the excitation light of the sample S according to the filter switching are required, and therefore the time required for acquiring the fluorescence image tends to be long. In contrast, in this embodiment, a multi-bandpass filter is used as the light source side filter set 9a and the camera side filter set 9b, and filter switching is not required. The excitation light source 7 can also switch the wavelength band of the excitation light in the same field of view, so that the time required for acquiring the fluorescence image can be significantly reduced. In this embodiment, a multiband filter is used to capture a fluorescence image, so that each fluorescence image reflects a mixture of fluorescence from multiple fluorescence bands. However, by performing an unmixing process, the images can be separated into K dye images with high accuracy.

[0065] Furthermore, in the calculation step of this embodiment, the statistical values ​​calculated when compressing the cluster matrix are based on the integrated value, mode, or median value of the fluorescence intensities of the pixel groups. In this case, unmixing can be performed based on the overall tendency of the fluorescence intensities of the L pixel groups of the fluorescence image in the cluster matrix, and the accuracy of the generated dye image can be improved.

[0066] Furthermore, in the image generating step of this embodiment, a mixing matrix A is obtained using the statistics of C fluorescent images for each of L cluster matrices, and the C fluorescent images are unmixed using the mixing matrix A. In this way, the mixing matrix A is obtained based on the statistics of L pixel groups of the clustered fluorescent images, and unmixing can be performed using the mixing matrix A, thereby improving the accuracy of the generated dye image. Conventionally, the mixing matrix A was derived by calculation based on reference information (fluorescence spectrum, absorption spectrum, etc.) of each dye. In this embodiment, even if such reference information is unknown, the mixing matrix A can be estimated from the matrix data Y obtained from the fluorescent image.

[0067] Furthermore, in the image generation step of this embodiment, a loss function is calculated based on the statistical values ​​of C fluorescent images for each of L cluster matrices using the NMF calculation method, and a mixing matrix A is obtained based on the loss function Los, which is the sum of the loss functions for each of L cluster matrices. In this way, a loss function is calculated based on the statistical values ​​of L pixel groups of the clustered fluorescent image, a mixing matrix A is obtained based on the sum of these loss functions, and unmixing can be performed using the mixing matrix A. This makes it possible to further improve the accuracy of the generated dye image even if there is a difference in the size of the distribution area of ​​each dye in the fluorescent image of the sample S. That is, when a loss function is calculated for a fluorescent image without clustering, emphasis is placed on the separation accuracy of dyes with relatively large distribution areas, which results in a decrease in separation accuracy of dyes with relatively small distribution areas, but in this embodiment, the separation accuracy of multiple dyes can be improved uniformly.

[0068] In addition, in the image generation step of this embodiment, the loss function for each of the L cluster matrices may be corrected by dividing it by coefficients a, b, and c based on the statistical value of the cluster matrix, and the mixing matrix A may be calculated based on the sum of the corrected loss functions. With this configuration, the loss function is calculated based on the statistical value of the L pixel groups of the clustered fluorescent image, and each loss function is corrected based on the statistical value when the sum of the loss functions is calculated. This makes it possible to generate a dye image with high accuracy even if there is a difference in the fluorescence intensity of each dye in the fluorescent image of the sample S. That is, when the loss function Los is calculated without correcting the L loss functions, emphasis is placed on the separation accuracy of dyes with relatively large fluorescence intensity, which results in a decrease in separation accuracy of dyes with relatively small fluorescence intensity. However, in this embodiment, the separation accuracy of multiple dyes can be improved uniformly.

[0069] In addition, in this embodiment, the calculation time for generating a dye image can be significantly reduced by performing clustering and unmixing using matrix data Y' based on statistical values. For example, when a fluorescent image including 660 tiles of 2048×2048 images is processed and the number of bands C of the excitation light is set to "10", the calculation time is about 17,000 seconds when unmixing is performed using matrix data Y as is without clustering. In contrast, according to the method of this embodiment, the number of processed data is significantly reduced, and the calculation time is significantly reduced to about 0.06 seconds when the number of clusters L is set to "6". In addition, according to the method of this embodiment, the accuracy of the dye image can be maintained by unmixing using statistical values.

[0070] 11 shows an example of a dye image generated by the dye image acquisition system 1 according to this embodiment. As described above, according to this embodiment, the distribution of rhodamine, which is a dye contained in the sample S, can be obtained with high accuracy. It was found that a fluorescent image of equal accuracy can be obtained compared to a conventional fluorescent imaging method using a bandpass filter.

[0071] Various embodiments of the present invention have been described above, but the present invention is not limited to the above-described embodiments, and may be modified or applied to other things without departing from the spirit of the invention as described in each claim.

[0072] For example, in the image processing device 5 of this embodiment, the number L of pixel groups to be clustered and the number K of dye images are set in advance as parameters according to the number of dyes contained in the sample S, but the image processing device 5 may repeatedly generate dye images by sequentially changing the parameters L and K. For example, if unmixing is performed with the number of clusters L=C-1 and the number of dyes K=C-1 to generate dye images, and the accuracy of the obtained dye images for each dye is poor, the number of clusters L and the number of dyes K may be sequentially changed to C-2, C-3, ... and unmixing may be repeated.

[0073] Furthermore, the dye image acquisition system 1 of this embodiment may generate a dye image by performing unmixing using the C sets of separated fluorescent images obtained by the first camera 15a and the second camera 15b as they are. In this case, the number of fluorescent images to be unmixed is 2×C. Furthermore, the dye image acquisition system 1 may switch between M bandpass filters that transmit one fluorescent band as the camera-side filter set 9b, and perform unmixing on the M×C fluorescent images obtained as a result.

[0074] Furthermore, the dye image acquisition system 1 may include a plurality of excitation light sources 7 that simultaneously irradiate excitation light of a plurality of wavelength bands onto the sample S, and may acquire C fluorescence images while irradiating excitation light of C types of wavelength distributions by changing the intensity ratio between the excitation light of the plurality of wavelength bands. In this case as well, dye images for each of the plurality of dyes can be obtained with high accuracy.

[0075] When generating matrix data Y before clustering, the image processing device 5 according to this embodiment may generate the data in which N pixels constituting the image are arranged in the row direction according to a specified rule, or may generate the data in which N pixels constituting the image are arranged in the row direction according to a random rule. However, the data of the fluorescence image of C rows constituting one piece of matrix data Y is set as data in which N pixels are arranged according to the same rule. Even if matrix data Y generated by random arrangement is used, the same matrix data Y' can be regenerated by clustering.

[0076] Furthermore, when generating the matrix data Y before clustering, the image processing device 5 according to this embodiment may generate the matrix data Y by excluding background pixels (pixels in which no dye is present) contained in the fluorescence image.

[0077] As a method for clustering a group of pixels in the image processing device 5, a method using machine learning such as the K-means method, a method using deep learning, or the like may be adopted.

[0078] In addition, as a method of clustering a pixel group in the image processing device 5, in addition to the K-means method, a method using machine learning such as a decision tree, a support vector machine, KNN (K nearest neighbor), a self-organizing map, a spectral clustering, a Gaussian mixture model, DBSCAN, affinity propagation, MeanShift, Ward, agglomerative clustering, OPTICS, or BIRCH, or a method using deep learning may be adopted. In addition, preprocessing may be performed on the matrix data Y before applying clustering. For example, the dimension of data of each pixel C may be reduced by phasor analysis, principal component analysis, singular value decomposition, independent component analysis, linear discriminant analysis, t-SNE, UMAP, or other machine learning.

[0079] In the image acquisition device 3 of this embodiment, in addition to the inclined dichroic mirror having wavelength characteristics in which the transmittance changes linearly with respect to the wavelength as described above, a single-band dichroic mirror that transmits one wavelength band, or a multi-band dichroic mirror that transmits multiple wavelength bands may be used as the wavelength information acquisition optical system 13. In this case, the above-mentioned multi-bandpass filter or a single-bandpass filter that transmits one wavelength band may be used as the camera-side filter set 9b.

[0080] Here, when a single bandpass filter is adopted as the camera side filter set 9b, and an inclined dichroic mirror having a transmittance characteristic t(λ)=a1λ+b1 with respect to the wavelength λ is adopted as the wavelength information acquisition optical system 13, the wavelength information acquisition unit 202 of the image processing device 5 calculates the transmittance t(λ)=a1λ+b1 using the following formula:

number

[0081] The wavelength information acquisition optical system 13 is not limited to a dichroic mirror, and may be a filter set having similar wavelength characteristics, or a beam splitter (such as a polarizing beam splitter) that splits the fluorescence. Filters having different wavelength characteristics may be switched and used to capture images multiple times with one camera to obtain a set of separated fluorescence images. The fluorescence of the two components separated by the wavelength information acquisition optical system 13 may be captured by one camera with a divided field of view.

[0082] In addition, the wavelength information acquisition unit 202 of the image acquisition device 3 of this embodiment may process a fluorescent image acquired using a camera capable of detecting at least two or more fluorescent wavelengths in order to acquire wavelength information related to the fluorescent wavelength. Examples of such cameras include a color sensor (color camera) and a multiband sensor (multiband camera). For example, when a color sensor is used, the wavelength information acquisition unit 202 can calculate and acquire information related to the fluorescent wavelength by comparing three intensity values ​​of R pixels, G pixels, and B pixels acquired from the color sensor. In addition, in the case of a multiband sensor, the wavelength information acquisition unit 202 can calculate and acquire information related to the fluorescent wavelength by comparing intensity values ​​for different wavelengths acquired from the multiband sensor. In this case, the wavelength information is also acquired based on a fluorescent image capturing the fluorescent light, so that the wavelength information can be analyzed with high accuracy. As a result, the accuracy of separation of the dye image can be improved.

[0083] In the dye image acquisition method of the present disclosure, it is preferable that the clustering step clusters N pixels based on distribution information of intensity values ​​for each of the C excitation lights. In the dye image acquisition device of the present disclosure, it is preferable that the image processing device clusters N pixels based on distribution information of intensity values ​​for each of the C excitation lights. This makes it possible to cluster pixels according to the fluorescence intensity for each excitation light, and by unmixing based on the clustering results, it is possible to improve the accuracy of separation of dye images.

[0084] In addition, the dye image acquisition method of the present disclosure preferably further comprises a wavelength information acquisition step of acquiring wavelength information related to the fluorescence wavelength corresponding to each pixel of the fluorescence image, and in the clustering step, N pixels are clustered based on the wavelength information corresponding to each pixel. In addition, in the dye image acquisition device of the present disclosure, it is also preferable that the image processing device further acquires wavelength information related to the fluorescence wavelength corresponding to each pixel of the fluorescence image, and clusters N pixels based on the wavelength information corresponding to each pixel. In this way, pixels can be clustered using the wavelength information corresponding to each pixel of the fluorescence image, and by unmixing based on the clustering result, the accuracy of separation of the dye image can be improved.

[0085] In addition, in the dye image acquisition method of the present disclosure, in the wavelength information acquisition step, it is also preferable that the sample is irradiated with excitation light having one of C wavelength distributions, the fluorescence from the sample is separated through a wavelength information acquisition optical system that separates the fluorescence with different wavelength characteristics, the separated fluorescence is imaged to obtain a plurality of separated fluorescence images, and the wavelength information is acquired based on the plurality of separated fluorescence images. In addition, in the dye image acquisition device of the present disclosure, it is also preferable that the image acquisition device irradiates the sample with excitation light having one of C wavelength distributions, the fluorescence from the sample is separated through a wavelength information acquisition optical system that separates the fluorescence with different wavelength characteristics, the separated fluorescence is imaged to obtain a plurality of separated fluorescence images, and the image processing device acquires wavelength information based on the plurality of separated fluorescence images. In this case, the fluorescence from the sample generated by the irradiation of the excitation light is separated with different wavelength characteristics, and the wavelength information is acquired based on the plurality of separated fluorescence images obtained by imaging the separated fluorescence, so that the wavelength information can be analyzed with high accuracy. As a result, the accuracy of separation of the dye image can be improved.

[0086] In addition, in the dye image acquisition method of the present disclosure, it is preferable that in the wavelength information acquisition step, the excitation light having one of the wavelength distributions of C wavelengths is irradiated onto the sample, the fluorescence from the sample is captured by a camera capable of detecting at least two or more fluorescent wavelengths to obtain a fluorescence image, and the wavelength information is acquired based on the fluorescence image. In addition, in the dye image acquisition device of the present disclosure, it is preferable that the image acquisition device irradiates the sample with the excitation light having one of the wavelength distributions of C wavelengths, captures the fluorescence from the sample by a camera capable of detecting at least two or more fluorescent wavelengths to obtain a fluorescence image, and acquires the wavelength information based on the fluorescence image. In this case, too, the wavelength information is acquired based on a fluorescent image capturing the fluorescence from the sample generated by the irradiation of the excitation light, so that the wavelength information can be analyzed with high accuracy. As a result, the accuracy of separation of the dye image can be improved.

[0087] Furthermore, in the dye image acquisition method of the present disclosure, it is also preferable that in the calculation step, the statistical value is calculated based on an integrated value, a mode, or an intermediate value of the intensity values ​​of the pixel groups. Furthermore, in the dye image acquisition device of the present disclosure, it is also preferable that the image processing device calculates the statistical value based on an integrated value, a mode, or an intermediate value of the intensity values ​​of the pixel groups. In this case, unmixing can be performed based on the overall tendency of the intensity values ​​of the L pixel groups of the clustered fluorescent image, and the accuracy of the generated dye image can be improved.

[0088] In the dye image acquisition method or dye image acquisition device of the present disclosure, outliers or some pixels may be excluded before calculating the statistical values. Furthermore, each pixel may be included in different clusters in a duplicated manner. In this case, the pixel is used in the calculation of the statistical values ​​in each cluster.

[0089] Furthermore, in the dye image acquisition method of the present disclosure, it is also preferable that in the image generation step, a mixing matrix is ​​obtained using statistics of C fluorescent images for each of L cluster matrices, and unmixing is performed using the mixing matrix. Furthermore, in the dye image acquisition device of the present disclosure, it is also preferable that the image processing device obtains a mixing matrix using statistics of C fluorescent images for each of L cluster matrices, and unmixing is performed using the mixing matrix. In this way, a mixing matrix is ​​obtained based on statistics of L pixel groups of the clustered fluorescent image, and unmixing can be performed using the mixing matrix, thereby improving the accuracy of the generated dye image.

[0090] Furthermore, in the dye image acquisition method of the present disclosure, it is also preferable that in the image generation step, a loss value is calculated based on the statistical values ​​of C fluorescent images for each of L cluster matrices using non-negative matrix factorization, and a mixing matrix is ​​obtained based on the sum of the loss values. Furthermore, in the dye image acquisition device of the present disclosure, it is also preferable that the image processing device calculates a loss value based on the statistical values ​​of C fluorescent images for each of L cluster matrices using non-negative matrix factorization, and a mixing matrix is ​​obtained based on the sum of the loss values. In this way, a loss value is calculated based on the statistical values ​​of L pixel groups of the clustered fluorescent image, a mixing matrix is ​​obtained based on the sum of the loss values, and unmixing can be performed using the mixing matrix. This makes it possible to improve the accuracy of the generated dye image even if there is a difference in the size of the distribution area of ​​each dye in the fluorescent image of the sample.

[0091] In addition, in the dye image acquisition method of the present disclosure, it is also preferable that in the image generation step, the loss value for each of the L cluster matrices is corrected based on the statistical value, and a mixing matrix is ​​obtained based on the sum of the corrected loss values. In addition, in the dye image acquisition device of the present disclosure, it is also preferable that the image processing device corrects the loss value for each of the L cluster matrices based on the statistical value, and a mixing matrix is ​​obtained based on the sum of the corrected loss values. With this configuration, the loss value is calculated based on the statistical value of the L pixel groups of the clustered fluorescent image, and each loss value is corrected based on the statistical value when the sum of the loss values ​​is obtained. As a result, even if there is a difference in the fluorescent intensity of each dye in the fluorescent image of the sample, a dye image can be generated with high accuracy. [Explanation of symbols]

[0092] 1...dye image acquisition system, 3...image acquisition device, 5...image processing device, 7...excitation light source, 9a...light source side filter set, 9b...camera side filter set, 11...dichroic mirror, 15a, 15b...camera, 13...wavelength information acquisition optical system, 201...image acquisition section, 202...wavelength information acquisition section, 203...clustering section, 204...statistical value calculation section, 205...image generation section, C1, C2, C3...dyes, GC1 to GC6...fluorescence images, PGr 01 ~PGr 03 ,PGr1~PGr6...pixel groups, S...sample.

Claims

1. an excitation light source capable of switching between and irradiating excitation light of multiple wavelength bands; a fluorescence image acquisition unit that acquires a fluorescence image by capturing fluorescence generated by irradiating a sample with the excitation light; a wavelength information acquisition optical system that can be arranged on an optical path of the fluorescence and that acquires wavelength information of the fluorescence; a clustering unit that generates clustering data by clustering each pixel of the fluorescence image based on the wavelength information of the fluorescence; a dye image generating unit that performs unmixing on the fluorescent image using the clustering data to generate a dye image that indicates the distribution of dyes in the sample, the wavelength information acquisition optical system is disposed on an optical path of the fluorescence when acquiring wavelength information of the fluorescence, and is removed from the optical path of the fluorescence when acquiring the fluorescence image. Dye image acquisition device.

2. 2. The dye image acquisition device according to claim 1, wherein the wavelength information acquisition optical system includes a filter having transmittance characteristics in which the transmittance changes linearly with wavelength.

3. 3. The dye image obtaining apparatus according to claim 1, further comprising a multi-bandpass filter, disposed on the optical path of the excitation light, for transmitting light in a plurality of wavelength bands.

4. 3. The dye image obtaining apparatus according to claim 1, further comprising a multi-bandpass filter, disposed on an optical path of the fluorescence, for transmitting light in a plurality of wavelength bands.

5. 5. The dye image acquisition device according to claim 4, wherein the multi-bandpass filter is set in accordance with wavelength bands of fluorescence generated by dyes that may be contained in the sample.

6. The dye image acquisition device according to claim 1 , wherein the wavelength information of the fluorescence is a centroid fluorescence wavelength of the fluorescence.

7. an excitation light irradiation step of switching between excitation lights of a plurality of wavelength bands and irradiating the excitation light; a fluorescence image acquisition step of acquiring a fluorescence image by capturing fluorescence generated by irradiating the sample with the excitation light; a wavelength information acquisition step of acquiring wavelength information of the fluorescence using a wavelength information acquisition optical system; a clustering step of clustering each pixel of the fluorescence image based on the wavelength information of the fluorescence to generate clustering data; a dye image generation step of performing unmixing on the fluorescence image using the clustering data to generate a dye image showing the distribution of dyes in the sample, In the wavelength information acquisition step, the wavelength information acquisition optical system is disposed on an optical path of the fluorescence, In the fluorescence image acquisition step, the wavelength information acquisition optical system is removed from the optical path of the fluorescence. Dye image acquisition method.

8. 8. The dye image acquisition method according to claim 7, wherein the wavelength information acquisition optical system includes a filter having transmittance characteristics in which the transmittance changes linearly with wavelength.

9. 9. The dye image acquisition method according to claim 7, wherein in the excitation light irradiation step, the excitation light is irradiated onto the sample after passing through a multi-bandpass filter that is provided on the optical path of the excitation light and transmits light of multiple wavelength bands.

10. 9. The dye image acquisition method according to claim 7, wherein the fluorescence image acquisition step captures the fluorescence that has passed through a multi-bandpass filter that is provided on an optical path of the fluorescence and transmits light in a plurality of wavelength bands.

11. The dye image acquisition method according to claim 10 , wherein the multi-bandpass filter is set in accordance with wavelength bands of fluorescence generated by dyes that may be contained in the sample.

12. The dye image acquisition method according to claim 7 or 8, wherein the wavelength information of the fluorescence is a centroid fluorescence wavelength of the fluorescence.

Images