Separated image acquisition method, separated image acquisition device, and separated image acquisition program

Abstract

To improve throughput in acquiring separated images by unmixing.SOLUTION: A fluorescent dye image acquisition system 1 includes: an excitation light source 7 that irradiates a sample S with excitation light having a plurality of wavelength distributions; a camera 15 for acquiring a target fluorescence image under a plurality of optical states with different wavelength characteristics via a camera-side filter set 9b with respect to a plurality of fluorescence lights generated from the sample S by each of the excitation light having a plurality of wavelengths; and an image processing device 5. The image processing device 5 stores a plurality of mixing matrices for obtaining fluorescent dye images from which fluorescence light is separated, and generates the fluorescence dye image for each of the plurality of fluorescent lights based on a target fluorescence image acquired in one optical state by the camera 15 and on the selected mixing matrix.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the embodiment relates to a separated image acquiring method, a separated image acquiring device, and a separated image acquiring program. [Background technology]

[0002] Conventionally, a multiple staining method has been used in which a sample such as a biological tissue is simultaneously stained with a plurality of substances in the sample. In order to observe the substances in the sample that have been multiple stained, the sample is irradiated with excitation light to obtain a fluorescent image. For example, Non-Patent Document 1 below discloses the application of a nonnegative matrix factorization (NMF) method to blind unmix a fluorescent image in which fluorescence in a plurality of wavelength ranges is observed to obtain a separated image for each substance in the sample. In addition, Non-Patent Document 2 below discloses a method of unmixing a fluorescent image by clustering the fluorescent image, extracting the maximum value of the fluorescent intensity for each clustered pixel group, and generating a separated image based on the maximum value. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Binjie Qin et al., “Target / Background ClassificationRegularized Nonnegative Matrix Factorization for Fluorescence Unmixing”, IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL.65, NO.4, APRIL2016 [Non-Patent Document 2] Tristan D. McRae et al., “Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning”, PLOSONE, December 2,2019 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional methods described above, the calculation time for unmixing increases depending on the type of excitation light or the number of fluorescent dyes to be observed, which tends to result in a decrease in throughput.

[0005] Therefore, one aspect of the embodiments has been made in consideration of such problems, and has an objective of providing a separated image acquisition method, a separated image acquisition device, and a separated image acquisition program that are capable of improving throughput when acquiring separated images by unmixing. [Means for solving the problem]

[0006] A separation image acquisition method according to a first aspect of an embodiment includes a selection step of selecting desired image separation information from a storage unit that stores multiple pieces of image separation information for obtaining a fluorescence separation image by separating a fluorescence image, the image separation information being acquired based on multiple reference fluorescence images in multiple optical states having different wavelength characteristics; an irradiation step of irradiating a sample with each of the multiple wavelengths of excitation light; an acquisition step of acquiring a target fluorescence image in at least one optical state of the multiple optical states for each of the multiple fluorescences generated from the sample by each of the multiple wavelengths of excitation light, via a fluorescence filter unit having multiple reflection wavelength ranges and multiple transmission wavelength ranges; and a generation step of generating a fluorescence separation image for each of the multiple fluorescences based on the target fluorescence image acquired in the acquisition step and the image separation information selected in the selection step.

[0007] Alternatively, a separation image acquisition device according to a second aspect of the embodiment includes a memory unit that stores image separation information for obtaining a fluorescence separation image by separating a fluorescence image, the fluorescence image being acquired based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions; an image acquisition device that acquires a target fluorescence image in at least one of the plurality of optical states for each of a plurality of fluorescences generated from the sample by each of the excitation light having the plurality of wavelengths, via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and an image processing device that processes the fluorescence images, wherein the image processing device selects desired image separation information from the plurality of image separation information stored in the memory unit, and generates a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired by the image acquisition device and the selected image separation information.

[0008] Alternatively, a separation image acquisition program according to a third aspect of the embodiment is a separation image acquisition program for generating a fluorescence separation image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, which are acquired via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges for each of a plurality of fluorescences generated from the sample by each of the excitation light of a plurality of wavelengths by irradiating the sample with each of the excitation light of a plurality of wavelength distributions, and causes a computer to execute a storage process of storing a plurality of pieces of image separation information for obtaining a fluorescence separation image obtained by separating the fluorescence, which has been acquired based on the plurality of reference fluorescence images, a selection process of selecting desired image separation information from the plurality of stored image separation information, and a generation process of generating a fluorescence separation image based on the selected image separation information and a target fluorescence image acquired in at least one optical state out of the plurality of optical states for each of the plurality of fluorescences.

[0009] According to the first, second, or third aspect, desired image separation information is selected from a plurality of image separation information acquired based on a plurality of reference fluorescent images, a target fluorescent image is acquired by capturing a fluorescent image of the sample through a fluorescent filter unit in the same optical state as one of the plurality of reference fluorescent images using excitation light with a different wavelength distribution, and a fluorescent separation image is generated using the acquired target fluorescent image and the selected image separation information. As a result, even when the type of excitation light used for observation or the number of fluorescent dyes to be observed is large, unmixing can be performed by utilizing image separation information acquired in the past, and the amount of calculation required for unmixing can be reduced. As a result, it is possible to improve the throughput when acquiring separated images by unmixing. Effect of the Invention

[0010] According to one aspect of the present invention, it is possible to improve the throughput when obtaining separated images by unmixing. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a fluorescent dye image acquisition system 1 according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing a configuration of an image acquisition device 3 in FIG. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of the image processing device 5 in FIG. 1. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of the image processing device 5 of FIG. [Diagram 5] FIG. 5 is a diagram showing an image of a group of pixels clustered by a first clustering function of matrix acquisition unit 203 in FIG. 4. [Figure 6] 2 is a graph showing wavelength characteristics of the absorptance of excitation light for multiple fluorescent dyes contained in sample S. [Figure 7] 5 is a graph showing the distribution of centroid fluorescence wavelengths identified by the matrix acquisition unit 203 of FIG. 4. [Figure 8]FIG. 5 is a diagram showing an image of a group of pixels clustered by the second clustering function of matrix acquisition unit 203 in FIG. 4. [Figure 9] 5 is a diagram showing an image of matrix data Y' and fluorescent dye matrix data X' regenerated by the matrix acquisition unit 203 in FIG. 4. FIG. [Figure 10] 4 is a flowchart showing the procedure of a separated image acquisition method according to the embodiment. [Figure 11] 1 is a diagram showing an example of a fluorescent dye image generated based on a target fluorescent dye image by the fluorescent dye image acquisition system 1 according to the present embodiment. FIG. [Figure 12] FIG. 2 is a diagram showing an example of a fluorescent dye image generated based on a target fluorescent dye image without mixing matrix correction by the fluorescent dye image acquisition system 1 according to the present embodiment. [Figure 13] FIG. 13 is a block diagram showing a functional configuration of an image processing device 5A according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description, the same elements or elements having the same functions will be denoted by the same reference numerals, and duplicated description will be omitted.

[0013] FIG. 1 is a schematic diagram of a fluorescent dye image acquisition system 1, which is a separation image acquisition device according to an embodiment. The fluorescent dye image acquisition system 1 is a device for generating a fluorescent dye image (fluorescence separation image) for identifying the distribution of a fluorescent dye in a sample such as a biological tissue to be observed. The image generated by the fluorescent dye image acquisition system 1 is used for the purpose of developing medicines and examining treatment methods through the analysis of the image. For this reason, the fluorescent dye image acquisition system 1 is required to generate an image capable of identifying the distribution of many substances (fluorochromes) contained in a sample quantitatively with high throughput. The fluorescent dye image acquisition system 1 includes an image acquisition device 3 that irradiates the sample S with excitation light and acquires an image of the fluorescence generated in response to the irradiation, and an image processing device 5 that processes the image acquired by the image acquisition device 3. The image acquisition device 3 and the image processing device 5 may be configured to be capable of transmitting and receiving image data between them using wired or wireless communication, or may be configured to be capable of inputting and outputting image data via a recording medium.

[0014] Figure 2 is a perspective view showing the configuration of the image acquisition device 3 in Figure 1. In Figure 2, the optical path of the excitation light is indicated by a dotted line with an arrow, and the optical path of the fluorescence is indicated by a solid line with an arrow. The image acquisition device 3 is configured to include an excitation light source (illumination device) 7, a light source side filter set 9a, a dichroic mirror 11, a camera side filter set (fluorescence filter section) 9b, a wavelength information acquisition optical system (optical filter) 13, and a camera (image acquisition device) 15.

[0015] The excitation light source 7 is a light source capable of switching between and irradiating excitation light of a plurality of wavelength bands (wavelength distributions), and is, for example, an LED (Light Emitting Diode) light source, a light source consisting of a plurality of monochromatic laser light sources, or a light source combining a white light source and a wavelength selection optical element. The light source side filter set 9a is a multi-bandpass filter provided on the optical path of the excitation light from the excitation light source 7 and having a property of transmitting light of a plurality of predetermined wavelength bands. The transmission wavelength band of this light source side filter set 9a is set according to the plurality of wavelength bands of the excitation light that can be used. The dichroic mirror 11 is an optical member provided between the light source side filter set 9a and the sample S and having a property of reflecting the excitation light toward the sample S and transmitting the fluorescence emitted from the sample S in response to the reflection. The camera side filter set 9b is a multi-bandpass filter provided on the optical path of the fluorescence transmitted by the dichroic mirror 11 and having a property of transmitting light of a plurality of predetermined wavelength bands. The transmission wavelength band of this camera side filter set 9b is set according to the wavelength band of the fluorescence generated in the fluorescent dye that can be contained in the sample S to be observed. In detail, the camera-side filter set 9b has, as its wavelength characteristics, transmission wavelength bands (transmission wavelength ranges) corresponding to a plurality of fluorescence wavelength bands, and reflection wavelength bands (reflection wavelength ranges) between these plurality of transmission wavelength bands.

[0016] The wavelength information acquisition optical system 13 is an optical system that is detachably supported on the optical path of the fluorescence transmitted by the camera-side filter set 9b and that acquires wavelength information of the fluorescence. That is, the wavelength information acquisition optical system 13 is provided so as to be switchable between two states: a state in which the wavelength information acquisition optical system 13 is disposed on the optical path of the fluorescence from the sample S (first optical state) and a state in which the wavelength information acquisition optical system 13 is removed from the optical path of the fluorescence (second optical state). Note that the wavelength information acquisition optical system 13 may be, for example, a fluorescence filter set including two or more fluorescence filters having different wavelength characteristics, as long as it can realize a plurality of optical states having different wavelength characteristics. In the first optical state, the wavelength information acquisition optical system 13 transmits the fluorescence generated in the sample S and transmitted through the dichroic mirror 11 and the camera-side filter set 9b toward the camera 15 with a predetermined wavelength characteristic. In the second optical state different from the first optical state, the wavelength information acquisition optical system 13 causes the fluorescence transmitted through the dichroic mirror 11 and the camera-side filter set 9b to be incident on the camera 15 in the original optical state (without transmitting through the wavelength information acquisition optical system 13). For example, a dichroic mirror (also called a gradient filter) having wavelength characteristics of transmittance such that the transmittance increases linearly as the wavelength increases is used as the wavelength information acquisition optical system 13. Alternatively, a dichroic mirror (gradient filter) having wavelength characteristics of transmittance such that the transmittance decreases linearly as the wavelength decreases may be used as the wavelength information acquisition optical system 13. The wavelength information acquisition optical system 13 using such a dichroic mirror can cause the fluorescence to enter the camera 15 with two different wavelength transmission characteristics.

[0017] The camera 15 is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer equal to or greater than 2, for example, 2048×2048), and is a camera that captures the fluorescence transmitted through the wavelength information acquisition optical system 13 to acquire a first fluorescence image when the wavelength information acquisition optical system 13 is switched onto the optical path of the fluorescence (in the first optical state). In addition, the camera 15 captures the fluorescence that does not transmit through the wavelength information acquisition optical system 13 to acquire a second fluorescence image when the wavelength information acquisition optical system 13 is removed from the optical path of the fluorescence (in the second optical state). The camera 15 acquires a first fluorescence image and a second fluorescence image for each of the multiple fluorescences generated from the sample S by each of the multiple excitation light beams in the multiple wavelength bands irradiated by the excitation light source 7. The camera 15 outputs the acquired first fluorescence image and second fluorescence image to the image processing device 5 using communication or via a recording medium.

[0018] Next, the configuration of the image processing device 5 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 5, and Fig. 4 is a block diagram showing the functional configuration of the image processing device 5.

[0019] 3, the image processing device 5 is physically a computer or the like including a processor such as a CPU (Central Processing Unit) 101, a recording medium such as a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, each of which is electrically connected. Note that the image processing device 5 may include, as input / output devices, a display, a keyboard, a mouse, a touch panel display, etc., or may include a data recording device such as a hard disk drive or a semiconductor memory. The image processing device 5 may also be composed of multiple computers.

[0020] As shown in FIG. 4, the image processing device 5 includes, as functional components, an information search unit (selection unit) 201, an image acquisition unit 202, a matrix acquisition unit 203, a matrix correction unit 204, an image generation unit 205, and an image separation information storage unit (storage unit) 206. The information search unit 201 and the image separation information storage unit 206 may be external devices connected to the image processing device 5. Each functional unit of the image processing device 5 shown in FIG. 4 is realized by reading a program (a separated image acquisition program according to the embodiment) onto hardware such as the CPU 101 and the RAM 102, thereby operating the communication module 104 and the input / output module 106 under the control of the CPU 101, and reading and writing data in the RAM 102. The CPU 101 of the image processing device 5 executes this computer program to cause each functional unit of FIG. 4 to function, and sequentially executes processes corresponding to a separated image acquisition method described later.

[0021] The CPU 101 may be a standalone hardware, or may be implemented in a programmable logic such as an FPGA, like a software processor. The RAM and ROM may also be standalone hardware, or may be built into a programmable logic such as an FPGA. All of the various data required for the execution of this computer program and the various data generated by the execution of this computer program are stored in built-in memories such as the ROM 103 and RAM 102, or in storage media such as a hard disk drive. The functions of the functional components of the image processing device 5 will be described in detail below.

[0022] The information search unit 201 searches (selects) a desired mixing matrix from the data stored in the image separation information storage unit 206 based on the image acquisition conditions when the first or second fluorescent image of the sample S is acquired, information (estimation processing information) on the estimation processing of the mixing matrix (image separation information) based on the first and second fluorescent images, and storage setting information when the mixing matrix is ​​stored in the image separation information storage unit 206. The information search unit 201 may set the image acquisition conditions, estimation processing information, storage setting information, etc. used for the search based on information input by the user via the input / output module 106 of the image processing device 5, based on information transmitted from an external device such as the image acquisition device 3, or based on information referenced from the RAM 102 or ROM 103 of the image processing device 5.

[0023] The image acquisition conditions include the type of sample S, barcode information attached to the sample S, type of fluorescent dye contained in the sample S, date and time of staining of the sample S, date and time of measurement of the sample S, type of excitation light, intensity of each excitation light, type or number of filters (presence or absence of a gradient filter, etc.), type of camera, type or magnification of objective lens, exposure time of each excitation light, environmental temperature, data of the acquired fluorescent image itself, etc. The estimation processing information includes version information of the software used in the estimation processing, algorithm, parameters or options used in the estimation processing, time required for the estimation processing, etc. The saved setting information includes the saved user name, project name, assay name, mixing matrix name, tag information (freely added keywords, SNS hashtags), etc.

[0024] In the image separation information storage unit 206, a mixing matrix acquired by the matrix acquisition unit 203 described later is stored in association with the image acquisition conditions when acquiring a plurality of fluorescent images (for example, a first fluorescent image and a second fluorescent image) on which the mixing matrix is ​​based, the estimated processing information related to the estimated processing used when acquiring the mixing matrix, and the storage setting information when the mixing matrix is ​​saved. The information search unit 201 searches (selects) a mixing matrix associated with information that matches or corresponds (similar) to a search key including the image acquisition conditions, the estimated processing information, or the storage setting information set by a user, an external device, or the like. Then, the information search unit 201 transfers the search result of the mixing matrix to the image acquisition unit 202 and the matrix acquisition unit 203. At this time, the information search unit 201 searches for a mixing matrix using at least one item of the above-mentioned image acquisition conditions, the estimated processing information, and the storage setting information as a search key. The information search unit 201 searches for a mixing matrix using a search key that includes at least one item of the image acquisition condition.

[0025] If the search process by the information search unit 201 does not find a mixing matrix that matches or corresponds (similar) to the search key such as the image acquisition conditions, the image acquisition unit 202 acquires a first fluorescence image (first reference fluorescence image) and a second fluorescence image (second reference fluorescence image) from the image acquisition device 3 in order to newly acquire a mixing matrix.

[0026] Specifically, if the information retrieval unit 201 cannot find a mixing matrix, the image acquisition unit 202 acquires pre-specified C (C is an integer equal to or greater than 2) first reference fluorescence images of the sample S from the image acquisition device 3. These C first reference fluorescence images are fluorescence images composed of N pixels that are generated by irradiating the sample S with excitation light of C wavelength bands in the first optical state and capturing the fluorescence generated from the sample S in response to the irradiation. At this time, the number C of first reference fluorescence images to be acquired (the number C of wavelength bands of excitation light irradiated onto the sample S) is pre-specified to be equal to or greater than the maximum number of fluorescent dyes that can be contained in the sample S. Similarly, the image acquisition unit 202 acquires C second reference fluorescence images in the second optical state.

[0027] On the other hand, if the mixing matrix is ​​found by the information search unit 201, the image acquisition unit 202 does not acquire a fluorescence image in the first optical state, but acquires C fluorescence images of the sample S in the second optical state as C target fluorescence images for acquiring a fluorescent dye image, in the same manner as described above.

[0028] The matrix acquisition unit 203 acquires a mixing matrix based on the C sets of the first and second reference fluorescent images acquired by the image acquisition unit 202. The function of acquiring this mixing matrix includes a clustering function and a statistical value calculation function.

[0029] The clustering function of the matrix acquisition unit 203 will be described.

[0030] The matrix acquisition unit 203 estimates a centroid fluorescence wavelength indicating the center of gravity of the wavelength distribution of the fluorescence by calculating the ratio of the fluorescence intensity (brightness value) of one fluorescence image to the fluorescence intensity of the other fluorescence image for each of the first and second reference fluorescence images of set C. At this time, the matrix acquisition unit 203 calculates the average value of the fluorescence intensity of one fluorescence image to the average value of the fluorescence intensity of the other fluorescence image for a pixel group clustered by the clustering process described below, and calculates the ratio of these average values. The matrix acquisition unit 203 acquires the estimated centroid fluorescence wavelength as wavelength information related to the fluorescence wavelength.

[0031] The matrix acquisition unit 203 performs clustering on N pixels constituting the C first reference fluorescence images and the C second reference fluorescence images, based on the C first reference fluorescence images and the C second reference fluorescence images acquired by the image acquisition unit 202, and the acquired wavelength information. Prior to the clustering process, the matrix acquisition unit 203 generates matrix data Y in which the fluorescence intensity values ​​of the N pixels constituting each of the C first reference fluorescence images and the C second reference fluorescence images are arranged in parallel in a one-dimensional manner.

[0032] The matrix acquisition unit 203 has a function (first clustering function) of clustering N pixels of the second reference fluorescence image into C pixel groups based on distribution information of excitation light in C wavelength bands of fluorescence intensity. In detail, the matrix acquisition unit 203 clusters pixels having the same wavelength band of excitation light with the highest fluorescence intensity into the same pixel group. FIG. 5 shows an image of pixel groups clustered by the first clustering function of the matrix acquisition unit 203, and FIG. 6 shows wavelength characteristics of excitation light absorption rates of multiple fluorescent dyes contained in the sample S. As shown in FIG. 5, assuming that the fluorescent dyes contained in the sample S are three types of fluorescent dyes C1, C2, and C3, and six second reference fluorescence images are obtained using excitation light in six wavelength bands, the matrix acquisition unit 203 clusters N pixels contained in the six second reference fluorescence images GC1 to GC6 into six pixel groups PGr1 to PGr6. As shown in FIG. 6, different types of fluorescent dyes generally have different wavelength characteristics of absorptance, and the three types of fluorescent dyes C1, C2, and C3 also have wavelength characteristics CW1, CW2, and CW3 with different peak wavelengths. Therefore, in six wavelength bands EW1, EW2, EW3, EW4, EW5, and EW6 of excitation light, the fluorescent dye with the highest absorptance is determined to be one of the three fluorescent dyes C1, C2, and C3. For example, the fluorescent dye C1 has the highest absorptance of excitation light in the wavelength band EW1, the fluorescent dye C1 has the highest absorptance of excitation light in the wavelength band EW2, and the fluorescent dye C2 has the highest absorptance of excitation light in the wavelength band EW3. Using this property, the matrix acquisition unit 203 can cluster N pixels into pixel groups in the range where the same fluorescent dye is distributed by the first clustering function. However, the six pixel groups PGr1 to PGr6 clustered by the first clustering function do not correspond one-to-one to the three types of fluorescent dyes C1, C2, and C3.

[0033] In addition, the matrix acquisition unit 203 has a function (second clustering function) of further clustering the C pixel groups clustered by the first clustering function into L pixel groups (L is an integer between 2 and N-1) based on the wavelength information. Here, the number L of pixel groups to be clustered is set in advance as a parameter stored in the image processing device 5, for example, corresponding to the number of types of fluorescent dyes that may be present in the sample S. The number L of pixel groups may be determined according to the type of excitation light or the number C of wavelength distributions of the excitation light, or may be determined independently of the type of excitation light or the number C of wavelength distributions of the excitation light. That is, for each of the C pixel groups clustered by the first clustering function, the matrix acquisition unit 203 specifies a centroid fluorescence wavelength estimated for the wavelength band of the excitation light corresponding to the pixel group. More specifically, the matrix acquisition unit 203 acquires wavelength information for the pixel groups clustered as having the highest absorptance in a certain wavelength band, and specifies the centroid fluorescence wavelength based on the acquired wavelength information. At this time, wavelength information is acquired using the average value of the fluorescence intensity in pixel groups of a pair of the first and second reference fluorescence images obtained corresponding to that wavelength band. Furthermore, the matrix acquisition unit 203 clusters the C pixel groups into L pixel groups by determining the distance (closeness of values) between the centroid fluorescence wavelengths identified for each of the C pixel groups. Then, the matrix acquisition unit 203 divides and regenerates matrix data Y, in which the fluorescence intensity values ​​of the pixels of the C first reference fluorescence image and the fluorescence intensity values ​​of the pixels of the C second reference fluorescence image are arranged in parallel in a one-dimensional manner, into cluster matrices for each of the L pixel groups.

[0034] 7 shows the distribution of the centroid fluorescence wavelengths identified by the matrix acquisition unit 203, and FIG. 8 shows an image of pixel groups clustered by the second clustering function by the matrix acquisition unit 203. In the example shown in FIG. 7 and FIG. 8, centroid fluorescence wavelengths FW1 to FW6 are identified for each of six pixel groups PGr1 to PGr6 clustered by the first clustering function, and pixel group PGr1 and pixel group PGr2, which are close to each other in distance of centroid fluorescence wavelengths, are grouped into a new pixel group PGr 01 Similarly, pixel group PGr3 and pixel group PGr4 are clustered into pixel group PGr 02The pixel groups PGr5 and PGr6 are clustered into the pixel group PGr 03 As a result, the pixels of the C fluorescent images can be divided into L pixel groups corresponding to the distribution of the fluorescent dyes assumed to be contained in the sample S. However, the number of divisions L by the second clustering function is set to be equal to or less than the number C of fluorescent images (the number C of wavelength bands of excitation light).

[0035] Next, the statistical value calculation function of the matrix acquisition unit 203 will be described.

[0036] The matrix acquisition unit 203 obtains a mixing matrix A for generating K fluorescent dye images showing the respective distributions of K (K is an integer between 2 and C) fluorescent dyes from the C first reference fluorescent images and the C second reference fluorescent images, based on the L cluster matrices obtained for the sample S. Generally, according to a non-negative matrix factorization (NMF) calculation method, the relationship between matrix data Y, which is an observation value matrix, and fluorescent dye matrix data X, in which the K fluorescent dye images are arranged in parallel in a one-dimensional manner for each pixel, is expressed by the following formula using the mixing matrix A: Y=AX Here, Y is matrix data of C × 2 rows and N columns, A is matrix data of C × 2 rows and K columns, and X is matrix data of K rows and N columns. Conversely, once the value of mixing matrix A is found, the fluorescent dye matrix data X can be expressed as the inverse matrix A of mixing matrix A. -1 and matrix data Y are used to obtain the following equation: X=A -1 Y (This process is called unmixing.)

[0037] Here, the matrix acquisition unit 203 regenerates the matrix data Y' by compressing the matrix data Y generated by the clustering function in units of pixel groups clustered by the clustering function. In detail, the matrix acquisition unit 203 calculates a statistical value for each pixel group of the clustered cluster matrix for the fluorescence intensity of each row of the matrix data Y, and compresses the pixel group of each row into one pixel having the calculated statistical value. In this way, the matrix acquisition unit 203 regenerates the matrix data Y' which is matrix data of C × 2 rows and L columns. The matrix acquisition unit 203 may calculate an average value based on the integrated value of the fluorescence intensity, may calculate the most frequent value of the fluorescence intensity, or may calculate the median value of the fluorescence intensity, as the statistical value.

[0038] Furthermore, the matrix acquisition unit 203 also compresses the reproduced matrix data Y' and the fluorescent dye matrix data X' compressed in the same manner from the fluorescent dye matrix data X, using the following formula including the mixing matrix A: Y'=AX' Using the property that the above holds, the mixing matrix A is derived based on the matrix data Y'. FIG. 9 shows an image of the matrix data Y' regenerated by the matrix acquisition unit 203 and the corresponding fluorescent dye matrix data X'. One square shown in FIG. 9 represents one element of the matrix data. In this way, the three pixel groups PGr 01 ~PGr 03 The fluorescent dye matrix data X and matrix data Y are divided into the pixel group PGr 01 ~PGr 03 The statistical values ​​for each sample are used as representative values, and the data is compressed into three columns of fluorescent dye matrix data X' and matrix data Y'.

[0039] The matrix acquisition unit 203 derives the mixing matrix A based on the matrix data Y' as follows. That is, the matrix acquisition unit 203 sets an initial value to the mixing matrix A, calculates the following loss function (loss value) Loss while sequentially changing the value of the mixing matrix A, and derives the mixing matrix A that reduces the value of the loss function Loss. Note that a regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree to which the regularization term is emphasized) may be added to this loss function.

number

[0040] As described above, the matrix acquisition unit 203 calculates a loss function for each of the L cluster matrices divided by the clustering function by referring to the statistical values ​​of the C×2 matrix data Y', calculates the loss function Loss based on the sum of the L loss functions, and obtains the mixing matrix A based on the loss function Loss. At this time, the matrix acquisition unit 203 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values ​​a, b, and c of the statistical values ​​of the C×2 matrix data Y', and then obtains the loss function Loss by calculating the sum of the corrected loss functions. Note that the matrix acquisition unit 203 may calculate the loss function for each of the L cluster matrices by correcting the row components of the difference value Y'-AX' for each wavelength band of the excitation light by dividing them by the C×2 statistical values ​​corresponding to each wavelength band of the excitation light.

[0041] The above formula can also be generalized as follows. That is, the matrix acquisition unit 203 derives the mixing matrix A and the fluorescent dye matrix data X' based on the matrix data Y' as follows. That is, the matrix acquisition unit 203 sets initial values ​​to the mixing matrix A and the fluorescent dye matrix data X', calculates a loss function (loss value) Loss using the following formula while sequentially changing the values ​​of the mixing matrix A and the fluorescent dye matrix data X', and derives the mixing matrix A and the fluorescent dye matrix data X' that reduce the value of the loss function Loss. A regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree to which the regularization term is emphasized) may be added to this loss function. Also, the calculation may be performed with a constraint that the mixing matrix A and the fluorescent dye matrix data X' are non-negative values.

number

[0042] As described above, the matrix acquisition unit 203 calculates a loss function by referring to the statistical values ​​of the C×2 matrix data Y′ for each of the L cluster matrices divided by the clustering function, and obtains the L loss functions Los i The loss function Los is calculated based on the sum of the L cluster matrices, and the mixing matrix A is obtained based on the loss function Los. imay be calculated by correcting by dividing the row components of the difference values ​​Y'-AX' for each wavelength band of the excitation light by C x 2 statistical values ​​corresponding to each wavelength band of the excitation light.

[0043] The matrix acquisition unit 203 stores the mixing matrix (image separation information for obtaining a fluorescent dye image by separating fluorescence) acquired using the above-mentioned clustering function and statistical value calculation function in the image separation information storage unit 206 by linking it to the image acquisition conditions, estimated processing information, save setting information, etc. The image acquisition conditions, estimated processing information, and save setting information linked to the mixing matrix are information corresponding to the acquisition of the first fluorescent image or the second fluorescent image, information corresponding to the estimation of the mixing matrix, or information corresponding to the saving of the mixing matrix.

[0044] When the information search unit 201 finds a mixing matrix, the matrix correction unit 204 acquires a corrected mixing matrix based on the mixing matrix and the C target fluorescent images acquired by the image acquisition unit 202. That is, the matrix correction unit 204 generates matrix data Y2 in which the values ​​of the fluorescent intensities of the pixels of the C target fluorescent images are arranged in parallel in one dimension. The matrix correction unit 204 also extracts matrix data of C rows and K columns corresponding to the rows of the second fluorescent image from the elements of the searched mixing matrix A as a mixing matrix A0. Furthermore, the matrix correction unit 204 acquires a corrected mixing matrix (corrected image separation information) A2 using the matrix data Y2 and the mixing matrix A0. Here, the mixing matrix A2 is matrix data of C rows and K columns including only the rows corresponding to the rows of the second fluorescent image among the elements of the mixing matrix A.

[0045] For example, the acquisition of a corrected mixing matrix is ​​executed as follows. The matrix correction unit 204 regenerates matrix data Y2' by clustering the matrix data Y2 using a clustering function and compressing it in units of pixel groups. This clustering is performed in the same manner as the process in the matrix acquisition unit 203, for example, based on the distribution of fluorescence intensity for each of C wavelength bands. Then, the matrix correction unit 204 uses the mixing matrix A0 as an initial value and calculates the following equation; L=||Y2'-A2X'|| 2 Here, a mixing matrix A2 and fluorescent dye matrix data X' that minimize the Euclidean distance calculated by the following formula may be used instead of the above formula, where a formula with a regularization term added, such as the following formula: L=||Y2'-A2X'|| 2 +λ||A2-A0|| 2 By adding the regularization term, it is possible to control the deviation from the initial value so that it does not become too large. In addition to the above regularization term, a term such as L1 norm ||A2||1 may be added to impose a sparsity constraint on the matrix A2. Also, values ​​that are expected to be correct among the elements of the mixing matrix A0 may be fixed, and the values ​​of the other elements may be corrected. Also, instead of the Euclidean distance, a loss function such as Kullback-Leibler divergence or Itakura-Saito divergence may be used.

[0046] If the information search unit 201 does not find a mixing matrix, the image generation unit 205 acquires K fluorescent dye images by unmixing the C second reference fluorescent images acquired of the sample S to be observed using the mixing matrix A newly acquired by the matrix acquisition unit 203. Specifically, the image generation unit 205 extracts the mixing matrix A2 from the mixing matrix A, and adds the inverse matrix A2 of the mixing matrix A2 to the matrix data Y2 generated based on the C second reference fluorescent images. -1 Calculate the fluorescent dye matrix data X by applying

[0047] When the information search unit 201 finds a mixing matrix, the image generation unit 205 calculates fluorescent dye matrix data X by unmixing the C target fluorescent images obtained of the sample S to be observed in the same manner as described above, using the mixing matrix A2 corrected by the matrix correction unit 204.

[0048] Then, the image generating unit 205 reconstructs K fluorescent dye images from the fluorescent dye matrix data X, and outputs the reconstructed K fluorescent dye images. The output destination at this time may be an output device of the image processing device 5, such as a display or a touch panel display, or may be an external device connected to the image processing device so as to be able to communicate data with the image processing device.

[0049] Next, a procedure for an observation process of the sample S using the fluorescent dye image acquisition system 1 according to this embodiment, that is, a flow of a separation image acquisition method according to this embodiment, will be described. Fig. 10 is a flowchart showing the procedure for the observation process by the fluorescent dye image acquisition system 1.

[0050] First, the information retrieval unit 201 of the image processing device 5 uses the set image acquisition conditions and the like as a search key to search for a mixing matrix A from among the data stored in the image separation information storage unit 206 (step S1). If the mixing matrix is ​​not found as a result of the search (step S2; No), the image acquisition unit 202 of the image processing device 5 acquires C first reference fluorescent images and C second reference fluorescent images that are the results of the fluorescent observation of the sample S (step S3).

[0051] Next, the matrix acquisition unit 203 of the image processing device 5 acquires a mixing matrix A based on the C first reference fluorescent images and the C second reference fluorescent images (step S4). After that, the acquired mixing matrix A is linked to the image acquisition conditions and the like by the matrix acquisition unit 203 and stored in the image separation information storage unit 206 (step S5; storage step). Furthermore, the image generation unit 205 acquires fluorescent dye matrix data X based on matrix data Y2 in which N pixels of the C second reference fluorescent images are arranged in parallel and matrix A2 extracted from the mixing matrix A (step S6).

[0052] On the other hand, if the mixing matrix is ​​found as a result of the search (step S2; Yes), the image acquisition unit 202 of the image processing device 5 acquires C target fluorescent images that are the results of the fluorescent observation of the sample S (step S7; acquisition step). Next, the matrix correction unit 204 of the image processing device 5 extracts (selects) the mixing matrix A0 from the searched mixing matrix A (step S8). Furthermore, the matrix correction unit 204 acquires the mixing matrix A2 corrected using the matrix data Y2 generated from the C target fluorescent images and the mixing matrix A0 (step S9). Then, the image generation unit 205 acquires the fluorescent dye matrix data X based on the matrix data Y2 and the mixing matrix A2 (step S10; generation step). The corrected mixing matrix A2 may be stored in the image separation information storage unit 206.

[0053] Finally, the image generating unit 205 of the image processing device 5 reproduces K fluorescent dye images from the fluorescent dye matrix data X acquired in step S6 or step S10, and outputs these K fluorescent dye images (step S11; generating step). With the above, the observation process for the sample S is completed.

[0054] According to the fluorescent dye image acquisition system 1 described above, a desired mixing matrix is ​​selected from a plurality of mixing matrices for obtaining a fluorescent dye image acquired based on C first reference fluorescent images and C second reference fluorescent images, C target fluorescent images captured in the same optical state as either the first reference fluorescent image or the second reference fluorescent image are acquired, and a fluorescent dye image is generated using the acquired C target fluorescent images and the selected mixing matrix. As a result, even when the type of excitation light used for observation or the number of fluorescent dyes to be observed is large, unmixing can be performed using a mixing matrix acquired in the past, and the amount of calculation required for unmixing can be reduced. As a result, the throughput when acquiring a fluorescent dye image by unmixing can be improved.

[0055] In this embodiment, a mixing matrix that corresponds to the image acquisition conditions is selected from among the mixing matrices generated based on previously acquired fluorescent images, and the selected mixing matrix is ​​used for unmixing. As a result, a mixing matrix suitable for the target fluorescent image is selected, which improves the separation accuracy of the acquired fluorescent dye images.

[0056] Furthermore, in this embodiment, a corrected mixing matrix is ​​obtained based on the target fluorescent image and the searched mixing matrix. As a result, the previously obtained mixing matrix is ​​corrected using the target fluorescent image, so that unmixing suitable for the target fluorescent image is performed, and the separation accuracy of the obtained fluorescent dye images can be further improved.

[0057] Furthermore, in this embodiment, a fluorescent dye image is generated using the corrected mixing matrix and the target fluorescent image. In this case, unmixing suitable for the target fluorescent image is performed, and the separation accuracy of the acquired fluorescent dye image can be further improved. In other words, a mixing matrix generated in the past may contain errors due to the influence of noise in the past fluorescent image used to estimate the mixing matrix. In this case, if the mixing matrix is ​​used as is for the target fluorescent image, the error of the fluorescent dye image generated from the target fluorescent image containing noise of a different nature tends to be large. In contrast, in this embodiment, a mixing matrix corrected based on the target fluorescent image is used, and therefore errors in the generated fluorescent dye image can be suppressed.

[0058] Fig. 11 shows an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 according to this embodiment. Fig. 12 shows an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 according to this embodiment without correcting the mixing matrix. The same fluorescent dye images are shown in corresponding positions in Figs. 11 and 12. Thus, it was found that, according to this embodiment, the image quality of some fluorescent dye images (particularly the fluorescent dye image in the upper right) was greatly improved by correcting the mixing matrix.

[0059] Various embodiments of the present invention have been described above, but the present invention is not limited to the above-described embodiments, and may be modified or applied to other things without departing from the spirit of the invention as described in each claim.

[0060] For example, in the image acquisition device 3 of this embodiment, the first fluorescence image and the second fluorescence image are acquired by attaching and detaching the wavelength information acquisition optical system 13. Instead, the image acquisition device 3 may be configured to include two cameras and acquire fluorescence images in two or more different optical states by detecting the fluorescence that transmits through the wavelength information acquisition optical system 13 and the fluorescence that is reflected by the wavelength information acquisition optical system 13 with each camera.

[0061] Alternatively, the image acquisition device 3 may include a color camera such as an RGB color camera, and acquire a reference fluorescence image and a target fluorescence image based on image data for each wavelength characteristic output from the color camera. Even in such a case, the image acquisition device 3 is configured to acquire fluorescence images in two or more different optical states. Alternatively, the image acquisition device 3 may include a hyperspectral camera, and acquire fluorescence images based on image data for each wavelength characteristic output from the hyperspectral camera. Even in such a case, the image acquisition device 3 is configured to acquire fluorescence images in two or more different optical states.

[0062] Furthermore, the matrix correction unit 204 of the image processing device 5 functions to correct the mixing matrix A when the mixing matrix is ​​found by the information search unit 201. In contrast to this, the matrix correction unit 204 of the image processing device 5 may function to always correct the mixing matrix A regardless of the search result of the information search unit 201 for a mixing matrix.

[0063] 13 shows the functional configuration of an image processing device 5A according to a modified example. The image processing device 5A differs from the image processing device 5 in that, instead of the matrix correction unit 204 and the image generation unit 205, the image processing device 5A includes an image correction unit 204A and an image output unit 205A, and has a function of correcting the fluorescent dye matrix data X without correcting the mixing matrix.

[0064] The image correction unit 204A performs unmixing for the C target fluorescent images by directly using the mixing matrix A0 extracted from the mixing matrix A searched for by the information search unit 201, thereby generating fluorescent dye matrix data X. Then, the image correction unit 204A generates fluorescent dye matrix data X corrected using the matrix data Y2 and the mixing matrix A0. C Generate.

[0065] For example, the corrected fluorescent dye matrix data X C The acquisition of is executed as follows: The image correction unit 204A uses the mixing matrix A0 and the initially generated fluorescent dye matrix data X as initial values, and calculates the following equation: L=||Y2-A2X|| 2 It is possible to obtain a mixing matrix A2 and fluorescent dye matrix data X that minimize the Euclidean distance calculated by the following formula. Here, instead of the above formula, a formula to which a regularization term is added, such as the following formula, may be used. L=||Y2-A2X|| 2 +λ(|Δ x X|+|Δ y X|) This regularization term is a regularization term of Total Variation Loss, and is a term obtained by calculating the L1 norm of each of the differentials in the x and y directions with the matrix data X as an image. In addition, an equation in which a regularization term related to the mixing matrix A is combined with the above equation may be used.

[0066] The image output unit 205A outputs the fluorescent dye matrix data X C Then, K fluorescent dye images are reconstructed from the K fluorescent dye images, and the reconstructed K fluorescent dye images are output.

[0067] According to the above modification, a fluorescent dye image can be obtained that is corrected based on a previously acquired mixing matrix and a target fluorescent image, and the separation accuracy of the acquired fluorescent dye image can be further improved.

[0068] As another modified example, the image correction section 204A of the image processing device 5A may obtain corrected fluorescent dye matrix data X by one of the following two methods.

[0069] As a first method, the image correction unit 204A generates fluorescent dye matrix data X by unmixing the C target fluorescent images directly using the mixing matrix A0 extracted from the mixing matrix A searched for by the information search unit 201. Then, the image correction unit 204A inputs the generated fluorescent dye matrix data X to a trained inference model, and converts the output of the trained inference model into the corrected fluorescent dye matrix data X C Note that, when learning the inference model, the image correction unit 204A can construct a learned inference model by using, as training data, a combination of the fluorescent dye matrix data X derived by unmixing in the matrix acquisition unit 203 and the fluorescent dye matrix data X obtained by unmixing the C target fluorescent images using the mixing matrix A0 as is.

[0070] According to the first method, the generated fluorescent dye image can be input to an inference model, and a corrected fluorescent dye image can be obtained based on the output of the inference model. As a result, the separation accuracy of the obtained fluorescent dye image can be further improved.

[0071] As a second method, the image correction unit 204A inputs matrix data Y2 based on C target fluorescent images and a mixing matrix A0 extracted from the mixing matrix A searched for by the information search unit 201 to a trained inference model, and converts the output of the trained inference model into corrected fluorescent dye matrix data X CNote that, when learning the inference model, the image correction unit 204A can construct a learned inference model by using, as training data, a combination of the fluorescent dye matrix data X derived by unmixing in the matrix acquisition unit 203, the mixing matrix A0 extracted from the searched mixing matrix A, and the matrix data Y2 based on the C target fluorescent images.

[0072] According to the second method, the target fluorescent image and the mixing matrix are input to the inference model, and a corrected fluorescent dye image can be obtained based on the output of the inference model. As a result, the separation accuracy of the obtained fluorescent dye image can be further improved.

[0073] In the first aspect, it is preferable that the image separation information is stored in association with the image acquisition conditions under which the multiple reference fluorescent images were acquired, and that the selection step selects the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions under which the target fluorescent image was acquired. In the second aspect, it is preferable that the storage unit stores the image separation information in association with the image acquisition conditions under which the multiple reference fluorescent images were acquired, and that the image processing device selects the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions under which the target fluorescent image was acquired. In this way, image separation information corresponding to the image acquisition conditions is selected from the image separation information generated based on previously acquired fluorescent images, and unmixing is performed using the selected image separation information. As a result, the separation accuracy of the acquired separated images can be improved.

[0074] In the first aspect, it is also preferable that the generating step acquires corrected image separation information which is image separation information corrected based on the target fluorescent image and the selected image separation information. In the second aspect, it is also preferable that the image processing device acquires corrected image separation information which is image separation information corrected based on the target fluorescent image and the selected image separation information in generating the fluorescence separation image. In this way, the image separation information acquired in the past is corrected using the target fluorescent image, so that unmixing suitable for the target fluorescent image is performed and the separation accuracy of the acquired separation image can be further improved.

[0075] In the first aspect, it is also preferable that in the generating step, the fluorescence separation image is generated using the corrected image separation information and the target fluorescence image. In the second aspect, it is also preferable that in generating the fluorescence separation image, the image processing device generates the fluorescence separation image using the corrected image separation information and the target fluorescence image. In this case, unmixing suitable for the target fluorescence image is performed, and the separation accuracy of the obtained separation image can be further improved.

[0076] In addition, in the first aspect, it is preferable that in the generating step, the fluorescence separation image is corrected based on the target fluorescence image and the image separation information. In addition, in the second aspect, it is preferable that the image processing device corrects the fluorescence separation image based on the target fluorescence image and the image separation information in generating the fluorescence separation image. In this case as well, it is possible to obtain a fluorescence separation image that has been corrected based on the image separation information and target fluorescence image previously obtained, and it is possible to further improve the separation accuracy of the obtained separation image.

[0077] Furthermore, in the first aspect, it is also preferable that in the generating step, the fluorescence separation image generated based on the target fluorescent image and the image separation information is input to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescence separation image. Furthermore, in the second aspect, it is also preferable that in generating the fluorescence separation image, the image processing device inputs the fluorescence separation image generated based on the target fluorescent image and the image separation information to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescence separation image. In this case, the generated fluorescence separation image can be input to the inference model, and a fluorescence separation image corrected based on the output of the inference model can be obtained. As a result, the separation accuracy of the obtained separation image can be further improved.

[0078] Furthermore, in the first aspect, it is also preferable that in the generating step, the target fluorescent image and image separation information are input to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescent separation image. Furthermore, it is also preferable that in generating the fluorescent separation image, the image processing device inputs the target fluorescent image and image separation information to a trained inference model, and obtains the output of the trained inference model as a corrected fluorescent separation image. In this way, the target fluorescent image and image separation information can be input to the inference model, and a corrected fluorescent separation image can be obtained based on the output of the inference model. As a result, the separation accuracy of the obtained separation image can be further improved.

[0079] Furthermore, in the first aspect, it is also preferable that the plurality of optical states includes at least a first optical state in which the fluorescence is transmitted through the optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter. Furthermore, in the second aspect, it is also preferable that the plurality of optical states includes at least a first optical state in which the fluorescence is transmitted through the optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter. In this way, it is possible to obtain wavelength information of the fluorescence based on the fluorescence images in the two optical states, and it is possible to store image separation information obtained as a result of clustering the pixel groups of the fluorescence images based on the wavelength information. This makes it possible to obtain a highly accurate separation image while reducing the overall calculation time.

[0080] In the first aspect, it is also preferable that in the generating step, the fluorescence separation image is generated based on the image separation information and the target fluorescence image acquired in the second optical state. In the second aspect, it is also preferable that in generating the fluorescence separation image, the image processing device generates the fluorescence separation image based on the image separation information and the target fluorescence image acquired in the second optical state. With this configuration, it is possible to obtain a highly accurate separation image by unmixing using the target separation image in the second optical state.

[0081] The separated image acquisition method of the embodiment is a separated image acquisition method comprising: [1] "a selection step of selecting desired image separation information from a storage unit storing multiple pieces of image separation information for obtaining a fluorescence separation image by separating a fluorescence image, the image separation information being acquired based on multiple reference fluorescence images in multiple optical states having different wavelength characteristics; an irradiation step of irradiating a sample with each of multiple wavelengths of excitation light; an acquisition step of acquiring a target fluorescence image in at least one optical state of the multiple optical states for each of a plurality of fluorescences generated from the sample by each of the multiple wavelengths of excitation light, via a fluorescence filter unit having multiple reflection wavelength ranges and multiple transmission wavelength ranges; and a generation step of generating a fluorescence separation image for each of the multiple fluorescences based on the target fluorescence image acquired in the acquisition step and the image separation information selected in the selection step."

[0082] The embodiment of the separated image acquisition method may be the separated image acquisition method described in [1] above, [2] "in which the image separation information is stored in association with the image acquisition conditions when the plurality of reference fluorescent images were acquired, and in the selection step, the image separation information linked to the image acquisition conditions corresponding to the image acquisition conditions when the target fluorescent image was acquired is selected."

[0083] The separated image acquisition method of the embodiment may be [3] "the separated image acquisition method described in [1] or [2] above, in which in the generation step, corrected image separation information is acquired, which is image separation information corrected based on the target fluorescent image and the selected image separation information."

[0084] The separated image acquisition method of the embodiment may be [4] "the separated image acquisition method described in the above [3], in which, in the generating step, the fluorescent separated image is generated using the corrected image separation information and the target fluorescent image."

[0085] The separated image acquisition method of the embodiment may be [5] "the separated image acquisition method described in [1] or [2] above, in which, in the generating step, the fluorescence separated image is corrected based on the target fluorescent image and the image separation information."

[0086] The separated image acquisition method of the embodiment may be [6] "the separated image acquisition method described in [1] or [2] above, in which, in the generation step, the fluorescence separated image generated based on the target fluorescence image and the image separation information is input to a trained inference model, and the output of the trained inference model is obtained as the corrected fluorescence separated image."

[0087] The separated image acquisition method of the embodiment may be [7] "the separated image acquisition method described in [1] or [2] above, in which, in the generation step, the target fluorescent image and the image separation information are input into a trained inference model, and the output of the trained inference model is obtained as the corrected fluorescent separation image."

[0088] The embodiment of the separated image acquisition method may be [8] "the separated image acquisition method described in any one of the above [1] to [7], wherein the plurality of optical states include at least a first optical state in which the fluorescence is transmitted through an optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter."

[0089] The separated image acquisition method of the embodiment may be [9] "the separated image acquisition method described in [8] above, in which, in the generation step, a fluorescent separated image is generated based on the image separation information and the target fluorescent image acquired in the second optical state."

[0090] A separation image acquisition device of an embodiment is

[10] a separation image acquisition device comprising: a storage unit that stores image separation information for obtaining a fluorescence separation image obtained by separating a fluorescence image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions; an image acquisition device that acquires a target fluorescence image in at least one optical state of the plurality of optical states for each of a plurality of fluorescences generated from the sample by each of the excitation light of a plurality of wavelengths via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and an image processing device that processes the fluorescence images, wherein the image processing device selects desired image separation information from the plurality of image separation information stored in the storage unit, and generates a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired by the image acquisition device and the selected image separation information.

[0091] The embodiment of the separated image acquisition device may be

[11] "the separated image acquisition device described in

[10] above, wherein the memory unit stores the image separation information in association with the image acquisition conditions when the multiple reference fluorescent images were acquired, and the image processing device selects the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions when the target fluorescent image was acquired."

[0092] The embodiment of the separated image acquisition device may be

[12] "the separated image acquisition device described in

[10] or

[11] above, wherein the image processing device acquires corrected image separation information, which is image separation information corrected based on the target fluorescent image and the selected image separation information, in generating the fluorescent separation image."

[0093] The embodiment of the separated image acquisition device may be

[13] "the separated image acquisition device described in

[12] above, wherein the image processing device generates the fluorescent separation image by using the corrected image separation information and the target fluorescent image in generating the fluorescent separation image."

[0094] The embodiment of the separated image acquisition device may be

[14] "the separated image acquisition device described in

[10] or

[11] above, wherein the image processing device corrects the fluorescent separation image based on the target fluorescent image and the image separation information in generating the fluorescent separation image."

[0095] The separated image acquisition device of the embodiment may be

[15] "the separated image acquisition device described in

[10] or

[11] above, wherein in generating the fluorescence separation image, the image processing device inputs the fluorescence separation image generated based on the target fluorescence image and the image separation information to a trained inference model, and obtains the output of the trained inference model as the corrected fluorescence separation image."

[0096] The embodiment of the separated image acquisition device may be

[16] "the separated image acquisition device described in

[10] or

[11] above, wherein, in generating the fluorescence separation image, the image processing device inputs the target fluorescent image and the image separation information into a trained inference model, and obtains the output of the trained inference model as the corrected fluorescent separation image."

[0097] The embodiment of the separated image acquisition device may be

[17] "a separated image acquisition device according to any one of

[10] to

[16] above, wherein the plurality of optical states include at least a first optical state in which the fluorescence is transmitted through an optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter."

[0098] The embodiment of the separated image acquisition device may be

[18] "the separated image acquisition device described in

[17] above, wherein the image processing device generates the fluorescent separation image based on the image separation information and the target fluorescent image acquired in the second optical state in generating the fluorescent separation image."

[0099] The separated image acquisition program of the embodiment is

[19] "a separated image acquisition program for generating a fluorescence separation image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, which are acquired through a fluorescence filter unit having a plurality of reflection wavelength bands and a plurality of transmission wavelength bands for each of a plurality of fluorescences generated from the sample by each of the excitation light of a plurality of wavelengths by irradiating the sample with each of the excitation light of a plurality of wavelength distributions, the separated image acquisition program causing a computer to execute a storage process for storing a plurality of pieces of image separation information for obtaining a fluorescence separation image obtained by separating the fluorescence, which is acquired based on the plurality of reference fluorescence images, a selection process for selecting desired image separation information from the plurality of stored image separation information, and a generation process for generating a fluorescence separation image based on a target fluorescence image acquired in at least one of the plurality of optical states for each of the plurality of fluorescences and the selected image separation information." [Explanation of symbols]

[0100] 1... fluorescent dye image acquisition system, 3... image acquisition device, 5, 5A... image processing device, 7... excitation light source (irradiation device), 9a... light source side filter set, 9b... camera side filter set (fluorescence filter section), 11... dichroic mirror, 15... camera (image acquisition device), 13... wavelength information acquisition optical system (optical filter), 201... information search section (selection section), 202... image acquisition section, 203... matrix acquisition section, 204... matrix correction section, 204A... image correction section, 205... image generation section, 205A... image output section, 206... image separation information storage section (storage section), C1, C2, C3... fluorescent dyes, GC1 to GC6... fluorescent images, PGr 01 ~PGr 03 ,PGr1~PGr6...pixel groups, S...sample.

Claims

1. a selection step of selecting desired image separation information from a storage unit storing a plurality of pieces of image separation information for obtaining a fluorescence separation image obtained by separating a fluorescence image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation step of irradiating a sample with each of the excitation lights having a plurality of wavelengths; an acquisition step of acquiring a target fluorescence image in at least one optical state among the plurality of optical states for each of a plurality of fluorescences generated from the sample by each of the excitation light of the plurality of wavelengths via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; a generating step of generating a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired in the acquiring step and the image separation information selected in the selecting step; A method for acquiring a separated image comprising:

2. a storage unit that stores image separation information for obtaining a fluorescence separation image by separating a fluorescence image, the fluorescence image being obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions; an image acquisition device that acquires a target fluorescence image in at least one optical state among the plurality of optical states for each of a plurality of fluorescences generated from the sample by each of the excitation light of a plurality of wavelengths via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; an image processing device for processing the fluorescent light image, The image processing device includes: Selecting a desired image separation information from the plurality of image separation information stored in the storage unit; generating a fluorescence separation image for each of the plurality of fluorescent lights based on the target fluorescent image acquired by the image acquisition device and the selected image separation information; Separation image acquisition device.

3. A separated image acquisition program for generating a fluorescence separated image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, the reference fluorescence images being acquired through a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges for each of a plurality of fluorescences generated from the sample by each of the excitation light beams having a plurality of wavelengths, by irradiating the sample with each of the excitation light beams having a plurality of wavelength distributions, the program comprising: For computers, a storage process for storing a plurality of pieces of image separation information for obtaining a fluorescence separation image obtained by separating the fluorescence, the image separation information being acquired based on a plurality of reference fluorescence images; A selection process for selecting a desired image separation information from the plurality of stored image separation information; a generation process for generating a fluorescence separation image based on a target fluorescence image acquired in at least one optical state among the plurality of optical states for each of the plurality of fluorescences and the selected image separation information; A separate image acquisition program that executes the above.

Images