Separated image acquisition method, separated image acquisition device, and separated image acquisition program

Abstract

A fluorescent dye image acquisition system (1) is provided with: an excitation light source (7) that irradiates a sample (S) with excitation light having a plurality of wavelength distributions; a camera (15) that acquires, via a camera-side filter bank (9b), target fluorescence images in a plurality of optical states having different wavelength characteristics with respect to a plurality of fluorescent lights generated from the sample (S) by excitation lights having a plurality of wavelengths; and an image processing device (5) that stores a plurality of mixing matrices for obtaining fluorescent dye images from which the fluorescence is separated, and generates a fluorescent dye image for each of the plurality of fluorescence on the basis of the target fluorescence image acquired in one optical state by the camera (15) and the selected mixing matrices.

Description

Technical Field

[0001] One aspect of the embodiment relates to a separate image acquisition method, a separate image acquisition device, and a separate image acquisition program. Background Art

[0002] Currently, multiple staining methods are used to simultaneously stain multiple substances within a sample, such as biological tissue. Furthermore, in order to observe the substances within the multi-stained sample, the sample is irradiated with excitation light to obtain a fluorescence image. For example, the following non-patent document 1 discloses a method for applying non-negative matrix factorization (NMF) to blindly unmix a fluorescence image in which fluorescence in multiple wavelength regions is observed to obtain a separated image of each substance within the sample. Furthermore, the following non-patent document 2 discloses a method for unmixing a fluorescence image, clustering the fluorescence image, extracting the maximum value of the fluorescence intensity in each clustered pixel group, and generating a separated image based on this maximum value.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-patent literature 1: Binjie Qin et al., "Target / Background ClassificationRegularized Nonnegative Matrix Factorization for Fluorescence Unmixing", IEEETRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT,VOL.65,NO.4,APRIL 2016

[0006] Non-patent literature 2: Tristan D.McRae et al., "Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning", PLOS ONE, December 2, 2019 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] In the conventional method described above, the calculation time for demixing tends to increase depending on the type of excitation light, the number of fluorescent dyes to be observed, and the like, leading to a decrease in throughput.

[0009] Therefore, one aspect of the embodiment is developed in view of this technical problem, and its technical problem is to provide a separated image acquisition method, a separated image acquisition device, and a separated image acquisition program that can achieve improved throughput when obtaining separated images by demixing.

[0010] Technical means to solve the problem

[0011] The first aspect of the embodiment involves a separation image acquisition method comprising: a selection step of selecting desired image separation information from a storage unit, the storage unit storing a plurality of image separation information for obtaining a fluorescence separation image after the fluorescence image is separated, which is obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation step of irradiating a sample with excitation light of a plurality of wavelengths; an acquisition step of acquiring, for each of a plurality of fluorescences generated from the sample by the excitation light of the plurality of wavelengths, a target fluorescence image in at least one of a plurality of optical states via a fluorescence filter unit having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions; and a generation step of generating, for each of the plurality of fluorescences, a fluorescence separation image based on the target fluorescence image acquired in the acquisition step and the image separation information selected in the selection step.

[0012] Alternatively, the second aspect of the embodiment involves a separation image acquisition device comprising: a storage unit, which stores image separation information for obtaining a fluorescence separation image after the fluorescence image is separated, which is obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device, which irradiates the sample with excitation lights of a plurality of wavelength distributions; an image acquisition device, which acquires, for each of a plurality of fluorescences generated from the sample by the excitation lights of a plurality of wavelengths, a target fluorescence image in at least one of a plurality of optical states via a fluorescence filter unit having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions; and an image processing device, which processes the fluorescence image, the image processing device selecting desired image separation information from the plurality of image separation information stored in the storage unit, and generating a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired in the image acquisition device and the selected image separation information.

[0013] Alternatively, the separation image acquisition program involved in the third aspect of the embodiment is a separation image acquisition program for generating a fluorescence separation image for each of the multiple fluorescences generated from the sample by the excitation light of multiple wavelengths, based on multiple reference fluorescence images in multiple optical states with different wavelength characteristics obtained through a fluorescence filter unit having multiple reflection wavelength regions and multiple transmission wavelength regions, by irradiating the sample with excitation light of multiple wavelength distributions. The program causes the computer to execute: storage processing of multiple image separation information obtained based on the multiple reference fluorescence images for obtaining a fluorescence separation image after separating the fluorescence; selection processing of selecting desired image separation information from the stored multiple image separation information; and generation processing of a fluorescence separation image for each of the multiple fluorescences based on the object fluorescence image obtained under at least one optical state of the multiple optical states and the selected image separation information.

[0014] According to the first, second, or third aspects, desired image separation information is selected from a plurality of image separation information acquired based on a plurality of reference fluorescence images. Excitation light having different wavelength distributions is used, via a fluorescence filter unit, to acquire a target fluorescence image capturing a fluorescence image of a specimen in the same optical state as any of the plurality of reference fluorescence images. A fluorescence separation image is generated using the acquired target fluorescence image and the selected image separation information. Consequently, even when a large number of excitation light types or a large number of fluorochromes are used for observation, previously acquired image separation information can be effectively utilized for demixing, thereby reducing the computational complexity of demixing. Consequently, the throughput of acquiring separated images through demixing can be improved.

[0015] Effects of the Invention

[0016] According to one aspect of the present invention, it is possible to improve the throughput when obtaining separated images by demixing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 1 is a schematic configuration diagram of a fluorescent dye image acquisition system 1 according to an embodiment.

[0018] Figure 2 Yes Figure 1 A perspective view of the structure of the image acquisition device 3.

[0019] Figure 3 Yes Figure 1 FIG. 5 is a block diagram showing an example of the hardware configuration of the image processing device 5 .

[0020] Figure 4 Yes Figure 1 A block diagram of the functional structure of the image processing device 5 is shown.

[0021] Figure 5 Yes means through Figure 4 FIG. 1 is a diagram of an image of a pixel group clustered by the first clustering function of the matrix acquisition unit 203 .

[0022] Figure 6 This is a graph showing the wavelength characteristics of the absorptivity of excitation light by a plurality of fluorescent dyes contained in the sample S.

[0023] Figure 7 Yes means through Figure 4 The matrix acquisition unit 203 is a diagram of the distribution of the centroid fluorescence wavelength specified.

[0024] Figure 8 Yes means through Figure 4 FIG. 1 is a diagram of an image of a pixel group clustered by the second clustering function of the matrix acquisition unit 203 .

[0025] Figure 9 It means by Figure 4 FIG. 1 is a diagram showing an image of the matrix data Y′ and the fluorescent dye matrix data X′ regenerated by the matrix acquisition unit 203 .

[0026] Figure 10 This is a flowchart showing the procedure of the separation image acquisition method according to the embodiment.

[0027] Figure 11 1 is a diagram showing an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 according to the present embodiment.

[0028] Figure 12 This is a diagram showing an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 according to the present embodiment without performing correction of the mixing matrix.

[0029] Figure 13 1 is a block diagram showing the functional configuration of an image processing device 5A according to a modified example. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description, the same elements or elements having the same functions are denoted by the same reference numerals, and repeated descriptions are omitted.

[0031] Figure 1This is a schematic diagram of the structure of a fluorescent dye image acquisition system 1, which is a separation image acquisition device according to an embodiment. The fluorescent dye image acquisition system 1 is a device for generating fluorescent dye images (fluorescence separation images) that are used to identify the distribution of fluorescent dyes within a sample such as a biological tissue being observed. The images generated by the fluorescent dye image acquisition system 1 are used for purposes such as analyzing the images, developing pharmaceutical agents, and studying therapeutic methods. Therefore, the fluorescent dye image acquisition system 1 is required to generate images that can quantitatively identify the distribution of a large amount of substances (fluorochromes) contained in the sample with high throughput. The fluorescent dye image acquisition system 1 includes an image acquisition device 3 that irradiates the sample S with excitation light and acquires an image of the corresponding fluorescence generated, and an image processing device 5 that performs data processing on the image acquired by the image acquisition device 3. The image acquisition device 3 and the image processing device 5 can be configured to transmit and receive image data between them using wired or wireless communication, or can be configured to input and output image data via a recording medium.

[0032] Figure 2 Yes Figure 1 A perspective view of the structure of the image acquisition device 3. Figure 2 In FIG, the dotted line with an arrow indicates the optical path of the excitation light, and the solid line with an arrow indicates the optical path of the fluorescence. The image acquisition device 3 is configured to include an excitation light source (irradiation device) 7, a light source-side filter group 9a, a dichroic mirror 11, a camera-side filter group (fluorescence filter unit) 9b, a wavelength information acquisition optical system (optical filter) 13, and a camera (image acquisition device) 15.

[0033] The excitation light source 7 is a light source capable of switching between multiple wavelength bands (wavelength distributions) for irradiation. Examples include an LED (Light Emitting Diode) light source, a light source composed of multiple monochromatic laser light sources, or a light source combining a white light source and a wavelength-selective optical element. The light source-side filter group 9a is a multi-bandpass filter, positioned in the optical path of the excitation light from the excitation light source 7, and has the property of transmitting light of multiple predetermined wavelength bands. The transmission wavelength band of the light source-side filter group 9a is set based on the multiple wavelength bands of the excitation light that can be used. The dichroic mirror 11 is an optical component positioned between the light source-side filter group 9a and the sample S, and has the property of reflecting the excitation light toward the sample S while transmitting the corresponding fluorescence emitted from the sample S. The camera-side filter group 9b is a multi-bandpass filter, positioned in the optical path of the fluorescence transmitted by the dichroic mirror 11, and has the property of transmitting light of multiple predetermined wavelength bands. The transmission wavelength band of the camera-side filter group 9b is set based on the wavelength band of the fluorescence generated by the fluorescent pigment in the sample S that may be included in the observation object. Specifically, the camera-side filter group 9 b has, as its wavelength characteristics, transmission bands (transmission wavelength regions) corresponding to a plurality of fluorescence wavelength bands, and reflection bands (reflection wavelength regions) between these plurality of transmission bands.

[0034] The wavelength information acquisition optical system 13 is an optical system detachably supported in the optical path of fluorescence transmitted by the camera-side filter set 9b and used to acquire wavelength information of the fluorescence. Specifically, the wavelength information acquisition optical system 13 is configured to switch between two states: a state in which it is positioned in the optical path of the fluorescence from the specimen S (a first optical state) and a state in which it is removed from the optical path of the fluorescence (a second optical state). Furthermore, the wavelength information acquisition optical system 13 can be configured in any number of optical states, as long as it can achieve multiple different wavelength characteristics. For example, it can be a fluorescence filter set comprising two or more fluorescence filters with different wavelength characteristics. In the first optical state, the wavelength information acquisition optical system 13 transmits the fluorescence generated in the specimen S and transmitted through the dichroic mirror 11 and the camera-side filter set 9b toward the camera 15 with predetermined wavelength characteristics. In the second optical state, which is different from the first optical state, the wavelength information acquisition optical system 13 allows the fluorescence transmitted through the dichroic mirror 11 and the camera-side filter set 9b to enter the camera 15 in its original optical state (without transmitting through the wavelength information acquisition optical system 13). For example, a dichroic mirror (also called a tilt filter) having a wavelength characteristic in which the transmittance increases linearly with increasing wavelength can be used as the wavelength information acquisition optical system 13. Alternatively, a dichroic mirror (or tilt filter) having a wavelength characteristic in which the transmittance decreases linearly with decreasing wavelength can be used as the wavelength information acquisition optical system 13. The wavelength information acquisition optical system 13 using such a dichroic mirror can cause fluorescence with two different wavelength transmittance characteristics to enter the camera 15.

[0035] The camera 15 is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer greater than or equal to 2, for example, 2048×2048). When the wavelength information acquisition optical system 13 is switched to the fluorescence optical path (in the first optical state), the camera 15 captures the fluorescence transmitted through the wavelength information acquisition optical system 13 to obtain a first fluorescence image. Furthermore, when the wavelength information acquisition optical system 13 is disconnected from the fluorescence optical path (in the second optical state), the camera 15 captures the fluorescence that has not transmitted through the wavelength information acquisition optical system 13 to obtain a second fluorescence image. The camera 15 captures first and second fluorescence images for each of the multiple fluorescence images generated from the specimen S by the excitation light of multiple wavelength bands emitted by the excitation light source 7. The camera 15 outputs the acquired first and second fluorescence images to the image processing device 5 using communication or via a recording medium.

[0036] Next, refer to Figure 3 and Figure 4 The configuration of the image processing device 5 will be described. Figure 3 is a block diagram showing an example of the hardware configuration of the image processing device 5. Figure 4 It is a block diagram showing the functional structure of the image processing device 5 .

[0037] like Figure 3 As shown, the image processing device 5 is a computer that physically includes a CPU (Central Processing Unit) 101 as a processor, a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103 as a recording medium, a communication module 104, and an input / output module 106, etc., all of which are electrically connected. Furthermore, the image processing device 5 may include a display, keyboard, mouse, touch panel, monitor, etc. as input / output devices, and may also include a data recording device such as a hard disk drive or semiconductor memory. Furthermore, the image processing device 5 may be composed of multiple computers.

[0038] like Figure 4 As shown, the image processing device 5 includes, as functional components, an information search unit (selection unit) 201, an image acquisition unit 202, a matrix acquisition unit 203, a matrix correction unit 204, an image generation unit 205, and an image separation information storage unit (storage unit) 206. Furthermore, the information search unit 201 and the image separation information storage unit 206 may be external devices connected to the image processing device 5. Figure 4 Each functional unit of the image processing device 5 shown is realized by reading a program (separated image acquisition program of the embodiment) into the hardware such as the CPU 101 and the RAM 102, and operating the communication module 104 and the input / output module 106 under the control of the CPU 101, and reading and writing data in the RAM 102. The CPU 101 of the image processing device 5 executes the computer program to Figure 4 Each functional unit plays a role and sequentially performs processing corresponding to the separation image acquisition method described later.

[0039] Furthermore, CPU 101 may be a single piece of hardware or a component incorporated into a programmable logic device such as an FPGA, as in the case of a soft processor. RAM and ROM may also be single pieces of hardware or components incorporated into a programmable logic device such as an FPGA. Various data required to execute the computer program, as well as various data generated by executing the computer program, are all stored in internal memory such as ROM 103 and RAM 102, or in storage media such as a hard disk drive. The functions of the functional components of image processing device 5 are described in detail below.

[0040] The information search unit 201 searches (selects) a desired mixing matrix from the data stored in the image separation information storage unit 206 based on the image acquisition conditions for acquiring the first or second fluorescence image of the sample S, information related to the estimation process (estimation process information) based on the mixing matrix (image separation information) of the first and second fluorescence images, and storage setting information for storing the mixing matrix in the image separation information storage unit 206. The information search unit 201 may set the image acquisition conditions, estimation process information, and storage setting information used for the search based on information input by a user via the input / output module 106 of the image processing device 5, may set the image acquisition conditions, estimation process information, and storage setting information used for the search based on information transmitted from an external device such as the image acquisition device 3, or may set the image acquisition conditions, estimation process information, and storage setting information used for the search based on information referenced from the RAM 102 or ROM 103 of the image processing device 5.

[0041] As the above-mentioned image acquisition conditions, the type of sample S, the information of the barcode attached to the sample S, the type of fluorescent pigment contained in the sample S, the date of staining of the sample S, the date of measurement of the sample S, the type of excitation light, the intensity of each excitation light, the type or number of filters (the presence or absence of tilt filters, etc.), the type of camera, the type or magnification of the objective lens, the exposure time of each excitation light, the ambient temperature, the data of the acquired fluorescence image itself, etc. As the above-mentioned estimation processing information, the version information of the software used for estimation processing, the algorithm, parameters, or options used for estimation processing, the time required for estimation processing, etc. can be cited. As the above-mentioned saved setting information, the saved user name, project name, detection name, mixing matrix name, tag information (with free keywords, hash tags for SNS), etc. can be cited.

[0042] In the image separation information storage unit 206, the mixing matrix acquired by the matrix acquisition unit 203 (described later) is stored in association with the image acquisition conditions used when acquiring the multiple fluorescence images (e.g., the first and second fluorescence images) that form the basis of the mixing matrix, the estimation processing information related to the estimation process used when acquiring the mixing matrix, and the storage setting information used when storing the mixing matrix. The information search unit 201 searches (selects) for a mixing matrix associated with information that matches or corresponds to (is similar to) a search key containing the image acquisition conditions, estimation processing information, or storage setting information set by a user, an external device, or the like. The information search unit 201 then passes the search results for the mixing matrix to the image acquisition unit 202 and the matrix acquisition unit 203. At this time, the information search unit 201 searches for the mixing matrix using at least one of the aforementioned image acquisition conditions, estimation processing information, and storage setting information as a search key. The information search unit 201 searches for the mixing matrix using a search key containing at least one of the image acquisition conditions.

[0043] When no mixing matrix that is consistent with or corresponds to (similar to) the search key such as the image acquisition condition is found in the retrieval process based on the information retrieval unit 201, in order to re-acquire the mixing matrix, the image acquisition unit 202 acquires the first fluorescent image (first reference fluorescent image) and the second fluorescent image (second reference fluorescent image) from the image acquisition device 3.

[0044] Specifically, if the information retrieval unit 201 cannot see the mixing matrix, the image acquisition unit 202 acquires C (C is an integer greater than or equal to 2) pre-specified first reference fluorescence images of the specimen S from the image acquisition device 3. These C first reference fluorescence images are fluorescence images consisting of N pixels, generated by irradiating the specimen S with excitation light of C wavelength bands in a first optical state and capturing the corresponding fluorescence generated from the specimen S. The number C of first reference fluorescence images acquired (the number C of wavelength bands of excitation light irradiated on the specimen S) is pre-specified to be greater than the maximum number of fluorochromes that can be contained in the specimen S. Similarly, the image acquisition unit 202 acquires C second reference fluorescence images in a second optical state.

[0045] On the other hand, when the mixing matrix is ​​seen by the information retrieval unit 201, the image acquisition unit 202 does not acquire the fluorescence image in the first optical state, but acquires C fluorescence images of the sample S as the object in the second optical state as C object fluorescence images for acquiring the fluorescent pigment image, as described above.

[0046] The matrix acquisition unit 203 acquires a mixing matrix based on the first reference fluorescent image and the second reference fluorescent image of group C acquired by the image acquisition unit 202. The function of acquiring the mixing matrix includes a clustering function and a statistical value calculation function.

[0047] The clustering function of the matrix acquisition unit 203 will be described.

[0048] The matrix acquisition unit 203 calculates the ratio of the fluorescence intensity (brightness value) of the first reference fluorescence image and the second reference fluorescence image of group C to estimate the centroid fluorescence wavelength, which represents the centroid of the fluorescence wavelength distribution. The matrix acquisition unit 203 calculates the average fluorescence intensity of the first fluorescence image and the average fluorescence intensity of the second fluorescence image for the pixel groups clustered by the clustering process described below, and then calculates the ratio of these average values. The matrix acquisition unit 203 acquires the estimated centroid fluorescence wavelength as wavelength information related to the fluorescence wavelength.

[0049] The matrix acquisition unit 203 performs clustering on the N pixels constituting the C first reference fluorescence images and the C second reference fluorescence images, based on the wavelength information obtained by the image acquisition unit 202. Prior to the clustering process, the matrix acquisition unit 203 generates matrix data Y in which the fluorescence intensity values ​​of the N pixels constituting the C first reference fluorescence images and the C second reference fluorescence images are arranged in parallel in a one-dimensional manner.

[0050] The matrix acquisition unit 203 has the following function (first clustering function): based on the distribution information of each excitation light in the C wavelength bands of fluorescence intensity, the matrix acquisition unit 203 clusters the N pixels of the second reference fluorescence image into C pixel groups. Specifically, the matrix acquisition unit 203 clusters the pixels with the same wavelength band of excitation light having the highest fluorescence intensity into the same pixel group. Figure 5 The image of the pixel group clustered by the first clustering function based on the matrix acquisition unit 203 is shown in FIG. Figure 6 The wavelength characteristics of the absorption rate of the excitation light of the multiple fluorescent pigments contained in the sample S are shown in FIG. Figure 5 As shown in FIG. 1 , the fluorescent dyes contained in the sample S are three types of fluorescent dyes, namely, fluorescent dye C1, fluorescent dye C2, and fluorescent dye C3. Assuming that six second reference fluorescent images are obtained using excitation light of six wavelength bands, the matrix acquisition unit 203 clusters the N pixels contained in the six second reference fluorescent images GC1 to GC6 into six pixel groups PGr1 to PGr6. Figure 6As shown, different types of fluorescent pigments typically have different wavelength characteristics for their absorption rates. The three fluorescent pigments C1, C2, and C3 also have wavelength characteristics with different peak wavelengths, CW1, CW2, and CW3. Therefore, within the six excitation light wavelength bands EW1, EW2, EW3, EW4, EW5, and EW6, the fluorescent pigment with the highest absorption rate is determined to be one of the three fluorescent pigments C1, C2, and C3. For example, fluorescent pigment C1 has the highest absorption rate for excitation light in wavelength band EW1, fluorescent pigment C1 has the highest absorption rate for excitation light in wavelength band EW2, and fluorescent pigment C2 has the highest absorption rate for excitation light in wavelength band EW3. Leveraging this property, the matrix acquisition unit 203 uses the first clustering function to cluster N pixels into pixel groups with the same fluorescent pigment distribution range. However, the six pixel groups PGr1 to PGr6 clustered by the first clustering function do not correspond one-to-one with the three fluorescent pigments C1, C2, and C3.

[0051] Furthermore, the matrix acquisition unit 203 has the following function (second clustering function): based on wavelength information, the C pixel groups clustered by the first clustering function are further clustered into L pixel groups (L is an integer greater than 2 and less than N-1). The number of clustered pixel groups L corresponds to, for example, the number of types of fluorescent pigments that may be present in the sample S and is pre-set as a parameter stored in the image processing device 5. The number of pixel groups L can be determined based on the type of excitation light or the number C of wavelength distributions of the excitation light, or can be determined independently of the type of excitation light or the number C of wavelength distributions of the excitation light. Specifically, the matrix acquisition unit 203 specifies, for each of the C pixel groups clustered by the first clustering function, a centroid fluorescence wavelength estimated based on the wavelength band of the excitation light corresponding to that pixel group. More specifically, the matrix acquisition unit 203 obtains wavelength information for the pixel groups clustered based on the maximum absorbance in a certain wavelength band and specifies the centroid fluorescence wavelength based on the obtained wavelength information. At this time, wavelength information is acquired using the average value of the fluorescence intensities in the pixel groups of the first reference fluorescence image and the second reference fluorescence image, obtained corresponding to the wavelength band. Furthermore, the matrix acquisition unit 203 determines the distance (the degree of similarity in values) between the specific centroid fluorescence wavelengths of each of the C pixel groups, thereby clustering the C pixel groups into L pixel groups. The matrix acquisition unit 203 then divides the matrix data Y, which contains the fluorescence intensity values ​​of the C pixels of the first reference fluorescence image and the C pixels of the second reference fluorescence image, arranged one-dimensionally in parallel, into cluster matrices for each of the L pixel groups, and regenerates the matrix.

[0052] exist Figure 7 The distribution of the centroid fluorescence wavelength specified by the matrix acquisition unit 203 is shown in FIG. Figure 8 , an image of a pixel group clustered by the second clustering function of the matrix acquisition unit 203 is shown. Figure 7 and Figure 8 In the example shown, according to the specific centroid fluorescence wavelengths FW1 to FW6 of the six pixel groups PGr1 to PGr6 clustered by the first clustering function, the pixel group PGr1 and the pixel group PGr2 whose centroid fluorescence wavelengths are close to each other are clustered into a new pixel group PGr 01 Similarly, the pixel groups PGr3 and PGr4 are clustered into the pixel group PGr 02 , clustering pixel groups PGr5 and PGr6 into pixel group PGr 03 Thus, the pixels of the C fluorescence images can be divided into L pixel groups corresponding to the distribution of the fluorescent pigment assumed to be contained in the sample S. However, the number of divisions L based on the second clustering function is set to be less than the number C of fluorescence images (the number C of wavelength bands of excitation light).

[0053] Next, the statistical value calculation function of the matrix acquisition unit 203 will be described.

[0054] The matrix acquisition unit 203 obtains a mixing matrix A for generating K fluorescent pigment images representing the distribution of K fluorescent pigments (K is an integer greater than or equal to 2 and less than or equal to C) from the C first reference fluorescent images and the C second reference fluorescent images, based on the L cluster matrices obtained for the sample S. Typically, the relationship between the observation matrix, i.e., matrix data Y, and fluorescent pigment matrix data X, which is a one-dimensional array of K fluorescent pigment images arranged in parallel for each pixel, is expressed using the mixing matrix A using the following equation:

[0055] Y=AX

[0056] Here, Y is the matrix data of C×2 rows and N columns, A is the matrix data of C×2 rows and K columns, and X is the matrix data of K rows and N columns. Conversely, if the value of the mixing matrix A is obtained, the fluorescent dye matrix data X can be obtained using the inverse matrix A of the mixing matrix A. -1 And matrix data Y, through the following formula:

[0057] X=A -1 Y

[0058] Derivation (This process is called demixing).

[0059] Here, the matrix acquisition unit 203 compresses the matrix data Y generated by the clustering function, per pixel group, to regenerate matrix data Y'. Specifically, the matrix acquisition unit 203 calculates a statistical value for each pixel group in the cluster matrix, clustered based on the fluorescence intensity of each row of the matrix data Y, and compresses the pixel group in each row into a single pixel having the calculated statistical value. Thus, the matrix acquisition unit 203 regenerates matrix data Y', which consists of C×2 rows and L columns. The statistical value can be calculated as an average value based on the cumulative value of the fluorescence intensity, the mode of the fluorescence intensity, or the median of the fluorescence intensity.

[0060] Furthermore, the matrix acquisition unit 203 uses the following equation containing the mixing matrix A in the regenerated matrix data Y′ and the fluorescent dye matrix data X′ compressed in the same manner from the fluorescent dye matrix data X:

[0061] Y'=AX'

[0062] The same property holds true. Based on the matrix data Y', the mixing matrix A is derived. Figure 9 Matrix data Y' regenerated by the matrix acquisition unit 203 and the corresponding fluorescent dye matrix data X' are shown in FIG. Figure 9 Each square in the figure represents an element of the matrix data. 01 ~PGr 03 The fluorescent dye matrix data X and matrix data Y are based on each pixel group PGr 01 ~PGr 03 The statistical values ​​are representative values ​​and are compressed into three columns of fluorescent dye matrix data X' and matrix data Y'.

[0063] The matrix acquisition unit 203 derives the mixing matrix A based on the matrix data Y' as follows. Specifically, the matrix acquisition unit 203 sets an initial value for the mixing matrix A, calculates the loss function (loss value) Los described below while sequentially changing the value of the mixing matrix A, and derives the mixing matrix A such that the value of the loss function Los decreases. Furthermore, a regularization term such as the L1 norm λ|A| (λ represents a coefficient indicating the degree of emphasis on the regularization term) may be added to this loss function.

[0064] [Mathematical formula 1]

[0065]

[0066] In the above formula, j is a parameter indicating the position of a row of matrix data (corresponding to the wavelength band of excitation light for each of the two fluorescence images). The matrix subscript 1j represents the matrix data for the jth row of the first cluster matrix, the matrix subscript 2j represents the matrix data for the jth row of the second cluster matrix, and the matrix subscript 3j represents the matrix data for the jth row of the third cluster matrix. Furthermore, the parameters a, b, and c represent the average of the statistical values ​​for each column of the matrix data Y'.

[0067] As described above, the matrix acquisition unit 203 calculates a loss function for each of the L cluster matrices divided by the clustering function, referring to the statistical values ​​of the C×2 matrix data Y'. The loss function Los is calculated based on the sum of the L loss functions, and the mixing matrix A is obtained based on this loss function Los. In this case, the matrix acquisition unit 203 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values ​​a, b, and c of the statistical values ​​of the C×2 matrix data Y'. The loss function Los is then calculated by summing the corrected loss functions. Alternatively, the matrix acquisition unit 203 can calculate the loss function for each of the L cluster matrices by dividing the row component of each wavelength band of the excitation light of the differential value Y'-AX' by the C×2 statistical values ​​corresponding to each wavelength band of the excitation light and performing the correction.

[0068] Furthermore, the above formula can also be generalized as follows. That is, the matrix acquisition unit 203 derives the mixing matrix A and the fluorescent pigment matrix data X' based on the matrix data Y' as follows. Specifically, the matrix acquisition unit 203 sets initial values ​​for the mixing matrix A and the fluorescent pigment matrix data X', and while sequentially changing the values ​​of the mixing matrix A and the fluorescent pigment matrix data X', calculates the loss function (loss value) Los using the following formula, thereby deriving the mixing matrix A and the fluorescent pigment matrix data X' that reduce the value of the loss function Los. Furthermore, a regularization term such as the L1 norm λ|A| (λ is a coefficient indicating the degree of emphasis on the regularization term) can be added to this loss function. Furthermore, the calculation can be performed under the constraint that the mixing matrix A and the fluorescent pigment matrix data X' are non-negative values.

[0069] [Mathematical formula 2]

[0070]

[0071] In the above formula, j is a parameter indicating the position of the row of the matrix data (corresponding to the wavelength band of the excitation light), and i is a parameter indicating the position of the column of the matrix data (corresponding to the i-th cluster). ij The weight of each element in the matrix data can be calculated based on the value of each element or its standard deviation. ijAll are set to the same value regardless of the weight of each element. In addition, the average value of the statistical value of each column of the matrix data Y' in the above formula is set to a, b, c, ..., and replaced by w 1j =1 / a, w 2j =1 / b, w 3j =1 / c is the same as the formula of the loss function Los shown above.

[0072] As described above, the matrix acquisition unit 203 calculates the loss function for each of the L cluster matrices divided by the clustering function, referring to the statistical value of the C×2 matrix data Y', and calculates the loss function based on the L loss functions Los i The loss function Los is calculated by summing the difference values ​​Y'-AX' and the mixing matrix A is obtained based on the loss function Los. In addition, the matrix acquisition unit 203 can calculate the loss function Los of each of the L cluster matrices by dividing the row component of each wavelength band of the excitation light of the difference value Y'-AX' by C×2 statistical values ​​corresponding to each wavelength band of the excitation light and performing correction. i .

[0073] The matrix acquisition unit 203 stores the mixing matrix (image separation information for obtaining the fluorescent dye image after fluorescence separation) obtained using the aforementioned clustering function and statistical value calculation function in the image separation information storage unit 206, in association with image acquisition conditions, estimation processing information, and storage setting information. The image acquisition conditions, estimation processing information, and storage setting information associated with the mixing matrix are information corresponding to when acquiring the first or second fluorescent image, information corresponding to when estimating the mixing matrix, or information corresponding to when storing the mixing matrix.

[0074] When the information retrieval unit 201 receives the mixing matrix, the matrix correction unit 204 obtains a corrected mixing matrix, or a corrected mixing matrix, based on the mixing matrix and the C target fluorescence images obtained by the image acquisition unit 202. Specifically, the matrix correction unit 204 generates matrix data Y2, which arranges the fluorescence intensity values ​​of the pixels of the C target fluorescence images in parallel in a one-dimensional manner. Furthermore, the matrix correction unit 204 extracts, from the elements of the retrieved mixing matrix A, matrix data consisting of C rows and K columns corresponding to the rows of the second fluorescence image, as the mixing matrix A0. Furthermore, the matrix correction unit 204 uses the matrix data Y2 and the mixing matrix A0 to obtain a corrected mixing matrix A2 (corrected image separation information). Here, the mixing matrix A2 is matrix data consisting of only the C rows and K columns of the elements of the mixing matrix A corresponding to the rows of the second fluorescence image.

[0075] For example, the corrected mixing matrix is ​​obtained as follows. The matrix correction unit 204 clusters the matrix data Y2 using a clustering function and compresses it in pixel groups, thereby regenerating matrix data Y2'. This clustering is performed, for example, based on the distribution of fluorescence intensity in each of the C wavelength bands, in the same manner as the processing in the matrix acquisition unit 203. Then, the matrix correction unit 204 uses the mixing matrix A0 as an initial value and obtains a matrix that satisfies the following equation:

[0076] L=||Y2'-A2X'|| 2

[0077] The calculated Euclidean distance is minimized between the mixing matrix A2 and the fluorescent dye matrix data X'. Here, instead of the above equation, an equation to which a regularization term such as the following equation is added may be used.

[0078] L=||Y2'-A2X'|| 2 +λ||A2-A0|| 2

[0079] By adding regularization terms, deviations from the initial values ​​can be controlled to be small. In addition to the regularization terms mentioned above, terms such as the L1 norm ||A2||1 can be added to impose sparsity constraints on matrix A2. Furthermore, expected positive values ​​in the mixing matrix A0 can be fixed, and the values ​​of other elements can be corrected. Furthermore, loss functions such as the Kullback-Leibler divergence and the Itakura-Saito divergence can be used instead of the Euclidean distance.

[0080] When the information retrieval unit 201 cannot see the mixing matrix, the image generation unit 205 uses the mixing matrix A newly obtained by the matrix acquisition unit 203 to demix the C second reference fluorescence images obtained for the observation target sample S, thereby obtaining K fluorescent dye images. Specifically, the image generation unit 205 extracts the mixing matrix A2 from the mixing matrix A, and applies the inverse matrix A2 of the mixing matrix A2 to the matrix data Y2 generated based on the C second reference fluorescence images. -1 , thereby calculating the fluorescent pigment matrix data X.

[0081] When the information retrieval unit 201 sees the mixing matrix, the image generation unit 205 uses the mixing matrix A2 corrected by the matrix correction unit 204 to demix the C object fluorescence images obtained with the observation object sample S as the object in the same manner as the above-mentioned procedure, thereby calculating the fluorescent pigment matrix data X.

[0082] The image generator 205 then regenerates K fluorescent pigment images based on the fluorescent pigment matrix data X and outputs the regenerated K fluorescent pigment images. The output destination may be an output device of the image processing device 5, such as a display or a touch panel display, or an external device connected to the image processing device for data communication.

[0083] Next, the procedure of observation processing for the sample S using the fluorescent dye image acquisition system 1 of the present embodiment, that is, the flow of the separation image acquisition method of the present embodiment, will be described. Figure 10 1 is a flowchart showing the procedure of observation processing performed by the fluorescent dye image acquisition system 1 .

[0084] First, the information search unit 201 of the image processing device 5 uses the set image acquisition conditions and the like as search keys to search for the mixing matrix A from the data stored in the image separation information storage unit 206 (step S1). If no mixing matrix is ​​found as a result of the search (step S2: No), the image acquisition unit 202 of the image processing device 5 acquires C first reference fluorescence images and C second reference fluorescence images as the results of fluorescence observation of the specimen S (step S3).

[0085] Next, the matrix acquisition unit 203 of the image processing device 5 acquires a mixing matrix A based on the C first reference fluorescence images and the C second reference fluorescence images (step S4). The mixing matrix A acquired by the matrix acquisition unit 203 is then stored in the image separation information storage unit 206 in association with image acquisition conditions and the like (step S5; storage step). Furthermore, the image generation unit 205 acquires fluorochrome matrix data X based on the matrix data Y2 of N pixels of the C second reference fluorescence images arranged in parallel and the matrix A2 extracted from the mixing matrix A (step S6).

[0086] On the other hand, if a mixing matrix is ​​found in the search results (step S2; yes), the image acquisition unit 202 of the image processing device 5 acquires C target fluorescence images as a result of fluorescence observation of the sample S (step S7; acquisition step). Next, the mixing matrix A0 is extracted (selected) from the mixing matrix A retrieved by the matrix correction unit 204 of the image processing device 5 (step S8). Furthermore, the matrix correction unit 204 acquires the mixing matrix A2, which is corrected using the matrix data Y2 generated from the C target fluorescence images and the mixing matrix A0 (step S9). Then, the image generation unit 205 acquires the fluorochrome matrix data X based on the matrix data Y2 and the mixing matrix A2 (step S10; generation step). Furthermore, the corrected mixing matrix A2 can be stored in the image separation information storage unit 206.

[0087] Finally, the image generation unit 205 of the image processing device 5 regenerates K fluorescent dye images based on the fluorescent dye matrix data X acquired in step S6 or step S10 and outputs these K fluorescent dye images (step S11; generation step). The above-described observation process for the sample S is completed.

[0088] According to the fluorescent pigment image acquisition system 1 described above, a desired mixing matrix is ​​selected from a plurality of mixing matrices obtained based on C first reference fluorescent images and C second reference fluorescent images for obtaining a fluorescent pigment image. C target fluorescent images captured in the same optical state as any of the first and second reference fluorescent images are acquired, and a fluorescent pigment image is generated using the acquired C target fluorescent images and the selected mixing matrix. Consequently, even when the types of excitation light used for observation or the number of fluorescent pigments to be observed are numerous, demixing can be effectively performed using previously acquired mixing matrices, thereby reducing the computational complexity of demixing. Consequently, the throughput of fluorescent pigment image acquisition through demixing can be improved.

[0089] Furthermore, in this embodiment, a mixing matrix corresponding to the image acquisition conditions is selected from mixing matrices generated based on previously acquired fluorescence images, and demixing is performed using the selected mixing matrix. As a result, a mixing matrix appropriate for the target fluorescence image is selected, thereby improving the separation accuracy of the acquired fluorescent pigment image.

[0090] Furthermore, in this embodiment, a corrected mixing matrix is ​​obtained based on the target fluorescence image and the retrieved mixing matrix. This corrects the previously obtained mixing matrix using the target fluorescence image, enabling demixing suitable for the target fluorescence image and further improving the separation accuracy of the obtained fluorochrome image.

[0091] Furthermore, in this embodiment, a fluorochrome image is generated using a corrected mixing matrix and a target fluorescence image. This allows for unmixing tailored to the target fluorescence image, further improving the separation accuracy of the resulting fluorochrome image. However, previously generated mixing matrices may contain errors due to the influence of noise in the previous fluorescence images used to estimate the mixing matrix. In this case, if the mixing matrix is ​​used as is for the target fluorescence image, the error in the fluorochrome image generated from the target fluorescence image containing noise of varying nature tends to increase. In contrast, this embodiment uses a mixing matrix corrected based on the target fluorescence image, thereby minimizing errors in the generated fluorochrome image.

[0092] exist Figure 11An example of a fluorescent dye image generated by the fluorescent dye image acquisition system 1 of this embodiment based on the target fluorescent image is shown in FIG. Figure 12 , an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 of this embodiment without performing correction of the mixing matrix is ​​shown. Figure 11 and Figure 12 The corresponding positions in represent the same fluorescent dye image. Thus, according to this embodiment, correction of the mixing matrix significantly improves the image quality of some fluorescent dye images (particularly the fluorescent dye image in the upper right).

[0093] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and can be modified or applied to other aspects within the scope of the spirit described in the claims.

[0094] For example, in the image acquisition device 3 of this embodiment, the first and second fluorescence images are acquired by attaching and detaching the wavelength information acquisition optical system 13. Alternatively, the image acquisition device 3 may be configured to include two cameras, each of which detects fluorescence transmitted through the wavelength information acquisition optical system 13 and fluorescence reflected by the wavelength information acquisition optical system 13, thereby acquiring fluorescence images under two or more different optical conditions.

[0095] Alternatively, the image acquisition device 3 may include a color camera, such as an RGB color camera, and acquire a reference fluorescence image and a target fluorescence image based on image data for each wavelength characteristic outputted from the color camera. In this case, fluorescence images under two or more different optical conditions are acquired. Alternatively, the image acquisition device 3 may include a hyperspectral camera and acquire fluorescence images based on image data for each wavelength characteristic outputted from the hyperspectral camera. In this case, fluorescence images under two or more different optical conditions are acquired.

[0096] Furthermore, when the information retrieval unit 201 sees the mixing matrix, the matrix correction unit 204 of the image processing device 5 functions to correct the mixing matrix A. Alternatively, the matrix correction unit 204 of the image processing device 5 may always function to correct the mixing matrix A regardless of the retrieval result of the mixing matrix by the information retrieval unit 201.

[0097] exist Figure 13 2 shows the functional configuration of an image processing device 5A according to a modified example. Image processing device 5A differs from image processing device 5 in that it includes an image correction unit 204A and an image output unit 205A in place of the matrix correction unit 204 and the image generation unit 205, and has the function of correcting the fluorescent dye matrix data X instead of correcting the mixing matrix.

[0098] The image correction unit 204A unmixes the C target fluorescence images directly using the mixing matrix A0 extracted from the mixing matrix A retrieved by the information retrieval unit 201, thereby generating the fluorescent pigment matrix data X. Then, the image correction unit 204A generates the fluorescent pigment matrix data X corrected using the matrix data Y2 and the mixing matrix A0. C .

[0099] For example, the corrected fluorochrome matrix data X C The image correction unit 204A can obtain the following equation by setting the mixing matrix A0 and the fluorescent dye matrix data X generated initially as initial values:

[0100] L=||Y2-A2X|| 2

[0101] The calculated Euclidean distance is minimized between the mixing matrix A2 and the fluorescent dye matrix data X. Here, an equation with a regularization term added, such as the following equation, may be used instead of the above equation.

[0102] L=||Y2-A2X|| 2 +λ(|Δ x X|+|Δ y X|)

[0103] This regularization term is a regularization term for the Total Variation Loss, and is a term that calculates the L1 norm of the differentials in the x and y directions, using the matrix data X as an image. Furthermore, the above equation can be combined with a regularization term related to the mixing matrix A.

[0104] The image output unit 205A outputs the fluorescent dye matrix data X corrected by the image correction unit 204A. C K fluorescent dye images are regenerated and the regenerated K fluorescent dye images are output.

[0105] According to the above-described modification, a fluorescent dye image corrected based on a mixing matrix and a target fluorescent image acquired in the past can be obtained, and the separation accuracy of the acquired fluorescent dye image can be further improved.

[0106] Furthermore, as another modified example, the image correction unit 204A of the image processing device 5A may obtain the corrected fluorescent dye matrix data X by using either of the following two methods.

[0107] As a first method, the image correction unit 204A unmixes the C target fluorescence images directly using the mixing matrix A0 extracted from the mixing matrix A retrieved by the information retrieval unit 201, thereby generating the fluorescent pigment matrix data X. The image correction unit 204A then inputs the generated fluorescent pigment matrix data X into the learned inference model and obtains the output of the learned inference model as the corrected fluorescent pigment matrix data X. C Furthermore, when learning the inference model, the image correction unit 204A can construct a learned inference model by using, as training data, a combination of the fluorescent pigment matrix data X derived by unmixing in the matrix acquisition unit 203 and the fluorescent pigment matrix data X obtained by directly unmixing C target fluorescence images using the mixing matrix A0.

[0108] According to this first method, the generated fluorescent dye image can be input into the inference model to obtain a fluorescent dye image corrected based on the output of the inference model. As a result, the separation accuracy of the acquired fluorescent dye image can be further improved.

[0109] Furthermore, as a second method, the image correction unit 204A inputs the matrix data Y2 based on the C target fluorescence images and the mixing matrix A0 extracted from the mixing matrix A retrieved by the information retrieval unit 201 into the learned inference model, and obtains the output of the learned inference model as the corrected fluorescent pigment matrix data X C In addition, when learning the inference model, the image correction unit 204A can construct a learned inference model by using a combination of the fluorescent pigment matrix data X derived by demixing in the matrix acquisition unit 203, the mixing matrix A0 extracted from the retrieved mixing matrix A, and the matrix data Y2 based on C object fluorescence images as training data.

[0110] According to the second method, the target fluorescence image and the mixing matrix can be input into the inference model to obtain a corrected fluorescence dye image based on the output of the inference model. As a result, the separation accuracy of the obtained fluorescence dye image can be further improved.

[0111] In the first aspect, preferably, the image separation information is stored in association with image acquisition conditions when acquiring multiple reference fluorescence images, and in the selection step, the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions when acquiring the target fluorescence image is selected. In the second aspect, preferably, the storage unit stores the image separation information in association with the image acquisition conditions when acquiring multiple reference fluorescence images, and the image processing device selects the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions when acquiring the target fluorescence image. Thus, the image separation information corresponding to the image acquisition conditions is selected from the image separation information generated based on previously acquired fluorescence images, and demixing is performed using the selected image separation information. As a result, the separation accuracy of the acquired separated images can be improved.

[0112] Furthermore, in the first aspect, preferably, the generating step acquires corrected image separation information, i.e., image separation information corrected based on the target fluorescence image and the selected image separation information. Furthermore, in the second aspect, preferably, the image processing device acquires corrected image separation information, i.e., image separation information corrected based on the target fluorescence image and the selected image separation information, when generating the fluorescence separation image. This corrects previously acquired image separation information using the target fluorescence image, thereby enabling demixing suitable for the target fluorescence image and further improving the separation accuracy of the acquired separation image.

[0113] Furthermore, in the first aspect, preferably, the generating step generates the fluorescence separated image using the corrected image separation information and the target fluorescence image. Furthermore, in the second aspect, preferably, the image processing device generates the fluorescence separated image using the corrected image separation information and the target fluorescence image. In this case, demixing suitable for the target fluorescence image can be performed, further improving the separation accuracy of the obtained separated image.

[0114] Furthermore, in the first aspect, preferably, in the generating step, the fluorescence separation image is corrected based on the target fluorescence image and the image separation information. Furthermore, in the second aspect, preferably, in generating the fluorescence separation image, the image processing device corrects the fluorescence separation image based on the target fluorescence image and the image separation information. In this case, a fluorescence separation image corrected based on previously acquired image separation information and the target fluorescence image can be obtained, further improving the separation accuracy of the acquired separation image.

[0115] Furthermore, in the first aspect, preferably, in the generating step, the fluorescence separation image generated based on the target fluorescence image and the image separation information is input into a learned inference model, and the output of the learned inference model is obtained as a corrected fluorescence separation image. Furthermore, in the second aspect, preferably, in generating the fluorescence separation image, the image processing device inputs the fluorescence separation image generated based on the target fluorescence image and the image separation information into a learned inference model, and the output of the learned inference model is obtained as a corrected fluorescence separation image. In this case, the generated fluorescence separation image can be input into the inference model, resulting in a fluorescence separation image corrected based on the output of the inference model. As a result, the separation accuracy of the obtained separation image can be further improved.

[0116] Furthermore, in the first aspect described above, preferably, in the generating step, the target fluorescence image and image separation information are input into a learned inference model, and the output of the learned inference model is obtained as a corrected fluorescence separation image. Furthermore, preferably, in generating the fluorescence separation image, the image processing device inputs the target fluorescence image and image separation information into the learned inference model, and the output of the learned inference model is obtained as a corrected fluorescence separation image. In this manner, the target fluorescence image and image separation information can be input into the inference model to obtain a fluorescence separation image corrected based on the output of the inference model. As a result, the separation accuracy of the obtained separation image can be further improved.

[0117] Furthermore, in the first aspect, the plurality of optical states preferably includes at least a first optical state in which fluorescence is transmitted through the optical filter, and a second optical state in which fluorescence is not transmitted through the optical filter. Furthermore, in the second aspect, the plurality of optical states preferably includes at least a first optical state in which fluorescence is transmitted through the optical filter, and a second optical state in which fluorescence is not transmitted through the optical filter. This allows fluorescence wavelength information to be obtained based on fluorescence images in the two optical states, and allows image separation information to be stored, resulting from clustering of pixel groups in the fluorescence image based on this wavelength information. This reduces overall computation time while enabling the acquisition of highly accurate separated images.

[0118] Furthermore, in the first aspect, preferably, in the generating step, the fluorescence separation image is generated based on the image separation information and the fluorescence image of the subject acquired in the second optical state. Furthermore, in the second aspect, preferably, in generating the fluorescence separation image, the image processing device generates the fluorescence separation image based on the image separation information and the fluorescence image of the subject acquired in the second optical state. With the above configuration, by performing demixing using the separation image of the subject acquired in the second optical state, a highly accurate separation image can be obtained.

[0119] The separation image acquisition method of the embodiment is [1] "a separation image acquisition method comprising: a selection step of selecting desired image separation information from a storage unit, the storage unit storing a plurality of image separation information for obtaining a fluorescence separation image after the fluorescence image is separated, which is obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation step of irradiating a sample with excitation light of a plurality of wavelengths; an acquisition step of acquiring, for each of a plurality of fluorescences generated from the sample by the excitation light of the plurality of wavelengths, a target fluorescence image in at least one of the plurality of optical states via a fluorescence filter unit having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions; and a generation step of generating, for each of the plurality of fluorescences, a fluorescence separation image based on the target fluorescence image acquired in the acquisition step and the image separation information selected in the selection step."

[0120] The separation image acquisition method of the embodiment may also be, [2] "according to the separation image acquisition method described in the above [1], wherein the image separation information is stored in association with the image acquisition conditions when acquiring the multiple reference fluorescence images, and in the selection step, the image separation information associated with the image acquisition conditions is selected, and the image acquisition conditions correspond to the image acquisition conditions when acquiring the object fluorescence image."

[0121] The separation image acquisition method of the embodiment may also be, [3] "the separation image acquisition method described in [1] or [2] above, wherein, in the generation step, image separation information corrected based on the object fluorescence image and the selected image separation information, i.e., corrected image separation information, is obtained."

[0122] The separation image acquisition method of the embodiment may also be, [4] "according to the separation image acquisition method described in the above [3], wherein, in the generation step, the fluorescence separation image is generated using the correction image separation information and the object fluorescence image."

[0123] The separation image acquisition method of the embodiment may also be, [5] "according to the separation image acquisition method described in [1] or [2] above, wherein, in the generation step, the fluorescence separation image is corrected based on the object fluorescence image and the image separation information."

[0124] The separation image acquisition method of the embodiment may also be, [6] "according to the separation image acquisition method described in [1] or [2] above, wherein, in the generation step, the fluorescence separation image generated based on the object fluorescence image and the image separation information is input into the learned inference model, and the output of the learned inference model is obtained as the corrected fluorescence separation image."

[0125] The separation image acquisition method of the embodiment may also be, [7] "according to the separation image acquisition method described in [1] or [2] above, wherein, in the generation step, the object fluorescence image and the image separation information are input into the learned inference model, and the output of the learned inference model is obtained as the corrected fluorescence separation image."

[0126] The separation image acquisition method of the embodiment may also be, [8] "a separation image acquisition method according to any one of the above [1] to [7], wherein the multiple optical states include at least a state in which the fluorescence is allowed to pass through the optical filter, i.e., a first optical state, and a state in which the fluorescence is not allowed to pass through the optical filter, i.e., a second optical state."

[0127] The separation image acquisition method of the embodiment may also be, [9] "according to the separation image acquisition method described in the above [8], wherein, in the generation step, a fluorescence separation image is generated based on the image separation information and the object fluorescence image obtained in the second optical state."

[0128] The separation image acquisition device of the embodiment is,

[10] "a separation image acquisition device comprising: a storage unit storing image separation information for obtaining a fluorescence separation image after the separation fluorescence image is obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device irradiating a sample with excitation light of a plurality of wavelength distributions; an image acquisition device acquiring, for each of a plurality of fluorescences generated from the sample by the excitation light of a plurality of wavelengths, a target fluorescence image in at least one of the plurality of optical states via a fluorescence filter unit having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions; and an image processing device processing the fluorescence image, the image processing device selecting desired image separation information from the plurality of image separation information stored in the storage unit, and generating a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired in the image acquisition device and the selected image separation information".

[0129] The separation image acquisition device of the embodiment may also be,

[11] "the separation image acquisition device according to the above-mentioned

[10] , wherein the storage unit stores the image separation information in association with the image acquisition conditions when acquiring the plurality of reference fluorescence images, and the image processing device selects the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions when acquiring the object fluorescence image."

[0130] The separation image acquisition device of the embodiment may also be,

[12] "the separation image acquisition device according to the above-mentioned

[10] or

[11] , wherein the image processing device obtains image separation information corrected based on the object fluorescence image and the selected image separation information, i.e., corrected image separation information, in the generation of the fluorescence separation image."

[0131] The separation image acquisition device of the embodiment may also be,

[13] "the separation image acquisition device according to the above-mentioned

[12] , wherein the image processing device generates the fluorescence separation image using the correction image separation information and the object fluorescence image in generating the fluorescence separation image."

[0132] The separation image acquisition device of the embodiment may also be,

[14] "the separation image acquisition device according to the above-mentioned

[10] or

[11] , wherein the image processing device corrects the fluorescence separation image based on the object fluorescence image and the image separation information during the generation of the fluorescence separation image."

[0133] The separation image acquisition device of the embodiment may also be,

[15] "the separation image acquisition device described in

[10] or

[11] above, wherein the image processing device inputs the fluorescence separation image generated based on the object fluorescence image and the image separation information into the learned inference model in generating the fluorescence separation image, and obtains the output of the learned inference model as the corrected fluorescence separation image."

[0134] The separation image acquisition device of the embodiment may also be,

[16] "the separation image acquisition device according to the above-mentioned

[10] or

[11] , wherein the image processing device inputs the object fluorescence image and the image separation information into the learned inference model in generating the fluorescence separation image, and obtains the output of the learned inference model as the corrected fluorescence separation image."

[0135] The separation image acquisition device of the embodiment may also be,

[17] "a separation image acquisition device according to any one of the above

[10] to

[16] , wherein the multiple optical states include at least a state in which the fluorescence passes through the optical filter, i.e., a first optical state, and a state in which the fluorescence does not pass through the optical filter, i.e., a second optical state."

[0136] The separation image acquisition device of the embodiment may also be,

[18] "the separation image acquisition device according to the above-mentioned

[17] , wherein the image processing device generates a fluorescence separation image based on the image separation information and the object fluorescence image obtained in the second optical state in the generation of the fluorescence separation image."

[0137] The separation image acquisition program of the embodiment is,

[19] "a separation image acquisition program for generating a fluorescence separation image for each of a plurality of fluorescences generated from the sample by the excitation light of a plurality of wavelengths, based on a plurality of reference fluorescence images in a plurality of optical states with different wavelength characteristics obtained through a fluorescence filter unit having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions, by irradiating each of the sample with excitation light of a plurality of wavelength distributions, and causing a computer to execute: a storage process for storing a plurality of image separation information obtained based on the plurality of reference fluorescence images for obtaining a fluorescence separation image after separating the fluorescence; a selection process for selecting desired image separation information from the plurality of stored image separation information; and a generation process for generating a fluorescence separation image for each of the plurality of fluorescences based on the object fluorescence image obtained in at least one of the plurality of optical states and the selected image separation information".

[0138] Explanation of symbols

[0139] 1... Fluorescent dye image acquisition system, 3... Image acquisition device, 5, 5A... Image processing device, 7... Excitation light source (irradiation device), 9a... Light source side filter group, 9b... Camera side filter group (fluorescence filter unit), 11... Dichroic mirror, 15... Camera (image acquisition device), 13... Wavelength information acquisition optical system (optical filter), 201... Information retrieval unit (selection unit), 202... Image acquisition unit, 203... Matrix acquisition unit, 204... Matrix correction unit, 204A... Image correction unit, 205... Image generation unit, 205A... Image output unit, 206... Image separation information storage unit (storage unit), C1, C2, C3... Fluorescent dye, GC1 to GC6... Fluorescent image, PGr 01 ~PGr 03 , PGr1~PGr6…pixel group, S…sample.

Claims

1. A method for obtaining a separated image, wherein: have: a selecting step of selecting desired image separation information from a storage unit storing a plurality of image separation information for obtaining a fluorescence separation image after separating the fluorescence image, the image separation information being obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation step of irradiating the sample with each of a plurality of wavelengths of excitation light; an acquisition step of acquiring, for each of a plurality of fluorescences generated from the sample by the excitation light of the plurality of wavelengths, a fluorescence image of the object in at least one of the plurality of optical states via a fluorescence filter portion having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions; as well as A generating step of generating a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired in the acquiring step and the image separation information selected in the selecting step.

2. The separation image acquisition method according to claim 1, wherein: The image separation information is stored in association with image acquisition conditions when acquiring the plurality of reference fluorescence images. In the selecting step, the image separation information associated with an image acquisition condition corresponding to an image acquisition condition when acquiring the target fluorescence image is selected.

3. The separation image acquisition method according to claim 1 or 2, wherein: In the generating step, corrected image separation information is acquired, which is image separation information corrected based on the target fluorescence image and the selected image separation information.

4. The separation image acquisition method according to claim 3, wherein: In the generating step, the fluorescence separated image is generated using the corrected image separation information and the target fluorescence image.

5. The separation image acquisition method according to claim 1 or 2, wherein: In the generating step, the fluorescence separated image is corrected based on the target fluorescence image and the image separation information.

6. The separation image acquisition method according to claim 1 or 2, wherein: In the generating step, the fluorescence separation image generated based on the target fluorescence image and the image separation information is input to a learned inference model, and an output of the learned inference model is obtained as the corrected fluorescence separation image.

7. The separation image acquisition method according to claim 1 or 2, wherein: In the generating step, the object fluorescence image and the image separation information are input into a learned inference model, and an output of the learned inference model is obtained as the corrected fluorescence separation image.

8. The separation image acquisition method according to any one of claims 1 to 7, wherein: The plurality of optical states include at least a first optical state in which the fluorescence is transmitted through the optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter.

9. The method for obtaining a separated image according to claim 8, wherein: In the generating step, a fluorescence separation image is generated based on the image separation information and the subject fluorescence image acquired in the second optical state.

10. A separation image acquisition device, wherein: have: a storage unit storing image separation information for obtaining a fluorescence separation image after separation of the fluorescence image, which is obtained based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device for irradiating the sample with each of a plurality of excitation lights having a wavelength distribution; an image acquisition device for acquiring, for each of a plurality of fluorescences generated from the sample by the excitation light of a plurality of wavelengths, a fluorescence image of the subject in at least one of the plurality of optical states via a fluorescence filter portion having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions; as well as an image processing device for processing the fluorescent image, The image processing device, selecting desired image separation information from a plurality of image separation information stored in the storage unit, For each of the plurality of fluorescences, a fluorescence separation image is generated based on the target fluorescence image acquired by the image acquisition device and the selected image separation information.

11. The separation image acquisition device according to claim 10, wherein: The storage unit stores the image separation information in association with image acquisition conditions when acquiring the plurality of reference fluorescence images. The image processing device selects the image separation information associated with an image acquisition condition corresponding to an image acquisition condition when acquiring the target fluorescence image.

12. The separation image acquisition device according to claim 10 or 11, wherein: The image processing device acquires corrected image separation information, which is image separation information corrected based on the target fluorescence image and the selected image separation information, when generating the fluorescence separation image.

13. The separation image acquisition device according to claim 12, wherein: The image processing device generates the fluorescence separation image by using the corrected image separation information and the target fluorescence image.

14. The separation image acquisition device according to claim 10 or 11, wherein: The image processing device corrects the fluorescence separation image based on the target fluorescence image and the image separation information when generating the fluorescence separation image.

15. The separation image acquisition device according to claim 10 or 11, wherein: In generating the fluorescence separation image, the image processing device inputs the fluorescence separation image generated based on the target fluorescence image and the image separation information into a learned inference model, and obtains an output of the learned inference model as the corrected fluorescence separation image.

16. The separation image acquisition device according to claim 10 or 11, wherein: In generating the fluorescence separation image, the image processing device inputs the target fluorescence image and the image separation information into a learned inference model, and obtains an output of the learned inference model as the corrected fluorescence separation image.

17. The separation image acquisition device according to any one of claims 10 to 16, wherein: The plurality of optical states include at least a first optical state in which the fluorescence is transmitted through the optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter.

18. The separation image acquisition device according to claim 17, wherein: The image processing device generates the fluorescence separation image based on the image separation information and the target fluorescence image acquired in the second optical state.

19. A separation image acquisition program, wherein: A separation image acquisition program for generating a fluorescence separation image for each of a plurality of fluorescences generated from the sample by the excitation light of a plurality of wavelengths by irradiating the sample with each of the excitation light of a plurality of wavelengths, based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, obtained through a fluorescence filter portion having a plurality of reflection wavelength regions and a plurality of transmission wavelength regions. Causes the computer to execute: a storage process for storing a plurality of image separation information obtained based on a plurality of reference fluorescence images and used to obtain a fluorescence separation image after fluorescence separation; a selection process of selecting desired image separation information from the plurality of stored image separation information; as well as A generation process is performed to generate a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired in at least one of the plurality of optical states and the selected image separation information.

Images