Generation method, generation device, and program

Abstract

To improve throughput when obtaining separated images by unmixing, and to improve separation accuracy. [Solution] The method for generating a separated image includes a selection step of selecting a desired library A from DB6 storing a plurality of libraries A for obtaining pre-correction dye images X obtained by separating fluorescence images of different wavelengths; an acquisition step of irradiating a sample S with each of excitation light of a plurality of wavelengths and acquiring an observation image Y relating to the fluorescence generated from the sample S for each of the excitation light of the plurality of wavelengths; a first conversion step of converting the plurality of observation images Y acquired in the acquisition step into a plurality of pre-correction dye images X based on the library A selected in the selection step; and an update process of the plurality of pre-correction dye images X obtained in the first conversion step so that they are highly independent of each other, thereby generating an updated separation matrix P. k and generating a generation step of generating:

Description

[Technical Field]

[0001] One aspect of the present disclosure relates to a generation method, a generation device, and a program. [Background technology]

[0002] Conventionally, a multi-staining technique has been used to simultaneously stain multiple substances within a sample, such as biological tissue. To observe the substances within a multi-stained sample, excitation light is irradiated onto the sample to acquire a fluorescence image. For example, Non-Patent Document 1 below discloses the application of a nonnegative matrix factorization (NMF) technique to blind unmix fluorescence images obtained by observing fluorescence in multiple wavelength ranges to obtain separate images of each substance within the sample. Furthermore, Non-Patent Document 2 below discloses a fluorescence image unmixing technique that involves clustering the fluorescence image, extracting the maximum fluorescence intensity value for each clustered pixel group, and generating a separate image based on this maximum value. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Binjie Qin et al., “Target / Background ClassificationRegularized Nonnegative Matrix Factorization for Fluorescence Unmixing”, IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL.65, NO.4, APRIL2016 [Non-patent document 2] Tristan D. McRae et al., “Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning”, PLOSONE, December 2,2019 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional methods described above, the calculation time for unmixing increases depending on the type of excitation light or the number of fluorescent dyes to be observed, which tends to result in a decrease in throughput. In addition, there is a demand for improving the accuracy of the generated separated images (separation accuracy).

[0005] One aspect of the present disclosure has been made in consideration of such problems, and aims to provide a method, a device, and a program for generating separated images that can improve throughput when obtaining separated images by unmixing and also improve separation accuracy. [Means for solving the problem]

[0006] (1) A method for generating a separated image according to one aspect of the present disclosure includes a selection step of selecting desired image separation information from a storage unit that stores multiple pieces of image separation information for obtaining separated fluorescence images obtained by separating fluorescence images of different wavelengths; an acquisition step of irradiating a sample with each of multiple wavelengths of excitation light and obtaining, for each of the multiple wavelengths of excitation light, an observation fluorescence image of the fluorescence generated from the sample; a first conversion step of converting the multiple observation fluorescence images obtained in the acquisition step into multiple separated fluorescence images based on the image separation information selected in the selection step; and a generation step of performing an update process on the multiple separated fluorescence images obtained in the first conversion step so that each image has high independence, thereby generating updated separation information.

[0007] In a method for generating a separated image according to an aspect of the present disclosure, a plurality of observation fluorescence images acquired for each of a plurality of wavelengths of excitation light are converted into a plurality of separated fluorescence images based on image separation information for obtaining the separated fluorescence images. In this manner, by performing unmixing using pre-stored image separation information, the amount of calculation required for unmixing can be reduced. As a result, throughput can be improved when acquiring separated images through unmixing. Furthermore, in a method for generating a separated image according to an aspect of the present disclosure, an update process is performed to increase the independence of each of the plurality of separated fluorescence images, and updated separation information is generated. By generating and utilizing such updated separation information, it becomes possible, for example, to correct the image separation information based on the updated separation information so that the independence of the separated fluorescence images is increased, thereby improving the separation accuracy of the separated fluorescence images.

[0008] (2) The generation method of (1) above may further include a first correction step of correcting the image separation information stored in the storage unit based on the updated separation information generated in the generation step, and acquiring corrected separation information, which is the image separation information after the correction. By correcting the image separation information and acquiring the corrected separation information in this way, it is possible to appropriately acquire information that allows for separating a plurality of separated fluorescent images with high accuracy.

[0009] (3) The generation method of (2) above may further include a second conversion step of converting the plurality of observation fluorescent images into a plurality of separated fluorescent images based on the corrected separation information acquired in the first correction step. With this configuration, a plurality of separated fluorescent images separated with high accuracy can be acquired.

[0010] (4) Any one of the generating methods (1) to (3) above may further include a second correction step of acquiring correction information for correcting the image separation information stored in the storage unit based on the updated separation information generated in the generating step. In this way, by acquiring the correction information for correcting the image separation information from the updated separation information, it becomes possible to correct the image separation information using the correction information as needed, thereby improving the separation accuracy of the separated fluorescent images.

[0011] (5) Any one of the generating methods (1) to (4) above may further include a third conversion step of converting the plurality of observation fluorescent images into a plurality of separated fluorescent images based on the updated separation information generated in the generating step and the image separation information selected in the selecting step. With this configuration, a plurality of separated fluorescent images separated with high accuracy can be obtained.

[0012] (6) In any one of the generation methods (1) to (5) above, the generation step may involve updating the multiple separated fluorescent images to generate updated separation information so as to minimize mutual information between the multiple separated fluorescent images. By updating the multiple separated fluorescent images to generate updated separation information so as to minimize mutual information, which is an index for measuring the degree of dependence between two random variables, the independence of the separated fluorescent images can be appropriately increased, and the separation accuracy of the separated fluorescent images can be improved.

[0013] (7) Any one of the generating methods (1) to (6) above may further include a first correction step of correcting the image separation information stored in the storage unit based on the updated separation information generated in the generating step to obtain corrected separation information, which is the image separation information after the correction, and the generating step may generate the updated separation information so that the corrected separation information does not become a negative value. This prevents the corrected separation information from becoming a negative value, and further improves the separation accuracy of the separated fluorescence image using the corrected separation information.

[0014] (8) Any one of the generating methods (1) to (7) above may further include a first correction step of correcting the image separation information stored in the storage unit based on the updated separation information generated in the generating step to obtain corrected separation information, which is the corrected image separation information, and the generating step may generate the updated separation information so that the difference between the corrected separation information and the image separation information falls within a predetermined range. This prevents the corrected separation information from deviating significantly from the image separation information before correction, and further improves the separation accuracy of the separated fluorescent image using the corrected separation information.

[0015] (9) In any one of the generating methods (1) to (8) above, the generating step may generate data-reduced images by reducing some of the data for each of the plurality of separated fluorescence images, and perform an update process to increase the independence of each of the plurality of data-reduced images, thereby generating updated separation information. Such a configuration can reduce the processing load.

[0016] (10) A method for generating a separated image according to one aspect of the present disclosure includes a selection step of selecting desired image separation information from a storage unit that stores multiple pieces of image separation information for obtaining separated fluorescence images obtained by separating fluorescence images of different wavelengths; an acquisition step of irradiating a sample with each of multiple wavelengths of excitation light to obtain an observation fluorescence image relating to the fluorescence generated from the sample for each of the multiple wavelengths of excitation light; a generation step of generating updated separation information based on the image separation information selected in the selection step and the multiple observation fluorescence images acquired in the acquisition step so that the multiple separated fluorescence images converted from the multiple observation fluorescence images have high independence from each other; and a conversion step of converting the multiple observation fluorescence images into multiple separated fluorescence images based on the updated separation information generated in the generation step and the image separation information selected in the selection step.

[0017] In a method for generating a separated image according to an aspect of the present disclosure, a plurality of observation fluorescence images acquired for each of a plurality of wavelengths of excitation light are converted into a plurality of separated fluorescence images based on image separation information for obtaining the separated fluorescence images. In this manner, by performing unmixing using pre-stored image separation information, the amount of calculation required for unmixing can be reduced. As a result, throughput can be improved when obtaining separated images through unmixing. Furthermore, in a method for generating a separated image according to an aspect of the present disclosure, conversion into separated fluorescence images is performed based on updated separation information and image separation information that are generated so that the independence of each of the plurality of separated fluorescence images is high, thereby improving the separation accuracy of the separated fluorescence images.

[0018] (11) A separated image generating device according to one aspect of the present disclosure includes a memory unit that stores multiple pieces of image separation information for obtaining separated fluorescence images obtained by separating fluorescence images of different wavelengths; an image acquisition device that irradiates a sample with excitation light of multiple wavelengths and acquires, for each of the excitation light of the multiple wavelengths, an observation fluorescence image relating to the fluorescence generated from the sample; and an image processing device that processes the multiple observation fluorescence images to generate multiple separated fluorescence images, wherein the image processing device is configured to select desired image separation information from the memory unit, convert the multiple observation fluorescence images acquired by the image acquisition device into multiple separated fluorescence images based on the selected image separation information, and perform an update process on the converted multiple separated fluorescence images so that each image has high independence, thereby generating updated separation information.

[0019] (12) A separated image generating device according to one aspect of the present disclosure includes a memory unit that stores multiple pieces of image separation information for obtaining separated fluorescence images obtained by separating fluorescence images of different wavelengths; an image acquisition device that irradiates a sample with excitation light of multiple wavelengths and acquires, for each of the excitation light of the multiple wavelengths, an observation fluorescence image relating to the fluorescence generated from the sample; and an image processing device that processes the multiple observation fluorescence images to generate multiple separated fluorescence images, wherein the image processing device is configured to: select desired image separation information from the memory unit; generate updated separation information based on the selected image separation information and the multiple observation fluorescence images acquired by the image acquisition device so that the multiple separated fluorescence images converted from the multiple observation fluorescence images have high independence from each other; and convert the multiple observation fluorescence images into multiple separated fluorescence images based on the generated updated separation information and the selected image separation information.

[0020] (13) A program according to one aspect of the present disclosure causes a computer to perform the following steps: a selection process for selecting desired image separation information from a storage unit that stores multiple pieces of image separation information for obtaining separated fluorescence images obtained by separating fluorescence images of different wavelengths; a conversion process for converting multiple observation fluorescence images acquired by an image acquisition device that irradiates a sample with excitation light of multiple wavelengths and acquires observation fluorescence images of fluorescence generated from the sample for each of the excitation light of the multiple wavelengths into multiple separated fluorescence images based on the image separation information selected in the selection process; and a generation process for updating the converted separated fluorescence images so that they are more independent of each other, thereby generating updated separation information.

[0021] (14) A program according to one aspect of the present disclosure causes a computer to perform the following steps: a selection process of selecting desired image separation information from a storage unit that stores multiple pieces of image separation information for obtaining separated fluorescence images obtained by separating fluorescence images of different wavelengths; a generation process of generating updated separation information based on the selected image separation information and multiple observation fluorescence images acquired by an image acquisition device that irradiates a sample with excitation light of multiple wavelengths and acquires, for each of the excitation light of multiple wavelengths, observation fluorescence images relating to fluorescence generated from the sample, so that the independence of each of the multiple separated fluorescence images converted from the multiple observation fluorescence images is increased; and a conversion process of converting the multiple observation fluorescence images into multiple separated fluorescence images based on the generated updated separation information and the selected image separation information. [Effects of the Invention]

[0022] According to one aspect of the present disclosure, it is possible to improve the throughput when obtaining separated images by unmixing, and also improve the separation accuracy. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a schematic diagram illustrating the configuration of a fluorescent dye image acquisition system according to an embodiment. [Figure 2] FIG. 2 is a perspective view showing the configuration of the image acquisition device of FIG. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of the image processing device in FIG. 1. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the image processing device in FIG. 1. [Figure 5] FIG. 10 is a diagram illustrating unmixing. [Figure 6] FIG. 1 is a diagram illustrating an overview of library estimation. [Figure 7] FIG. 1 is a diagram illustrating an overview of library correction. [Figure 8] FIG. 1 is a diagram illustrating an overview of library correction. [Figure 9] FIG. 10 is a diagram showing a processing pattern when library correction is performed after unmixing. [Figure 10] 10 is a flowchart showing a procedure for library correction processing. [Figure 11] FIG. 10 is a diagram showing an image of matrix data of a dye image. [Figure 12] FIG. 10 is a diagram illustrating mutual information. [Figure 13] 10 is a flowchart showing a procedure for processing related to mutual information. [Figure 14] FIG. 10 is a diagram illustrating updating of mutual information. [Figure 15] FIG. 10 is a diagram illustrating a grid search in the mutual information update process. [Figure 16] 10 is a flowchart showing the procedure of a separation image generation method according to the present embodiment. [Figure 17] FIG. 1 is a diagram illustrating the implementation conditions of a comparative example and an example. [Figure 18] FIG. 10 is a diagram showing an unmixing result according to a comparative example. [Figure 19] FIG. 10 is a diagram showing an unmixing result according to the embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of independence evaluation using a correlation coefficient, among examples of independence evaluation other than mutual information. [Figure 21] FIG. 10 is a diagram illustrating an update process when independence is evaluated using mutual information, a correlation coefficient, or HSIC. [Figure 22] FIG. 10 is a diagram illustrating an update process when independence evaluation is performed by independent component analysis. [Figure 23] FIG. 10 is a diagram showing a comparison between the unmixing result based on the correlation coefficient and the unmixing result based on the mutual information. [Figure 24] FIG. 10 is a diagram showing a comparison between the unmixing results by HSIC and the unmixing results by mutual information. [Figure 25] FIG. 10 is a diagram showing a comparison between the unmixing results by HSIC and the unmixing results by mutual information. [Figure 26] FIG. 10 is a diagram showing a comparison between the unmixing result based on independent component analysis and the unmixing result based on mutual information. [Figure 27] 10A and 10B are diagrams illustrating an update process in library correction according to a modified example. [Figure 28] 10A and 10B are diagrams illustrating an update process in library correction according to a modified example. [Figure 29] FIG. 10 is a diagram showing a processing pattern when unmixing is performed after library correction. [Figure 30] FIG. 10 is a diagram illustrating library correction in which the user selects data to be used. [Figure 31] 10 is a flowchart showing the procedure of a separation image generating method according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description, the same elements or elements having the same functions will be denoted by the same reference numerals, and redundant description will be omitted.

[0025] FIG. 1 is a schematic diagram of a fluorescent dye image acquisition system 1, which is an apparatus for generating separated images according to an embodiment. The fluorescent dye image acquisition system 1 is an apparatus for generating fluorescent dye images (separated fluorescent images) for identifying the distribution of fluorescent dyes in a sample, such as biological tissue, to be observed. The images generated by the fluorescent dye image acquisition system 1 are used for purposes such as drug development, pathological diagnosis, and treatment method research through image analysis. Therefore, the fluorescent dye image acquisition system 1 is required to generate images capable of quantitatively identifying the distribution of many substances (fluorochromes) contained in the sample with high throughput. The fluorescent dye image acquisition system 1 includes an image acquisition device 3 that irradiates the sample S with excitation light and acquires images of the fluorescence generated in response to the irradiation, an image processing device 5 that processes the images acquired by the image acquisition device 3, and a DB (storage unit) 6 that stores image separation information for obtaining separated fluorescent images by separating fluorescent images of different wavelengths. The image acquisition device 3, the image processing device 5, and the DB 6 may be configured to transmit and receive information such as image data between them via wired or wireless communication, or may be configured to input and output information such as image data via a recording medium.

[0026] 2 is a perspective view showing the configuration of the image acquisition device 3 of FIG. 1. In FIG. 2, the optical path of the excitation light is indicated by a dotted line with an arrow, and the optical path of the fluorescence is indicated by a solid line with an arrow. The image acquisition device 3 executes an acquisition step of irradiating the sample S with excitation light of each of multiple wavelengths and acquiring, for each of the excitation light of the multiple wavelengths, an observation image (observation fluorescence image) of the fluorescence generated from the sample S. The image acquisition device 3 is configured to include an excitation light source 7, a light source-side filter set 9a, a dichroic mirror 11, a camera-side filter set 9b, a wavelength information acquisition optical system 13, and a camera 15.

[0027] The excitation light source 7 is a light source capable of switching between and irradiating excitation light in multiple wavelength bands (wavelength distributions), such as a multicolor LED (Light Emitting Diode) light source, a light source consisting of multiple monochromatic laser light sources, or a light source combining a white light source and a wavelength-selective optical element. The light-source-side filter set 9a is provided on the optical path of the excitation light from the excitation light source 7 and is a multi-bandpass filter that transmits light in multiple predetermined wavelength bands. The transmission wavelength bands of this light-source-side filter set 9a are set according to the multiple wavelength bands of the excitation light that may be used. The dichroic mirror 11 is provided between the light-source-side filter set 9a and the sample S and is an optical element that reflects the excitation light toward the sample S and transmits the fluorescence emitted from the sample S in response. The camera-side filter set 9b is provided on the optical path of the fluorescence transmitted by the dichroic mirror 11 and is a multi-bandpass filter that transmits light in multiple predetermined wavelength bands. The transmission wavelength bands of this camera-side filter set 9b are set according to the wavelength bands of fluorescence generated by fluorescent dyes that may be contained in the sample S to be observed. In detail, the camera-side filter set 9b has, as its wavelength characteristics, transmission wavelength bands (transmission wavelength ranges) corresponding to multiple fluorescence wavelength bands, and reflection wavelength bands (reflection wavelength ranges) between these multiple transmission wavelength bands.

[0028] The wavelength information acquisition optical system 13 is detachably supported on the optical path of the fluorescence transmitted by the camera-side filter set 9b and is an optical system for acquiring wavelength information of the fluorescence. That is, the wavelength information acquisition optical system 13 is provided so as to be switchable between two states: a state in which it is disposed on the optical path of the fluorescence from the sample S (first optical state) and a state in which it is removed from the optical path of the fluorescence (second optical state). Note that the wavelength information acquisition optical system 13 may be any optical system capable of realizing a plurality of optical states with different wavelength characteristics, and may be, for example, a fluorescence filter set including two or more fluorescence filters with different wavelength characteristics. In the first optical state, the wavelength information acquisition optical system 13 transmits the fluorescence generated in the sample S and transmitted through the dichroic mirror 11 and the camera-side filter set 9b toward the camera 15 with predetermined wavelength characteristics. In a second optical state different from the first optical state, the wavelength information acquisition optical system 13 allows the fluorescence transmitted through the dichroic mirror 11 and the camera-side filter set 9b to enter the camera 15 in its original optical state (without transmitting through the wavelength information acquisition optical system 13). For example, an LRG (Linear Reflectance Gradient on the wavelength axis) filter (also referred to as a gradient filter) having wavelength characteristics of transmittance such that the transmittance increases linearly as the wavelength increases may be used as wavelength information acquisition optical system 13. Alternatively, an LRG filter (gradient filter) having wavelength characteristics of transmittance such that the transmittance decreases linearly as the wavelength decreases may be used as wavelength information acquisition optical system 13. Wavelength information acquisition optical system 13 using such a gradient filter can cause fluorescence to enter camera 15 with two different wavelength transmission characteristics.

[0029] The camera 15 is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer greater than or equal to 2, for example, 2048 × 2048). When the wavelength information acquisition optical system 13 is switched onto the optical path of the fluorescence (in the first optical state), the camera captures the fluorescence that has passed through the wavelength information acquisition optical system 13 to acquire a first reference fluorescence image. When the wavelength information acquisition optical system 13 is removed from the optical path of the fluorescence (in the second optical state), the camera 15 captures the fluorescence that has not passed through the wavelength information acquisition optical system 13 to acquire a second reference fluorescence image. The camera 15 acquires a first reference fluorescence image and a second reference fluorescence image for each of the multiple fluorescences generated from the sample S by the excitation light of the multiple wavelength bands irradiated by the excitation light source 7. The camera 15 outputs the acquired first reference fluorescence image and second reference fluorescence image to the image processing device 5 via communication or a recording medium.

[0030] Next, the configuration of the image processing device 5 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 5, and Fig. 4 is a block diagram showing the functional configuration of the image processing device 5.

[0031] 3, the image processing device 5 is physically a computer or the like including a processor such as a CPU (Central Processing Unit) 101, a recording medium such as a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, all of which are electrically connected to one another. Note that the image processing device 5 may include input / output devices such as a display, a keyboard, a mouse, a touch panel display, or a data recording device such as a hard disk drive or semiconductor memory. The image processing device 5 may also be composed of multiple computers.

[0032] FIG. 5 is a diagram illustrating unmixing. As shown in FIG. 5(a), for example, it is assumed that a sample S contains three types of dyes ("Dye 1," "Dye 2," and "Dye 3"). Then, as shown in FIG. 5(b), it is assumed that an image acquisition device 3 irradiates the sample S with excitation light of three wavelength bands ("blue," "green," and "red"), and acquires each observed image Y. Here, as shown in FIGS. 5(b) and 5(c), fluorescence associated with multiple dyes is mixed in each observed image Y. The observed image Y obtained when irradiated with excitation light in the blue wavelength band contains the fluorescence of "pigment 1" (1.0) and the fluorescence of "pigment 2" (0.2), the observed image Y obtained when irradiated with excitation light in the green wavelength band contains the fluorescence of "pigment 1" (0.1), "pigment 2" (1.1), and "pigment 3" (0.3), and the observed image Y obtained when irradiated with excitation light in the red wavelength band contains the fluorescence of "pigment 2" (0.1) and "pigment 3" (0.9). If the degree of mixing of each dye can be calculated with high accuracy in advance, unmixing can be performed based on a library of mixing matrices to obtain the separated fluorescence image, dye image X (Figure 5(d)).

[0033] FIG. 6 is a diagram illustrating an overview of library estimation. As described above, if a library consisting of a mixing matrix has been calculated, it is possible to acquire a dye image X, which is a separated fluorescent image, based on the observed image Y and the library, taking into consideration the degree of mixing of each dye. Here, an example of a method for estimating (acquiring) such a library will be described. Note that the library estimation method is not limited to the following method.

[0034] Assume now that pre-specified C (C is an integer greater than or equal to 2) first reference fluorescence images and C second reference fluorescence images of sample S have been acquired by the image acquisition device 3. The C first reference fluorescence images are fluorescence images composed of N pixels generated by irradiating sample S with excitation light of C wavelength bands, respectively, in the first optical state and capturing the fluorescence emitted from sample S in response. The C second reference fluorescence images are fluorescence images composed of N pixels generated by irradiating sample S with excitation light of C wavelength bands, respectively, in the second optical state and capturing the fluorescence emitted from sample S in response. In this case, the number C (the number C of wavelength bands of excitation light irradiated onto sample S) is pre-specified to be equal to or greater than the maximum number of fluorescent dyes that can be contained in sample S. In FIG. 6, the multiple observation images labeled "with gradient filter" are first reference fluorescence images, and the multiple observation images labeled "without gradient filter" are second reference fluorescence images.

[0035] Clustering (division into regions by dye) is performed on these images. First, for each of C sets of first and second reference fluorescent images, the ratio of the fluorescence intensity (brightness value) of one fluorescent image to the fluorescence intensity of the other fluorescent image is calculated to estimate the centroid fluorescent wavelength, which indicates the centroid of the fluorescence wavelength distribution. The estimated centroid fluorescent wavelength is acquired as wavelength information related to the fluorescence wavelength. Clustering is performed on N pixels constituting the C first and C second reference fluorescent images based on the C first and C second reference fluorescent images and the acquired wavelength information. A first clustering is performed to cluster the N pixels of the second reference fluorescent image into C pixel groups based on distribution information of the fluorescence intensity for each excitation light in the C wavelength bands. In particular, pixels having the same wavelength band of excitation light with the greatest fluorescence intensity are clustered into the same pixel group.

[0036] Furthermore, a second clustering is performed to further cluster the C pixel groups clustered by the first clustering into L pixel groups (L is an integer between 2 and N-1). Here, the number L of pixel groups to be clustered is preset as a parameter stored in the image processing device 5, corresponding to, for example, the number of types of fluorescent dyes that can be present in the sample S. The number L of pixel groups may be determined according to the type of excitation light or the number C of wavelength distributions of the excitation light, or may be determined independently of the type of excitation light or the number C of wavelength distributions of the excitation light. For each of the C pixel groups clustered by the first clustering, a centroid fluorescence wavelength estimated for the wavelength band of the excitation light corresponding to that pixel group is identified. More specifically, wavelength information is acquired for a pixel group clustered as having the highest absorptance in a certain wavelength band, and the centroid fluorescence wavelength is identified based on the acquired wavelength information. In this case, the wavelength information is acquired using the average value of the fluorescence intensity of the pixel groups in a set of the first and second reference fluorescent images obtained corresponding to that wavelength band. Furthermore, by determining the distance (closeness of values) between the centroid fluorescence wavelengths identified for each of the C pixel groups, the C pixel groups are clustered into L pixel groups. In the example shown in Figure 6, the C pixel groups are clustered into "Cluster 1," "Cluster 2," and "Cluster 3."

[0037] After clustering, the degree of mixing for each dye is calculated by mathematical processing, and a mixing matrix, library A, is estimated. Based on the L cluster matrices obtained for sample S, a mixing matrix (library A) is obtained for generating K fluorescent dye images showing the respective distributions of K (K is an integer between 2 and C) fluorescent dyes from C first reference fluorescent images and C second reference fluorescent images. Generally, according to the nonnegative matrix factorization (NMF) calculation method, the relationship between matrix data (observed image Y), which is an observation matrix, and fluorescent dye matrix data (dye image X), in which K fluorescent dye images are arranged in parallel in one dimension for each pixel, is expressed using the mixing matrix (library A) as follows: Y = AX (1) Here, Y is matrix data of C × 2 rows and N columns, A is matrix data of C × 2 rows and K columns, and X is matrix data of K rows and N columns. Conversely, once the value of the mixing matrix of library A is determined, the fluorescent dye matrix data (dye image X) can be calculated by the inverse matrix A of the mixing matrix (library A). -1 and matrix data (observed image Y), the following equation (2): X=A -1 Y···(2) (This process is called unmixing.)

[0038] DB6 (see FIG. 1) stores a plurality of libraries A which are image separation information for obtaining dye images X, which are separated fluorescent images obtained by separating fluorescent images of different wavelengths.

[0039] FIG. 7 is a diagram illustrating an overview of library correction. In the method for generating separated images according to this embodiment, the accuracy of unmixing is improved by correcting a library A prepared in advance and stored in the DB 6. As shown in FIG. 7(a), for example, suppose that a sample S contains three types of dyes ("Dye 1," "Dye 2," and "Dye 3"). As shown in FIG. 7(b), the image acquisition device 3 irradiates the sample S with excitation light of six wavelength bands, thereby acquiring respective observed images Y. In this case, based on the library A (library A before correction) read from the DB 6, the multiple observed images Y are converted into multiple separated fluorescent images, dye images X (specifically, dye images X before correction) (FIG. 7(c)). That is, unmixing is first performed based on the library A before correction. In the method for generating separated images according to this embodiment, for example, library A is corrected, and unmixing is performed based on the corrected library A' (hereinafter referred to as corrected library A'), thereby enabling conversion to a more highly accurate corrected dye image X' (hereinafter referred to as corrected dye image X') (FIG. 7(d)). The library is corrected so as to increase the independence of the multiple observed images Y (details will be described later).

[0040] Figure 8 is a diagram for explaining the outline of library correction in more detail. When excitation light of six wavelength bands is irradiated onto a sample S and observation images Y are acquired for each (Figure 8(a)), the inverse matrix A of the mixing matrix (library A) is -1 and matrix data (observed image Y), and a plurality of pre-correction dye images X are obtained (FIG. 8(b)). Then, the plurality of pre-correction dye images X are updated so that they are more independent of each other, and a separation matrix P k (Updated separation information) is calculated. Separation matrix P k As will be described later, the calculated separation matrix P may be used to correct the library A to generate a corrected library A', or to generate a correction matrix B (correction information) for correcting the library A, or to correct (update) the dye image X before correction to derive the corrected dye image X'. k The corrected dye image X' is repeatedly derived based on the (updated separation information) and the final corrected dye image X' is calculated (FIGS. 8(c) and 8(d)).

[0041] The following formula (3) holds true for the dye images X and X' before and after updating. X´=P k P k-1 …P2P1X···(3) P=P k P k-1 ...If P2P1X, the following equations (4) and (5) hold. X´=PX (4) X=P -1 X´···(5) When equation (5) is substituted into equation (2), which is the unmixing relation, the following equation (6) is obtained. Y=AP -1 X´···(6) A´=AP -1 Then, Y=A'X', and A' is the corrected library.

[0042] An example of the processing of library correction and image updating (generation of corrected dye image X') after unmixing as described above will be described with reference to FIG. 9. As shown in FIG. 9, first, unmixing is performed based on the pre-correction library A and observed image Y to obtain pre-correction dye image X. As the processing of library correction and image updating after unmixing, for example, "Pattern 1" and "Pattern 2" shown in FIG. 9 are conceivable. However, the processing is not limited to the processing shown in FIG. 9. In Pattern 1, library correction is performed on the pre-correction dye image X so that each image is highly independent, and a separation matrix P k (updated separation information) is calculated, and the separation matrix P k Based on the above, a correction matrix B (correction information) for correcting the library A before correction stored in DB6 is acquired, and a corrected dye image X' is also acquired. In this case, for example, the correction matrix B and the corrected dye image X' are output. In pattern 2, the library correction is performed on the dye image X before correction so that each image is highly independent, and the separation matrix P k (updated separation information) is calculated, and the separation matrix P k Based on this, the pre-correction library A stored in DB6 is corrected to obtain a post-correction library A' (post-correction separation information) and a post-correction dye image X'. In this case, for example, the post-correction library A' and the post-correction dye image X' are output.

[0043] Returning to Fig. 4, the image processing device 5 includes, as functional components, a selection unit 201, an acquisition unit 202, a conversion unit 203, a generation unit 204, and a correction unit 205. The functional units of the image processing device 5 shown in Fig. 4 are realized by loading a program (a separated image acquisition program according to the embodiment) onto hardware such as the CPU 101 and RAM 102 shown in Fig. 3, thereby operating the communication module 104, the input / output module 106, etc., and reading and writing data from and to the RAM 102 under the control of the CPU 101. The CPU 101 of the image processing device 5 executes this computer program to cause the functional units of Fig. 4 to function, and sequentially executes processes corresponding to a separated image generation method described below.

[0044] The CPU 101 may be a standalone piece of hardware, or may be implemented in a programmable logic device such as an FPGA, like a software processor. The RAM and ROM may also be standalone pieces of hardware, or may be built into a programmable logic device such as an FPGA. All of the various data required to execute this computer program and all of the various data generated by the execution of this computer program are stored in built-in memories such as the ROM 103 and RAM 102, or in storage media such as a hard disk drive. The functions of the functional components of the image processing device 5 will be described in detail below.

[0045] The selection unit 201 executes a selection step of selecting a desired library A from DB6, which stores multiple libraries A, each of which is image separation information for obtaining a dye image X, which is a separated fluorescent image obtained by separating fluorescent images of different wavelengths. Library A includes a mixing matrix. The selection unit 201 searches (selects) a desired mixing matrix (library A) from the data stored in DB6 based on, for example, the image acquisition conditions used when acquiring a first fluorescent image or a second fluorescent image of sample S, information (estimation processing information) related to the estimation process of a mixing matrix (library A, image separation information) based on the first fluorescent image and the second fluorescent image, and storage setting information used when saving the mixing matrix to DB6. The selection unit 201 may set the image acquisition conditions, estimation processing information, and storage setting information used for the search based on information input by a user via the input / output module 106 of the image processing device 5, information transmitted from an external device such as the image acquisition device 3, or information accessed from the RAM 102, ROM 103, or the like of the image processing device 5.

[0046] The image acquisition conditions include the type of sample S, barcode information attached to the sample S, the type of fluorescent dye contained in the sample S, the date and time of staining the sample S, the date and time of measurement of the sample S, the type of excitation light, the intensity of each excitation light, the type or number of filters (such as the presence or absence of a gradient filter), the type of camera, the type or magnification of the objective lens, the exposure time of each excitation light, the ambient temperature, and the data of the acquired fluorescent image itself.The estimation processing information includes the version information of the software used for the estimation processing, the algorithm, parameters, or options used for the estimation processing, the time required for the estimation processing, etc.The saved setting information includes the saved user name, project name, assay name, mixing matrix name, tag information (user-selected keywords, social media hashtags), etc.

[0047] The DB6 stores the acquired mixing matrix in association with the image acquisition conditions used to acquire the multiple fluorescence images (e.g., the first fluorescence image and the second fluorescence image) that formed the basis of the mixing matrix, estimation processing information related to the estimation processing used to acquire the mixing matrix, and storage setting information used to save the mixing matrix. The selection unit 201 searches for (selects) a mixing matrix associated with information that matches or corresponds (is similar to) a search key including the image acquisition conditions, estimation processing information, or storage setting information set by a user, an external device, or the like. The selection unit 201 then transfers the search results for the mixing matrix (library A) to the conversion unit 203. At this time, the selection unit 201 searches for a mixing matrix using at least one item from the image acquisition conditions, estimation processing information, and storage setting information as a search key. The selection unit 201 searches for a mixing matrix using a search key that includes at least one item of the image acquisition conditions.

[0048] The acquisition unit 202 acquires a plurality of observation images Y from the image acquisition device 3. The plurality of observation images Y are a plurality of observation fluorescence images acquired by the image acquisition device 3, and are images relating to fluorescence generated from the sample S by irradiating the sample S with excitation light of a plurality of wavelengths. The acquisition unit 202 passes the acquired plurality of observation images Y to the conversion unit 203.

[0049] The conversion unit 203 executes a first conversion step (performs unmixing) of converting the multiple observed images Y acquired by the acquisition unit 202 into multiple pre-corrected dye images X based on the library A selected by the selection unit 201. The unmixing is executed based on, for example, the above-described formula (2), and the pre-corrected dye images X are acquired from the library A and the observed images Y. The conversion unit 203 passes the multiple pre-corrected dye images X to the generation unit 204.

[0050] The generation unit 204 performs an update process on the plurality of pre-correction dye images X obtained in the first conversion step by the conversion unit 203 so that the independence of each of the dye images X is increased, and generates a separation matrix P kThe generation unit 204 performs an update process on the plurality of pre-correction dye images X so as to minimize the mutual information between the plurality of pre-correction dye images X, and generates a separation matrix P k The generation unit 204 generates the generated separation matrix P k is passed to the correction unit 205.

[0051] The correction unit 205 corrects the separation matrix P generated by the generation unit 204. k Based on the corrected library A' acquired by the correction unit 205, the conversion unit 203 executes a first correction step of correcting the uncorrected library A stored in the DB6 and acquiring the corrected library A' (details will be described later). Then, based on the corrected library A' acquired by the correction unit 205, the conversion unit 203 executes a second conversion step of converting the multiple observed images Y into multiple corrected dye images X' (performing unmixing). Such second unmixing may be carried out, for example, by substituting X in the above-mentioned formula (2) for the corrected dye images X', A -1 After correction, the inverse matrix of library A' is A' -1 and a corrected dye image X' is obtained from the corrected library A' and the observed image Y.

[0052] The correction unit 205 corrects the separation matrix P generated by the generation unit 204. k The conversion unit 203 may execute a second correction step of acquiring a correction matrix B (correction information) for correcting the uncorrected library A stored in the DB 6 based on the above (details will be described later). k and the pre-correction library A selected by the selection unit 201, a third conversion step may be performed to convert the plurality of observed images Y into a plurality of post-correction dye images X′.

[0053] Fig. 10 is a flowchart showing an example of the procedure for library correction processing. As shown in Fig. 10, an uncorrected dye image X is obtained from library A and observed image Y based on the above-mentioned formula (2) (step S1). Now, as shown in Fig. 11, assume that dye image X is represented by a size C × N (C is the number of dye images, and N is the number of data points (pixels)), and is a matrix in which C dye images are each arranged horizontally in one dimension. In Fig. 11, data for N pixels for each of three dye images X is arranged, with C = 3.

[0054] Next, the separation matrix P k is initialized with an identity matrix (step S2). Here, the separation matrix P k is the identity matrix. Among the C data (columns), the dye image X i ,X j In step S3, the separation matrix P k is shown, and i and j are selected in the range of i=1,2,3, j=1,2,3 (where i≠j).

[0055] Next, multiple dye images X i ,X j The updated separation matrix P k A coefficient α related to the derivation of is calculated (step S4).

[0056] FIG. 12 is a diagram illustrating mutual information. FIG. 12(a) is a diagram illustrating the distribution of brightness values ​​when the mutual information between two images ("Image 1" and "Image 2") is large, and FIG. 12(b) is a diagram illustrating the distribution of brightness values ​​when the mutual information between two images ("Image 1" and "Image 2") is small. Mutual information indicates a measure of the interdependence (correlation) between two information sources. Mutual information here is an index that measures the dependency between two images using, for example, a probability distribution (histogram). A large amount of mutual information between images indicates low independence of each image. A small amount of mutual information between images indicates high independence of each image. A mutual information of 0 indicates that each image is completely independent.

[0057] In the example shown in Figures 12(a) and 12(b), histograms (horizontal axis: brightness value, vertical axis: number of pixels) of "Image 1" and "Image 2" are shown, and the distribution of brightness values ​​of "Image 1" and "Image 2" is shown in the lower left. In the graph showing the distribution of brightness values ​​(graph at the lower left), the horizontal axis represents the brightness value of "Image 1" and the vertical axis represents the brightness value of "Image 2," and the distribution of brightness values ​​for each pixel in "Image 1" and "Image 2" is shown. In the example shown in Figure 12(a), the correlation between the brightness values ​​of each pixel in "Image 1" and "Image 2" is extremely high, indicating that the mutual information between "Image 1" and "Image 2" is large. On the other hand, in the example shown in Figure 12(b), the correlation between the brightness values ​​of each pixel in "Image 1" and "Image 2" is low, indicating that the mutual information between "Image 1" and "Image 2" is small.

[0058] 13 is a flowchart showing the procedure of the process related to the mutual information. i ,X j Assume that the dye image X is input (step S11). i ,X j Minimizing the mutual information of X i and X j -αX i This means minimizing the mutual information of the brightness values ​​between the two. First, an initial value α0 is set for α. Here, the initial value α0 is set in advance to an appropriate value within the search range of α.

[0059] Next, for the brightness value of each pixel, X j -αX i is calculated (step S13). Since the initial value α0 is set as the value of α, X j -α0X i Then, a histogram is created (step S14). Specifically, as shown in FIG. 14(a), the horizontal axis is X j The luminance value of each pixel is the vertical axis, X j -αX i A two-dimensional histogram is created as the brightness value of each pixel.

[0060] Next, the mutual information I(α) is calculated (step S15). The mutual information I(α) is calculated based on the following equation (7): j The center coordinates of each bin in the two-dimensional histogram are x bin , X j -αX i The center coordinates of each bin in the two-dimensional histogram are y bin Also, X j and X j -αX i The simultaneous probability of p(x bin ,y bin ) and X j The marginal probability of p(x bin ), X j -αX i The marginal probability of p(y bin )

[0061]

number

[0062] Next, the value of α is updated (step S16), and X in step S13 is executed again. j -αX i For example, in the example shown in FIG. 14, the mutual information I(α=α0=0 in this case) is first calculated as I(α=α0)=0.66 (see FIG. 14(a)), and then the value of α is updated and I(α=0.2)=0.49 is calculated (see FIG. 14(b)). Thereafter, the process of steps S13 to S16 is repeated while updating the value of α until the search for α is completed, and the α that minimizes the mutual information I(α) is calculated. min is determined (step S17). The value of α may be updated randomly (random search), or may be searched for sequentially while shifting the value at equal intervals (grid search). In the case of grid search, as shown in FIG. 15, the value of α is shifted by a predetermined value (here, 0.05) to find the α that minimizes the mutual information I(α). min (Here, α min=0.15) is determined. Finally, the α that minimizes the mutual information I(α) is determined. min is output (step S18).

[0063] Returning to FIG. 10, in step S4, the mutual information I(α) is minimized. min Once calculated, the calculated α(α=α min ) to obtain the dye image X i ,X j The separation matrix P k (j, i) is updated (step S5). Specifically, the separation matrix P k (j,i) is updated. In the following equation (8), γ is the separation matrix P k Coefficient for updating (j,i), α is the calculated α min is. Separation matrix P k (j,i)=-γα (8)

[0064] The processes in steps S3 to S5 are repeated for all possible combinations of i and j. Then, the updated separating matrix P k The dye image is updated using the formula (9) below, and a corrected dye image X' is obtained (step S6). Corrected dye image X´=P k X···(9)

[0065] The acquisition of the corrected dye image X' is repeated until the value of the corrected dye image X' converges within a predetermined range or until a set number of times has been reached.

[0066] Finally, the correction matrix B and the corrected dye image X' are output (step S7). The correction matrix B is, for example, the updated separation matrix P k or separation matrix P k For example, P = P k P k-1 ...P2P1X, the correction matrix B may be calculated by the following equation (10) or (11). -1 is the corrected library A'. B=P -1 ···(10) B=AP -1 -A···(11) Although it has been explained here that the correction matrix B is calculated, if the correction matrix B is not calculated and the separation matrix P k Alternatively, only the separation matrix P k The corrected library A' may be calculated based on the above.

[0067] Next, an example of the procedure of the separation image generation method according to this embodiment will be described with reference to Fig. 16. Note that the procedure of the generation method shown in Fig. 16 is merely an example, and the present invention is not limited to this. That is, here, the updated separation matrix P k The following description will be given assuming that the library A is corrected based on the updated separation matrix P k The correction matrix B may be calculated based on the following.

[0068] As shown in Figure 16, first, a desired pre-correction library A is selected from DB6, which stores multiple pre-correction libraries A for obtaining pre-correction dye images X (separated fluorescent images) obtained by separating fluorescent images of different wavelengths (step S101, selection step).

[0069] Next, an observation image Y (observation fluorescence image) of fluorescence generated from the sample S by irradiating the sample S with each of the excitation light beams of the multiple wavelengths is acquired for each of the excitation light beams of the multiple wavelengths (step S102, acquisition step).

[0070] Next, based on the library A selected in the selection step, the plurality of observation images Y (observation fluorescent images) acquired in the acquisition step are converted into a plurality of pre-correction dye images X (step S103, first conversion step).

[0071] Next, the multiple uncorrected dye images X obtained in the first conversion step are updated so that they are more independent of each other, and the updated separation matrix P kis generated (step S104, generation step).

[0072] And the updated separation matrix P k Based on this, the uncorrected library A stored in DB6 is corrected to obtain a corrected library A' (step S105, first correction step). Furthermore, based on the corrected library A', a plurality of observed images Y (observed fluorescent images) are converted into a plurality of corrected dye images X' (step S106, second conversion step).

[0073] Next, the effects of the separation image generation method according to this embodiment will be described.

[0074] The method for generating a separated image according to this embodiment includes a selection step of selecting a desired library A from DB6 storing a plurality of libraries A (image separation information) for obtaining pre-correction dye images X (separated fluorescent images) obtained by separating fluorescent images of different wavelengths; an acquisition step of irradiating a sample S with each of the excitation lights of a plurality of wavelengths and acquiring, for each of the excitation lights of the plurality of wavelengths, an observation image Y (observation fluorescent image) relating to the fluorescence generated from the sample S; a first conversion step of converting the plurality of observation images Y acquired in the acquisition step into a plurality of pre-correction dye images X based on the library A selected in the selection step; and an update process of the plurality of pre-correction dye images X obtained in the first conversion step so that they are highly independent of each other, thereby obtaining an updated separation matrix P. k and a generating step of generating (updated separation information).

[0075] In the separation image generation method according to this embodiment, a plurality of observation images Y acquired for each of a plurality of wavelengths of excitation light are converted into a plurality of pre-correction dye images X based on a library A for obtaining a pre-correction dye image X. In this way, by performing unmixing using the pre-stored library A, the amount of calculation required for unmixing can be reduced. As a result, it is possible to improve the throughput when obtaining separation images by unmixing. Furthermore, in the separation image generation method according to this embodiment, an update process is performed so that the independence of each of the plurality of pre-correction dye images X is increased, and the separation matrix P k Such a separation matrix P k is generated and utilized, for example, the separation matrix P k Based on this, it becomes possible to correct library A so that the independence of dye image X before correction is increased, thereby improving the separation accuracy of separated fluorescent images.

[0076] The method for generating the separated image is as follows: k The method may further include a first correction step of correcting library A stored in DB6 based on the above, and acquiring corrected library A'. In this way, by correcting library A and acquiring corrected library A', it is possible to appropriately acquire information that allows multiple pre-correction dye images X to be separated with high accuracy.

[0077] The above-described method for generating separated images may further include a second conversion step of converting the plurality of observed images Y into a plurality of corrected dye images X' based on the corrected library A' acquired in the first correction step. With this configuration, a plurality of corrected dye images X' separated with high accuracy can be acquired.

[0078] 17 to 19, the separation accuracy of the separation image generation method (Example) according to this embodiment will be described in comparison with a comparative example. The Example here refers to a separation image generation method that obtains a corrected dye image X' via the second conversion step described above, while the Comparative Example refers to a separation image generation method that obtains a pre-correction dye image X via only the first conversion step. FIG. 17 is a diagram illustrating the implementation conditions for the Comparative Example and Example. As shown in FIG. 17(a), in the Comparative Example and Example, a sample containing fluorescent tape in four colors (red, orange, green, and blue) is irradiated with seven excitation lights as shown in FIG. 17(b), and an observation image Y is obtained for each sample. In the Comparative Example, the pre-correction dye image X is obtained without any correction of Library A, while in the Example, the corrected dye image X' is obtained using the corrected Library A'.

[0079] FIGS. 18(a) to 18(d) show the unmixing results (dye image X before correction) of the comparative example, and FIGS. 19(a) to 19(d) show the unmixing results (dye image X' after correction) of the example. As shown in FIGS. 18(a) to 18(d), in the unmixing results of the comparative example, the orange dye is slightly mixed with the green (FIG. 18(b)) and red (FIG. 18(d)). In contrast, as shown in FIGS. 19(a) to 19(d), in the unmixing results of the example, the orange dye is not mixed with the green (FIG. 19(b)) and red (FIG. 19(d)), improving the separation accuracy of each dye.

[0080] The method for generating the separated image is as follows: k The method may further include a second correction step of acquiring a correction matrix B for correcting the library A stored in the DB 6 based on the separation matrix P k By obtaining a correction matrix B for correcting library A from the matrix B, library A can be corrected using correction matrix B as needed, thereby improving separation accuracy.

[0081] The method for generating the separated image is as follows: k and a third conversion step of converting the plurality of observed images Y into a plurality of post-correction dye images X' based on the pre-correction library A selected in the selection step. With this configuration, a plurality of post-correction dye images X' separated with high accuracy can be obtained.

[0082] In the generation step of the separation image generation method, the plurality of pre-correction dye images X are updated so as to minimize the mutual information between the plurality of pre-correction dye images X, and the updated separation matrix P k The multiple uncorrected dye images X are updated so that the mutual information, which is an index for measuring the degree of dependence between two random variables, is minimized, and the updated separation matrix P k By generating the above formula, the independence of the dye images can be appropriately increased, and the separation accuracy can be improved.

[0083] Various embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments, and may be modified or applied to other things within the scope that does not change the gist of the claims.

[0084] For example, in the generation step, the separation matrix P after updating is generated so that the mutual information between the multiple uncorrected dye images X is minimized. k However, the present invention is not limited to this, and an index of independence evaluation other than mutual information may be used. Specifically, a correlation coefficient, HSIC (Hilbert-Schmidt Independence Criteria), or independent component analysis may be used as an index of independence evaluation of a plurality of pre-correction dye images X.

[0085] The correlation coefficient is, for example, a statistic related to second-order correlation (correlation between two variables), and may be, for example, Pearson's product-moment correlation coefficient. In the example shown in FIG. 20(a), for the correlation between two variables x and y, it can be determined that the closer the absolute value of the correlation coefficient is to 1, the lower the independence, and the closer the absolute value is to 0, the higher the independence. However, even if the correlation coefficient is 0, it does not necessarily mean that the independence is high. For example, if two variables x and y have a correlation as shown in FIG. 20(b), even if the absolute value of the correlation coefficient is 0, it does not mean that the independence is high. Independence evaluation using the correlation coefficient allows for high-speed processing.

[0086] HSIC is a statistic related to higher-order correlation. When HSIC=0, it can be determined that the data are completely independent. However, (number of samples) 2 This requires a large amount of memory and increases the amount of calculation. Independent component analysis allows for high-speed processing because it performs optimization calculations for all dye images simultaneously, rather than for each pair of dye images.

[0087] Fig. 21 is a diagram explaining the process of updating a separation matrix when independence evaluation is performed using mutual information, correlation coefficient, or HSIC. Fig. 22 is a diagram explaining the process of updating when independence evaluation is performed using independent component analysis. As shown in Fig. 21, in mutual information, correlation coefficient, and HSIC, coefficients are calculated between two pieces of data (dye images X before correction) to perform the update process, and an updated separation matrix is ​​calculated. As shown in Fig. 22, in independent component analysis, all images (dye images X before correction) are processed together to calculate an updated separation matrix.

[0088] Figure 23 shows a comparison of the unmixing results using the correlation coefficient (Figure 23(a)) and the unmixing results using the mutual information (Figure 23(b)). Note that the results shown in Figures 23 to 26 are the unmixing results when seven colors of excitation light were irradiated onto a sample containing fluorescent tape of four colors (red, orange, green, and blue), as in the experiment described in Figure 17. As shown in Figure 23(a), unmixing using the correlation coefficient could be performed quickly, but the separation accuracy was worse than the unmixing results using the mutual information (Figure 23(b)).

[0089] Fig. 24 shows a comparison of the unmixing results using HSIC (Fig. 24(a)) and the unmixing results using mutual information (Fig. 24(b)). As shown in Fig. 24(a), unmixing using HSIC was able to achieve separation accuracy as high as that using mutual information, but the processing time was extremely long at 39 minutes. In this regard, for example, as shown in Fig. 25(a), by reducing the number of samples (to 1 / 16 in this case), it was possible to reduce the processing time to 12 seconds, which is the same as that using mutual information, while maintaining a certain degree of separation accuracy.

[0090] Fig. 26 shows a comparison between the unmixing results based on independent component analysis (Fig. 26(a)) and the unmixing results based on mutual information (Fig. 26(b)). As shown in Fig. 26(a), unmixing based on independent component analysis achieved high separation accuracy and an extremely short processing time of 0.14 seconds.

[0091] 27 and 28 are diagrams illustrating the update process in library correction according to a modified example. In the example shown in FIG. 27, of the multiple pre-correction dye images X shown in FIG. 27(a), corrected dye images X' shown in FIG. 27(b) are acquired for only some of the dye images X. For example, in the example shown in FIG. 27, the dye image of "Dye 3" has sufficiently high accuracy (it is separated from the other dyes), so it is not targeted for acquisition of corrected dye images X'. The user may select the dyes that require updating, so that corrected dye images X' may be acquired for only some of the dye images X. In this case, as shown in FIG. 27(c), the separation matrix P k The update range is also limited to the areas corresponding to "Pigment 1" and "Pigment 2," enabling high-speed processing.

[0092] In the example shown in FIG. 28, a case is assumed in which it is clear that "pigment 3" has leaked into "pigment 1" and "pigment 2" (and there is no leak between "pigment 1" and "pigment 2"), as shown in FIG. 28(a). In this case, the user may select only the dye images of "pigment 1" and "pigment 2" into which "pigment 3" has leaked, thereby obtaining a corrected dye image X'. In this case, as shown in FIG. 28(c), the separation matrix P k The update range is also limited to the parts of "Pigment 1," "Pigment 2," and "Pigment 3" that correspond to each other, enabling high-speed processing.

[0093] In addition, in the generation step, the separation matrix P is set so that the corrected library A' does not become a negative value. k This prevents the corrected library A' from becoming a negative value and the unmixing result from becoming ∞, thereby further improving the accuracy of dye image separation using the corrected library A'.

[0094] In the generation step, the separation matrix P is calculated so that the difference between the corrected library A' and the uncorrected library A falls within a predetermined range. kThis prevents the corrected library A' from deviating significantly from the uncorrected library A, and can further improve the accuracy of dye image separation using the corrected library A'.

[0095] Furthermore, although the embodiment has been described as performing library correction and image updating after unmixing, this is not limiting, and unmixing may be performed after library correction. That is, the method of generating separated images includes a selection step of selecting a desired pre-correction library A from DB6, which stores a plurality of pre-correction libraries A for obtaining dye images obtained by separating fluorescence images of different wavelengths, an acquisition step of irradiating a sample S with each of a plurality of excitation lights of a plurality of wavelengths and acquiring an observation image Y relating to the fluorescence generated from the sample S for each of the excitation lights of a plurality of wavelengths, and a separation matrix P based on the pre-correction library A selected in the selection step and the plurality of observation images Y acquired in the acquisition step, so as to increase the independence of each of the plurality of dye images converted from the plurality of observation images Y. k and a generation step for generating the separation matrix P k and a conversion step of converting the plurality of observed images Y into a plurality of dye images based on the uncorrected library A selected in the selection step.

[0096] FIG. 29 is a diagram showing processing patterns when unmixing is performed after library correction. As the processing of library correction before unmixing, for example, "Pattern 1," "Pattern 2," and "Pattern 3" shown in FIG. 29 are conceivable. Note that the processing is not limited to these. In Pattern 1, a separation matrix P is set based on the pre-correction library A and the observed image Y so that the independence of each of the multiple dye images converted from the multiple observed images Y is increased. k is calculated, and the separation matrix P kIn pattern 2, the uncorrected library A stored in DB6 is corrected based on the uncorrected library A and the observed image Y, and a separation matrix P is calculated based on the uncorrected library A and the observed image Y so that the independence of each of the multiple dye images converted from the multiple observed images Y is increased. k is calculated, and the separation matrix P k In pattern 3, a correction matrix B for correcting the uncorrected library A stored in DB6 is acquired based on the uncorrected library A and the observed image Y. -1 Y), library correction is performed to increase the independence of each, and the separation matrix P k is calculated, and the separation matrix P k Based on this, a correction matrix B for correcting the uncorrected library A stored in the DB 6 is acquired. Then, in the unmixing process, for example, the separation matrix P k A corrected dye image X' is obtained based on the corrected library A' based on the above and the observed image Y.

[0097] In addition, for example, in the library correction before unmixing shown in FIG. 29, the observed image Y used may be one whose data has been reduced by, for example, selecting the data to be used by the user, or by thinning, binning, clustering, etc. In this case, the observed image Y only needs to have enough data points to perform independence evaluation. Also, as shown in FIG. 30, for example, the user may select a range common to each dye image (such as an area with spillover), thereby performing data reduction and efficient processing. That is, in the generation step, a data-reduced image is generated by reducing some of the data for each of the multiple dye images, and an update process is performed so that the independence of each of the multiple data-reduced images is increased, and a separation matrix P k According to this configuration, the processing load can be reduced.

[0098] Fig. 31 is a flowchart showing the procedure for a method for generating separated images according to a modified example, specifically the procedure for performing unmixing after library correction. As shown in Fig. 31, first, a desired pre-correction library A is selected from DB6, which stores multiple pre-correction libraries A for obtaining pre-correction dye images X (separated fluorescent images) obtained by separating fluorescent images of different wavelengths (step S201).

[0099] Next, an observation image Y (observation fluorescence image) of fluorescence generated from the sample S by irradiating the sample S with each of the excitation light beams of the multiple wavelengths is acquired for each of the excitation light beams of the multiple wavelengths (step S202).

[0100] Next, based on the library A selected in the selection step and the observed image Y (observed fluorescent image), the updated separation matrix P k is generated (step S203).

[0101] And the updated separation matrix P k Based on this, the uncorrected library A stored in DB6 is corrected to obtain a corrected library A' (step S204). Finally, based on the corrected library A', a plurality of observed images Y (observed fluorescent images) are converted into a plurality of corrected dye images X' (step S205). [Explanation of symbols]

[0102] 1... Fluorescent dye image acquisition system (generation device), 3... Image acquisition device, 5... Image processing device, A... Library (image separation information), A'... Corrected library (corrected separation information), B... Correction matrix (correction information), I... Mutual information, P k ...separation matrix (updated separation information), S... sample, X... dye image (separated fluorescent image), X'... corrected dye image (separated fluorescent image), Y... observed image (observed fluorescent image).

Claims

1. a selection step of selecting desired image separation information from a storage unit storing a plurality of pieces of image separation information for obtaining separated fluorescent images obtained by separating fluorescent images of different wavelengths; an acquisition step of irradiating a sample with each of excitation light beams having a plurality of wavelengths and acquiring an observation fluorescence image of fluorescence generated from the sample for each of the excitation light beams having the plurality of wavelengths; a first conversion step of converting the plurality of observation fluorescent images acquired in the acquisition step into a plurality of separated fluorescent images based on the image separation information selected in the selection step; a generating step of performing an update process on the plurality of separated fluorescence images obtained in the first converting step so as to increase the independence of each of the separated fluorescence images, and generating updated separation information; A method for generating a separated image comprising:

2. The generation method according to claim 1, further comprising a first correction step of correcting the image separation information stored in the memory unit based on the updated separation information generated in the generation step, and obtaining corrected separation information, which is the image separation information after the correction.

3. The generating method according to claim 2 , further comprising a second conversion step of converting the plurality of observation fluorescent images into a plurality of separated fluorescent images based on the corrected separation information acquired in the first correction step.

4. The generation method according to claim 1 , further comprising a second correction step of acquiring correction information for correcting the image separation information stored in the memory unit based on the updated separation information generated in the generation step.

5. 2. The generating method according to claim 1, further comprising a third conversion step of converting the plurality of observation fluorescence images into a plurality of separated fluorescence images based on the updated separation information generated in the generating step and the image separation information selected in the selecting step.

6. The generating method according to any one of claims 1 to 5, wherein the generating step performs an update process on the plurality of separated fluorescence images to generate the updated separation information so as to minimize mutual information between the plurality of separated fluorescence images.

7. a first correction step of correcting the image separation information stored in the storage unit based on the updated separation information generated in the generation step, and acquiring corrected separation information that is the image separation information after the correction; The generating method according to claim 6 , wherein in said generating step, said updated separation information is generated so that said corrected separation information does not become a negative value.

8. a first correction step of correcting the image separation information stored in the storage unit based on the updated separation information generated in the generation step, and acquiring corrected separation information that is the image separation information after the correction; The generating method according to claim 6 , wherein in the generating step, the updated separation information is generated so that a difference between the corrected separation information and the image separation information falls within a predetermined range.

9. The generating method according to any one of claims 1 to 5, wherein the generating step generates a data-reduced image by reducing some of the data for each of the plurality of separated fluorescence images, performs an update process to increase the independence of each of the plurality of data-reduced images, and generates the updated separation information.

10. a selection step of selecting desired image separation information from a storage unit storing a plurality of pieces of image separation information for obtaining separated fluorescent images obtained by separating fluorescent images of different wavelengths; an acquisition step of irradiating a sample with each of excitation light beams having a plurality of wavelengths and acquiring an observation fluorescence image of fluorescence generated from the sample for each of the excitation light beams having the plurality of wavelengths; a generating step of generating updated separation information based on the image separation information selected in the selecting step and the plurality of observation fluorescent images acquired in the acquiring step, so that the independence of each of the plurality of separated fluorescent images converted from the plurality of observation fluorescent images is increased; a conversion step of converting the plurality of observation fluorescent images into a plurality of separated fluorescent images based on the updated separation information generated in the generation step and the image separation information selected in the selection step; A method for generating a separated image comprising:

11. a storage unit that stores a plurality of pieces of image separation information for obtaining separated fluorescent images obtained by separating fluorescent images of different wavelengths; an image acquisition device that irradiates a sample with excitation light of each of a plurality of wavelengths and acquires an observation fluorescence image of fluorescence generated from the sample for each of the excitation light of the plurality of wavelengths; an image processing device that processes the plurality of observation fluorescent images to generate a plurality of separated fluorescent images; The image processing device includes: selecting the desired image separation information from the storage unit; converting the plurality of observation fluorescent images acquired by the image acquisition device into a plurality of separated fluorescent images based on the selected image separation information; performing an update process on the converted separated fluorescent images so that the images are more independent of each other, thereby generating updated separated information; A device for generating a separated image configured to perform the following:

12. a storage unit that stores a plurality of pieces of image separation information for obtaining separated fluorescent images obtained by separating fluorescent images of different wavelengths; an image acquisition device that irradiates a sample with excitation light of each of a plurality of wavelengths and acquires an observation fluorescence image of fluorescence generated from the sample for each of the excitation light of the plurality of wavelengths; an image processing device that processes the plurality of observation fluorescent images to generate a plurality of separated fluorescent images; The image processing device includes: selecting the desired image separation information from the storage unit; generating updated separation information based on the selected image separation information and the plurality of observation fluorescent images acquired by the image acquisition device so that the independence of each of the plurality of separated fluorescent images converted from the plurality of observation fluorescent images is increased; converting the plurality of observation fluorescent images into a plurality of separated fluorescent images based on the generated updated separation information and the selected image separation information; A device for generating a separated image configured to perform the following:

13. For computers, a selection process for selecting desired image separation information from a storage unit that stores a plurality of pieces of image separation information for obtaining separated fluorescent images obtained by separating fluorescent images of different wavelengths; a conversion process of converting a plurality of observation fluorescence images acquired by an image acquisition device that irradiates a sample with excitation light of a plurality of wavelengths and acquires observation fluorescence images relating to fluorescence generated from the sample for each of the excitation light of the plurality of wavelengths into a plurality of separated fluorescence images based on the image separation information selected in the selection process; a generation process of performing an update process on the converted separated fluorescent images so as to increase the independence of each of the separated fluorescent images, and generating updated separated information; A program that executes the following.

14. For computers, a selection process for selecting desired image separation information from a storage unit that stores a plurality of pieces of image separation information for obtaining separated fluorescent images obtained by separating fluorescent images of different wavelengths; a generation process for generating updated separation information based on the selected image separation information and a plurality of observation fluorescence images acquired by an image acquisition device that irradiates a sample with excitation light of a plurality of wavelengths and acquires, for each of the excitation light of a plurality of wavelengths, an observation fluorescence image relating to fluorescence generated from the sample, so that each of the plurality of separated fluorescence images converted from the plurality of observation fluorescence images has high independence; a conversion process for converting the plurality of observation fluorescent images into a plurality of separated fluorescent images based on the generated updated separation information and the selected image separation information; A program that executes the following.

Images