Information processing device, information processing method, program, microscope system, and analysis system

Abstract

The present invention improves analysis accuracy in a fluorescent observation. An information processing device according to an embodiment is provided with: a separation unit (132) that separates a fluorescent image observed from a specimen labeled with one or more fluorescent dyes into a fluorescent component image containing one or more fluorescent components and an autofluorescent component image containing one or more autofluorescent components; a generation unit (133) that generates an autofluorescent component correction image using a reference spectrum of each of one or more autofluorescent substances contained in the specimen and the autofluorescent component image; and a processing unit (130) that processes the fluorescent component image on the basis of the autofluorescent component correction image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, a program, a microscope system, and an analysis system. [Background technology]

[0002] In recent years, there is an increasing need to detect and evaluate the phenotype molecules of immune cells, which are classified into numerous subsets due to the development of cancer immunotherapy, etc. Multicolor imaging analysis methods using fluorescent dyes as markers are considered to be an effective means of elucidating the relationship between their localization and function. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 8200013 Summary of the Invention [Problem to be solved by the invention]

[0004] In general, when observing a stained tissue section stained with a fluorescent dye antibody, the accuracy of the measurement of the fluorescence intensity reflecting the amount of the antibody can be affected by autofluorescence, etc. Even when the autofluorescence component is removed from the captured image of the stained tissue section by color separation using the spectrum, it is impossible to completely eliminate the autofluorescence component remaining in the stained fluorescent image obtained by this color separation.

[0005] In this way, if autofluorescence components remain in a stained fluorescent image, problems arise such as difficulties in quantifying the amount of dye antibodies distributed in the area to be analyzed in the stained fluorescent image, which prevents improvements in analytical accuracy in fluorescent observation.

[0006] Therefore, the present disclosure has been made in consideration of the above circumstances, and proposes an information processing device, an information processing method, a program, a microscope system, and an analysis system that make it possible to improve the analysis accuracy in fluorescence observation. [Means for solving the problem]

[0007] An information processing device according to an embodiment of the present disclosure includes a separation unit that separates a fluorescence image observed from a specimen labeled with one or more fluorescent dyes into a fluorescence component image containing one or more fluorescent components and an autofluorescence component image containing one or more autofluorescence components, a generation unit that generates an autofluorescence component corrected image using reference spectra for each of the one or more autofluorescent substances contained in the specimen and the autofluorescence component image, and a processing unit that processes the fluorescence component image based on the autofluorescence component corrected image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment. [Diagram 2] 5A and 5B are diagrams showing specific examples of fluorescence spectra acquired by a fluorescence signal acquisition unit. [Diagram 3] FIG. 1 is a diagram illustrating an overview of nonnegative matrix factorization. [Figure 4] FIG. 1 is a diagram for explaining an overview of clustering. [Diagram 5] 1 is a block diagram showing an example of the configuration of a microscope system in the case where the information processing system according to the first embodiment is realized as a microscope system. [Figure 6] 4 is a flowchart showing an example of a basic operation of the information processing system according to the first embodiment. [Figure 7] 6 is a flowchart showing an example of an operation for generating a pseudo autofluorescence component image according to the first embodiment. [Figure 8] FIG. 7 is a diagram showing an example of an autofluorescence component image generated in step S103 of FIG. 6. [Figure 9]FIG. 7 is a diagram showing an example of an autofluorescence reference spectrum included in the specimen information acquired in step S102 in FIG. 6. [Figure 10] FIG. 8 is a diagram for explaining step S112 in FIG. 7. [Figure 11] FIG. 8 is a diagram for explaining step S114 in FIG. 7. [Figure 12] 11 is a flowchart for explaining an extraction process of an analysis target region and its analysis according to a comparative example of the first embodiment. [Figure 13] 13 is a schematic diagram for explaining the extraction process of the analysis target region and the analysis thereof according to the comparative example shown in FIG. 12. [Figure 14] 4 is a flowchart for explaining an extraction process of an analysis target region and its analysis according to the first embodiment. [Figure 15] 15 is a schematic diagram for explaining the extraction process of the analysis target region according to the embodiment shown in FIG. 14 and the analysis thereof. FIG. [Figure 16] 10 is a flowchart for explaining an extraction process of an analysis target region and its analysis according to the second embodiment. [Figure 17] 17A and 17B are schematic diagrams for explaining the extraction process of the analysis target region according to the present embodiment shown in FIG. 16 and the analysis thereof. [Figure 18] 13 is a flowchart for explaining a generation process of a spectral intensity ratio image and its analysis according to the third embodiment. [Figure 19] FIG. 13 is a diagram for explaining a first generation method according to the third embodiment. [Figure 20] FIG. 13 is a diagram for explaining a second generation method according to the third embodiment. [Figure 21] 13 is a flowchart for explaining a process for generating a fluorescence component image using machine learning according to a fourth embodiment. [Figure 22] FIG. 22 is a schematic diagram for explaining the generation process of the fluorescence component image using machine learning according to the embodiment shown in FIG. 21. [Figure 23] FIG. 2 is a diagram illustrating an example of a measurement system of the information processing system according to the embodiment. [Figure 24]FIG. 13 is a diagram for explaining a method for calculating the number of fluorescent molecules (or the number of antibodies) in one pixel according to an embodiment. [Diagram 25] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device according to each embodiment and modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] The explanation will be given in the following order. 0. Introduction 1. First embodiment 1.1. Configuration Example 1.2.Application example to microscope system 1.3.About least squares method 1.4. Nonnegative Matrix Factorization (NMF) 1.5. Characteristic Configuration of the First Embodiment 1.5.1. Basic operation example 1.5.2. Example of autofluorescence correction image generation flow 1.6. Extraction and analysis of the analysis area 1.7. Actions and Effects 2. Second embodiment 2.1. Extraction of the analysis area and its analysis 2.2. Actions and Effects 3. Third embodiment 3.1. Generation and analysis of spectral intensity ratio images 3.2.Spectral intensity image generation method 3.2.1 First generation method 3.2.2 Second generation method 3.3. Actions and Effects 4. Fourth embodiment 4.1.Generating fluorescent component images using machine learning 4.2. Actions and Effects 5. Example of measurement system configuration 6. Calculation method for the number of fluorescent molecules (or antibodies) 7. Hardware configuration example

[0011] <0. Introduction> In recent years, there is an increasing need to detect and evaluate the phenotype molecules of immune cells, which are classified into many subsets due to the development of cancer immunotherapy, etc. A multicolor imaging analysis method using fluorescent dyes as markers is considered to be an effective means for determining the relationship between their localization and function, and is also adopted in the following embodiments.

[0012] The following embodiment relates to image processing after color separation processing is performed on a multichannel image (number of pixels x wavelength channel (CH)) obtained by exciting an immunohistochemically stained pathological specimen section with excitation light of multiple wavelengths and capturing the image with an imaging device that separates the fluorescence.

[0013] In addition, the multichannel image in the present disclosure may include various images having a data cube structure composed of image data of multiple wavelength channels (however, a single wavelength channel is not excluded), such as a stained specimen image, a fluorescent component image, a fluorescent component-corrected image, an autofluorescent component image, and an autofluorescent component-corrected image, which will be described later. Therefore, the fluorescent component image, the fluorescent component-corrected image, the autofluorescent component image, and the autofluorescent component-corrected image are not limited to image data of fluorescent components or autofluorescent components of a single wavelength channel, but may be spectral images composed of fluorescent components or autofluorescent components of multiple wavelength channels. In addition, the fluorescent components and autofluorescent components in this description refer to the wavelength components of fluorescence or autofluorescence in a multichannel image obtained by imaging with an imaging device.

[0014] As described above, when observing the fluorescence of a stained tissue section stained with a fluorescent dye antibody, the accuracy of the measurement of the fluorescence intensity reflecting the amount of antibody may be affected by autofluorescence, etc. Furthermore, even when a process (hereinafter referred to as color separation process) is performed to remove the spectral information (hereinafter referred to as autofluorescence components or autofluorescence spectra) derived from the autofluorescent substance from a captured image (hereinafter referred to as a stained specimen image) of a stained tissue section (hereinafter referred to as a stained section) by color separation using a spectrum, it is impossible to completely remove the autofluorescence components contained in the stained specimen image (hereinafter referred to as a fluorescent component image) after the color separation process, which causes problems such as an impediment to the quantification of the amount of dye antibody distributed in the analysis target region in the stained specimen image.

[0015] In the following embodiment, in order to solve these problems, a pseudo autofluorescence component image (autofluorescence component corrected image) is generated from a color-separated image of a stained section (spectral image of each fluorescent component and / or each autofluorescence component, i.e., a fluorescent component image and / or an autofluorescence component image) and a method is proposed for image processing using this image, the method having the following characteristics.

[0016] As a first feature, in the following embodiment, an image (autofluorescence component corrected image) is generated in which the spectral intensity of the autofluorescence is calculated from a fluorescence component image (also called an antibody number image) composed of autofluorescence components extracted from a stained specimen image by color separation processing and spectral information derived from fluorescent substances (fluorescence components, also called fluorescence spectra) obtained by color separation processing. By using such an autofluorescence component corrected image, it is possible to achieve effects such as reducing the effort required to separately obtain an autofluorescence component image obtained by imaging an unstained section, and improving quantitativeness by using spectral information linked to position information.

[0017] The second feature is that the autofluorescence-corrected image generated from the stained specimen image is used to perform brightness threshold processing of the stained specimen image, which makes it possible to extract areas that have specific signals of fluorescent substances (also called dyeing fluorescent dyes) that can be distinguished from autofluorescence.

[0018] As a third feature, for fluorescent substances as well, an image (fluorescence component corrected image) is generated by calculating the fluorescence spectral intensity from the fluorescent substance-derived fluorescent component and the fluorescent component image obtained by color separation processing, and the relative ratio between the spectral intensity derived from the fluorescent substance and the spectral intensity derived from the autofluorescent substance is calculated. This makes it possible to evaluate the color separation accuracy and the designed fluorescent dye antibody panel.

[0019] In addition, the above-mentioned Patent Document 1 proposes a method of extracting a target by setting a threshold based on the difference in pixel brightness values, but since there is no correlation between the difference in brightness values ​​and the presence or absence of molecular expression, it is possible that the extraction of a target using such a method is inappropriate. In contrast, in the following embodiment, the target region is extracted using an autofluorescence component corrected image in which a correlation is recognized between the difference in brightness values ​​and the presence or absence of molecular expression, making it possible to appropriately extract the target region.

[0020] In addition, in silico labeling technology (Eric M. Christiansen et al., “In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images”, Cell, 173, 792-803, April 19, 2018) utilizes deep learning to predict the target cell region from a transmission image (autofluorescence component image), and it is possible to reproduce a highly accurate fluorescent label image. However, in this technology, it is necessary to prepare an image that is actually fluorescently labeled as learning data as an input image, but it is not possible to prepare images of the same field of view as the two input images (autofluorescence component image and fluorescent label image), and images of consecutive sections are used, which results in a spatial difference between the two input images. In contrast, in the following embodiment, a pseudo autofluorescence component image is generated from a stained specimen image obtained by imaging a stained section, so that quantitativeness can be discussed using data from the same population.

[0021] Furthermore, it is also conceivable to specify the region using serial sections of the same tissue block, but even when serial sections are used, images of regions shifted by about several μm in the optical axis direction are used, and therefore there is a possibility that unstained and stained sections will be affected by spatial differences in the distribution morphology and signal intensity of cells and autofluorescent substances.In contrast, in the following embodiment, an autofluorescent component corrected image is generated based on spectral information of the autofluorescent substance extracted from the stained section itself, making it possible to quantify the stained specimen image under conditions where the variation of the autofluorescent substance, etc. is spatially uniform.

[0022] <1. First embodiment> First, a first embodiment according to the present disclosure will be described.

[0023] (1.1. Configuration example) An example of the configuration of an information processing system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the information processing system according to this embodiment includes an information processing device 100 and a database 200, and a fluorescent reagent 10, a specimen 20, and a fluorescently stained specimen 30 exist as inputs to the information processing system.

[0024] (Fluorescent Reagent 10) The fluorescent reagent 10 is a chemical used to stain the specimen 20. The fluorescent reagent 10 is, for example, a fluorescent antibody (including a primary antibody used for direct labeling, or a secondary antibody used for indirect labeling), a fluorescent probe, or a nuclear staining reagent, but the type of the fluorescent reagent 10 is not limited to these. The fluorescent reagent 10 is managed by being assigned identification information (hereinafter referred to as "reagent identification information 11") that can identify the fluorescent reagent 10 (or the manufacturing lot of the fluorescent reagent 10). The reagent identification information 11 is, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited to this. Even if the fluorescent reagent 10 is the same product, its properties differ for each manufacturing lot depending on the manufacturing method, the state of the cells from which the antibody was obtained, etc. For example, the fluorescent reagent 10 has different wavelength spectrum of fluorescence (fluorescence spectrum), quantum yield, or fluorescent labeling rate for each manufacturing lot. Therefore, in the information processing system according to this embodiment, the fluorescent reagent 10 is managed for each manufacturing lot by being assigned the reagent identification information 11. This allows the information processing device 100 to perform fluorescence separation while taking into consideration slight differences in properties that appear for each manufacturing lot.

[0025] (specimen 20) The specimen 20 is prepared from a specimen or tissue sample taken from a human body for the purpose of pathological diagnosis or the like. The specimen 20 may be a tissue slice, a cell, or a microparticle, and the type of tissue (e.g., an organ, etc.) used, the type of disease to be treated, the attributes of the subject (e.g., age, sex, blood type, race, etc.), or the lifestyle of the subject (e.g., diet, exercise, smoking, etc.) of the specimen 20 are not particularly limited. The tissue slice may include, for example, a slice of a tissue slice to be stained (hereinafter simply referred to as a slice) before staining, a slice adjacent to a stained slice, a slice different from the stained slice in the same block (sampled from the same location as the stained slice), a slice in a different block (sampled from a different location from the stained slice) in the same tissue, a slice taken from a different patient, etc. The specimens 20 are managed with identification information (hereinafter referred to as "specimen identification information 21") that can identify each specimen 20. The specimen identification information 21, like the reagent identification information 11, is, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited thereto. The properties of the specimen 20 differ depending on the type of tissue used, the type of disease to be treated, the attributes of the subject, or the lifestyle of the subject. For example, the measurement channel or the wavelength spectrum of autofluorescence (autofluorescence spectrum) of the specimen 20 differs depending on the type of tissue used, etc. In the information processing system according to this embodiment, the specimens 20 are managed individually by being assigned specimen identification information 21. This allows the information processing device 100 to perform fluorescence separation while taking into consideration slight differences in properties that appear for each specimen 20.

[0026] (Fluorescent stained specimen 30) The fluorescent stained specimen 30 is produced by staining the specimen 20 with a fluorescent reagent 10. In this embodiment, the fluorescent stained specimen 30 is produced on the assumption that the specimen 20 is stained with one or more fluorescent reagents 10, and the number of fluorescent reagents 10 used for staining is not particularly limited. The staining method is determined by the respective combinations of the specimen 20 and the fluorescent reagents 10, and is not particularly limited.

[0027] (Information processing device 100) 1, the information processing device 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160. The information processing device 100 may be, for example, a fluorescent microscope, but is not limited thereto and may include various devices. For example, the information processing device 100 may be a PC (Personal Computer), etc.

[0028] (Acquisition part 110) The acquiring section 110 is configured to acquire information used for various processes of the information processing device 100. As shown in FIG.

[0029] (Information acquisition unit 111) The information acquisition unit 111 is configured to acquire information on the fluorescent reagent 10 (hereinafter referred to as "reagent information") and information on the specimen 20 (hereinafter referred to as "specimen information"). More specifically, the information acquisition unit 111 acquires the reagent identification information 11 attached to the fluorescent reagent 10 used to generate the fluorescent stained specimen 30, and the specimen identification information 21 attached to the specimen 20. For example, the information acquisition unit 111 acquires the reagent identification information 11 and the specimen identification information 21 using a barcode reader or the like. Then, the information acquisition unit 111 acquires the reagent information based on the reagent identification information 11 and the specimen information based on the specimen identification information 21 from the database 200. The information acquisition unit 111 stores the acquired information in the information storage unit 121, which will be described later.

[0030] In this embodiment, the specimen information includes information on the autofluorescence spectra (hereinafter also referred to as autofluorescence reference spectra) of one or more autofluorescent substances in the specimen 20, and the reagent information includes information on the fluorescence spectra (hereinafter also referred to as fluorescence reference spectra) of fluorescent substances in the fluorescently stained specimen 30. The autofluorescence reference spectrum and the fluorescence reference spectrum are each or collectively referred to as the "reference spectra."

[0031] (Fluorescence signal acquisition unit 112) The fluorescent signal acquisition unit 112 is configured to acquire a plurality of fluorescent signals corresponding to a plurality of excitation lights having different wavelengths when the fluorescent stained specimen 30 (created by staining the specimen 20 with the fluorescent reagent 10) is irradiated with the plurality of excitation lights having different wavelengths. More specifically, the fluorescent signal acquisition unit 112 receives light and outputs a detection signal according to the amount of received light, thereby acquiring a data cube (hereinafter, referred to as a stained specimen image) composed of the fluorescent spectrum of the fluorescent stained specimen 30 based on the detection signal. Here, the content of the excitation light (including the excitation wavelength, intensity, etc.) is determined based on reagent information, etc. (in other words, information, etc. related to the fluorescent reagent 10). Note that the fluorescent signal referred to here is not particularly limited as long as it is a signal derived from fluorescence, and the fluorescent spectrum is merely one example. In this description, a case in which the fluorescent signal is a fluorescent spectrum will be illustrated.

[0032] 2A to 2D are specific examples of the fluorescence spectrum acquired by the fluorescence signal acquisition unit 112. In FIG. 2A to 2D, the fluorescence stained specimen 30 contains four kinds of fluorescent substances, namely, DAPI, CK / AF488, PgR / AF594, and ER / AF647, and specific examples of the fluorescence spectrum acquired when the specimen is irradiated with excitation light having excitation wavelengths of 392 [nm] (FIG. 2A), 470 [nm] (FIG. 2B), 549 [nm] (FIG. 2C), and 628 [nm] (FIG. 2D) are shown. It should be noted that the fluorescence wavelength is shifted to the longer wavelength side than the excitation wavelength due to the release of energy for fluorescence emission (Stokes shift). In addition, the fluorescent substances contained in the fluorescence stained specimen 30 and the excitation wavelength of the irradiated excitation light are not limited to the above. The fluorescence signal acquisition unit 112 stores the stained specimen image consisting of the acquired fluorescence spectrum in the fluorescence signal storage unit 122 described later.

[0033] (Preservation section 120) The storage unit 120 is configured to store information used for various processes of the information processing device 100 or information output by various processes. As shown in FIG. 1, the storage unit 120 includes an information storage unit 121 and a fluorescent signal storage unit 122.

[0034] (Information storage section 121) The information storage unit 121 is configured to store the reagent information and specimen information acquired by the information acquisition unit 111 .

[0035] (Fluorescence signal storage unit 122) The fluorescent signal storage unit 122 is configured to store the fluorescent signal of the fluorescent stained specimen 30 acquired by the fluorescent signal acquisition unit 112 .

[0036] (Processing unit 130) The processing unit 130 is configured to perform various processes including color separation processing. As shown in FIG.

[0037] (Separation processing unit 132) The separation processing unit 132 is configured to separate the stained specimen image into fluorescence spectra for each fluorescent substance, and as described below, extracts an autofluorescence spectrum from the input stained specimen image and generates an autofluorescence component corrected image using the extracted autofluorescence spectrum (generation unit). The separation processing unit 132 then executes color separation processing of the stained specimen image using the generated autofluorescence component corrected image (separation unit). This separation processing unit 132 can function as a generation unit, separation unit, correction unit, and image generation unit in the claims.

[0038] The color separation process may use, for example, the least squares method (LSM), the weighted least squares method (WLSM), etc. Furthermore, the extraction of the autofluorescence spectrum and / or the fluorescence spectrum may use, for example, the non-negative matrix factorization (NMF), the singular value decomposition (SVD), the principal component analysis (PCA), etc.

[0039] (Operation unit 160) The operation unit 160 is configured to receive operation input from the operator. More specifically, the operation unit 160 includes various input means such as a keyboard, a mouse, a button, a touch panel, or a microphone, and the operator can perform various inputs to the information processing device 100 by operating these input means. Information regarding the operation input performed via the operation unit 160 is provided to the control unit 150.

[0040] (Database 200) The database 200 is a device that manages reagent information, specimen information, etc. More specifically, the database 200 manages the reagent identification information 11 and the reagent information, and the specimen identification information 21 and the specimen information, by linking them together. This allows the information acquisition unit 111 to acquire the reagent information based on the reagent identification information 11 of the fluorescent reagent 10, and the specimen information based on the specimen identification information 21 of the specimen 20, from the database 200.

[0041] The reagent information managed by the database 200 is assumed to be information including a measurement channel and a fluorescence reference spectrum specific to the fluorescent substance possessed by the fluorescent reagent 10 (although it is not necessarily limited to this). The "measurement channel" is a concept indicating the fluorescent substance contained in the fluorescent reagent 10. Since the number of fluorescent substances varies depending on the fluorescent reagent 10, the measurement channel is linked to each fluorescent reagent 10 and managed as reagent information. Also, the fluorescence reference spectrum included in the reagent information is, as described above, the fluorescence spectrum of each fluorescent substance contained in the measurement channel.

[0042] Moreover, it is assumed that the specimen information managed by the database 200 includes (but is not necessarily limited to) a measurement channel and an autofluorescence reference spectrum specific to the autofluorescence substance of the specimen 20. The "measurement channel" is a concept indicating an autofluorescence substance contained in the specimen 20, and in the example of FIG. 8, it is a concept indicating Hemoglobin, Archidonic Acid, Catalase, Collagen, FAD, NADPH, and ProLongDiamond. Since the number of autofluorescence substances varies depending on the specimen 20, the measurement channel is linked to each specimen 20 and managed as specimen information. Furthermore, the autofluorescence reference spectrum included in the specimen information is, as described above, the autofluorescence spectrum of each autofluorescence substance included in the measurement channel. Note that the information managed by the database 200 is not necessarily limited to the above.

[0043] The configuration example of the information processing system according to the present embodiment has been described above. Note that the above configuration described with reference to Fig. 1 is merely an example, and the configuration of the information processing system according to the present embodiment is not limited to this example. For example, the information processing device 100 does not necessarily have to include all of the configuration shown in Fig. 1, and may include configurations not shown in Fig. 1.

[0044] Here, the information processing system according to the present embodiment may include an imaging device (including, for example, a scanner) that acquires a fluorescence spectrum, and an information processing device that performs processing using the fluorescence spectrum. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 may be realized by the imaging device, and the other components may be realized by the information processing device. In addition, the information processing system according to the present embodiment may include an imaging device that acquires a fluorescence spectrum, and software used for processing using the fluorescence spectrum. In other words, the information processing system may not be provided with a physical component (for example, a memory, a processor, etc.) that stores or executes the software. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 may be realized by the imaging device, and the other components may be realized by the information processing device in which the software is executed. The software is provided to the information processing device via a network (for example, from a website or a cloud server, etc.), or is provided to the information processing device via any storage medium (for example, a disk, etc.). In addition, the information processing device in which the software is executed may be various servers (for example, a cloud server, etc.), a general-purpose computer, a PC, a tablet PC, etc. Note that the method in which the software is provided to the information processing device and the type of the information processing device are not limited to the above. Furthermore, please note that the configuration of the information processing system according to this embodiment is not necessarily limited to the above, and any configuration that a person skilled in the art can conceive of based on the technical level at the time of use may be applied.

[0045] (1.2. Application example to microscope system) The information processing system described above may be realized, for example, as a microscope system. Next, a configuration example of a microscope system in which the information processing system according to the present embodiment is realized as a microscope system will be described with reference to FIG.

[0046] As shown in FIG. 5, the microscope system according to this embodiment includes a microscope 101 and a data processing unit 107.

[0047] The microscope 101 includes a stage 102 , an optical system 103 , a light source 104 , a stage driving unit 105 , a light source driving unit 106 , and a fluorescent signal acquiring unit 112 .

[0048] The stage 102 has a mounting surface on which the fluorescently stained specimen 30 can be placed, and is movable in a direction parallel to the mounting surface (xy plane direction) and a direction perpendicular to the mounting surface (z-axis direction) by driving a stage driving unit 105. The fluorescently stained specimen 30 has a thickness in the Z direction, for example, of several μm to several tens of μm, and is sandwiched between a slide glass SG and a cover glass (not shown) and fixed by a predetermined fixing method.

[0049] An optical system 103 is disposed above the stage 102. The optical system 103 includes an objective lens 103A, an imaging lens 103B, a dichroic mirror 103C, an emission filter 103D, and an excitation filter 103E. The light source 104 is, for example, a light bulb such as a mercury lamp or an LED (Light Emitting Diode), and is driven by a light source drive unit 106 to irradiate excitation light onto the fluorescent label attached to the fluorescent stained specimen 30.

[0050] When obtaining a fluorescent image of the fluorescent stained specimen 30, the excitation filter 103E generates excitation light by transmitting only light of an excitation wavelength that excites the fluorescent dye from the light emitted from the light source 104. The dichroic mirror 103C reflects the excitation light that is transmitted through the excitation filter and incident thereon, and guides it to the objective lens 103A. The objective lens 103A focuses the excitation light on the fluorescent stained specimen 30. The objective lens 103A and the imaging lens 103B then magnify the image of the fluorescent stained specimen 30 to a predetermined magnification, and form the magnified image on the imaging plane of the fluorescent signal acquisition unit 112.

[0051] When the fluorescent stained specimen 30 is irradiated with excitation light, the staining agent bound to each tissue of the fluorescent stained specimen 30 emits fluorescence. This fluorescence passes through the dichroic mirror 103C via the objective lens 103A, and reaches the imaging lens 103B via the emission filter 103D. The emission filter 103D absorbs the light that has been magnified by the objective lens 103A and transmitted through the excitation filter 103E, and transmits only a portion of the colored light. The image of the colored light from which the external light has been removed is magnified by the imaging lens 103B as described above, and is imaged on the fluorescent signal acquisition unit 112.

[0052] The data processing unit 107 is configured to drive the light source 104, acquire a fluorescent image of the fluorescent stained specimen 30 using the fluorescent signal acquisition unit 112, and perform various processes using the acquired image. More specifically, the data processing unit 107 can function as a part or all of the information acquisition unit 111, the storage unit 120, the processing unit 130, the display unit 140, the control unit 150, the operation unit 160, or the database 200 of the information processing device 100 described with reference to FIG. 1. For example, the data processing unit 107 functions as the control unit 150 of the information processing device 100 to control the driving of the stage driving unit 105 and the light source driving unit 106, and to control the acquisition of a spectrum by the fluorescent signal acquisition unit 112. In addition, the data processing unit 107 functions as the processing unit 130 of the information processing device 100 to generate a fluorescent spectrum, separate the fluorescent spectrum into fluorescent substances, and generate image information based on the separation results.

[0053] The above describes an example of the configuration of a microscope system in the case where the information processing system according to the present embodiment is realized as a microscope system. Note that the above configuration described with reference to Fig. 5 is merely an example, and the configuration of the microscope system according to the present embodiment is not limited to this example. For example, the microscope system does not necessarily have to include all of the configurations shown in Fig. 5, and may include configurations that are not shown in Fig. 5.

[0054] (1.3. About the least squares method) Here, the least square method used in the color separation process by the separation processing unit 132 will be described. The least square method calculates the color mixing ratio by fitting a reference spectrum to a fluorescence spectrum, which is the pixel value of each pixel in the input stained specimen image. The color mixing ratio is an index indicating the degree to which each substance is mixed. The following formula (1) is a formula that represents a residual obtained by subtracting a reference spectrum (St) from a fluorescence spectrum (Signal). The fluorescence reference spectrum and the autofluorescence reference spectrum mixed at a color mixing ratio a. Note that "Signal (1 x number of channels)" in formula (1) indicates that the fluorescence spectrum (Signal) exists in the number of wavelength channels. For example, Signal is a matrix that represents one or more fluorescence spectra. Also, "St (number of substances x number of channels)" indicates that the reference spectrum exists in the number of wavelength channels for each substance (fluorescent substance and autofluorescent substance). For example, St is a matrix that represents one or more reference spectra. In addition, "a (1 × number of substances)" indicates that a color mixing ratio a is set for each substance (fluorescent substance and autofluorescent substance). For example, a is a matrix representing the color mixing ratio of each reference spectrum in the fluorescence spectrum.

[0055]

number

[0056] Then, the separation processing unit 132 calculates the color mixing ratio a of each material that minimizes the square sum of the residual formula (1). The sum of squares of the residual becomes minimum when the result of partial differentiation of the color mixing ratio a for the formula (1) expressing the residual is 0, so the separation processing unit 132 calculates the color mixing ratio a of each material that minimizes the sum of squares of the residual by solving the following formula (2). Note that "St'" in formula (2) indicates the transposed matrix of the reference spectrum St. Also, "inv(St*St')" indicates the inverse matrix of St*St'.

[0057]

number

[0058] Here, specific examples of the values ​​of the above formula (1) are shown in the following formulas (3) to (5). The examples of formulas (3) to (5) show a case where reference spectra (St) of three types of substances (the number of substances is three) are mixed at different color mixing rates a in a fluorescence spectrum (Signal).

[0059]

number

[0060]

number

[0061]

number

[0062] A specific example of the calculation result of the above formula (2) using the values ​​of formulas (3) and (5) is shown in the following formula (6). As shown in formula (6), it can be seen that the calculation result is correctly calculated as "a = (3 2 1)" (i.e., the same value as the above formula (4)).

[0063]

number

[0064] As described above, the separation processing unit 132 may extract the spectrum of each fluorescent substance from the fluorescence spectrum by performing calculations related to the weighted least squares method instead of the least squares method. In the weighted least squares method, weighting is performed to emphasize errors at low signal levels by utilizing the fact that noise in the fluorescence spectrum (signal), which is a measured value, has a Poisson distribution. However, the upper limit value at which weighting is not performed in the weighted least squares method is set as the offset value. The offset value is determined by the characteristics of the sensor used for the measurement, and when an imaging element is used as the sensor, it needs to be optimized separately. When the weighted least squares method is performed, the reference spectrum St in the above formulas (1) and (2) is replaced with St_ represented by the following formula (7). Note that the following formula (7) means that St_ is calculated by dividing each element (each component) of St expressed in a matrix by each corresponding element (each component) in the "signal+offset value" also expressed in a matrix (in other words, element division).

[0065]

number

[0066] Here, when the Offset value is 1 and the values ​​of the reference spectrum St and the fluorescence spectrum Signal are expressed by the above equations (3) and (5), respectively, a specific example of St_ expressed by the above equation (7) is shown in the following equation (8).

[0067]

number

[0068] A specific example of the calculation result of the color mixing rate a in this case is shown in the following formula (9): As shown in formula (9), it can be seen that "a=(3 2 1)" is correctly calculated as the calculation result.

[0069]

number

[0070] (1.4. Nonnegative Matrix Factorization (NMF)) Next, a description will be given of non-negative matrix factorization (NMF) used by the separation processing unit 132 to extract the autofluorescence spectrum and / or the fluorescence spectrum. However, the method is not limited to non-negative matrix factorization (NMF), and singular value decomposition (SVD), principal component analysis (PCA), etc. may also be used.

[0071] FIG. 3 is a diagram for explaining an overview of NMF. As shown in FIG. 3, NMF decomposes a non-negative matrix A with N rows and M columns (N×M) into a non-negative matrix W with N rows and k columns (N×k) and a matrix H with k rows and M columns (k×M). Matrices W and H are determined so that a mean square residual D between matrix A and the product (W*H) of matrix W and matrix H is minimized. In this embodiment, matrix A corresponds to the spectrum before the autofluorescence reference spectrum is extracted (N is the number of pixels, and M is the number of wavelength channels), and matrix H corresponds to the extracted autofluorescence reference spectrum (k is the number of autofluorescence reference spectra (in other words, the number of autofluorescent materials), and M is the number of wavelength channels). Here, the mean square residual D is expressed by the following formula (10). Note that "norm(D,'fro')" refers to the Frobenius norm of the mean square residual D.

[0072]

number

[0073] Factorization in NMF uses an iterative method starting with random initial values ​​for matrices W and H. In NMF, the value of k (the number of autofluorescence reference spectra) is required, but the initial values ​​of matrices W and H are not required and can be set as an option, and the solution becomes constant when the initial values ​​of matrices W and H are set. On the other hand, if the initial values ​​of matrices W and H are not set, these initial values ​​are set randomly and the solution does not become constant.

[0074] The specimen 20 differs in nature and autofluorescence spectrum according to the type of tissue used, the type of disease targeted, the attributes of the subject, or the lifestyle of the subject. Therefore, as described above, the information processing apparatus 100 according to the second embodiment can realize a more accurate color separation process by actually measuring the autofluorescence reference spectrum for each specimen 20.

[0075] Note that the matrix A, which is the input of NMF, is a matrix consisting of the same number of rows as the number of pixels N (= Hpix × Vpix) of the stained specimen image and the same number of columns as the number of wavelength channels M. Therefore, when the number of pixels of the stained specimen image is large or the number of wavelength channels M is large, the matrix A becomes a very large matrix, increasing the calculation cost of NMF and lengthening the processing time.

[0076] In such a case, for example, as shown in FIG. 4, by clustering into the number of classes N (<Hpix × Vpix) with the number of pixels N (= Hpix × Vpix) of the stained image specified, it is possible to suppress the redundancy of the processing time due to the enlargement of the matrix A.

[0077] In clustering, for example, among the stained images, spectra similar in the wavelength direction or intensity direction are classified into the same class. As a result, an image with a smaller number of pixels than the stained image is generated, and it becomes possible to reduce the scale of the matrix A' with this image as the input.

[0078] (Image generation unit 133) The image generating unit 133 is configured to generate image information based on the result of separation of the fluorescence spectra by the separation processing unit 132. For example, the image generating unit 133 can generate image information using fluorescence spectra corresponding to one or more fluorescent substances, or generate image information using autofluorescence spectra corresponding to one or more autofluorescent substances. The number and combination of fluorescent substances (molecules) or autofluorescent substances (molecules) used by the image generating unit 133 to generate image information are not particularly limited. In addition, when various processes (e.g., segmentation or calculation of S / N value) are performed using the fluorescence spectra or autofluorescence spectra after separation, the image generating unit 133 may generate image information indicating the results of those processes.

[0079] (Display section 140) The display unit 140 is configured to present the image information generated by the image generation unit 133 to the practitioner by displaying it on a display. The type of display used as the display unit 140 is not particularly limited. Although not described in detail in this embodiment, the image information generated by the image generation unit 133 may be presented to the practitioner by being projected by a projector or printed by a printer (in other words, the method of outputting the image information is not particularly limited).

[0080] (Control unit 150) The control unit 150 is a functional configuration that comprehensively controls the overall processing performed by the information processing device 100. For example, the control unit 150 controls the start and end of various processes as described above (for example, a process of adjusting the placement position of the fluorescent stained specimen 30, a process of irradiating the fluorescent stained specimen 30 with excitation light, a process of acquiring a spectrum, a process of generating an image corrected for autofluorescence components, a color separation process, a process of generating image information, and a process of displaying image information, etc.) based on an operation input by an operator performed via the operation unit 160. The contents of the control by the control unit 150 are not particularly limited. For example, the control unit 150 may control processes (for example, processes related to an OS (Operating System)) that are generally performed in a general-purpose computer, a PC, a tablet PC, etc.

[0081] (1.5. Characteristic Configuration of the First Embodiment) The configuration and application examples of the information processing system according to the present embodiment have been described above. Next, the characteristic configuration of the present embodiment will be described in detail below with reference to the drawings.

[0082] As described above, in this embodiment, in order to distinguish fluorescent signals derived from autofluorescent substances that are problematic when photographing a tissue sample from fluorescent signals derived from the fluorescent substance to be analyzed, a pseudo autofluorescent component image (autofluorescent component corrected image) is generated using spectral information derived from the autofluorescent substance (autofluorescent component (spectrum)) obtained when performing color separation processing on the stained specimen image, and this is used to enable quantitative analysis of the stained specimen image.

[0083] More specifically, in this embodiment, an autofluorescence component corrected image is generated based on the autofluorescence components extracted from the stained specimen image using the corresponding autofluorescence reference spectrum, and the stained specimen image is processed using the autofluorescence component corrected image to generate a fluorescence component image with higher color separation. The generated fluorescence component image may be displayed on the display unit 140, or a predetermined process (analysis process, etc.) may be performed by the processing unit 130 or other components (e.g., an analysis device connected via a network, etc.). The predetermined process may be, for example, a process of detecting specific cells.

[0084] (1.5.1. Basic operation example) 6 is a flowchart showing an example of a basic operation of the information processing system according to this embodiment. Note that the following operation is executed by each unit operating under the control of the control unit 150, for example.

[0085] 6, in the basic operation according to the present embodiment, first, the information acquiring unit 111 of the acquiring unit 110 acquires an image of the stained specimen by capturing an image of the fluorescent stained specimen 30 (step S101). The stained specimen image acquired in this way is stored in the information storage unit 121 in the storage unit 120, for example.

[0086] The information acquiring unit 111 also acquires the reagent information and specimen information from the database 200 connected via the network (step S102). As described above, the specimen information includes information on the autofluorescence reference spectra of one or more autofluorescent substances in the specimen 20, and the reagent information includes information on the fluorescence reference spectra of fluorescent substances in the fluorescent stained specimen 30. The acquired reagent information and specimen information are stored in the information storage unit 121 in the storage unit 120, for example.

[0087] Next, the separation processing unit 132 acquires the stained specimen image, the reagent information, and the specimen information stored in the information storage unit 121, and performs color separation processing of the stained specimen image by fitting the autofluorescence reference spectrum to the acquired stained specimen image using, for example, the least squares method (step S103). This color separation processing generates a fluorescent component image and an autofluorescent component image.

[0088] Next, the separation processing unit 132 generates an autofluorescence component corrected image using the autofluorescence component image generated in step S103 and the autofluorescence reference spectrum included in the specimen information acquired in step S102 (step S104). The generation of the autofluorescence component corrected image will be described in more detail later.

[0089] Next, the separation processing unit 132 processes the stained specimen image using the autofluorescence component corrected image to generate a fluorescence component image (step S105). In this way, by removing the autofluorescence components contained in the stained specimen image using the autofluorescence component corrected image generated using the autofluorescence reference spectrum, it is possible to generate a fluorescence component image with improved color separation accuracy, that is, with a reduced amount of remaining autofluorescence components (correction unit).

[0090] Then, separation processing section 132 transmits the generated fluorescent component image to image generating section 133, an external server, etc. (Step S106), after which this operation ends.

[0091] (1.5.2. Example of flow for generating autofluorescence correction image) Next, an example of the operation when generating the autofluorescence component image described in step S104 of Fig. 6 will be described. Fig. 7 is a flowchart showing an example of the operation when generating a pseudo autofluorescence component image according to this embodiment. Also, Fig. 8 is a diagram showing an example of the autofluorescence component image generated in step S103 of Fig. 6, and Fig. 9 is a diagram showing an example of the autofluorescence reference spectrum included in the specimen information acquired in step S102 of Fig. 6. Furthermore, Fig. 10 is a diagram for explaining step S112 of Fig. 7, and Fig. 11 is a diagram for explaining step S114 of Fig. 7.

[0092] 7, in the flow for generating an autofluorescence component corrected image, the separation processing unit 132 first selects (step S111) one of the autofluorescence component images (see FIG. 8) generated in step S103 in FIG 6 that has not yet been selected (this image is designated as an autofluorescence component image of an autofluorescence channel CHn (n is a natural number)). Note that the autofluorescence channel may be identification information assigned to each type of autofluorescence.

[0093] Next, as shown in Fig. 10, the separation processing unit 132 generates a spectral image for the autofluorescence channel CHn from the autofluorescence component image for the autofluorescence channel CHn selected in step S111 and the autofluorescence reference spectrum for the autofluorescence channel CHn among the autofluorescence reference spectra (see Fig. 9) included in the specimen information acquired in step S102 of Fig. 6 (step S112). The above-mentioned NMF can be used to generate this spectral image.

[0094] Next, the separation processing unit 132 determines whether or not all autofluorescence component images have been selected in step S111 (step S113), and if not (NO in step S113), returns to step S111, selects an unselected autofluorescence component image, and executes the subsequent operations.

[0095] On the other hand, if all autofluorescence component images have been selected in step S111 (YES in step S113), the separation processing unit 132 adds together the spectral images of the autofluorescence channels CH generated in the repeated step S112 (step S114), as shown in Fig. 11. This generates an autofluorescence component corrected image. Thereafter, the separation processing unit 132 ends the operation shown in Fig. 7.

[0096] (1.6. Extraction and analysis of the analysis target area) Next, the extraction process and analysis of an analysis target region from a fluorescent component image according to this embodiment will be described with reference to a comparative example. Fig. 12 is a flow chart for explaining the extraction process and analysis of an analysis target region according to the comparative example, and Fig. 13 is a schematic diagram for explaining the extraction process and analysis of an analysis target region according to the comparative example shown in Fig. 12. Fig. 14 is a flow chart for explaining the extraction process and analysis of an analysis target region according to this embodiment, and Fig. 15 is a schematic diagram for explaining the extraction process and analysis of an analysis target region according to this embodiment shown in Fig. 14.

[0097] (Comparative Example) 12 and 13, in the comparative example, first, a stained section is imaged to obtain a stained specimen image, and an unstained tissue section adjacent or close to the stained section (hereinafter also referred to as an unstained tissue section or unstained section) is imaged to obtain an image of the unstained section (hereinafter referred to as an unstained specimen image) (step S901). Next, a color separation process is performed using the obtained stained specimen image and unstained specimen image to generate a fluorescent component image and an autofluorescent component image (step S902).

[0098] Next, a threshold value for the number of antibodies is set for extracting candidate regions (regions to be extracted) for the region to be analyzed (regions to be analyzed) based on the autofluorescence component image (step S903). Note that the threshold value for the number of antibodies may be, for example, a threshold value for the pixel values ​​of each pixel in the fluorescence component image. Furthermore, the region to be extracted may be the region hatched with dots in step S903 of FIG. 13.

[0099] Next, the pixel value of each pixel in the stained specimen image is compared with a threshold value, and a mask image for extracting an area to be analyzed is generated based on the comparison result (step S904). As a result, a binary mask is generated as the mask image, in which, for example, if the pixel value is equal to or greater than the threshold value, it is set to '1', and if the pixel value is less than the threshold value, it is set to '0'. Note that a pixel or area to which '1' is assigned may be a pixel or area that is a candidate for the analysis object, and a pixel or area to which '0' is assigned may be a pixel or area excluded from the analysis object.

[0100] Next, a logical AND operation is performed between each pixel of the mask image generated in step S904 and each pixel of the fluorescence component image to extract an analysis target region from the fluorescence component image (step S905). This results in an extracted image in which the region labeled with the fluorescent dye antibody (morphological information of the analysis target) is extracted using the value extracted from the autofluorescence component image as a threshold value.

[0101] Thereafter, for example, the region to be analyzed is sent to an analysis device such as an external server and quantitatively evaluated (step S906).

[0102] (Present embodiment) 14 and 15, in this embodiment, first, a stained specimen image is obtained by imaging a stained section (step S1). That is, in this embodiment, it is not necessary to image an unstained section to obtain an unstained specimen image.

[0103] Next, in this embodiment, a color separation process is performed on the stained specimen image to generate a fluorescence component image (step S2). Also, in this embodiment, an autofluorescence component corrected image is generated using the autofluorescence component image generated by the color separation process and the autofluorescence reference spectrum included in the acquired specimen information (step S3).

[0104] Next, based on the autofluorescence component corrected image, a threshold value for the number of antibodies is set for extracting an extraction target region that is a candidate for an analysis target region (step S4). As in the explanation using Figures 12 and 13, the threshold value for the number of antibodies may be, for example, a threshold value for the pixel values ​​of each pixel in the fluorescence component image, and the extraction target region may be the region hatched with dots in step S4 of Figure 15.

[0105] 12 and 13, a mask image is generated by comparing the pixel value of each pixel in the stained specimen image with a threshold value (step S5), and then a logical AND operation is performed between the mask image generated in step S5 and the fluorescent component image to extract an analysis target region from the fluorescent component image (step S6). As a result, an extracted image is obtained in which the region labeled with the fluorescent dye antibody (morphological information of the analysis target) is extracted using the value extracted from the autofluorescence component corrected image as the threshold value. The extracted analysis target region is then sent to an analysis device such as an external server and quantitatively evaluated (step S7).

[0106] (1.7. Actions and Effects) As described above, according to this embodiment, in the process of extracting the area to be analyzed, a value defined by the user from the autofluorescence component corrected image or an objective numerical value is set as a threshold, and the area to be analyzed is extracted using a mask image (binary mask) generated based on the magnitude relationship between the pixel values ​​in the stained specimen image and the threshold.

[0107] In this way, by using the autofluorescence component corrected image to set the threshold, it is possible to eliminate the need to separately image an unstained section to obtain an unstained specimen image and the need to generate an autofluorescence component image. In addition, the threshold can be set based on the autofluorescence component corrected image which retains spatial information in the stained specimen image (the signal distribution due to autofluorescence components and noise is the same), in other words, which is correlated with the stained specimen image and the fluorescent component image, making it possible to achieve the effect of setting a threshold which is correlated with the stained specimen image and the fluorescent component image.

[0108] <2. Second embodiment> Next, a second embodiment will be described. In the above-mentioned first embodiment, a unique threshold value was set for all pixels of the stained specimen image from the generated autofluorescence component corrected image. In contrast, in the second embodiment, a threshold distribution image is generated in which threshold values ​​that are not necessarily the same are spatially distributed by setting a different threshold value for each pixel. Note that in this embodiment, the configuration and basic operation of the information processing system may be similar to the configuration and basic operation of the information processing system according to the first embodiment, and therefore detailed description thereof will be omitted here.

[0109] (2.1. Extraction of the analysis area and its analysis) FIG. 16 is a flowchart for explaining the process of extracting an analysis target area and its analysis according to this embodiment, and FIG. 17 is a schematic diagram for explaining the process of extracting an analysis target area and its analysis according to this embodiment shown in FIG. 16.

[0110] As shown in FIGS. 16 and 17 , in this embodiment, first, similar to the operations described in the first embodiment using steps S1 to S3 in FIGS. 14 and 15 , a stained specimen image is obtained by imaging a stained section (step S21), a color separation process is performed on the stained specimen image to generate a fluorescence component image (step S22), and an autofluorescence component corrected image is generated using the autofluorescence component image generated by the color separation process and the autofluorescence reference spectrum included in the acquired specimen information (step S23).

[0111] Next, in this embodiment, a threshold distribution image in which a different antibody count threshold is set for each pixel is generated based on the autofluorescence component corrected image (step S24). A method for generating a threshold distribution image in which a different threshold is set for each pixel can be used, for example, by multiplying the pixel value of each pixel in the autofluorescence component corrected image by a coefficient that has been set in advance. Here, the coefficient can be determined, for example, within a numerical range that is considered appropriate. As a specific example, a value based on the maximum or average pixel value in the autofluorescence component corrected image can be used as the coefficient.

[0112] Next, in this embodiment, the pixel value of each pixel in the stained specimen image is compared with the threshold value of the corresponding pixel in the threshold distribution image, and a mask image for extracting an analysis target area is generated based on the comparison result (step S25). As a result, a binary mask is generated as the mask image, in which, for example, if the pixel value is equal to or greater than the threshold, it is set to '1', and if the pixel value is less than the threshold, it is set to '0'. Note that, as in the first embodiment, a pixel or area to which '1' is assigned may be a pixel or area that is a candidate for the analysis target, and a pixel or area to which '0' is assigned may be a pixel or area that is excluded from the analysis target. Also, in the comparison result for each pixel shown in step S25 of FIG. 17, a pixel hatched with dots may be, for example, a pixel whose pixel value is equal to or greater than the corresponding threshold.

[0113] 14 and 15, a logical AND operation is performed between the mask image generated in S25 and the fluorescence component image to extract an analysis target region from the fluorescence component image (step S26). As a result, an extracted image is obtained in which the region labeled with the fluorescent dye antibody (morphological information of the analysis target) is extracted using the value extracted from the autofluorescence component corrected image as a threshold. The extracted analysis target region is then sent to an analysis device such as an external server and quantitatively evaluated (step S27).

[0114] (2.2. Actions and Effects) As described above, in this embodiment, when generating a binary mask for extracting only a target region from a stained specimen image, a threshold distribution image in which different thresholds can be spatially distributed is generated, instead of uniquely setting a threshold from an autofluorescence component corrected image as in the first embodiment. This makes it possible to set a threshold using pixel values ​​of an autofluorescence component corrected image corresponding to a stained specimen image, and therefore makes it possible to extract an analysis target region using a threshold distribution image that holds spatial information on noise that may occur in the system, such as the distribution of autofluorescence components for each tissue region and the effects of hardware. As a result, it becomes possible to further improve the accuracy of analysis in fluorescence observation.

[0115] Other configurations, operations, and effects may be similar to those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0116] <3. Third embodiment> Next, a third embodiment will be described. In the above-described embodiment, a case where an autofluorescence component corrected image is generated from a stained specimen image is exemplified. In contrast, in this embodiment, in addition to the autofluorescence component corrected image, a pseudo fluorescence component image (hereinafter referred to as a fluorescence component corrected image) is also generated, and the ratio of the spectral intensity of the fluorescent dye to the autofluorescence is obtained as information by calculating the ratio. Note that in this embodiment, the configuration and basic operation of the information processing system may be similar to those of the information processing system according to the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0117] (3.1. Generation and analysis of spectral intensity ratio images) In this embodiment, the spectral intensity ratio is generated, for example, as spatially distributed information, that is, image data (spectral intensity ratio image). Fig. 18 is a flowchart for explaining the generation process of the spectral intensity ratio image according to this embodiment and its analysis.

[0118] As shown in FIG. 18, in this embodiment, first, an autofluorescence component corrected image is generated by executing operations similar to steps S101 to S104 in FIG. 6 in the first embodiment (step S104).

[0119] Next, in this embodiment, for example, a fluorescent component-corrected image is generated by applying the same operations as when generating an autofluorescence component-corrected image (step S301). Note that the fluorescent component-corrected image generation flow can be implemented by replacing the autofluorescence component image in the operations described in the first embodiment with FIG. 7 with a fluorescent component image and replacing the autofluorescence reference spectrum with a fluorescent reference spectrum, and therefore a detailed description thereof will be omitted here.

[0120] Next, the separation processing unit 132 calculates the ratio (spectral intensity ratio) between corresponding pixels in the fluorescent component corrected image and the autofluorescent component corrected image to generate a spectral intensity ratio image (step S302).

[0121] Thereafter, for example, the spectral intensity ratio image is sent to an analysis device such as an external server and quantitatively evaluated (step S303).

[0122] (3.2. Spectral Intensity Image Generation Method) In this embodiment, the spectral intensity ratio image may be generated, for example, by generating a spectral intensity image showing the spectral intensity of each pixel of the fluorescence component corrected image and a spectral intensity image showing the spectral intensity of each pixel of the autofluorescence component corrected image, and calculating the ratio of the spectral intensities between corresponding pixels.

[0123] Here, the method for generating the spectral intensity images of the fluorescent component corrected image and the autofluorescent component corrected image is not particularly limited, and two generation methods will be exemplified below.

[0124] (3.2.1 First generation method) Fig. 19 is a diagram for explaining the first generation method. As shown in Fig. 19, in the first generation method, a spectral intensity image is generated by adding up, in the wavelength direction, pixel values ​​for each wavelength channel of the fluorescence component corrected image / autofluorescence component corrected image having a data cube structure (step S31).

[0125] (3.2.2 Second generation method) Fig. 20 is a diagram for explaining the second generation method. As shown in Fig. 20, in the second generation method, a spectral intensity image is generated by extracting the maximum pixel value in the wavelength direction from the pixel values ​​for each wavelength channel of the fluorescence component corrected image / autofluorescence component corrected image in a data cube structure (step S32).

[0126] (3.3. Actions and Effects) As described above, in this embodiment, even for the fluorescent component, a fluorescent component corrected image is generated by multiplying each pixel of the fluorescent component image by the fluorescent reference spectrum extracted by color separation processing, and spectral intensity images are obtained for each of the fluorescent component corrected image and the autofluorescence component corrected image. Then, a spectral intensity ratio image is generated, which is information that spatially indicates the ratio of fluorescent dye intensity to autofluorescence, from the spectral intensity ratio between each pixel of the fluorescent component corrected image and each pixel of the autofluorescence component corrected image. The spectral intensity ratio image generated in this manner can be used for performance evaluation of a measurement system (corresponding to acquisition unit 110) in an information processing system, evaluation of color separation accuracy in the processing unit 130, and further for designing / evaluating fluorescent reagent panels, etc.

[0127] According to the present embodiment, for example, the spectral intensity ratio image generated as described above may be recorded in the storage unit 120 or the like in association with image acquisition conditions such as the photographing conditions when the acquisition unit 110 acquires the stained specimen image and the labeling conditions (staining conditions) of the fluorescent reagent 10 for the specimen 20, and may be used as reference information when acquiring a stained specimen image using the same acquisition unit 110. In addition, the spectral intensity ratio image and the image acquisition conditions may be stored in a server on a network and shared with other information processing systems or information processing devices 100, and may be used as reference information when acquiring a stained specimen image using the same model of acquisition unit 110 in another information processing system. Specifically, for example, the spectral intensity ratio image and the image acquisition conditions stored in the storage unit 120, server, or the like may be used for inter-instrument baseline correction, calibration curve correction, or the like when using the same model of acquisition unit 110.

[0128] Other configurations, operations, and effects may be similar to those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0129] <4. Fourth embodiment> Next, a fourth embodiment will be described. In the above-described embodiments, for example, NMF, SVD, PCA, etc. are used to extract the autofluorescence spectrum and / or the fluorescence spectrum. In contrast, in this embodiment, a case where machine learning is used to extract the autofluorescence spectrum and / or the fluorescence spectrum instead of these will be described. Note that in this embodiment, the configuration and basic operation of the information processing system may be similar to the configuration and basic operation of the information processing system according to the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0130] (4.1. Generation of fluorescent component images using machine learning) FIG. 21 is a flowchart for explaining the process of generating a fluorescence component image using machine learning according to this embodiment, and FIG. 22 is a schematic diagram for explaining the process of generating a fluorescence component image using machine learning according to this embodiment shown in FIG. 21.

[0131] As shown in FIGS. 21 and 22, in this embodiment, first, an autofluorescence component corrected image is generated by executing operations similar to steps S1 to S3 in FIG. 14 in the second embodiment (steps S401 to S403).

[0132] Next, in this embodiment, the fluorescent component image generated by the color separation process in step S401 and the pseudo autofluorescence component image generated in step S403 are input to the machine learning unit 401, and the machine learning unit 401 executes unsupervised deep learning using the fluorescent component image and the pseudo autofluorescence component image as input images to extract features such as the signal intensity and distribution derived from the autofluorescence component (e.g., autofluorescence spectrum) from the stained specimen image (step S404). Note that various machine learning methods such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), and RNN (Recurrent Neural Network) can be applied as the deep learning. Furthermore, the machine learning unit 401 may be implemented in, for example, the processing unit 130 in the information processing device 100, or in a cloud server connected to the information processing device 100 via a predetermined network.

[0133] Then, in this embodiment, the feature amounts extracted in step S404 are used to generate a more accurate fluorescent component image in which the remaining amount of the autofluorescent component is further reduced (step S405), and this operation ends.

[0134] (4.2. Actions and Effects) As described above, in this embodiment, a fluorescent component image generated by color separation processing of a stained specimen image and an autofluorescence component-corrected image generated therefrom are used as input images, and unsupervised machine learning is applied to extract feature amounts such as signal intensity and distribution derived from the autofluorescence component from the stained specimen image, and a fluorescent component image with higher accuracy is generated based on these extracted feature amounts. In this way, by performing machine learning that is derived from a stained specimen image, in other words, that uses an autofluorescence component-corrected image generated from a stained specimen image as input, it becomes easy to link the input image and output image of the machine learning, which makes it possible to achieve the effect of easily improving the learning effect.

[0135] Other configurations, operations, and effects may be similar to those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0136] <5. Measurement system configuration example> Next, a configuration example of the measurement system in the information processing device 100 according to the above-mentioned embodiment (hereinafter simply referred to as the embodiment) will be described. FIG. 23 is a diagram showing an example of the measurement system of the information processing system according to the embodiment. FIG. 23 shows an example of the measurement system when capturing a wide field of a fluorescent stained specimen 30 (or a specimen 20 that is an unstained specimen), such as WSI (Whole Slide Imaging). However, the measurement system according to the embodiment is not limited to the measurement system shown in FIG. 23, and may be modified in various ways as long as it is a measurement system capable of capturing image data (hereinafter referred to as wide-field image data) of the entire photographed region or the region of interest with sufficient resolution, such as a measurement system that captures the entire photographed region or a necessary region therein (also referred to as a region of interest) at once, or a measurement system that captures an image of the entire photographed region or the region of interest by line scanning.

[0137] As shown in FIG. 23, the measurement system according to the embodiment includes, for example, an information processing device 100, an XY stage 501, an excitation light source 510, a beam splitter 511, an objective lens 512, a spectroscope 513, and a photodetector 514.

[0138] The XY stage 501 is a stage on which the fluorescent stained specimen 30 (or specimen 20) to be analyzed is placed, and may be, for example, a stage that is movable in a plane (XY plane) parallel to the surface on which the fluorescent stained specimen 30 (or specimen 20) is placed.

[0139] The excitation light source 510 is a light source for exciting the fluorescent stained specimen 30 (or the specimen 20), and for example, emits a plurality of excitation light beams having mutually different wavelengths along a predetermined optical axis.

[0140] The beam splitter 511 is composed of, for example, a dichroic mirror, and reflects the excitation light from the excitation light source 510 and transmits the fluorescence from the fluorescently stained specimen 30 (or specimen 20).

[0141] The objective lens 512 irradiates the excitation light reflected by the beam splitter 511 onto the fluorescent stained specimen 30 (or specimen 20) on the XY stage 501.

[0142] The spectroscope 513 is configured using one or more prisms, lenses, etc., and splits the fluorescence emitted from the fluorescent stained specimen 30 (or specimen 20) and transmitted through the objective lens 512 and beam splitter 511 into beams in predetermined directions.

[0143] The photodetector 514 detects the light intensity for each wavelength of the fluorescence dispersed by the spectroscope 513, and inputs the thus obtained fluorescence signal (fluorescence spectrum and / or autofluorescence spectrum) to the fluorescence signal acquisition unit 112 of the information processing device 100.

[0144] In the above configuration, when the entire photographing area exceeds the area that can be photographed in one shot (hereinafter referred to as the field of view), such as WSI, the XY stage 501 is moved for each photographing to move the field of view, and photographing of each field of view is performed sequentially. Then, wide-field image data of the entire photographing area is generated by tiling the image data (hereinafter referred to as the field of view image data) obtained by photographing each field of view. The generated wide-field image data is stored, for example, in the fluorescent signal storage unit 122. Note that the tiling of the field of view image data may be performed in the acquisition unit 110 of the information processing device 100, in the storage unit 120, or in the processing unit 130.

[0145] Then, the processing unit 130 according to the embodiment performs the above-mentioned processing on the obtained wide-field image data to obtain the coefficient C, i.e., a fluorescence separation image for each fluorescent molecule (or an autofluorescence separation image for each autofluorescent molecule).

[0146] 6. How to calculate the number of fluorescent molecules (or antibodies) Next, a method for calculating the number of fluorescent molecules (or the number of antibodies) in one pixel in the above-mentioned embodiment will be described. FIG. 24 is a schematic diagram for explaining a method for calculating the number of fluorescent molecules (or the number of antibodies) in one pixel in the embodiment. In the example shown in FIG. 24, when the image sensor and the sample are arranged via an objective lens, the size of the bottom surface of the sample corresponding to image sensor 1 [pixel] is assumed to be 13 / 20 (μm) × 13 / 20 (μm). Also, the thickness of the sample is assumed to be 10 (μm). In that case, the volume (m 3 ) is expressed as 13 / 20(μm) × 13 / 20(μm) × 10(μm). The volume (liters) is expressed as 13 / 20(μm) × 13 / 20(μm) × 10(μm) × 10 3 It is expressed as:

[0147] If the concentration of the antibodies (which may be the number of fluorescent molecules) contained in the sample is uniform and is 300 (nM), the number of antibodies per pixel is expressed by the following formula (11).

[0148]

number

[0149] In this way, the number of fluorescent molecules or the number of antibodies in the fluorescently stained specimen 30 is calculated as a result of the fluorescence separation process, allowing the practitioner to compare the numbers of fluorescent molecules between multiple fluorescent substances or to compare data captured under different conditions. Furthermore, while brightness (or fluorescence intensity) is a continuous value, the number of fluorescent molecules or the number of antibodies is a discrete value, so the information processing device 100 according to the modified example can reduce the amount of data by outputting image information based on the number of fluorescent molecules or the number of antibodies.

[0150] <7. Hardware configuration example> The embodiments and modifications of the present disclosure have been described above. Next, a hardware configuration example of the information processing device 100 according to each embodiment and modification will be described with reference to Fig. 25. Fig. 25 is a block diagram showing a hardware configuration example of the information processing device 100. Various processes performed by the information processing device 100 are realized by cooperation between software and hardware described below.

[0151] 25, the information processing device 100 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing device 100 also includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing device 100 may include a processing circuit such as a DSP or an ASIC instead of or in addition to the CPU 901.

[0152] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing device 100 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. The CPU 901 may embody at least the processing unit 130 and the control unit 150 of the information processing device 100, for example.

[0153] The CPU 901, ROM 902, and RAM 903 are connected to one another by a host bus 904a including a CPU bus, etc. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 904. Note that the host bus 904a, bridge 904, and external bus 904b do not necessarily need to be configured separately, and these functions may be implemented on a single bus.

[0154] The input device 906 is realized by a device into which information is input by the implementer, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever. The input device 906 may be, for example, a remote control device using infrared or other radio waves, or an external connection device such as a mobile phone or a PDA that supports the operation of the information processing device 100. Furthermore, the input device 906 may include, for example, an input control circuit that generates an input signal based on information input by the implementer using the above-mentioned input means and outputs it to the CPU 901. The implementer can input various data to the information processing device 100 and instruct processing operations by operating this input device 906. The input device 906 may embody at least the operation unit 160 of the information processing device 100, for example.

[0155] The output device 907 is formed by a device capable of visually or audibly notifying the operator of acquired information. Such devices include display devices such as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, and a lamp, audio output devices such as a speaker and a headphone, a printer, etc. The output device 907 may embody at least the display unit 140 of the information processing device 100, for example.

[0156] The storage device 908 is a device for storing data. The storage device 908 is realized by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. This storage device 908 stores the programs and various data executed by the CPU 901, and various data acquired from the outside. The storage device 908 may embody at least the storage unit 120 of the information processing device 100, for example.

[0157] The drive 909 is a reader / writer for a storage medium, and is built into or externally attached to the information processing device 100. The drive 909 reads information recorded on a removable storage medium, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903. The drive 909 can also write information to the removable storage medium.

[0158] The connection port 911 is an interface connected to an external device, and is a connection port for connecting to an external device capable of transmitting data via, for example, a USB (Universal Serial Bus) or the like.

[0159] The communication device 913 is, for example, a communication interface formed by a communication device for connecting to the network 920. The communication device 913 is, for example, a communication card for a wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 913 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 913 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP.

[0160] In this embodiment, the sensor 915 includes a sensor capable of acquiring a spectrum (e.g., an image sensor, etc.), but may also include other sensors (e.g., an acceleration sensor, a gyro sensor, a geomagnetic sensor, a pressure sensor, a sound sensor, a distance measurement sensor, etc.) The sensor 915 may embody at least the fluorescent light signal acquisition unit 112 of the information processing device 100, for example.

[0161] The network 920 is a wired or wireless transmission path for information transmitted from devices connected to the network 920. For example, the network 920 may include public line networks such as the Internet, telephone lines, and satellite communication networks, as well as various types of LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The network 920 may also include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0162] The above shows an example of a hardware configuration capable of realizing the functions of the information processing device 100. Each of the above components may be realized using a general-purpose member, or may be realized by hardware specialized for the function of each component. Therefore, the hardware configuration to be used can be changed as appropriate depending on the technical level at the time of implementing the present disclosure.

[0163] It is possible to create a computer program for implementing each function of the information processing device 100 as described above and install it in a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium includes, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, and the like. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0164] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0165] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0166] Note that the following configurations also fall within the technical scope of the present disclosure. (1) a separation unit that separates a fluorescence image observed from a specimen labeled with one or more fluorescent dyes into a fluorescence component image containing one or more fluorescent components and an autofluorescence component image containing one or more autofluorescence components; a generation unit that generates an autofluorescence component corrected image using the autofluorescence component image and reference spectra of one or more autofluorescence substances contained in the specimen; a processing unit that processes the fluorescence component image based on the autofluorescence component corrected image; An information processing device comprising: (2) The information processing device according to (1), wherein the processing unit generates a fluorescent component corrected image by correcting the fluorescent component image using the autofluorescence component corrected image. (3) The information processing device according to (1) or (2), wherein the generation unit generates the autofluorescence component corrected image by adding a result of multiplying the autofluorescence component image by the reference spectrum for each of the autofluorescence substances. (4) The information processing device according to any one of (1) to (3), wherein the processing unit generates area information indicating an area labeled with the fluorescent dye from the autofluorescence component corrected image, and processes the fluorescence component image using the generated area information. (5) The information processing device according to (4), wherein the region information is a binary mask indicating the region labeled with the fluorescent dye. (6) The information processing device according to (5), wherein the processing unit generates the binary mask based on a magnitude relationship between a threshold value set based on the autofluorescence component corrected image and the pixel value of each pixel in the fluorescence image. (7) The information processing device described in (5), wherein the processing unit sets a threshold value for each pixel based on a pixel value of each pixel in the autofluorescence component corrected image, and generates the binary mask based on a magnitude relationship between the threshold value for each pixel and the pixel value of each pixel in the fluorescence image. (8) The information processing device according to any one of (1) to (7), wherein the generating unit further generates a fluorescent component corrected image using the reference spectrum of each of the one or more fluorescent dyes and the fluorescent component image. (9) The information processing device according to (8), further comprising an evaluation unit that evaluates, based on the fluorescence component corrected image and the autofluorescence component corrected image, at least one of the following: a measurement accuracy of a measurement system that acquires the fluorescence image, a separation performance between a fluorescence component image and an autofluorescence component image by the separation unit, and a staining performance of a fluorescent reagent panel constituted by the one or more fluorescent dyes. (10) an image generating unit that generates a spectral intensity ratio image representing a ratio between a spectral intensity of each pixel in the fluorescence component corrected image and a spectral intensity of each pixel in the autofluorescence component corrected image; a storage unit that stores the imaging conditions when the specimen is imaged to obtain the fluorescence image and the spectral intensity ratio image in association with each other; The information processing device according to (8) or (9), further comprising: (11) a correction unit that corrects the fluorescence component image based on the fluorescence image, the fluorescence component image, and the autofluorescence component corrected image, The processing unit processes the corrected fluorescence component image based on the autofluorescence component corrected image. The information processing device according to any one of (1) to (10). (12) The information processing device according to (11), wherein the correction unit estimates a feature amount of the autofluorescent substance in the fluorescence image from the fluorescence component image and the autofluorescence component corrected image, and corrects the fluorescence component image based on the estimated feature amount. (13) The information processing device according to (12), wherein the correction unit estimates the feature amount by using a trained model that has the fluorescence component image and the autofluorescence component corrected image as input. (14) The information processing device according to (13), wherein the learned model is a model trained by unsupervised learning. (15) The information processing device according to any one of (1) to (14), further comprising an acquisition unit configured to acquire the fluorescent image by imaging the specimen. (16) The information processing device according to any one of (1) to (15), wherein the processing unit detects a specific cell included in the fluorescent image based on the processed fluorescent component image. (17) A fluorescent image observed from a specimen labeled with one or more fluorescent dyes is separated into a fluorescent component image containing one or more fluorescent components and an autofluorescence component image containing one or more autofluorescence components; generating an autofluorescence component corrected image using a reference spectrum of each of the one or more autofluorescence substances contained in the specimen and the autofluorescence component image; processing the fluorescence component image based on the autofluorescence component corrected image; An information processing method comprising the steps of: (18) A process of separating a fluorescence image observed from a specimen labeled with one or more fluorescent dyes into a fluorescence component image including one or more fluorescent components and an autofluorescence component image including one or more autofluorescence components; generating an autofluorescence component corrected image using the autofluorescence component image and reference spectra of one or more autofluorescence substances contained in the specimen; processing the fluorescence component image based on the autofluorescence component corrected image; A program for causing a computer to execute the following. (19) A microscope system comprising: a light source that irradiates light onto a specimen labeled with one or more fluorescent dyes; an imaging device that observes fluorescence emitted from the specimen irradiated with the light; and a program for executing processing on a fluorescent image acquired by the imaging device, The program is executed on a computer, A process of separating the fluorescence image into a fluorescence component image including one or more fluorescence components and an autofluorescence component image including one or more autofluorescence components; generating an autofluorescence component corrected image using the autofluorescence component image and reference spectra of one or more autofluorescence substances contained in the specimen; processing the fluorescence component image based on the autofluorescence component corrected image; causing the computer to execute Microscope system. (20) The information processing device according to (1), an analysis device connected to the information processing device via a predetermined network and configured to analyze the fluorescent component image processed by the information processing device; An analysis system comprising: [Explanation of symbols]

[0167] 10 Fluorescent Reagents 11 Reagent Identification Information 20 specimens 21 Specimen Identification Information 30 Fluorescent stained specimens 100 Information processing device 110 Acquisition Department 111 Information Acquisition Department 112 Fluorescence signal acquisition unit 120 Preservation Department 121 Information Storage Department 122 Fluorescence signal storage unit 130 Processing section 132 Separation Processing Unit 133 Image Generation Unit 140 Display section 150 Control section 160 Operation section 200 Databases 401 Machine Learning Department

Claims

1. a separation unit that separates a fluorescence image observed from a specimen labeled with one or more fluorescent dyes into a fluorescence component image including one or more fluorescent components and an autofluorescence component image including one or more autofluorescence components; a generation unit that generates an autofluorescence component corrected image using the autofluorescence component image and reference spectra of one or more autofluorescence substances contained in the sample; a processing unit that processes the fluorescence component image based on the autofluorescence component corrected image; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the processing unit generates a fluorescence component corrected image by correcting the fluorescence component image using the autofluorescence component corrected image.

3. The information processing apparatus according to claim 1 , wherein the generating section generates the autofluorescence component corrected image by adding a result of multiplying the autofluorescence component image by the reference spectrum for each of the autofluorescence substances.

4. The information processing apparatus according to claim 1 , wherein the processing unit generates region information indicating a region labeled with the fluorescent dye from the autofluorescence component corrected image, and processes the fluorescence component image using the generated region information.

5. The information processing apparatus according to claim 4 , wherein the region information is a binary mask indicating the region labeled with the fluorescent dye.

6. The information processing apparatus according to claim 5 , wherein the processing unit generates the binary mask based on a magnitude relationship between a threshold value set based on the autofluorescence component corrected image and the pixel value of each pixel in the fluorescence image.

7. 6. The information processing device according to claim 5, wherein the processing unit sets the threshold value for each pixel based on a pixel value of each pixel in the autofluorescence component corrected image, and generates the binary mask based on a magnitude relationship between the threshold value for each pixel and the pixel value of each pixel in the fluorescence image.

8. The information processing apparatus according to claim 1 , wherein the generating section further generates a fluorescence component corrected image by using a reference spectrum of each of the one or more fluorescent dyes and the fluorescence component image.

9. 9. The information processing device according to claim 8, further comprising an evaluation unit that evaluates, based on the fluorescence component-corrected image and the autofluorescence component-corrected image, at least one of the following: a measurement accuracy of a measurement system that acquires the fluorescence image, a separation performance between a fluorescence component image and an autofluorescence component image by the separation unit, and a staining performance of a fluorescent reagent panel constituted by the one or more fluorescent dyes.

10. an image generating unit that generates a spectral intensity ratio image representing a ratio between a spectral intensity of each pixel in the fluorescence component corrected image and a spectral intensity of each pixel in the autofluorescence component corrected image; a storage unit that stores the imaging conditions when the specimen is imaged to obtain the fluorescence image and the spectral intensity ratio image in association with each other; The information processing device according to claim 8 , further comprising:

11. a correction unit that corrects the fluorescence component image based on the fluorescence image, the fluorescence component image, and the autofluorescence component corrected image, The processing unit processes the corrected fluorescence component image based on the autofluorescence component corrected image. The information processing device according to claim 1 .

12. The information processing device according to claim 11 , wherein the correction section estimates a feature amount of the autofluorescent substance in the fluorescence image from the fluorescence component image and the autofluorescence component corrected image, and corrects the fluorescence component image based on the estimated feature amount.

13. The information processing device according to claim 12 , wherein the correction unit estimates the feature amount by using a trained model that receives as input the fluorescence component image and the autofluorescence component corrected image.

14. The information processing device according to claim 13 , wherein the learned model is a model trained by unsupervised learning.

15. The information processing device according to claim 1 , further comprising an acquisition unit configured to acquire the fluorescent image by capturing an image of the specimen.

16. The information processing apparatus according to claim 1 , wherein the processing unit detects a specific cell contained in the fluorescent image based on the processed fluorescent component image.

17. A fluorescent image observed from a specimen labeled with one or more fluorescent dyes is separated into a fluorescent component image containing one or more fluorescent components and an autofluorescence component image containing one or more autofluorescence components; generating an autofluorescence component corrected image using the autofluorescence component image and a reference spectrum of each of the one or more autofluorescence substances contained in the specimen; processing the fluorescence component image based on the autofluorescence component corrected image; An information processing method comprising the steps of:

18. A process of separating a fluorescence image observed from a specimen labeled with one or more fluorescent dyes into a fluorescence component image including one or more fluorescent components and an autofluorescence component image including one or more autofluorescence components; generating an autofluorescence component corrected image using the autofluorescence component image and reference spectra of one or more autofluorescence substances contained in the sample; processing the fluorescence component image based on the autofluorescence component corrected image; A program for causing a computer to execute the following.

19. A microscope system comprising: a light source that irradiates light onto a specimen labeled with one or more fluorescent dyes; an imaging device that observes fluorescence emitted from the specimen irradiated with the light; and a program for executing processing on a fluorescent image acquired by the imaging device, The program is executed on a computer, A process of separating the fluorescence image into a fluorescence component image including one or more fluorescence components and an autofluorescence component image including one or more autofluorescence components; generating an autofluorescence component corrected image using the autofluorescence component image and reference spectra of one or more autofluorescence substances contained in the sample; processing the fluorescence component image based on the autofluorescence component corrected image; causing the computer to execute Microscope system.

20. The information processing device according to claim 1 ; an analysis device connected to the information processing device via a predetermined network and configured to analyze the fluorescent component image processed by the information processing device; An analysis system comprising:

Images