Information Processing Apparatus and Information Processing System

Abstract

The present invention enables execution of more accurate fluorescence separation. An information processing apparatus (100) according to an embodiment comprises a separation unit (2311) that separates, on the basis of a fluorescence image of a biological sample including a cell, a reference spectrum originating from the biological sample or a fluorescence reagent, and form information of the cell, a fluorescence signal originating from the fluorescence reagent from the fluorescence image.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and an information processing system.

Background Art

[0002] In recent years, with the development of cancer immunotherapy and the like, fluorescence and multiplex labeling of immunohistochemistry have been progressing. For example, a technique has been performed in which an autofluorescence spectrum is extracted from an unstained section of the same tissue block, and fluorescence separation of a stained section is performed using the autofluorescence spectrum.

[0003] Also, for example, in Patent Document 1 below, a fluorescence spectrum obtained by irradiating microparticles multiply labeled with a plurality of fluorescent dyes with excitation light is approximated by a linear sum of single-staining spectra obtained with microparticles individually labeled with each fluorescent dye. A technique is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, in recent years, with the spread of cancer immunotherapy and the like, fluorescence and multi-markerization of immunohistochemistry have begun to progress. In order to use more types of fluorescent dyes in multicolorization, both fluorescence separation between stained fluorescences and fluorescence separation between stained fluorescence and autofluorescence are required to be accurate.

[0006] Therefore, the present disclosure has been made in view of the above circumstances, and provides a novel and optimized information processing apparatus and information processing system capable of performing more accurate fluorescence separation.

Means for Solving the Problems

[0007] An information processing apparatus according to an embodiment of the present disclosure includes a separation unit that separates a fluorescence signal derived from a fluorescent reagent from a fluorescence image based on a fluorescence image of a biological sample containing cells, a reference spectrum derived from the biological sample or a fluorescent reagent, and morphological information of the cells.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Mode for Carrying Out the Invention

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0010] The description will be made in the following order. 1. First 2. First Embodiment 2.1. Configuration Example 2.2. Application Example to Microscope System 2.3. Processing Flow 2.4. Fluorescence Separation Processing 2.5. Training of Inference Model 2.6. Function and Effect 3. Second Embodiment 3.1. Fluorescence Separation Processing 3.2. Fluorescence Separation Processing Using Least Squares Method 3.3. Training of Inference Model 3.4. Function and Effect 4. Third Embodiment 4.1. Example of First Procedure 4.2. Example of Second Procedure 5. Fourth Embodiment 6. Fifth Embodiment 6.1. Method for Fixing Staining Fluorescence Spectrum in Minimization of Mean Squared Residual D Using Recurrence Formula 6.2. Method for Fixing Staining Fluorescence Spectrum in Minimization of Mean Squared Residual D Using DFP Method, BFGS Method, etc. 7. Example of Hardware Configuration 8. Remarks 9. Modification Example of System Configuration 10. Application Example 1 11. Application Example 2

[0011] <1. Introduction> First of all, the following embodiments according to the present disclosure propose an information processing apparatus and an information processing system that excite multiply-fluorescently stained cells (fixed cells or floating cells) with excitation light of multiple wavelengths and separate the fluorescence.

[0012] In the fluorescence separation of multiply-fluorescently stained cells, accurate fluorescence separation (including fluorescence separation between stained fluorescences and fluorescence separation between stained fluorescence and autofluorescence) is required. However, in imaging, there is a problem that the autofluorescence spectrum varies between pixels. If extraction of the autofluorescence spectrum and fluorescence separation are performed relying only on the spectrum information for each pixel, although an image with high contrast can be obtained for each pixel, artifacts such as autofluorescence and noise are easily affected on a pixel-by-pixel basis. As a result, even morphologically identical cells may obtain completely different fluorescence separation results for each pixel. For example, variations may occur in both the luminance and wavelength directions within a single cell region, and variations in the luminance and wavelength directions may also occur between cells of the same morphology.

[0013] Therefore, in the following embodiments, by using an inference model of machine learning that inputs, in addition to the pre-fluorescence-separation image information (corresponding to the fluorescence signal (also referred to as a fluorescence-stained image) described later) obtained by imaging a specimen stained with a fluorescent dye (corresponding to the fluorescence-stained specimen described later) and the spectrum information for each molecule contained in the fluorescent dye and the specimen (corresponding to the reference spectrum described later), morphological information of cells, tissues, etc. that are the specimens (not limited to fluorescence. For example, an expression map of an antigen, etc.), a more accurate fluorescence separation result (for example, a two-dimensional image for each fluorescent dye) with the autofluorescence region clarified and noise reduced is output.

[0014] Also, in another embodiment, by using an inference model of machine learning that inputs the pre-fluorescence-separation image information and staining information such as a combination of fluorescent dyes (which may be antibody dyes), and outputs morphological information of cells, tissues, etc. that are the specimens, it becomes possible to perform fluorescence separation using morphological information in addition to spectrum information in the subsequent fluorescence separation process. Thereby, for example, it becomes possible to suppress variations between pixels in the fluorescence separation result in a single cell region.

[0015] <2. First Embodiment> First, the information processing apparatus and information processing system according to the first embodiment of the present disclosure will be described in detail with reference to the drawings.

[0016] (2.1. Configuration example) With reference to FIG. 1, a configuration example of the information processing system according to this embodiment will be described. As shown in FIG. 1, the information processing system according to this embodiment includes an information processing apparatus 100 and a database 200. As inputs to the information processing system, there are a fluorescent reagent 10, a specimen 20, and a fluorescently stained specimen 30.

[0017] (Fluorescent reagent 10) The fluorescent reagent 10 is a chemical used for staining the specimen 20, and may include, for example, an antibody labeled with a fluorescent dye. The fluorescent reagent 10 is, for example, a fluorescent antibody (including a primary antibody used for direct labeling or a secondary antibody used for indirect labeling), a fluorescent probe, or a nuclear staining reagent, etc., but the type of the fluorescent reagent 10 is not limited to these. Also, the fluorescent reagent 10 is managed with identification information (hereinafter referred to as "reagent identification information 11") that can identify the fluorescent reagent 10 (or the manufacturing lot of the fluorescent reagent 10). The reagent identification information 11 is, for example, barcode information etc. (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited to this. Even for the same product, the properties of the fluorescent reagent 10 differ for each manufacturing lot depending on the manufacturing method, the state of the cells from which the antibody was obtained, etc. For example, in the fluorescent reagent 10, the spectrum, quantum yield, or fluorescence labeling rate, etc. differ for each manufacturing lot. Therefore, in the information processing system according to this embodiment, the fluorescent reagent 10 is managed for each manufacturing lot by attaching the reagent identification information 11. Thereby, the information processing apparatus 100 can perform fluorescence separation in consideration of even the slight property differences that appear for each manufacturing lot.

[0018] (Specimen 20) The specimen 20 is prepared for pathological diagnosis or the like from a specimen or tissue sample collected from the human body. The specimen 20 may be a tissue section, a cell, or a microparticle. For the specimen 20, the type of tissue (such as an organ, etc.) used, the type of disease targeted, the attributes of the subject (such as age, gender, blood type, or race, etc.), or the lifestyle habits of the subject (such as eating habits, exercise habits, or smoking habits, etc.) are not particularly limited. Note that the tissue sections may include, for example, a section before staining of a tissue section to be stained (hereinafter also simply referred to as a section), a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), or a section in a different block (sampled from a location different from the stained section) in the same tissue, a section collected from a different patient, etc. In addition, the specimen 20 is managed with identification information (hereinafter referred to as "specimen identification information 21") that can identify each specimen 20. Similar to the reagent identification information 11, the specimen identification information 21 is, for example, barcode information etc. (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited thereto. The specimen 20 has different properties depending on the type of tissue used, the type of disease targeted, the attributes of the subject, or the lifestyle habits of the subject, etc. For example, in the specimen 20, the measurement channel or spectrum etc. are different depending on the type of tissue used, etc. Therefore, in the information processing system according to the present embodiment, the specimen 20 is individually managed by attaching the specimen identification information 21. Thereby, the information processing apparatus 100 can perform fluorescence separation in consideration of even the slight differences in properties that appear for each specimen 20.

[0019] (Fluorescently stained specimen 30) The fluorescently stained specimen 30 is created by staining the specimen 20 with the fluorescent reagent 10. In the present embodiment, the fluorescently stained specimen 30 assumes that the specimen 20 is stained with one or more fluorescent reagents 10, and the number of fluorescent reagents 10 used for staining is not particularly limited. Also, the staining method is determined by the combination of the specimen 20 and the fluorescent reagent 10, etc., and is not particularly limited.

[0020] (Information Processing Apparatus 100) As shown in FIG. 1, the information processing apparatus 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160. The information processing apparatus 100 can be, for example, a fluorescence microscope or the like, but is not necessarily limited thereto and may include various apparatuses. For example, the information processing apparatus 100 may be a PC (Personal Computer) or the like.

[0021] (Acquisition Unit 110) The acquisition unit 110 is configured to acquire information used for various processes of the information processing apparatus 100. As shown in FIG. 1, the acquisition unit 110 includes an information acquisition unit 111 and a fluorescence signal acquisition unit 112.

[0022] (Information Acquisition Unit 111) The information acquisition unit 111 is configured to acquire information regarding the fluorescent reagent 10 (hereinafter referred to as "reagent information") and information regarding the specimen 20 (hereinafter referred to as "specimen information"). More specifically, the information acquisition unit 111 acquires the reagent identification information 11 attached to the fluorescent reagent 10 used for generating the fluorescently stained specimen 30 and the specimen identification information 21 attached to the specimen 20. For example, the information acquisition unit 111 acquires the reagent identification information 11 and the specimen identification information 21 using a barcode reader or the like. Then, the information acquisition unit 111 acquires the reagent information based on the reagent identification information 11 and the specimen information based on the specimen identification information 21 from the database 200, respectively. The information acquisition unit 111 stores the acquired information in the information storage unit 121 described later.

[0023] (Fluorescence Signal Acquisition Unit 112) The fluorescence signal acquisition unit 112 is configured to acquire a plurality of fluorescence signals corresponding to a plurality of excitation lights when a plurality of excitation lights with different wavelengths are irradiated on a fluorescence-stained specimen 30 (created by staining the specimen 20 with the fluorescence reagent 10). More specifically, the fluorescence signal acquisition unit 112 receives light and outputs a detection signal corresponding to the received amount, and thereby acquires the fluorescence spectrum of the fluorescence-stained specimen 30 based on the detection signal. Here, the details of the excitation light (including the excitation wavelength, intensity, etc.) are determined based on reagent information or the like (in other words, information regarding the fluorescence reagent 10). Note that the fluorescence signal referred to here is not particularly limited as long as it is a signal derived from fluorescence, and may be, for example, a fluorescence spectrum.

[0024] A to D in FIG. 2 are specific examples of the fluorescence spectra acquired by the fluorescence signal acquisition unit 112. In A to D of FIG. 2, the fluorescence-stained specimen 30 contains four fluorescent substances, namely DAPI, CK / AF488, PgR / AF594, and ER / AF647, and specific examples of the fluorescence spectra acquired when excitation lights having excitation wavelengths of 392 [nm] (A in FIG. 2), 470 [nm] (B in FIG. 2), 549 [nm] (C in FIG. 2), and 628 [nm] (D in FIG. 2) are irradiated are shown. Note that it should be noted that due to the release of energy for fluorescence emission, the fluorescence wavelength is shifted to the longer wavelength side than the excitation wavelength (Stokes shift). Also, the fluorescent substances contained in the fluorescence-stained specimen 30 and the excitation wavelengths of the irradiated excitation lights are not limited to the above. The fluorescence signal acquisition unit 112 stores the acquired fluorescence spectrum in the fluorescence signal storage unit 122 described later.

[0025] (Storage unit 120) The storage unit 120 is configured to store information used in various processes of the information processing apparatus 100 or information output by various processes. As shown in FIG. 1, the storage unit 120 includes an information storage unit 121, a fluorescence signal storage unit 122, and a fluorescence separation result storage unit 123.

[0026] (Information storage unit 121) The information storage unit 121 is configured to store the reagent information and the specimen information acquired by the information acquisition unit 111.

[0027] (Fluorescence signal storage unit 122) The fluorescence signal storage unit 122 is configured to store the fluorescence signals of the fluorescently stained specimen 30 acquired by the fluorescence signal acquisition unit 112.

[0028] (Fluorescence separation result storage unit 123) The fluorescence separation result storage unit 123 is configured to store the results of the fluorescence separation process performed by the separation processing unit 131 described later. For example, the fluorescence separation result storage unit 123 stores the fluorescence signals for each fluorescent reagent or the autofluorescence signal of the specimen 20 separated by the separation processing unit 131. In addition, the fluorescence separation result storage unit 123 separately provides the results of the fluorescence separation process as teacher data in machine learning to the database 200 in order to improve the fluorescence separation accuracy by machine learning or the like. Note that the fluorescence separation result storage unit 123 may increase the free space by appropriately deleting the processing results it stores after providing the results of the fluorescence separation process to the database 200.

[0029] (Processing unit 130) The processing unit 130 is configured to perform various processes including the fluorescence separation process. As shown in FIG. 1, the processing unit 130 includes a separation processing unit 131, an image generation unit 132, and a model generation unit 133.

[0030] (Separation processing unit 131) The separation processing unit 131 is configured to execute a fluorescence separation process by using an inference model that takes as input image information, specimen information, reagent information, and the like.

[0031] For the image information, for example, fluorescence signals (two-dimensional images based on the fluorescence signals. Hereinafter referred to as fluorescence staining images) acquired by imaging the fluorescently stained specimen 30 with the fluorescence signal acquisition unit 112 may be used.

[0032] For specimen information, for example, the autofluorescence spectrum of each molecule contained in the specimen 20 specified from the specimen identification information 21 and the morphological information regarding the specimen 20 may be used. Note that the morphological information may be a bright-field image, an unstained image, or staining information of the same tissue block, and for example, may be an expression map of the target in the specimen 20.

[0033] Here, the expression map of the target may include, for example, information on the distribution (shape, etc.) of the target such as tissue, cells, or nuclei, information regarding the tissue in each region, information on which cells are located where, etc., and for example, may be a bright-field image obtained by imaging the target, or may be a binary mask representing the expression map of the target in binary.

[0034] Note that the target in this description may include nucleic acids in addition to antigens such as proteins and peptides. That is, in this embodiment, the type of the target is not limited, and it is possible to target various substances that can be targeted.

[0035] Also, the same tissue block may be a specimen that is the same as or similar to the specimen 20 or the fluorescently stained specimen 30.

[0036] Here, for a specimen that is the same as or similar to the specimen 20 or the fluorescently stained specimen 30, it is possible to use either an unstained section or a stained section. For example, when using an unstained section, a section before staining used as a stained section, a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), or a section in a different block (sampled from a location different from the stained section) in the same tissue, etc. can be used.

[0037] For the reagent information, for example, the fluorescence spectrum for each fluorescence reagent 10 used for staining the specimen 20 (hereinafter referred to as the standard spectrum or reference spectrum) may be used. For the reference spectrum of each fluorescence reagent 10, various fluorescence spectra may be applied, such as the fluorescence spectrum based on the catalog value provided by the reagent vendor, or the fluorescence spectrum for each fluorescence reagent 10 extracted from the image information obtained by imaging the same or similar fluorescence-stained specimens 30.

[0038] The separation processing unit 131 performs a process (fluorescence separation process) of separating the autofluorescence signal of each molecule contained in the specimen 20 and the fluorescence signal of each fluorescence reagent 10 from the image information by inputting the image information, specimen information, reagent information, etc. into a pre-prepared learned inference model. The details of the fluorescence separation process using the inference model and the learning of the inference model will be described in detail later.

[0039] In addition, the separation processing unit 131 may execute various processes using the fluorescence signal and autofluorescence signal obtained by the fluorescence separation process. For example, the separation processing unit 131 may perform a subtraction process (also referred to as "background subtraction process") on the image information of another specimen 20 using the separated autofluorescence signal, and execute a process of extracting the fluorescence signal from the image information of the other specimen 20.

[0040] When there are multiple specimens 20 that are identical or similar in terms of the tissue used for specimen 20, the type of disease targeted, the attributes of the subject, and the lifestyle habits of the subject, etc., the autofluorescence signals of these specimens 20 are likely to be similar. The similar specimens referred to here include, for example, the tissue section before staining of the tissue section to be stained (hereinafter referred to as the section), the section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), or a section in a different block (sampled from a location different from the stained section) in the same tissue, etc.), sections taken from different patients, etc. Therefore, when the autofluorescence signal can be extracted from a certain specimen 20, the separation processing unit 131 may extract the fluorescence signal from the image information of another specimen 20 by removing the autofluorescence signal from the image information of the other specimen 20. In this way, by using the background after removing the autofluorescence signal when calculating the S / N value using the image information of other specimens 20, it becomes possible to improve the S / N value in the two-dimensional image obtained by fluorescence separation.

[0041] In addition, in this description, the background may be an area not stained by the fluorescent reagent 10 or the signal value in that area. Therefore, the background may include autofluorescence signals and other noises, etc. before the background subtraction process is performed. Also, after the background subtraction process is performed, it may include autofluorescence signals and other noises, etc. that could not be completely removed.

[0042] In addition, the separation processing unit 131 can perform various processes using the separated fluorescence signal or autofluorescence signal in addition to the background subtraction process. For example, the separation processing unit 131 can use these signals to analyze the immobilization state of the specimen 20 or perform segmentation (or region division) to recognize the regions of objects (for example, cells, intracellular structures (cytoplasm, cell membrane, nucleus, etc.), or tissues (tumor part, non-tumor part, connective tissue, blood vessels, blood vessel walls, lymphatic vessels, fibrotic structures, necrosis, etc.)) included in the image information.

[0043] (Image generation unit 132) The image generation unit 132 is configured to generate (reconstruct) image information based on the fluorescence signal or autofluorescence signal separated by the separation processing unit 131. For example, the image generation unit 132 can generate image information containing only the fluorescence signal or image information containing only the autofluorescence signal. At this time, if the fluorescence signal is composed of a plurality of fluorescence components or the autofluorescence signal is composed of a plurality of autofluorescence components, the image generation unit 132 can generate image information in units of each component. Further, when the separation processing unit 131 performs various processes (for example, analysis of the immobilization state of the specimen 20, segmentation, or calculation of the S / N value, etc.) using the separated fluorescence signal or autofluorescence signal, the image generation unit 132 may generate image information indicating the results of those processes. According to this configuration, the distribution information of the fluorescent reagent 10 labeled on the target molecule or the like, that is, the two-dimensional spread, intensity, wavelength, and positional relationship of each fluorescence are visualized, and in particular, the visibility of the user, such as a doctor or researcher, can be improved in the tissue image analysis area where the information of the target substance is complex.

[0044] Also, the image generation unit 132 may control to distinguish the fluorescence signal from the autofluorescence signal based on the fluorescence signal or autofluorescence signal separated by the separation processing unit 131 and generate image information. Specifically, improving the brightness of the fluorescence spectrum of the fluorescent reagent 10 labeled on the target molecule or the like, extracting and discoloring only the fluorescence spectrum of the labeled fluorescent reagent 10, extracting the fluorescence spectra of two or more fluorescent reagents 10 from the specimen 20 labeled with two or more fluorescent reagents 10 and discoloring each to a different color, extracting only the autofluorescence spectrum of the specimen 20 and performing division or subtraction, improving the dynamic range, etc., to generate image information. Thereby, the user can clearly distinguish the color information derived from the fluorescent reagent bound to the target substance of interest, and the visibility of the user can be improved.

[0045] (Model generation unit 133) The model generation unit 133 is configured to generate an inference model used in the fluorescence separation process executed by the separation processing unit 131, or to update the parameters of the inference model by machine learning to improve the fluorescence separation accuracy.

[0046] (Display unit 140) The display unit 140 is configured to present the image information generated by the image generation unit 132 to the operator by displaying it on a display. Note that the type of display used as the display unit 140 is not particularly limited. Also, although not described in detail in this embodiment, the image information generated by the image generation unit 132 may be presented to the operator by being projected by a projector or printed by a printer (in other words, the output method of the image information is not particularly limited).

[0047] (Control unit 150) The control unit 150 is a functional configuration that comprehensively controls all the processes performed by the information processing apparatus 100. For example, based on the operation input by the user performed via the operation unit 160, the control unit 150 controls the start and end of various processes as described above (for example, the imaging process of the fluorescently stained specimen 30, the fluorescence separation process, various analysis processes, the generation process of image information (reconstruction process of image information), and the display process of image information, etc.). Note that the control content of the control unit 150 is not particularly limited. For example, the control unit 150 may control processes generally performed in a general-purpose computer, a PC, a tablet PC, etc. (for example, processes related to an OS (Operating System)).

[0048] (Operation unit 160) The operation unit 160 is configured to receive an operation input from the operator. More specifically, the operation unit 160 includes various input means such as a keyboard, a mouse, buttons, a touch panel, or a microphone, and the operator can perform various inputs to the information processing apparatus 100 by operating these input means. Information regarding the operation input performed via the operation unit 160 is provided to the control unit 150.

[0049] (Database 200) The database 200 is a device that accumulates and manages specimen information, reagent information, results of fluorescence separation processing, etc. More specifically, the database 200 manages by associating specimen identification information 21 with specimen information, and reagent identification information 11 with reagent information respectively. Thereby, the information acquisition unit 111 can acquire specimen information from the database 200 based on the specimen identification information 21 of the specimen 20 to be measured, and acquire reagent information based on the reagent identification information 11 of the fluorescent reagent 10.

[0050] As described above, the specimen information managed by the database 200 is information including the measurement channels and spectral information (autofluorescence spectrum) specific to the autofluorescent components contained in the specimen 20. However, in addition to these, the specimen information may also include target information for each specimen 20, specifically, information regarding the type of tissue used (such as organs, cells, blood, body fluids, ascites, pleural effusion, etc.), the type of disease targeted, the attributes of the subject (such as age, gender, blood type, or race, etc.), or the lifestyle habits of the subject (such as diet, exercise habits, or smoking habits, etc.). The information including the measurement channels and spectral information specific to the autofluorescent components contained in the specimen 20 and the target information may be associated for each specimen 20. Thereby, it is possible to easily trace the information including the measurement channels and spectral information specific to the autofluorescent components contained in the specimen 20 from the target information. For example, it is possible to cause the separation processing unit 131 to execute a similar separation process performed in the past based on the similarity of the target information in a plurality of specimens 20, and shorten the measurement time. Note that the "tissue used" is not particularly limited to the tissue collected from the subject, and may also include in-vivo tissues such as humans and animals, cell lines, solutions, solvents, solutes, and materials contained in the object to be measured.

[0051] In addition, as described above, the reagent information managed by the database 200 is information including the spectral information (fluorescence spectrum) of the fluorescent reagent 10. However, in addition to this, the reagent information may also include information regarding the manufacturing lot, fluorescent component, antibody, clone, fluorescence labeling rate, quantum yield, fading coefficient (information indicating the ease of reduction of the fluorescence intensity of the fluorescent reagent 10), and absorption cross section (or molar extinction coefficient) of the fluorescent reagent 10. Furthermore, the specimen information and the reagent information managed by the database 200 may be managed in different configurations, and in particular, the information regarding the reagent may be a reagent database that presents an optimal combination of reagents for the user.

[0052] Here, it is assumed that the specimen information and the reagent information are provided by a manufacturer or the like, or are independently measured within the information processing system according to the present disclosure. For example, in many cases, the manufacturer of the fluorescent reagent 10 does not measure and provide spectral information, fluorescence labeling rate, etc. for each manufacturing lot. Therefore, by independently measuring and managing these pieces of information within the information processing system according to the present disclosure, the separation accuracy between the fluorescence signal and the autofluorescence signal can be improved. Also, for the sake of simplifying management, the database 200 may use catalog values published by a manufacturer or the like, or literature values described in various documents as the specimen information and the reagent information (especially the reagent information). However, generally, since the actual specimen information and reagent information often differ from the catalog values and literature values, it is more preferable that the specimen information and the reagent information are independently measured and managed within the information processing system according to the present disclosure as described above.

[0053] In addition, the accuracy of the fluorescence separation process can be improved by using machine learning techniques that utilize the specimen information, reagent information, and the results of the fluorescence separation process managed in the database 200. In the present embodiment, the learning using machine learning techniques or the like is executed in the model generation unit 133. For example, the model generation unit 133 uses a neural network to perform machine learning on training data in which the separated fluorescence signal and autofluorescence signal are associated with the image information, specimen information, and reagent information used for the separation, to generate a classifier or estimator (inference model). Then, when new image information, specimen information, and reagent information are acquired, the model generation unit 133 can input this information into the inference model to predict and output the fluorescence signal and autofluorescence signal included in the image information.

[0054] In addition, similar fluorescence separation processes performed in the past (fluorescence separation processes using similar image information, specimen information, or reagent information) that are more accurate than the predicted fluorescence signal and autofluorescence signal are calculated, and the details of the processes (information and parameters used in the processes, etc.) in those processes are statistically or regressively analyzed, and a method for improving the fluorescence separation process of the fluorescence signal and autofluorescence signal may be output based on the analysis results. Note that the machine learning method is not limited to the above, and known machine learning techniques can be used. In addition, the fluorescence separation process of the fluorescence signal and autofluorescence signal may be performed by artificial intelligence. Also, not only the fluorescence separation process of the fluorescence signal and autofluorescence signal, but also various processes using the separated fluorescence signal or autofluorescence signal (for example, analysis of the immobilization state of the specimen 20, or segmentation, etc.) may be improved by machine learning techniques or the like.

[0055] The configuration example of the information processing system according to the present embodiment has been described above. Note that the above configuration described with reference to FIG. 1 is merely an example, and the configuration of the information processing system according to the present embodiment is not limited to such an example. For example, the information processing apparatus 100 does not necessarily have to include all of the configurations shown in FIG. 1, or may include configurations not shown in FIG. 1.

[0056] Here, the information processing system according to the present embodiment may include an imaging device that acquires a fluorescence spectrum (for example, including a scanner or the like) and an information processing device that performs processing using the fluorescence spectrum. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 can be realized by the imaging device, and the other configurations can be realized by the information processing device. Further, the information processing system according to the present embodiment may include an imaging device that acquires a fluorescence spectrum and software used for processing using the fluorescence spectrum. In other words, a physical configuration (for example, a memory, a processor, etc.) that stores or executes the software may not be provided in the information processing system. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 can be realized by the imaging device, and the other configurations can be realized by the information processing device on which the software is executed. Then, the software is provided to the information processing device via a network (for example, from a website, a cloud server, etc.) or via an arbitrary storage medium (for example, a disk, etc.). Further, the information processing device on which the software is executed can be various servers (for example, a cloud server, etc.), a general-purpose computer, a PC, or a tablet PC, etc. Note that the method by which the software is provided to the information processing device and the type of the information processing device are not limited to the above. Also, it should be noted that the configuration of the information processing system according to the present embodiment is not necessarily limited to the above, and configurations conceivable by those skilled in the art based on the technical level at the time of use can be applied.

[0057] (2.2. Application Example to a Microscope System) The information processing system described above may be realized as, for example, a microscope system. Therefore, subsequently, with reference to FIG. 3, a configuration example of the microscope system when the information processing system according to the present embodiment is realized as a microscope system will be described.

[0058] As shown in FIG. 3, the microscope system according to the present embodiment includes a microscope 101 and a data processing unit 107.

[0059] The microscope 101 includes a stage 102, an optical system 103, a light source 104, a stage drive unit 105, a light source drive unit 106, and a fluorescence signal acquisition unit 112.

[0060] The stage 102 has a placement surface on which the fluorescence-stained specimen 30 can be placed, and is movable in a parallel direction (x-y plane direction) and a perpendicular direction (z-axis direction) with respect to the placement surface by the drive of the stage drive unit 105. The fluorescence-stained specimen 30 has a thickness of, for example, several μm to several tens of μm in the Z direction, and is sandwiched between a slide glass SG and a cover glass (not shown) and fixed by a predetermined fixing method.

[0061] The optical system 103 is disposed above the stage 102. The optical system 103 includes an objective lens 103A, an imaging lens 103B, a dichroic mirror 103C, an emission filter 103D, and an excitation filter 103E. The light source 104 is, for example, a light bulb such as a mercury lamp or an LED (Light Emitting Diode), and irradiates excitation light for the fluorescent label attached to the fluorescence-stained specimen 30 by the drive of the light source drive unit 106.

[0062] When obtaining a fluorescence image of the fluorescence-stained specimen 30, the excitation filter 103E generates excitation light by transmitting only the light of the excitation wavelength that excites the fluorescent dye among the light emitted from the light source 104. The dichroic mirror 103C reflects the excitation light transmitted through the excitation filter and guides it to the objective lens 103A. The objective lens 103A condenses the excitation light onto the fluorescence-stained specimen 30. Then, the objective lens 103A and the imaging lens 103B magnify the image of the fluorescence-stained specimen 30 to a predetermined magnification and form the magnified image on the imaging surface of the fluorescence signal acquisition unit 112.

[0063] When the fluorescently stained specimen 30 is irradiated with excitation light, the stain bound to each tissue of the fluorescently stained specimen 30 emits fluorescence. This fluorescence passes through the dichroic mirror 103C via the objective lens 103A and reaches the imaging lens 103B via the emission filter 103D. The emission filter 103D absorbs the light that has passed through the excitation filter 103E and has been magnified by the objective lens 103A, and transmits only a part of the color-developed light. The image of the color-developed light from which the external light has been lost is magnified by the imaging lens 103B as described above and is formed on the fluorescence signal acquisition unit 112.

[0064] The data processing unit 107 is configured to drive the light source 104, acquire a fluorescence image of the fluorescently stained specimen 30 using the fluorescence signal acquisition unit 112, and perform various processes using this image. More specifically, the data processing unit 107 can function as part or all of the information acquisition unit 111, storage unit 120, processing unit 130, display unit 140, control unit 150, operation unit 160 of the information processing apparatus 100, or the database 200, as described with reference to FIG. 1. For example, by functioning as the control unit 150 of the information processing apparatus 100, the data processing unit 107 controls the driving of the stage driving unit 105 and the light source driving unit 106, or controls the acquisition of the spectrum by the fluorescence signal acquisition unit 112.

[0065] As described above, the configuration example of the microscope system when the information processing system according to the present embodiment is realized as a microscope system has been described. Note that the above configuration described with reference to FIG. 3 is merely an example, and the configuration of the microscope system according to the present embodiment is not limited to such an example. For example, the microscope system does not necessarily have to include all of the configurations shown in FIG. 3, or may include configurations not shown in FIG. 3.

[0066] (2.3. Processing Flow) Above, the configuration example of the information processing system according to the present embodiment has been described. Subsequently, with reference to FIG. 4, an example of the flow of various processes by the information processing apparatus 100 will be described.

[0067] In step S1000, the user determines the fluorescent reagent 10 and the specimen 20 to be used for analysis. In step S1004, the user creates a fluorescently stained specimen 30 by staining the specimen 20 with the fluorescent reagent 10.

[0068] In step S1008, the fluorescent signal acquisition unit 112 of the information processing apparatus 100 acquires image information (e.g., fluorescent staining image) and a part of the specimen information (e.g., morphological information) by imaging the fluorescently stained specimen 30. In step S1012, the information acquisition unit 111 acquires reagent information (e.g., fluorescent spectrum) and a part of the specimen information (e.g., autofluorescent spectrum) from the database 200 based on the reagent identification information 11 attached to the fluorescent reagent 10 used for generating the fluorescently stained specimen 30 and the specimen identification information 21 attached to the specimen 20.

[0069] In step S1016, the separation processing unit 131 acquires an inference model from the database 200 or the model generation unit 133. In step S1020, the separation processing unit 131 inputs the image information, the reagent information, and the specimen information into the inference model, and acquires image information (2D image) for each fluorescent reagent 10 (or fluorescent dye) as the fluorescence separation result that is the output thereof.

[0070] In step S1024, the image generation unit 132 displays the image information acquired by the separation processing unit 131. Thereby, a series of processing flows according to the present embodiment ends.

[0071] Note that each step in the flowchart of FIG. 4 does not necessarily need to be processed in time series in the described order. That is, each step in the flowchart may be processed in an order different from the described order or may be processed in parallel. Further, the information processing apparatus 100 may also execute processes not shown in FIG. 4. For example, the separation processing unit 131 may perform segmentation based on the acquired image information or may analyze the immobilization state of the specimen 20.

[0072] (2.4. Fluorescence Separation Processing) Next, the fluorescence separation process using the inference model according to this embodiment will be described in detail with reference to the drawings.

[0073] As described above, in the fluorescence separation process according to this embodiment, by using an inference model constructed by machine learning, the fluorescence separation accuracy is improved. Specifically, in addition to the fluorescence-stained image obtained from the fluorescence-stained specimen 30, the reference spectrum of each molecule contained in the specimen 20, and the reference spectrum of each fluorescent reagent 10, the morphological information of the specimen 20 is also used as an input to the inference model, and thus a fluorescence separation process is executed while considering the shape and type of cells and tissues constituting the specimen 20.

[0074] As the inference model, for example, a machine learning model using a multi-layer neural network such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), or RNN (Recurrent Neural Network) can be used.

[0075] FIG. 5 is a schematic diagram for explaining an example of the flow of the fluorescence separation process using the inference model according to this embodiment. As shown in FIG. 5, in this embodiment, morphological information is given as an input to the inference model 134 in addition to the fluorescence-stained image and the reference spectrum.

[0076] The fluorescence-stained image may be a two-dimensional image (spectral data) before fluorescence separation. The reference spectrum may be, as described above, the autofluorescence spectrum (for example, catalog value) of each molecule contained in the specimen 20 and the fluorescence spectrum (for example, catalog value) of each fluorescent reagent 10.

[0077] The morphological information may be, as described above, for example, a bright-field image obtained by imaging the specimen 20 before staining, the fluorescence-stained specimen 30 after staining, or a specimen similar thereto or the fluorescence-stained specimen with the fluorescence signal acquisition unit 112. This bright-field image may be, for example, a map showing the expression level of the target (hereinafter referred to as the target expression map) regardless of staining, non-staining, or fluorescence.

[0078] When the fluorescence staining image, the reference spectrum, and the morphological information are given to the input layer of the inference model 134, as a result of the fluorescence separation process, a two-dimensional image for each fluorescence is output from the output layer.

[0079] (2.5. Training of Inference Model) FIG. 6 is a diagram for explaining the training of the inference model according to the present embodiment. As shown in FIG. 6, in the training of the inference model 134, for the same specimen 20 (or fluorescence staining specimen 30), the fluorescence staining image (spectral spectrum data), the reference spectrum, and the morphological image (expression map of the target) are used, and the two-dimensional image for each color (fluorescence separation result) as the correct answer image of this fluorescence staining specimen 30 are input to the model generation unit 133 as teacher data (also referred to as training data or learning data). Based on the input teacher data, the model generation unit 133 learns and updates the parameters of each layer in the inference model 134 by machine learning, thereby executing the training of the inference model 134. As a result, the inference model 134 is updated so that the fluorescence separation accuracy is improved.

[0080] (2.6. Function and Effect) As described above, in the present embodiment, in addition to the fluorescence staining image (spectral spectrum data) and the reference spectrum before fluorescence separation, the morphological information of the specimen 20 (or fluorescence staining specimen 30) (expression map of the target not limited to fluorescence) is input and the fluorescence separation process is executed using the inference model. Therefore, it is possible to obtain a more accurate fluorescence separation result (two-dimensional image for each color) with the autofluorescence region clarified and noise reduced. As a result, it is possible to accurately acquire the target pathological information.

[0081] In the present embodiment, the case where the separation processing unit 131 executes the fluorescence separation process using the inference model 134 that inputs the morphological information of the specimen 20 (or fluorescence staining specimen 30) (expression map of the target not limited to fluorescence) is illustrated, but the present invention is not limited thereto. For example, the separation processing unit 131 may be configured to execute the fluorescence separation process by LSM or the like using the morphological information.

[0082] <3. Second Embodiment> Next, an information processing apparatus and an information processing system according to a second embodiment of the present disclosure will be described in detail with reference to the drawings.

[0083] The information processing system according to the present embodiment may have a configuration similar to that of the information processing system according to the first embodiment described above. However, in the present embodiment, the separation processing unit 131, the model generation unit 133, and the inference model 134 are replaced with a separation processing unit 231, a model generation unit 233, and an inference model 234, respectively, whereby the fluorescence separation processing according to the present embodiment is replaced with the processing content described later.

[0084] (3.1. Fluorescence Separation Processing) The inference model 234 according to the present embodiment is constructed as an inference model for generating morphological information of the specimen 20 (or the fluorescence-stained specimen 30), different from the inference model 134 for performing fluorescence separation processing.

[0085] FIG. 7 is a schematic diagram showing an example of the flow of fluorescence separation processing using the inference model according to the present embodiment. As shown in FIG. 7, in the present embodiment, first, in order to generate morphological information (for example, the expression map of the target) of the specimen 20 (or the fluorescence-stained specimen 30), for the same tissue block, a fluorescence-stained image (spectral spectrum data) before fluorescence separation or a fluorescence-stained image before fluorescence separation, a bright-field image (HE, DAB (immunostaining), etc.), and an image obtained by imaging a specimen identical or similar to the unstained specimen 20 with, for example, the fluorescence signal acquisition unit 112 (hereinafter referred to as an unstained image), and staining information (for example, a combination of a fluorescent dye and an antibody) are input to the inference model 234 of the model generation unit 233. As a result, morphological information regarding each combination of the fluorescent dye and the antibody is output as a binary mask from the inference model 234 (step S2000).

[0086] The morphological information generated in this way is input to the separation processing unit 231. Also, similar to the first embodiment, the fluorescence-stained image before fluorescence separation and the reference spectrum are also input to the separation processing unit 231. The separation processing unit 231 performs fluorescence separation on the input fluorescence-stained image based on algorithms such as the least squares method (LSM), weighted least squares method (WLSM), and constrained least squares method (CLSM) using the input morphological information and reference spectrum, thereby generating a two-dimensional image for each color (step S2004).

[0087] (Separation processing unit 231) FIG. 8 is a block diagram showing a more specific configuration example of the separation processing unit according to the present embodiment. As shown in FIG. 8, the separation processing unit 231 includes a fluorescence separation unit 2311 and a spectrum extraction unit 2312.

[0088] The fluorescence separation unit 2311 includes, for example, a first fluorescence separation unit 2311a and a second fluorescence separation unit 2311b, and separates the fluorescence spectrum of the fluorescence-stained image of the stained sample (hereinafter also simply referred to as the stained sample) input from the fluorescence signal storage unit 122 for each molecule.

[0089] The spectrum extraction unit 2312 is a configuration for optimizing the autofluorescence reference spectrum so as to obtain a more accurate fluorescence separation result, and adjusts the autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 to one that can obtain a more accurate fluorescence separation result based on the fluorescence separation result by the fluorescence separation unit 2311.

[0090] More specifically, the first fluorescence separation unit 2311a separates the fluorescence spectrum of the input stained sample into spectra for each molecule by performing fluorescence separation processing using the fluorescence reference spectrum included in the reagent information, the autofluorescence reference spectrum included in the specimen information, and the input morphological information, all of which are input from the information storage unit 121. Note that, for example, the least squares method (LSM), weighted least squares method (WLSM), etc. may be used for the fluorescence separation processing.

[0091] The spectrum extraction unit 2312 performs spectrum extraction processing on the autofluorescence reference spectrum input from the information storage unit 121 using the fluorescence separation result input from the first fluorescence separation unit 2311a, and optimizes the autofluorescence reference spectrum to obtain a more accurate fluorescence separation result by adjusting the autofluorescence reference spectrum based on the result. Note that, for example, non-negative matrix factorization (hereinafter also referred to as "NMF: Non-negative Matrix Factorization"), singular value decomposition (SVD), or the like may be used for the spectrum extraction processing.

[0092] The second fluorescence separation unit 2311b separates the fluorescence spectrum into spectra for each molecule by performing fluorescence separation processing on the fluorescence spectrum of the input stained sample using the adjusted autofluorescence reference spectrum input from the spectrum extraction unit 2312 and the morphological information. Note that, for example, the least squares method (LSM), the weighted least squares method (WLSM), or the like may be used for the fluorescence separation processing, similar to the first fluorescence separation unit 2311a.

[0093] In FIG. 8, the case where the adjustment of the autofluorescence reference spectrum is performed once is illustrated, but the present invention is not limited thereto. After inputting the fluorescence separation result by the second fluorescence separation unit 2311b to the spectrum extraction unit 2312 and repeating the process of re-executing the adjustment of the autofluorescence reference spectrum in the spectrum extraction unit 2312 one or more times, the final fluorescence separation result may be obtained.

[0094] (3.2. Fluorescence Separation Processing Using the Least Squares Method) Next, the fluorescence separation process using the least squares method will be described. The least squares method calculates the mixing ratio by fitting the fluorescence spectrum of the input stained sample to the reference spectrum. Note that the mixing ratio is an index indicating the degree to which each substance is mixed. The following formula (1) represents the residual obtained by subtracting from the fluorescence spectrum (Signal) the reference spectrum (St, fluorescence reference spectrum and autofluorescence reference spectrum) mixed at the mixing ratio a. Note that "Signal(1 × number of channels)" in formula (1) indicates that the fluorescence spectrum (Signal) exists for the number of wavelength channels (for example, Signal is a matrix representing the fluorescence spectrum). Also, "St(number of substances × number of channels)" indicates that the reference spectrum exists for the number of wavelength channels for each substance (fluorescent substance and autofluorescent substance) (for example, St is a matrix representing the reference spectrum). Also, "a(1 × number of substances)" indicates that the mixing ratio a is provided for each substance (fluorescent substance and autofluorescent substance) (for example, a is a matrix representing the mixing ratio of each reference spectrum in the fluorescence spectrum).

[0095]

Equation

[0096] Then, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b calculates the mixing ratio a of each substance at which the sum of the squares of the residuals of formula (1) is minimized. The sum of the squares of the residuals is minimized when the result of the partial derivative of formula (1) representing the residuals with respect to the mixing ratio a is 0. Therefore, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b calculates the mixing ratio a of each substance at which the sum of the squares of the residuals is minimized by solving the following formula (2). Note that "St'" in formula (2) indicates the transposed matrix of the reference spectrum St. Also, "inv(St*St')" indicates the inverse matrix of St*St'.

[0097]

Equation

[0098] Here, specific examples of each value in the above formula (1) are shown in the following formulas (3) to (5). In the examples of formulas (3) to (5), in the fluorescence spectrum (Signal), the case where the reference spectra (St) of three substances (the number of substances is 3) are mixed at different mixing ratios a is shown.

[0099]

Number

[0100]

Number

[0101]

Number

[0102] And, specific examples of the calculation results of the above formula (2) by each value in formulas (3) and (5) are shown in the following formula (6). As shown in formula (6), it can be seen that "a = (3 2 1)" (that is, the same value as the above formula (4)) is correctly calculated as the calculation result.

[0103]

Number

[0104] The first fluorescence separation unit 2311a / the second fluorescence separation unit 2311b can output a unique spectrum as a separation result by performing fluorescence separation processing using the reference spectra (autofluorescence reference spectrum and fluorescence reference spectrum) as described above (the separation results are not separated for each excitation wavelength). Therefore, the implementer can obtain the correct spectrum more easily. In addition, the reference spectrum (autofluorescence reference spectrum) regarding the autofluorescence used for separation is automatically acquired, and by performing the fluorescence separation processing, it becomes unnecessary for the implementer to extract the spectrum corresponding to the autofluorescence from an appropriate space of the non-stained section.

[0105] Note that, as described above, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b may extract the spectrum for each fluorescent substance from the fluorescence spectrum by performing calculations related to the weighted least square method instead of the least square method. In the weighted least square method, weights are assigned to emphasize the error at a low signal level by utilizing the fact that the noise of the fluorescence spectrum (Signal), which is a measured value, follows a Poisson distribution. However, an upper limit value at which no weighting is performed in the weighted least square method is defined as the Offset value. The Offset value is determined by the characteristics of the sensor used for measurement, and when an image sensor is used as the sensor, separate optimization is required. When the weighted least square method is performed, the reference spectrum St in the above equations (1) and (2) is replaced with St_ represented by the following equation (7). Note that the following equation (7) means that St_ is calculated by dividing each element (each component) of St represented by a matrix by the corresponding element (each component) in the "Signal + Offset value", which is also represented by a matrix (in other words, element-wise division).

[0106]

Equation

[0107] Here, when the Offset value is 1 and the values of the reference spectrum St and the fluorescence spectrum Signal are represented by the above equations (3) and (5) respectively, a specific example of St_ represented by the above equation (7) is shown in the following equation (8).

[0108]

Equation

[0109] And a specific example of the calculation result of the color mixing ratio a in this case is shown in the following equation (9). As shown in equation (9), it can be seen that "a = (3 2 1)" is correctly calculated as the calculation result.

[0110]

Number

[0111] By performing the fluorescence separation process using the least squares method as described above based on the attribute information of each pixel specified from the morphological information (for example, information on which region of the fluorescently stained specimen 30 the pixel belongs to), and making the correlation among the pixels having the same attribute information, it becomes possible to obtain a more accurate fluorescence separation result.

[0112] (3.3. Training of Inference Model) Also, FIG. 9 is a diagram for explaining the training of the inference model according to the present embodiment. As shown in FIG. 9, in the training of the inference model 234 for generating morphological information, for the same tissue block, a fluorescently stained image (spectral data) before fluorescence separation or a fluorescently stained image before fluorescence separation and a bright-field image (such as HE, DAB (immunostaining)), an unstained image of a specimen identical or similar to the unstained specimen 20, staining information (for example, a combination of a fluorescent dye and an antibody), and morphological information (binary mask) as the correct image of this specimen 20 (or fluorescently stained specimen 30) are input to the model generation unit 233 as teacher data. The model generation unit 233 executes the training of the inference model 234 by learning and updating the parameters of each layer in the inference model 234 by machine learning based on the input teacher data. Thereby, the inference model 234 is updated so that more accurate morphological information (binary mask of the target expression map) is generated.

[0113] In addition, when obtaining morphological information by inputting a fluorescence staining image (spectral data) before fluorescence separation into the inference model 234, in cases where the specimen 20 is fluorescence-stained using a plurality of different types of fluorescent reagents 10, etc., the morphology of cells, tissues, etc. in the specimen 20 may not appear in the morphological information. In such a case, before inputting into the inference model 234, fluorescence separation processing using LSM or the like on the fluorescence staining image may be executed first, and the inference model 234 may be trained using the fluorescence separation result.

[0114] (3.4. Action and Effect) As described above, according to the present embodiment, morphological information is generated using the inference model 234 of machine learning that inputs a fluorescence staining image (spectral data) before fluorescence separation and staining information such as a combination of antibody dyes and outputs morphological information. Therefore, it becomes possible to execute fluorescence separation processing using morphological information in addition to the reference spectrum in the subsequent fluorescence separation processing (S2004). Thereby, for example, it becomes possible to suppress a problem in which the fluorescence separation result differs for each pixel within a region where one cell is projected.

[0115] Other configurations, operations, and effects may be the same as those of the above-described first embodiment, and thus detailed description is omitted here. Further, in the present embodiment, the case where the fluorescence separation processing executed by the separation processing unit 231 is fluorescence separation processing using LSM or the like is illustrated, but it is not limited thereto. For example, similar to the first embodiment, it is also possible to perform fluorescence separation processing using the inference model 134.

[0116] <4. Third Embodiment> Generally, machine learning image recognition technologies include techniques such as classification (classifying whether an image is a cat or a dog), detection (finding the target with a bounding box), and segmentation (acquiring and labeling regions in units of one pixel). When generating morphological information in the form of a binary mask from an input image as in the second embodiment described above, it is necessary to adopt segmentation among the above-mentioned techniques. Therefore, in the third embodiment, several examples will be given to explain the procedure for constructing the inference model 234 according to the second embodiment using segmentation.

[0117] (4.1. First example of procedure) The first example of the procedure exemplifies the case of constructing the inference model 234 in one step. FIG. 10 is a schematic diagram for explaining a method of constructing an inference model according to the first example of the procedure of this embodiment. As shown in FIG. 10, in the first example of the procedure, first, a sample image and a correct answer image are input to the model generation unit 233.

[0118] Here, the sample image may be, for example, a fluorescence-stained image. This sample image may be either stained or unstained. Also, when it is stained, various stains such as HE staining and fluorescence antibody staining may be adopted for the staining.

[0119] Also, the correct answer image may be, for example, morphological information (binary mask of the target expression map). In this morphological information, regions such as tissues, cells, and nuclei, and regions such as combinations of antibodies and fluorescent dyes may be information represented by a binary mask.

[0120] The model generation unit 233 learns information such as tissues, cells, and nuclei in the sample image and information on combinations of antibodies and fluorescence by performing segmentation in the RNN, for example, to acquire and label regions in units of one pixel. As a result, an inference model 234 for outputting morphological information is constructed.

[0121] In such a one-step construction method, it is possible to obtain the merit that morphological information, which is the output of one inference model 234, can be obtained.

[0122] (4.2. Second Procedure Example) In the second procedure example, the case of constructing the inference model 234 in two steps is illustrated. FIG. 11 is a schematic diagram for explaining a method of constructing an inference model according to the second procedure example of the present embodiment. As shown in FIG. 11, in the second procedure example, a sample image and a correct answer image are input to the model generation unit 233, and in step S3000, an inference model 234A is constructed as its product, and an input image group composed of individual images such as tissues, cells, and nuclei, and correct answer label information are input to the model generation unit 233, and in step S3004, an inference model 234B is constructed as its product.

[0123] In step S3000, the model generation unit 233 acquires regions such as tissues, cells, and nuclei in the sample image by, for example, performing segmentation in the RNN to acquire and label regions in units of one pixel. As a result, an inference model 234A for outputting morphological information is constructed as its product.

[0124] Then, in step S3004, the model generation unit 233 learns information regarding the combination of the antibody and fluorescence by, for example, performing classification on each of the regions acquired in step S3000. As a result, an inference model 234B for outputting morphological information is constructed as its product.

[0125] According to such a two-step construction method, when changing information regarding the combination of the antibody and fluorescence, it is possible to obtain the merit that it is possible to cope by replacing only the second-stage inference model 234B. That is, there is a merit that it is easier to change the inference model as compared with the one-step construction method according to the first procedure example.

[0126] <5. Fourth Embodiment> In the above-described second embodiment, the case where the fluorescence separation process is performed using the autofluorescence reference spectrum (and the fluorescence reference spectrum) to extract the spectrum for each fluorescent substance from the fluorescence spectrum was exemplified. In contrast, in the fourth embodiment, the case where the fluorescence spectrum for each fluorescent substance is directly extracted from the stained section will be exemplified.

[0127] FIG. 12 is a block diagram showing a schematic configuration example of the separation processing unit according to the present embodiment. In the information processing apparatus 100 according to the present embodiment, the separation processing unit 131 is replaced with the separation processing unit 232 shown in FIG. 12.

[0128] As shown in FIG. 12, the separation processing unit 232 includes a color separation unit 2321, a spectrum extraction unit 2322, and a dataset creation unit 2323.

[0129] The color separation unit 2321 separates the fluorescence spectrum of the stained section (also referred to as a stained sample) input from the fluorescence signal storage unit 122 for each fluorescent substance.

[0130] The spectrum extraction unit 2322 is a configuration for improving the autofluorescence spectrum so as to obtain a more accurate color separation result, and adjusts the autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 to obtain a more accurate color separation result.

[0131] The dataset creation unit 2323 creates a dataset of the autofluorescence reference spectrum from the spectrum extraction result input from the spectrum extraction unit 2322.

[0132] More specifically, the spectrum extraction unit 2322 executes spectrum extraction processing using non-negative matrix factorization (NMF), singular value decomposition (SVD), etc. on the autofluorescence reference spectrum input from the information storage unit 121, and inputs the result to the dataset creation unit 2323. Note that in the spectrum extraction processing according to the present embodiment, for example, an autofluorescence reference spectrum for each cell tissue and / or type using a tissue microarray (TMA) is extracted.

[0133] Here, as a method for extracting the autofluorescence spectrum from an unstained section, generally principal component analysis (hereinafter referred to as "PCA: Principal Component Analysis") can be used. However, when the concatenated autofluorescence spectrum is used for processing as in the present embodiment, PCA is not necessarily suitable. Therefore, the spectrum extraction unit 1322 according to the present embodiment extracts the autofluorescence reference spectrum from the unstained section by performing non-negative matrix factorization (NMF) instead of PCA.

[0134] FIG. 13 is a diagram for explaining the outline of NMF. As shown in FIG. 13, NMF decomposes a non-negative N-row M-column (N×M) matrix A into a non-negative N-row k-column (N×k) matrix W and a non-negative k-row M-column (k×M) matrix H. The matrices W and H are determined such that the mean squared error D between the matrix A and the product (W*H) of the matrices W and H is minimized. In the present embodiment, the matrix A corresponds to the spectrum before the autofluorescence reference spectrum is extracted (N is the number of pixels and M is the number of wavelength channels), and the matrix H corresponds to the extracted autofluorescence reference spectrum (k is the number of autofluorescence reference spectra (in other words, the number of autofluorescent substances) and M is the number of wavelength channels). Here, the mean squared error D is represented by the following equation (10). Note that "norm(D, 'fro')" refers to the Frobenius norm of the mean squared error D.

[0135]

Equation

[0136] The factorization in NMF uses an iterative method that starts with random initial values for the matrices W and H. In NMF, the value of k (the number of autofluorescence reference spectra) is essential, but the initial values of the matrices W and H are not essential and can be set as an option. When the initial values of the matrices W and H are set, the solution becomes constant. On the other hand, when the initial values of the matrices W and H are not set, these initial values are set randomly and the solution does not become constant.

[0137] The specimen 20 differs in nature and autofluorescence spectrum according to the type of tissue used, the type of disease targeted, the attributes of the subject, or the lifestyle of the subject, etc. Therefore, as described above, the information processing apparatus 100 according to the second embodiment can realize a more accurate fluorescence separation process by actually measuring the autofluorescence reference spectrum for each specimen 20.

[0138] Note that the matrix A, which is the input of NMF, is a matrix consisting of the same number of rows as the number of pixels N (= Hpix × Vpix) of the specimen image and the same number of columns as the number of wavelength channels M. Therefore, when the number of pixels of the specimen image is large or the number of wavelength channels M is large, the matrix A becomes a very large matrix, increasing the calculation cost of NMF and lengthening the processing time.

[0139] In such a case, for example, as shown in FIG. 14, by clustering into the specified number of classes N (<Hpix × Vpix) of the number of pixels N (= Hpix × Vpix) of the specimen image, it is possible to suppress the redundancy of the processing time due to the enlargement of the matrix A.

[0140] In clustering, for example, among the specimen images, spectra similar in the wavelength direction or intensity direction are classified into the same class. As a result, an image with a smaller number of pixels than the specimen image is generated, and thus it is possible to reduce the scale of the matrix A' with this image as the input.

[0141] The dataset creation unit 2323 creates a dataset (hereinafter also referred to as an autofluorescence dataset) necessary for the color separation process by the color separation unit 2321 from the autofluorescence reference spectra for each cell tissue and / or type input from the spectrum extraction unit 2322, and inputs the created autofluorescence dataset to the color separation unit 2321.

[0142] The color separation unit 2321 separates the fluorescence spectrum into spectra for each molecule by performing color separation processing on the fluorescence spectrum of the stained sample input from the fluorescence signal storage unit 122, using the fluorescence reference spectrum and the autofluorescence reference spectrum input from the information storage unit 121 and the autofluorescence dataset input from the dataset creation unit 2323. Note that NMF or SVD can be used for the color separation processing.

[0143] For the NMF executed by the color separation unit 2321 according to the present embodiment, for example, the NMF (see FIG. 13 etc.) for extracting the autofluorescence spectrum from the unstained section described in the first embodiment can be used after being modified as follows.

[0144] That is, in the present embodiment, the matrix A corresponds to a plurality of specimen images (where N is the number of pixels and M is the number of wavelength channels) obtained from the stained section, the matrix H corresponds to the fluorescence spectra for each of the extracted fluorescent substances (where k is the number of fluorescence spectra (in other words, the number of fluorescent substances) and M is the number of wavelength channels), and the matrix W corresponds to the image of each fluorescent substance after fluorescence separation. Note that the matrix D is the mean squared residual.

[0145] Also, in the present embodiment, the initial value of the NMF may be random, for example. However, if the results differ for each execution count of the NMF, it is necessary to set an initial value in order to prevent this.

[0146] Note that when performing fluorescence separation processing using an algorithm such as NMF that rearranges the order of the corresponding spectra by a calculation algorithm, or an algorithm that requires rearranging the order of the spectra in order to improve the processing speed and the convergence of the results, which fluorescent dye each of the fluorescence spectra obtained as the matrix H corresponds to can be specified, for example, by obtaining the Pearson product-moment correlation coefficient (or cosine similarity) for each of all combinations.

[0147] Also, when using the default function (NMF) of MATLAB (registered trademark), even if an initial value is given, the output order changes. Although it is possible to fix this with a self-defined function, even if the order is changed using the default function, as described above, by using the Pearson product-moment correlation coefficient (or cosine similarity), it is possible to obtain the correct combination of substances and fluorescence spectra.

[0148] As described above, by configuring to solve NMF with the specimen image obtained from the stained section as matrix A, it becomes possible to directly extract the fluorescence spectra for each fluorescent substance from the stained section without the need for procedures such as photographing non-stained sections or generating autofluorescence reference spectra. Thereby, it becomes possible to significantly reduce the time and work cost required for the fluorescence separation process.

[0149] Furthermore, in this embodiment, since the fluorescence spectra for each fluorescent substance are extracted from the specimen image obtained from the same stained section, it is possible to obtain more accurate fluorescence separation results compared to the case of using, for example, the autofluorescence spectrum obtained from a non-stained section different from the stained section.

[0150] Other configurations, operations, and effects may be the same as those of the above-described embodiments, and thus detailed descriptions are omitted here.

[0151] In this embodiment, when extracting the fluorescence spectrum for each fluorescent substance, a concatenated fluorescence spectrum in which the fluorescence spectra for each fluorescent substance are concatenated may be used. When using the concatenated fluorescence spectrum, the extraction unit of the separation processing unit 131 concatenates a plurality of fluorescence spectra obtained by the fluorescence signal acquisition unit 112, and executes a process of extracting the fluorescence spectrum for each fluorescent substance with respect to the concatenated fluorescence spectrum generated thereby.

[0152] <6. Fifth Embodiment> In the above-described fourth embodiment, methods for enhancing the quantitativeness such as the concentration of the stained dye can include the following methods.

[0153] FIG. 15 is a flowchart for explaining the flow of NMF according to the fifth embodiment. FIG. 16 is a diagram for explaining the flow of processing in the first loop of the NMF shown in FIG. 15.

[0154] As shown in FIG. 15, in the NMF according to this embodiment, first, the variable i is reset to zero (step S401). The variable i indicates the number of times the factorization in NMF is repeated. Therefore, the matrix H0 shown in FIG. 16(a) corresponds to the initial value of the matrix H. In this example, for clarity, the position of the stained fluorescence spectrum in the matrix H is set as the bottom row, but it is not limited to this, and various changes such as the top row or the middle row are possible.

[0155] Next, in the NMF according to this embodiment, similar to the normal NMF, a non - negative N - row M - column (N×M) matrix A is divided by a non - negative N - row k - column (N×k) matrix W i to obtain a non - negative k - row M - column (k×M) matrix H i+1 (step S402). Thereby, for example, in the first loop, a matrix H1 as shown in FIG. 16(b) is obtained.

[0156] Next, the row of the fluorescence staining spectrum in the matrix H i+1 obtained in step S402 is replaced with the row of the fluorescence staining spectrum of the initial value, that is, the row of the stained fluorescence spectrum in the matrix H0 (step S403). That is, in this embodiment, the fluorescence staining spectrum in the matrix H is fixed to the initial value. For example, in the first loop, as shown in FIG. 16(c), by replacing the bottom row in the matrix H1 with the bottom row in the matrix H0, the stained fluorescence spectrum can be fixed.

[0157] Next, in the NMF according to this embodiment, the matrix A is divided by the matrix H i+1 obtained in step S403 to obtain the matrix W i+1 (step S404).

[0158] After that, in the NMF according to this embodiment, as in the normal NMF, it is determined whether or not the mean square error D satisfies a predetermined branching condition (step S405). If it is satisfied (YES in step S405), the finally obtained matrix H i+1 and W i+1 are used as solutions to end the NMF. On the other hand, if the predetermined branching condition is not satisfied (NO in step S405), after the variable i is incremented by 1 (step S406), the process returns to step S402 and the next loop is executed.

[0159] As described above, in the first method, in the spectrum extraction and color separation of a multi-stained pathological section image (specimen image), it is possible to directly color-separate a stained sample using NMF while ensuring the quantitativeness of the stained fluorescence, that is, while maintaining the spectrum of the stained fluorescence, without the need to photograph an unstained sample of the same tissue section for autofluorescence spectrum extraction. Thereby, for example, it is possible to achieve accurate color separation as compared with the case of using another specimen. It is also possible to reduce the labor of photographing another specimen.

[0160] Note that, as a method for minimizing the mean square error D, a method using a recurrence formula for minimizing D = |A - WH| 2 or a method using a quasi-Newton method (also called the DFP (Davidon-Fletcher-Powell) method) or the BFGS (Broyden-Fletcher-Goldfarb-Shanno) method, etc. can be considered. In those cases, as a method for fixing the stained fluorescence spectrum to the initial value, the following methods can be considered.

[0161] (6.1 Method for fixing the stained fluorescence spectrum in minimizing the mean square error D using a recurrence formula) D = |A - WH| 2 In the method of minimizing the mean square error D using a recurrence formula for minimizing, a loop process consisting of multiplicative update formulas as shown in the following formulas (11) and (12) is executed. In formulas (11) and (12), A = (a i,j ) N×Mand H = (h i,j ) k×M and W = (w i,j ) N×k . Also, t h, t w are the transposed matrices of the submatrices h and w, respectively. [Number] [Number]

[0162] In such a loop process, in order to fix the staining fluorescence spectrum to the initial value, a method of inserting the step of executing the following formula (13) between the step of executing formula (11) and the step of executing formula (12) can be used. Note that formula (13) indicates overwriting the submatrix corresponding to the staining fluorescence spectrum in the updated w i,j k+1 with the submatrix w i,j(part) k which is the initial value of the staining fluorescence spectrum. [Number]

[0163] (Method for fixing the staining fluorescence spectrum in minimizing the mean square error D using the 6.2 DFP method, BFGS method, etc.) Also, in the method of minimizing the mean square error D using the DFP method, BFGS method, etc., when the mean square error D to be minimized is D(x) and x is the coordinate (at the k-th update, x k = (a1, a2,..., an) k ), D(x) is minimized by going through the following steps. In the following steps, B represents the Hessian matrix. ·x k+1 = x k - αB k -1 Update the coordinate by D’(x k ) · New coordinate x k+1Displacement to the gradient in ·y k = D’(x k+1 ) - D’(x k ) to update the inverse Hessian matrix B k+1 -1

[0164] For the update of the Hessian matrix Bk+1, for example, various methods such as the DFP method shown in the following equation (14) and the BFGF method shown in equation (15) can be applied.

Equation

Equation

[0165] In the method of minimizing the mean squared error D using such methods as the DFP method and the BFGS method, there are several methods for fixing arbitrary coordinates, that is, for fixing the chromatic fluorescence spectrum to the initial value. For example, it is possible to fix the chromatic fluorescence spectrum to the initial value by executing the following process (1) or process (2) at the timing of updating the coordinates. (1) -αB k -1 D’(x k ) = 0, that is, replace the partial derivative D’(x k ) with zero (2) After calculating x k+1 after coordinate update, a part of the obtained coordinate x k+1 is forcibly replaced with x k (or a part thereof)

[0166] <7. Hardware configuration example> Subsequently, with reference to FIG. 17, a hardware configuration example of the information processing apparatus 100 according to each embodiment and modification will be described. FIG. 17 is a block diagram showing a hardware configuration example of the information processing apparatus 100. Various processes performed by the information processing apparatus 100 are realized by the cooperation of software and the hardware described below.

[0167] ​As shown in FIG. 17, the information processing apparatus 100 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing apparatus 100 also includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing apparatus 100 may have a processing circuit such as a DSP or an ASIC instead of or together with the CPU 901.

[0168] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operations within the information processing apparatus 100 according to various programs. The CPU 901 may be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. The CPU 901 can embody, for example, at least the processing unit 130 and the control unit 150 of the information processing apparatus 100.

[0169] The CPU 901, the ROM 902, and the RAM 903 are interconnected by a host bus 904a including a CPU bus or the like. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via the bridge 904. Note that it is not necessarily required to separately configure the host bus 904a, the bridge 904, and the external bus 904b, and these functions may be implemented in one bus.

[0170] The input device 906 is implemented by a device through which information is input by an implementer, such as a mouse, keyboard, touch panel, button, microphone, switch, lever, etc. Further, the input device 906 may be, for example, a remote control device using infrared rays or other radio waves, or an external connection device such as a mobile phone or PDA corresponding to the operation of the information processing device 100. Furthermore, the input device 906 may include, for example, an input control circuit that generates an input signal based on information input by the implementer using the above input means and outputs it to the CPU 901. By operating this input device 906, the implementer can input various data to the information processing device 100 or instruct a processing operation. The input device 906 can, for example, embody at least the operation unit 160 of the information processing device 100.

[0171] The output device 907 is formed of a device capable of visually or auditorily notifying the implementer of the acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, and lamps, acoustic output devices such as speakers and headphones, and printer devices. The output device 907 can, for example, embody at least the display unit 140 of the information processing device 100.

[0172] The storage device 908 is a device for storing data. The storage device 908 is realized, for example, by a magnetic storage unit device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. This storage device 908 stores programs executed by the CPU 901, various data, and various data acquired from the outside. The storage device 908 can, for example, embody at least the storage unit 120 of the information processing device 100.

[0173] Drive 909 is a reader / writer for a storage medium and is built into or externally attached to the information processing apparatus 100. Drive 909 reads information recorded on a removable storage medium such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to the RAM 903. Further, Drive 909 can also write information to the removable storage medium.

[0174] The connection port 911 is an interface for connecting to an external device, and is a connection port for an external device capable of data transmission, for example, by USB (Universal Serial Bus).

[0175] The communication device 913 is, for example, a communication interface formed of a communication device or the like for connecting to the network 920. The communication device 913 is, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). Further, the communication device 913 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. This communication device 913 can transmit and receive signals, for example, in accordance with a predetermined protocol such as TCP / IP, between the Internet and other communication devices.

[0176] In this embodiment, the sensor 915 includes a sensor capable of acquiring a spectrum (for example, an imaging element or the like), and may include other sensors (for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a pressure sensor, a sound sensor, or a distance measuring sensor). The sensor 915 can, for example, embody at least the fluorescence signal acquisition unit 112 of the information processing apparatus 100.

[0177] Note that the network 920 is a wired or wireless transmission path for information transmitted from devices connected to the network 920. For example, the network 920 may include public line networks such as the Internet, telephone line networks, satellite communication networks, and various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), etc. Further, the network 920 may include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0178] As described above, a hardware configuration example capable of realizing the functions of the information processing apparatus 100 has been shown. Each of the above-described components may be realized using general-purpose members, or may be realized by hardware specialized for the functions of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at the time of implementing the present disclosure.

[0179] Note that it is possible to create a computer program for realizing each function of the information processing apparatus 100 as described above and install it on a PC or the like. Further, it is possible to provide a computer-readable recording medium storing such a computer program. The recording medium includes, for example, magnetic disks, optical disks, magneto-optical disks, flash memories, and the like. Further, the above-described computer program may be distributed via a network, for example, without using a recording medium.

[0180] <8. Remarks> In the above, generally, since actual sample information and reagent information are often different from catalog values and literature values, it has been explained that it is more preferable that the sample information and reagent information be independently measured and managed within the information processing system according to the present disclosure. Therefore, as a remark, with reference to FIGS. 18 and 19, the fact that actual sample information and reagent information are different from catalog values and literature values will be explained.

[0181] FIG. 18 is a diagram showing the comparison result between the measured value and the catalog value of the spectral information of PE (Phycoerythrin), which is a kind of fluorescent component. Further, FIG. 19 is a diagram showing the comparison result between the measured value and the catalog value of the spectral information of BV421 (Brilliant Violet 421), which is a kind of fluorescent component. Note that as the measured values, the measurement results using samples in which these fluorescent components and the encapsulant are formulated are shown.

[0182] As shown in FIGS. 18 and 19, although the positions of the peaks in the spectral information are almost the same between the measured values and the catalog values, the shapes of the spectra on the longer wavelength side than the peak wavelength are different from each other. Therefore, when the catalog value is used as the spectral information, the separation accuracy between the fluorescence signal and the autofluorescence signal becomes low.

[0183] Note that not only the spectral information but also various information included in the specimen information and the reagent information are generally more preferably measured independently within the information processing system according to the present disclosure from the viewpoint of accuracy.

[0184] <9. Modification Example of System Configuration> Note that the information processing system (see FIG. 1) according to the above-described embodiment can also have a server-client type system configuration. FIG. 20 is a block diagram showing a schematic configuration example of an information processing system configured in a server-client type.

[0185] As shown in FIG. 20, the information processing system according to this modification example includes a client terminal 100A, a server device 100B, and a database 200, and these are connected to be communicable with each other via a predetermined network 300. Various networks such as a WAN (Wide Area Network) (including the Internet), a LAN (Local Area Network), a public line network, and a mobile communication network can be applied to the network 300.

[0186] The client terminal 100A is a terminal device used by doctors, researchers, etc. For example, it includes at least an acquisition unit 110, a display unit 140, a control unit 150, and an operation unit 160 in the configuration shown in FIG. 1.

[0187] On the other hand, the server device 100B is not limited to a single server and may be composed of a plurality of servers, or may be a cloud server. This server device 100B may include, for example, at least one of an information storage unit 121, a fluorescence signal storage unit 122, a fluorescence separation result storage unit 123, a separation processing unit 131, an image generation unit 132, and a model generation unit 133 in the configuration shown in FIG. 1. Among these configurations, the configurations that are not included in the server device 100B may be provided in the client terminal 100A.

[0188] The client terminals 100A connected to the same server device 100B are not limited to one and may be multiple. In that case, the multiple client terminals 100A may be introduced into different hospitals.

[0189] With such a system configuration, not only can a system with higher computing power be provided to users such as doctors and researchers, but also more information can be accumulated in the database 200. This suggests that it facilitates the expansion to a system that executes machine learning, etc. on the big data accumulated in the database 200.

[0190] However, it is not limited to the above configuration, and various changes are possible, for example, a configuration in which only the database 200 is shared by a plurality of information processing devices 100 via the network 300.

[0191] <10. Application Example 1> In the above-described embodiments, the case where the technology according to the present disclosure is applied to so-called multiple flow site imaging (MFI) for acquiring a two-dimensional fluorescence image of the fluorescence staining specimen 30 which is a multiply stained tissue section has been described. However, the present disclosure is not limited thereto, and the technology according to the present disclosure can also be applied to, for example, an imaging cytometer or the like for acquiring a two-dimensional fluorescence image of microparticles such as multiply stained cells.

[0192] <11. Application Example 2> In addition, the technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be applied to a pathological diagnosis system or its support system (hereinafter referred to as a diagnosis support system) for a doctor or the like to observe cells and tissues collected from a patient and diagnose a lesion. This diagnosis support system may be a WSI (Whole Slide Imaging) system for diagnosing or supporting a lesion based on an image acquired using digital pathology technology.

[0193] FIG. 21 is a diagram showing an example of a schematic configuration of a diagnosis support system 5500 to which the technology according to the present disclosure is applied. As shown in FIG. 21, the diagnosis support system 5500 includes one or more pathology systems 5510. Further, it may include a medical information system 5530 and a derivation device 5540.

[0194] Each of the one or more pathology systems 5510 is a system mainly used by a pathologist and is introduced into, for example, a research institute or a hospital. Each pathology system 5510 may be introduced into different hospitals and is connected to the medical information system 5530 and the derivation device 5540 via various networks such as a WAN (Wide Area Network) (including the Internet), a LAN (Local Area Network), a public line network, and a mobile communication network.

[0195] Each pathology system 5510 includes a microscope 5511, a server 5512, a display control device 5513, and a display device 5514.

[0196] The microscope 5511 has the function of an optical microscope, images an object to be observed stored on a glass slide, and acquires a pathological image which is a digital image. The object to be observed is, for example, tissue or cells collected from a patient, and may be a piece of an organ, saliva, blood, etc.

[0197] The server 5512 stores and preserves the pathological image acquired by the microscope 5511 in a storage unit (not shown). Also, when the server 5512 receives a browsing request from the display control device 5513, it searches for the pathological image from the storage unit (not shown) and sends the retrieved pathological image to the display control device 5513.

[0198] The display control device 5513 sends a browsing request for the pathological image received from the user to the server 5512. Then, the display control device 5513 causes the pathological image received from the server 5512 to be displayed on a display device 5514 using a liquid crystal, EL (Electro-Luminescence), CRT (Cathode Ray Tube), etc. Note that the display device 5514 may support 4K or 8K, and is not limited to one unit, and may be a plurality of units.

[0199] Here, when the object to be observed is a solid such as a piece of an organ, this object to be observed may be, for example, a stained thin section. The thin section may be produced, for example, by thinly slicing a block piece cut out from a specimen such as an organ. Also, when thinly slicing, the block piece may be fixed with paraffin or the like.

[0200] For staining of the thin section, various stains such as general stains showing the morphology of tissues such as HE (Hematoxylin-Eosin) staining and immunostains showing the immune state of tissues such as IHC (Immunohistochemistry) staining may be applied. At that time, one thin section may be stained using a plurality of different reagents, or two or more thin sections (also referred to as adjacent thin sections) continuously cut out from the same block piece may be stained using different reagents from each other.

[0201] The microscope 5511 may include a low-resolution imaging unit for imaging with low resolution and a high-resolution imaging unit for imaging with high resolution. The low-resolution imaging unit and the high-resolution imaging unit may have different optical systems or the same optical system. In the case of the same optical system, the resolution of the microscope 5511 may be changed according to the imaging target.

[0202] The glass slide on which the observation object is accommodated is placed on a stage located within the angle of view of the microscope 5511. First, the microscope 5511 acquires an overall image within the angle of view using the low-resolution imaging unit, and identifies the region of the observation object from the acquired overall image. Subsequently, the microscope 5511 divides the region where the observation object exists into a plurality of divided regions of a predetermined size, and acquires high-resolution images of the respective divided regions by sequentially imaging each divided region with the high-resolution imaging unit. In switching the target divided region, the stage may be moved, the imaging optical system may be moved, or both of them may be moved. Also, each divided region may overlap with an adjacent divided region in order to prevent the occurrence of an imaging omission region due to an unintended slip of the glass slide. Furthermore, the overall image may include identification information for associating the overall image with the patient. This identification information may be, for example, a character string, a QR code (registered trademark), or the like.

[0203] The high-resolution images acquired by the microscope 5511 are input to the server 5512. The server 5512 divides each high-resolution image into partial images (hereinafter referred to as tile images) of a smaller size. For example, the server 5512 divides one high-resolution image into a total of 100 tile images of 10×10 in the vertical and horizontal directions. In that case, if adjacent divided regions overlap, the server 5512 may perform stitching processing on the adjacent high-resolution images using a technique such as template matching. In that case, the server 5512 may also divide the entire high-resolution image bonded by the stitching processing to generate tile images. However, the generation of tile images from the high-resolution images may also be performed before the above-described stitching processing.

[0204] In addition, the server 5512 can generate tile images of a smaller size by further dividing the tile images. The generation of such tile images may be repeated until tile images of a size set as the minimum unit are generated.

[0205] When generating tile images of the minimum unit in this way, the server 5512 executes a tile synthesis process of generating one tile image by synthesizing a predetermined number of adjacent tile images for all tile images. This tile synthesis process can be repeated until finally one tile image is generated. Through such a process, a group of tile images having a pyramid structure in which each layer is composed of one or more tile images is generated. In this pyramid structure, the number of pixels of a tile image in a certain layer is the same as that of a tile image in a layer different from this layer, but their resolutions are different. For example, when synthesizing 4 tile images in total of 2×2 to generate one tile image in the upper layer, the resolution of the tile image in the upper layer is 1 / 2 times the resolution of the tile image in the lower layer used for the synthesis.

[0206] By constructing such a group of tile images having a pyramid structure, it becomes possible to switch the level of detail of the object to be observed displayed on the display device according to the layer to which the tile image to be displayed belongs. For example, when the tile image in the bottom layer is used, a narrow area of the object to be observed can be displayed in detail, and the wider the area of the object to be observed is, the coarser it can be displayed as the tile image in the upper layer is used.

[0207] The generated group of tile images having a pyramid structure is stored in a storage unit (not shown), for example, together with identification information (referred to as tile identification information) that can uniquely identify each tile image. When the server 5512 receives a request to acquire a tile image including tile identification information from another device (for example, the display control device 5513 or the derivation device 5540), the server 5512 transmits the tile image corresponding to the tile identification information to the other device.

[0208] Note that the tile images, which are pathological images, may be generated for each imaging condition such as the focal length and staining conditions. When tile images are generated for each imaging condition, other pathological images corresponding to imaging conditions different from the specific imaging condition, which are of the same region as the specific pathological image, may be arranged and displayed together with the specific pathological image. The specific imaging condition may be specified by the viewer. Also, when a plurality of imaging conditions are specified by the viewer, the pathological images of the same region corresponding to each imaging condition may be arranged and displayed together.

[0209] Further, the server 5512 may store the tile image group with a pyramid structure in another storage device other than the server 5512, for example, a cloud server or the like. Furthermore, part or all of the above tile image generation processing may be executed by a cloud server or the like.

[0210] The display control device 5513 extracts a desired tile image from the tile image group with a pyramid structure in response to an input operation from the user, and outputs this to the display device 5514. By such processing, the user can obtain a feeling as if observing the observation object while changing the observation magnification. That is, the display control device 5513 functions as a virtual microscope. The virtual observation magnification here actually corresponds to the resolution.

[0211] Note that any method may be used for the method of capturing a high-resolution image. A high-resolution image may be obtained by capturing divided regions while repeating the stop and movement of the stage, or a high-resolution image on a strip may be obtained by capturing divided regions while moving the stage at a predetermined speed. Also, the process of generating tile images from the high-resolution image is not an essential configuration, and an image with a stepwise changing resolution may be generated by stepwise changing the resolution of the entire high-resolution image stitched by the stitching process. Even in this case, it is possible to stepwise present to the user from a low-resolution image of a wide area to a high-resolution image of a narrow area.

[0212] The medical information system 5530 is a so-called electronic medical record system, which stores information for identifying patients, disease information of patients, examination information and image information used for diagnosis, diagnosis results, information related to diagnosis such as prescription drugs, etc. For example, a pathological image obtained by imaging an object to be observed of a certain patient can be once stored via the server 5512 and then displayed on the display device 5514 by the display control device 5513. A pathologist using the pathology system 5510 performs a pathological diagnosis based on the pathological image displayed on the display device 5514. The pathological diagnosis result performed by the pathologist is stored in the medical information system 5530.

[0213] The derivation device 5540 can execute an analysis including fluorescence separation processing on the pathological image. A learning model created by machine learning can be used for this analysis. The derivation device 5540 may derive, as the analysis result, a classification result of a specific region, an identification result of a tissue, etc. Further, the derivation device 5540 may derive identification results such as cell information, number, position, luminance information, etc. and scoring information for them. These information derived by the derivation device 5540 may be displayed on the display device 5514 of the pathology system 5510 as diagnostic support information.

[0214] Note that the derivation device 5540 may be a server system composed of one or more servers (including cloud servers), etc. Also, the derivation device 5540 may be configured to be incorporated into, for example, the display control device 5513 or the server 5512 within the pathology system 5510. That is, various analyses on the pathological image may be executed within the pathology system 5510.

[0215] The technology according to the present disclosure can be suitably applied to the entire diagnostic support system 5500 among the configurations described above. Specifically, the acquisition unit 110 corresponds to the microscope 5511, the display control device 5513 corresponds to the control unit 150, the display device 5514 corresponds to the display unit 140, the remaining configuration of the pathology system 5510 and the derivation device 5540 correspond to the remaining configuration of the information processing device 100, and the medical information system 5530 can correspond to the database 200. Thus, by applying the technology according to the present disclosure to the diagnostic support system 5500, effects such as enabling doctors and researchers to more accurately diagnose and analyze lesions can be achieved.

[0216] Note that the configuration described above is not limited to the diagnostic support system, and can also be applied to all biological microscopes such as confocal microscopes, fluorescence microscopes, and video microscopes. Here, the observation object may be a biological sample such as cultured cells, fertilized eggs, sperm, a biological material such as a cell sheet or three-dimensional cell tissue, or a living body such as a zebrafish or a mouse. Further, the observation object is not limited to a glass slide, and can also be observed in a state stored in a well plate, a petri dish, or the like.

[0217] Furthermore, a moving image may be generated from a still image of an observation object acquired using a microscope. For example, a moving image may be generated from still images captured continuously for a predetermined period, or an image sequence may be generated from still images captured at a predetermined interval. By generating a moving image from a still image in this way, it becomes possible to analyze dynamic characteristics of the observation object, such as the movement of cancer cells, nerve cells, myocardial tissue, sperm, etc., such as pulsation, elongation, migration, or the division process of cultured cells or fertilized eggs, using machine learning.

[0218] As described above, the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that those having ordinary knowledge in the technical field of the present disclosure can conceive of various modification examples or correction examples within the scope of the technical idea described in the claims, and these are also naturally understood to belong to the technical scope of the present disclosure.

[0219] Also, the effects described in this specification are illustrative or exemplary only and not limiting. That is, the technology according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description of this specification, together with or instead of the above effects.

[0220] Note that the following configurations also belong to the technical scope of the present disclosure. (1) An information processing apparatus including a separation unit that separates a fluorescence signal derived from the fluorescence reagent from the fluorescence image based on a fluorescence image of a biological sample containing cells, a reference spectrum derived from the biological sample or the fluorescence reagent, and morphological information of the cells. (2) The information processing apparatus according to (1), wherein the separation unit further separates a fluorescence signal derived from the biological sample from the fluorescence image. (3) The information processing apparatus according to (1), wherein the morphological information includes distribution information of a target in the biological sample. (4) The information processing apparatus according to (3), wherein the target is an antigen in the biological sample, and the distribution information includes the distribution of the expression level of the antigen. (5) The information processing apparatus according to (4), wherein the morphological information includes a binary mask image indicating the distribution of the expression level of the antigen. (6) The information processing apparatus according to any one of (1) to (5), wherein the fluorescence reagent includes an antibody labeled with a fluorescent dye. (7) The information processing apparatus according to any one of (1) to (6), further including an image generation unit that generates a fluorescence image corrected based on the separated fluorescence signal. (8) The information processing apparatus according to any one of (1) to (7), further including an extraction unit that optimizes a reference spectrum derived from the biological sample or the fluorescence reagent. (9) The separation unit is the information processing apparatus according to (8) above, which separates the fluorescence signal of the fluorescence image by using the least squares method, weighted least squares method, or restricted least squares method using the reference spectrum and the morphological information. (10) The separation unit is the information processing apparatus according to (8) above, which separates the fluorescence signal of the fluorescence image by inputting the fluorescence image, the reference spectrum, and the morphological information into a first image generation model. (11) The first image generation model is the learned model according to (10) above, which is learned with the color separation information obtained by separating the fluorescence signal of the fluorescence image as teacher data. (12) The separation unit is the information processing apparatus according to any one of (8) to (11) above, which further separates the fluorescence signal of the fluorescence image based on the bright-field image and the unstained image of the biological sample. (13) The separation unit is the information processing apparatus according to (12) above, which separates the fluorescence signal of the fluorescence image by inputting the fluorescence image, the bright-field image, the unstained image, and the staining information into a second inference model. (14) The fluorescent reagent contains an antibody labeled with a fluorescent dye. The staining information contains information on the combination of the antibody and the fluorescent dye of the fluorescent reagent, which is the information processing apparatus according to (13) above. (15) The second inference model is the learned model according to (13) or (14) above, which is learned with the morphological information generated as a binary mask image as teacher data. (16) The morphological information learned as the teacher data contains the region information of the biological sample, which is the information processing apparatus according to (15) above. (17) The morphological information learned as the teacher data contains the region information of the biological sample obtained by segmentation. The region information of the biological sample includes region information for at least one or more of tissue, cells, and nuclei, and the information processing apparatus according to (16) above. (18) The region information of the biological sample further includes the staining information, and the information processing apparatus according to (17) above. (19) The second inference model further includes a third inference model, The third inference model is a learned model that learns the staining information specified by classification as teacher data for the region information of each of the tissues or cells. The information processing apparatus according to (18) above. (20) An information processing apparatus that acquires a fluorescence image of a biological sample containing cells and a reference spectrum derived from the biological sample or a fluorescent reagent, A program for causing the information processing apparatus to execute a process of separating a fluorescence signal derived from the fluorescent reagent from the fluorescence image based on the fluorescence image, the reference spectrum, and the morphological information of the cells. An information processing system comprising:

Explanation of Signs

[0221] 10 Fluorescent reagent 11 Reagent identification information 20 Specimen 21 Specimen identification information 30 Fluorescently stained specimen 100 Information processing apparatus 110 Acquisition unit 111 Information acquisition unit 112 Fluorescence signal acquisition unit 120 Storage unit 121 Information storage unit 122 Fluorescence signal storage unit 123 Fluorescence separation result storage unit 130 Processing unit 131, 231 Separation processing unit 132 Image generation unit 133, 233 Model generation unit 134, 234, 234A Inference model 140 display unit 150 control unit 160 operation unit 200 database 2311 fluorescence separation unit 2311a first fluorescence separation unit 2311b second fluorescence separation unit 2312 spectrum extraction unit

Claims

1. A separation unit that separates a fluorescence signal derived from the fluorescent reagent from the fluorescence image based on a fluorescence image of a biological sample containing cells, a reference spectrum derived from the biological sample or the fluorescent reagent, and morphological information of the cells; An extraction unit that optimizes a reference spectrum derived from the biological sample or the fluorescent reagent; Comprising; The separation unit further separates the fluorescence signal of the fluorescence image based on a bright-field image and an unstained image of the biological sample. An information processing apparatus.

2. The information processing apparatus according to claim 1, wherein the separation unit further separates a fluorescence signal derived from the biological sample from the fluorescence image.

3. The information processing apparatus according to claim 1, wherein the morphological information includes distribution information of a target in the biological sample.

4. The information processing apparatus according to claim 3, wherein the target is an antigen in the biological sample, and the distribution information includes a distribution of the expression level of the antigen.

5. The information processing apparatus according to claim 4, wherein the morphological information includes a binary mask image indicating a distribution of the expression level of the antigen.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the fluorescent reagent includes an antibody labeled with a fluorescent dye.

7. The information processing apparatus according to any one of claims 1 to 6, further comprising an image generation unit that generates a fluorescence image corrected based on the separated fluorescence signal.

8. The information processing apparatus according to claim 1, wherein the separation unit separates the fluorescence signal of the fluorescence image by a least squares method, a weighted least squares method, or a restricted least squares method using the reference spectrum and the morphological information.

9. The information processing apparatus according to claim 1, wherein the separation unit separates the fluorescence signal of the fluorescence image by inputting the fluorescence image, the reference spectrum, and the morphological information into a first image generation model.

10. The information processing apparatus according to claim 9, wherein the first image generation model is a learned model that is learned using color separation information obtained by separating the fluorescence signal of the fluorescence image as teacher data.

11. The information processing apparatus according to claim 1, wherein the separation unit separates the fluorescence signal of the fluorescence image by inputting the fluorescence image, the bright-field image, the unstained image, and staining information into a second inference model.

12. The fluorescent reagent includes an antibody labeled with a fluorescent dye, The information processing apparatus according to claim 11, wherein the staining information includes information on a combination of the antibody and the fluorescent dye of the fluorescent reagent.

13. The information processing apparatus according to claim 11 or 12, wherein the second inference model is a learned model obtained by training the morphological information generated as a binary mask image using the teacher data.

14. The information processing apparatus according to claim 13, wherein the morphological information learned as the teacher data includes region information of the biological sample.

15. The morphological information learned as the teacher data includes region information of the biological sample obtained by segmentation, The information processing apparatus according to claim 14, wherein the region information of the biological sample includes region information for at least one of tissue, cells, and nuclei.

16. The information processing apparatus according to claim 15, wherein the region information of the biological sample further includes the staining information.

17. The second inference model further includes a third inference model, The third inference model is a learned model obtained by training the staining information specified by classification using the region information of each of the tissues or cells as teacher data. The information processing apparatus according to claim 16.

18. An information processing apparatus that acquires a fluorescence image of a biological sample containing cells and a reference spectrum derived from the biological sample or a fluorescent reagent, A program for causing the information processing apparatus to execute a process of separating a fluorescence signal derived from the fluorescent reagent from the fluorescence image based on the fluorescence image, the reference spectrum, and the morphological information of the cells, and a process of optimizing the reference spectrum derived from the biological sample or the fluorescent reagent, An information processing system comprising: In the process of separating the fluorescence signal, the fluorescence signal of the fluorescence image is further separated based on a bright-field image and an unstained image of the biological sample. Information processing system.

Images