Information processor and information processing system

Abstract

To enable more accurate fluorescence separation.SOLUTION: The information processor according to the embodiment includes a separation unit for separating a fluorescence signal derived from a fluorescent reagent from a fluorescence image on the basis of a fluorescence image of a biological sample containing cells, a reference spectrum derived from one of the biological sample and the fluorescent reagent, and morphological information of the cells.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and an information processing system. [Background technology]

[0002] In recent years, advances in cancer immunotherapy have led to advances in fluorescent and multi-labeled immunostaining. For example, a method is being developed in which the autofluorescence spectrum is extracted from an unstained section of the same tissue block, and then the autofluorescence spectrum is used to separate the fluorescence of a stained section.

[0003] Furthermore, for example, Patent Document 1 below discloses a technology for approximating a fluorescence spectrum obtained by irradiating excitation light onto a microparticle multiply labeled with multiple fluorescent dyes, by a linear sum of single-stained spectra obtained from microparticles individually labeled with each fluorescent dye. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-18108 [Patent Document 2] JP 2019-45540 A [Patent Document 3] Japanese Patent Application Publication No. 2018-185759 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, the spread of cancer immunotherapy has led to the use of fluorescent and multi-marker immunostaining. To use a wider variety of fluorescent dyes in multicolor staining, accurate separation of both the fluorescent light between stained fluorescent materials and the fluorescent light between stained fluorescent materials and autofluorescence is required.

[0006] Therefore, the present disclosure has been made in consideration of the above circumstances, and provides a new and optimized information processing device and information processing system that are capable of performing fluorescence separation more accurately. [Means for solving the problem]

[0007] An information processing device according to an embodiment of the present disclosure includes a separation unit that separates a fluorescent signal derived from a fluorescent reagent from a fluorescent image of a biological sample containing cells, based on a reference spectrum derived from the biological sample or a fluorescent reagent, and morphological information of the cells. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment. [Figure 2] 5A and 5B are diagrams showing specific examples of fluorescence spectra acquired by a fluorescence signal acquisition unit. [Figure 3] 1 is a block diagram showing an example of the configuration of a microscope system in the case where the information processing system according to the first embodiment is realized as a microscope system. [Figure 4] 4 is a flowchart showing an example of a processing flow of fluorescence separation performed by the information processing device according to the first embodiment. [Figure 5] A schematic diagram for explaining an example of the flow of fluorescence separation processing using an inference model related to the first embodiment. [Figure 6] FIG. 1 is a diagram for explaining training of an inference model according to the first embodiment. [Figure 7] A schematic diagram showing an example of the flow of fluorescence separation processing using an inference model related to the second embodiment. [Figure 8] FIG. 10 is a block diagram showing a more specific example of the configuration of a separation processing unit according to the second embodiment. [Figure 9] FIG. 10 is a diagram for explaining training of an inference model according to the second embodiment. [Figure 10] A schematic diagram for explaining a method of constructing an inference model relating to the first example procedure of the third embodiment. [Figure 11] A schematic diagram for explaining a method of constructing an inference model related to a second example procedure of the third embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of a schematic configuration of a separation processing unit according to a fourth embodiment. [Figure 13] FIG. 1 is a diagram illustrating an overview of non-negative matrix factorization. [Figure 14] FIG. 1 is a diagram illustrating an overview of clustering. [Figure 15] 13 is a flowchart illustrating the flow of NMF according to the fifth embodiment. [Figure 16] FIG. 16 is a diagram for explaining the processing flow in the first loop of NMF shown in FIG. [Figure 17] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device according to each embodiment. [Figure 18] FIG. 10 is a diagram for explaining that actual specimen information and reagent information differ from catalog values ​​and literature values. [Figure 19] FIG. 10 is a diagram for explaining that actual specimen information and reagent information differ from catalog values ​​and literature values. [Figure 20] FIG. 10 is a block diagram illustrating an example of the configuration of an information processing system according to a modified example of each embodiment. [Figure 21] FIG. 1 is a block diagram illustrating an example of a schematic configuration of a diagnosis support system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] The explanation will be given in the following order. 1. Introduction 2. First embodiment 2.1.Configuration Example 2.2.Application example to microscope system 2.3.Processing Flow 2.4.Fluorescence Separation Processing 2.5. Training the inference model 2.6. Actions and Effects 3. Second embodiment 3.1.Fluorescence Separation Processing 3.2. Fluorescence separation processing using the least squares method 3.3. Training the inference model 3.4. Actions and Effects 4. Third Embodiment 4.1. First Procedure Example 4.2. Second Procedure Example 5. Fourth Embodiment 6. Fifth Embodiment 6.1 Fixation of dye fluorescence spectrum in minimizing the mean square residual D using the recurrence formula 6.2 Fixation method of dye fluorescence spectrum in minimizing the root mean square residual D using DFP method, BFGS method, etc. 7. Hardware configuration example 8. Remarks 9. System Configuration Variations 10. Application Example 1 11. Application Example 2

[0011] <1. Introduction> First, the following embodiments of the present disclosure propose an information processing device and an information processing system that excites multiple fluorescently stained cells (whether fixed cells or floating cells) with excitation light of multiple wavelengths and separates the fluorescence.

[0012] Fluorescence separation of cells stained with multiple fluorescent dyes requires accurate fluorescence separation (including separation between dye fluorescence and between dye fluorescence and autofluorescence). However, imaging poses a problem: autofluorescence spectra vary from pixel to pixel. Relying solely on spectral information for each pixel to extract autofluorescence spectra and perform fluorescence separation can produce images with high contrast, but these images are susceptible to artifacts such as autofluorescence and noise. As a result, even for cells with identical morphology, completely different fluorescence separation results can be obtained for each pixel. For example, variations in both brightness and wavelength direction can occur within a single cell, and variations in brightness and wavelength direction can also occur between cells with the same morphology.

[0013] Therefore, in the following embodiment, a machine learning inference model is used to input image information (corresponding to a fluorescent signal (also called a fluorescent stained image) described below) before fluorescence separation obtained by capturing an image of a specimen stained with a fluorescent dye (corresponding to a fluorescent stained specimen described below), spectral information (corresponding to a reference spectrum described below) for each molecule contained in the fluorescent dye or specimen, as well as morphological information (not limited to fluorescence, such as an expression map of an antigen, etc.) of the specimen, thereby outputting more accurate fluorescence separation results (for example, two-dimensional images for each fluorescent dye) with clarification of autofluorescence regions and reduced noise.

[0014] In another embodiment, image information before fluorescence separation and staining information such as a combination of fluorescent dyes (or antibody dyes) are input, and a machine learning inference model is used to output morphological information of the specimen, such as cells or tissues, thereby enabling fluorescence separation using morphological information in addition to spectral information in the subsequent fluorescence separation process. This makes it possible to reduce pixel-to-pixel variations in the fluorescence separation results, for example, in a single cell region.

[0015] 2. First Embodiment First, an information processing device and an information processing system according to a first embodiment of the present disclosure will be described in detail with reference to the drawings.

[0016] (2.1. Configuration example) An example of the configuration of an information processing system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the information processing system according to this embodiment includes an information processing device 100 and a database 200, and a fluorescent reagent 10, a specimen 20, and a fluorescently stained specimen 30 exist as inputs to the information processing system.

[0017] (Fluorescent Reagent 10) The fluorescent reagent 10 is a chemical used to stain the specimen 20 and may include, for example, an antibody labeled with a fluorescent dye. The fluorescent reagent 10 may be, for example, a fluorescent antibody (including a primary antibody used for direct labeling or a secondary antibody used for indirect labeling), a fluorescent probe, or a nuclear staining reagent, but the type of fluorescent reagent 10 is not limited to these. The fluorescent reagent 10 is managed by being assigned identification information (hereinafter referred to as "reagent identification information 11") that can identify the fluorescent reagent 10 (or the manufacturing lot of the fluorescent reagent 10). The reagent identification information 11 may be, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited thereto. Even for the same fluorescent reagent 10, the properties of each manufacturing lot vary depending on the manufacturing method, the state of the cells from which the antibody was obtained, and other factors. For example, the spectrum, quantum yield, or fluorescent labeling rate of the fluorescent reagent 10 varies depending on the manufacturing lot. Therefore, in the information processing system according to this embodiment, the fluorescent reagent 10 is managed for each manufacturing lot by being assigned reagent identification information 11. This allows the information processing device 100 to perform fluorescence separation while taking into consideration slight differences in properties that appear between manufacturing lots.

[0018] (specimen 20) The specimen 20 is prepared from a specimen or tissue sample collected from a human body for purposes such as pathological diagnosis. The specimen 20 may be a tissue section, cells, or microparticles. The specimen 20 is not particularly limited in terms of the type of tissue (e.g., organ, etc.) used, the type of disease being treated, the subject's attributes (e.g., age, sex, blood type, race, etc.), or the subject's lifestyle (e.g., diet, exercise, smoking, etc.). Examples of tissue sections include a section of a tissue section to be stained (hereinafter simply referred to as a section) before staining, a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), a section in a different block of the same tissue (sampled from a different location than the stained section), or a section collected from a different patient. The specimens 20 are managed with identification information (hereinafter referred to as "specimen identification information 21") that identifies each specimen 20. Like the reagent identification information 11, the specimen identification information 21 is, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited to this. The properties of the specimen 20 vary depending on the type of tissue used, the type of disease being treated, the attributes of the subject, the lifestyle of the subject, etc. For example, the measurement channel or spectrum of the specimen 20 varies depending on the type of tissue used, etc. Therefore, in the information processing system according to this embodiment, the specimens 20 are individually managed by being assigned specimen identification information 21. This allows the information processing device 100 to perform fluorescence separation while taking into account slight differences in properties that appear for each specimen 20.

[0019] (30 fluorescent stained specimens) The fluorescently stained specimen 30 is produced by staining the specimen 20 with a fluorescent reagent 10. In this embodiment, the fluorescently stained specimen 30 is produced on the assumption that the specimen 20 is stained with one or more fluorescent reagents 10, and the number of fluorescent reagents 10 used for staining is not particularly limited. Furthermore, the staining method is determined by the respective combinations of the specimen 20 and the fluorescent reagents 10, and is not particularly limited.

[0020] (Information processing device 100) 1, the information processing device 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160. The information processing device 100 may be, for example, a fluorescence microscope, but is not limited to this and may include various devices. For example, the information processing device 100 may be a PC (Personal Computer), etc.

[0021] (Acquisition part 110) The acquiring section 110 is configured to acquire information used for various processes of the information processing device 100. As shown in FIG.

[0022] (Information acquisition unit 111) The information acquisition unit 111 is configured to acquire information related to the fluorescent reagent 10 (hereinafter referred to as "reagent information") and information related to the specimen 20 (hereinafter referred to as "specimen information"). More specifically, the information acquisition unit 111 acquires the reagent identification information 11 attached to the fluorescent reagent 10 used to generate the fluorescent stained specimen 30, and the specimen identification information 21 attached to the specimen 20. For example, the information acquisition unit 111 acquires the reagent identification information 11 and the specimen identification information 21 using a barcode reader or the like. Then, the information acquisition unit 111 acquires the reagent information based on the reagent identification information 11 and the specimen information based on the specimen identification information 21 from the database 200. The information acquisition unit 111 stores the acquired information in the information storage unit 121, which will be described later.

[0023] (Fluorescence signal acquisition unit 112) The fluorescence signal acquisition unit 112 is configured to acquire a plurality of fluorescence signals corresponding to each of the plurality of excitation lights when a fluorescence-stained specimen 30 (created by staining a specimen 20 with a fluorescence reagent 10) is irradiated with a plurality of excitation lights having mutually different wavelengths. More specifically, the fluorescence signal acquisition unit 112 receives light and outputs a detection signal according to the amount of received light, thereby acquiring a fluorescence spectrum of the fluorescence-stained specimen 30 based on the detection signal. Here, the content of the excitation light (including excitation wavelength, intensity, etc.) is determined based on reagent information, etc. (in other words, information, etc. related to the fluorescence reagent 10). Note that the fluorescence signal referred to here is not particularly limited as long as it is a signal derived from fluorescence, and may be, for example, a fluorescence spectrum.

[0024] 2A to 2D show specific examples of fluorescence spectra acquired by the fluorescence signal acquisition unit 112. Figures 2A to 2D show specific examples of fluorescence spectra acquired when a fluorescently stained specimen 30 contains four fluorescent substances, namely, DAPI, CK / AF488, PgR / AF594, and ER / AF647, and is irradiated with excitation light having excitation wavelengths of 392 nm (Figure 2A), 470 nm (Figure 2B), 549 nm (Figure 2C), and 628 nm (Figure 2D), respectively. Note that the fluorescence wavelength is shifted to a longer wavelength side than the excitation wavelength due to the release of energy for fluorescence emission (Stokes shift). The fluorescent substances contained in the fluorescently stained specimen 30 and the excitation wavelength of the irradiated excitation light are not limited to those described above. The fluorescence signal acquisition unit 112 stores the acquired fluorescence spectra in the fluorescence signal storage unit 122, which will be described later.

[0025] (Preservation section 120) The storage unit 120 is configured to store information used for various processes of the information processing device 100 or information output by various processes. As shown in Fig. 1, the storage unit 120 includes an information storage unit 121, a fluorescent light signal storage unit 122, and a fluorescent light separation result storage unit 123.

[0026] (Information storage section 121) The information storage unit 121 is configured to store the reagent information and specimen information acquired by the information acquisition unit 111 .

[0027] (Fluorescence signal storage unit 122) The fluorescent signal storage unit 122 is configured to store the fluorescent signal of the fluorescent stained specimen 30 acquired by the fluorescent signal acquisition unit 112 .

[0028] (Fluorescence separation result storage unit 123) The fluorescence separation result storage unit 123 is configured to store the results of the fluorescence separation process performed by the separation processing unit 131, which will be described later. For example, the fluorescence separation result storage unit 123 stores the fluorescence signals for each fluorescent reagent or the autofluorescence signals of the specimen 20, which have been separated by the separation processing unit 131. Furthermore, in order to improve the accuracy of the fluorescence separation by machine learning or the like, the fluorescence separation result storage unit 123 separately provides the results of the fluorescence separation process to the database 200 as training data for machine learning. After providing the results of the fluorescence separation process to the database 200, the fluorescence separation result storage unit 123 may increase free space by appropriately deleting the processing results it has stored.

[0029] (Processing unit 130) The processing unit 130 is configured to perform various processes including fluorescence separation processing. As shown in FIG.

[0030] (Separation processing unit 131) The separation processing unit 131 is configured to execute fluorescence separation processing by using an inference model that uses image information, specimen information, reagent information, and the like as input.

[0031] For example, the image information may be a fluorescent signal (a two-dimensional image based on the fluorescent signal, hereinafter referred to as a fluorescent stained image) acquired by capturing an image of the fluorescent stained specimen 30 with the fluorescent signal acquisition unit 112.

[0032] The specimen information may include, for example, the autofluorescence spectrum of each molecule contained in the specimen 20 identified from the specimen identification information 21, and morphological information about the specimen 20. Note that the morphological information may be a bright-field image or an unstained image of the same tissue block, as well as staining information, or may be, for example, an expression map of a target in the specimen 20.

[0033] Here, the expression map of a target may include, for example, information on the distribution (shape, etc.) of targets such as tissues, cells, and nuclei, information on the tissues in each region, and information on which cells are located where, and may be, for example, a bright-field image obtained by imaging the target, or a binary mask that represents the expression map of the target in binary.

[0034] In addition, the target in this description may include nucleic acids in addition to antigens such as proteins and peptides. That is, the type of target is not limited in this embodiment, and various substances that can be targeted can be targeted.

[0035] Furthermore, the same tissue block may be a specimen that is the same as or similar to specimen 20 or fluorescently stained specimen 30 .

[0036] Here, either an unstained section or a stained section can be used for a specimen that is the same as or similar to specimen 20 or fluorescently stained specimen 30. For example, when an unstained section is used, a section before staining that is used as the stained section, a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), or a section in a different block of the same tissue (sampled from a different location than the stained section) can be used.

[0037] The reagent information may use, for example, a fluorescence spectrum (hereinafter referred to as a standard spectrum or a reference spectrum) for each fluorescent reagent 10 used to stain the specimen 20. The reference spectrum for each fluorescent reagent 10 may be any of a variety of fluorescence spectra, such as a fluorescence spectrum based on a catalog value provided by a reagent vendor or a fluorescence spectrum for each fluorescent reagent 10 extracted from image information obtained by imaging the same or a similar fluorescently stained specimen 30.

[0038] The separation processing unit 131 inputs image information, specimen information, reagent information, etc. into a trained inference model prepared in advance, thereby performing a process (fluorescence separation process) of separating from the image information the autofluorescence signals of each molecule contained in the specimen 20 and the fluorescent signals of each fluorescent reagent 10. The content of the fluorescence separation process using the inference model and the learning of the inference model will be described in detail later.

[0039] Furthermore, the separation processing unit 131 may perform various processes using the fluorescent signal and the autofluorescent signal obtained by the fluorescence separation process. For example, the separation processing unit 131 may perform subtraction processing (also referred to as "background subtraction processing") on image information of another specimen 20 using the separated autofluorescent signal, thereby extracting the fluorescent signal from the image information of the other specimen 20.

[0040] When multiple specimens 20 are identical or similar in terms of the tissue used in the specimens 20, the type of disease being treated, the subject's attributes, and the subject's lifestyle, the autofluorescence signals of these specimens 20 are likely to be similar. Similar specimens include, for example, a tissue section (hereinafter referred to as a section) before staining, a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), a section in a different block of the same tissue (sampled from a different location from the stained section), or a section taken from a different patient. Therefore, if the separation processing unit 131 successfully extracts an autofluorescence signal from a specimen 20, it may extract a fluorescence signal from the image information of another specimen 20 by removing the autofluorescence signal from the image information of the other specimen 20. In this way, by using the background after removing the autofluorescence signal when calculating the S / N ratio using the image information of the other specimen 20, it is possible to improve the S / N ratio of the two-dimensional image obtained by fluorescence separation.

[0041] In this description, the background may refer to a region not stained with the fluorescent reagent 10 or the signal value in that region. Therefore, the background may include autofluorescence signals and other noises before background subtraction processing. Furthermore, the background may include autofluorescence signals and other noises that were not completely removed after background subtraction processing.

[0042] Furthermore, the separation processing unit 131 can perform various processes using the separated fluorescent signal or autofluorescent signal in addition to background subtraction processing. For example, the separation processing unit 131 can use these signals to analyze the fixation state of the specimen 20, or perform segmentation (or area division) to recognize areas of objects (e.g., cells, intracellular structures (cytoplasm, cell membrane, nucleus, etc.), or tissues (tumorous area, non-tumorous area, connective tissue, blood vessels, vascular walls, lymphatic vessels, fibrotic structures, necrosis, etc.)) contained in the image information.

[0043] (Image generation unit 132) The image generation unit 132 is configured to generate (reconstruct) image information based on the fluorescent signal or autofluorescent signal separated by the separation processing unit 131. For example, the image generation unit 132 can generate image information containing only the fluorescent signal or image information containing only the autofluorescent signal. In this case, if the fluorescent signal is composed of multiple fluorescent components or the autofluorescent signal is composed of multiple autofluorescent components, the image generation unit 132 can generate image information for each component. Furthermore, if the separation processing unit 131 performs various processes using the separated fluorescent signal or autofluorescent signal (e.g., analysis of the fixation state of the specimen 20, segmentation, calculation of the S / N value, etc.), the image generation unit 132 may generate image information indicating the results of those processes. This configuration visualizes distribution information of the fluorescent reagent 10 labeled with the target molecule, etc., i.e., the two-dimensional spread, intensity, wavelength, and relative positions of the fluorescence, thereby improving visibility for users such as doctors and researchers, particularly in tissue image analysis areas where information on target substances is complex.

[0044] The image generating unit 132 may also generate image information by controlling the separation processing unit 131 to distinguish between autofluorescent and fluorescent signals based on the fluorescent or autofluorescent signals separated by the separation processing unit 131. Specifically, the image generating unit 132 may generate image information by controlling the following: increasing the brightness of the fluorescent spectrum of the fluorescent reagent 10 labeled with a target molecule or the like; extracting and discoloring only the fluorescent spectrum of the labeled fluorescent reagent 10; extracting the fluorescent spectra of two or more fluorescent reagents 10 from a specimen 20 labeled with two or more fluorescent reagents 10 and discoloring each of them to a different color; extracting and dividing or subtracting only the autofluorescent spectrum of the specimen 20; improving the dynamic range; and so on. This allows the user to clearly distinguish color information derived from fluorescent reagents bound to target substances of interest, thereby improving user visibility.

[0045] (Model generation unit 133) The model generation unit 133 is configured to generate an inference model to be used in the fluorescence separation process performed by the separation processing unit 131, and to update the parameters of the inference model through machine learning to improve the accuracy of the fluorescence separation.

[0046] (Display section 140) The display unit 140 is configured to present the image information generated by the image generation unit 132 to the implementer by displaying it on a display. The type of display used as the display unit 140 is not particularly limited. Although not described in detail in this embodiment, the image information generated by the image generation unit 132 may be presented to the implementer by being projected by a projector or printed by a printer (in other words, the method of outputting the image information is not particularly limited).

[0047] (control unit 150) The control unit 150 is a functional configuration that comprehensively controls all of the processes performed by the information processing device 100. For example, the control unit 150 controls the start and end of various processes (e.g., imaging process of the fluorescently stained specimen 30, fluorescence separation process, various analysis processes, image information generation process (image information reconstruction process), image information display process, etc.) as described above, based on operation input by a user performed via the operation unit 160. Note that the content of control by the control unit 150 is not particularly limited. For example, the control unit 150 may control processes (e.g., processes related to an OS (Operating System)) that are generally performed in a general-purpose computer, a PC, a tablet PC, etc.

[0048] (Operation unit 160) The operation unit 160 is configured to receive operation input from the operator. More specifically, the operation unit 160 is provided with various input means such as a keyboard, a mouse, buttons, a touch panel, or a microphone, and the operator can perform various inputs to the information processing device 100 by operating these input means. Information regarding the operation input performed via the operation unit 160 is provided to the control unit 150.

[0049] (Database 200) The database 200 is a device that accumulates and manages specimen information, reagent information, and the results of fluorescence separation processing, etc. More specifically, the database 200 manages specimen identification information 21 and specimen information, and reagent identification information 11 and reagent information, by linking them together. This allows the information acquisition unit 111 to acquire specimen information from the database 200 based on the specimen identification information 21 of the specimen 20 that is the measurement target, and reagent information based on the reagent identification information 11 of the fluorescent reagent 10.

[0050] As described above, the specimen information managed by the database 200 includes measurement channels and spectral information (autofluorescence spectra) specific to the autofluorescence components contained in the specimens 20. However, in addition to this, the specimen information may also include target information about each specimen 20, specifically, the type of tissue used (e.g., organ, cell, blood, body fluid, ascites, pleural effusion, etc.), the type of disease being treated, the subject's attributes (e.g., age, sex, blood type, or race), or the subject's lifestyle (e.g., diet, exercise, or smoking habits). The information including the measurement channels and spectral information specific to the autofluorescence components contained in the specimens 20 and the target information may be linked for each specimen 20. This makes it possible to easily trace information including the measurement channels and spectral information specific to the autofluorescence components contained in the specimens 20 from the target information. For example, it is possible to cause the separation processing unit 131 to perform similar separation processing performed in the past based on the similarity of the target information in multiple specimens 20, thereby shortening the measurement time. Note that the "tissue to be used" is not particularly limited to tissue collected from a subject, but may also include in vivo tissues and cell lines of humans and animals, as well as solutions, solvents, solutes, and materials contained in the subject of measurement.

[0051] As described above, the reagent information managed by database 200 includes spectral information (fluorescence spectrum) of the fluorescent reagent 10. However, the reagent information may also include information related to the fluorescent reagent 10, such as the production lot, fluorescent component, antibody, clone, fluorescent labeling rate, quantum yield, bleaching coefficient (information indicating the ease with which the fluorescence intensity of the fluorescent reagent 10 decreases), and absorption cross section (or molar extinction coefficient). Furthermore, the specimen information and reagent information managed by database 200 may be managed in different configurations, and in particular, the information related to reagents may be a reagent database that presents the user with optimal reagent combinations.

[0052] Here, it is assumed that the specimen information and reagent information are provided by a manufacturer or measured independently within the information processing system according to the present disclosure. For example, manufacturers of fluorescent reagents 10 often do not measure and provide spectral information, fluorescent labeling rates, etc. for each production lot. Therefore, measuring and managing this information independently within the information processing system according to the present disclosure can improve the accuracy of separating fluorescent signals and autofluorescent signals. Furthermore, to simplify management, database 200 may use catalog values ​​published by manufacturers or literature values ​​found in various documents as specimen information and reagent information (particularly reagent information). However, because actual specimen information and reagent information generally differ from catalog values ​​and literature values, it is more preferable for specimen information and reagent information to be measured and managed independently within the information processing system according to the present disclosure, as described above.

[0053] Furthermore, the accuracy of the fluorescence separation process can be improved by machine learning techniques that use the specimen information, reagent information, and results of the fluorescence separation process managed in database 200. In this embodiment, learning using machine learning techniques is performed in the model generation unit 133. For example, the model generation unit 133 uses a neural network to generate a classifier or estimator (inference model) that has been machine-learned using learning data that links the separated fluorescence signals and autofluorescence signals with the image information, specimen information, and reagent information used for the separation. When new image information, specimen information, and reagent information are acquired, the model generation unit 133 inputs the information into the inference model, thereby being able to predict and output the fluorescence signals and autofluorescence signals contained in the image information.

[0054] Alternatively, similar previously performed fluorescence separation processes (fluorescence separation processes using similar image information, specimen information, or reagent information) that are more accurate than the predicted fluorescence and autofluorescence signals may be calculated, the details of those processes (such as information and parameters used in the processes) may be statistically or regressionally analyzed, and a method for improving the fluorescence separation process for the fluorescence and autofluorescence signals may be output based on the analysis results. The machine learning method is not limited to the above, and known machine learning techniques may also be used. Furthermore, the fluorescence separation process for the fluorescence and autofluorescence signals may be performed using artificial intelligence. Furthermore, not only the fluorescence separation process for the fluorescence and autofluorescence signals, but also various processes using the separated fluorescence or autofluorescence signals (e.g., analysis of the fixation state of the specimen 20 or segmentation) may be improved using machine learning techniques.

[0055] An example of the configuration of the information processing system according to this embodiment has been described above. Note that the above configuration described with reference to Fig. 1 is merely an example, and the configuration of the information processing system according to this embodiment is not limited to this example. For example, the information processing device 100 does not necessarily have to include all of the components shown in Fig. 1, and may include components not shown in Fig. 1.

[0056] The information processing system according to this embodiment may include an imaging device (e.g., including a scanner) that acquires a fluorescence spectrum and an information processing device that performs processing using the fluorescence spectrum. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 may be realized by the imaging device, and the other components may be realized by the information processing device. The information processing system according to this embodiment may also include an imaging device that acquires a fluorescence spectrum and software used for processing using the fluorescence spectrum. In other words, the information processing system may not include a physical component (e.g., a memory, a processor, etc.) that stores and executes the software. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 may be realized by the imaging device, and the other components may be realized by the information processing device that executes the software. The software may be provided to the information processing device via a network (e.g., from a website or cloud server, etc.) or via any storage medium (e.g., a disk, etc.). The information processing device that executes the software may be various servers (e.g., cloud servers, etc.), general-purpose computers, PCs, tablet PCs, etc. Note that the method by which the software is provided to the information processing device and the type of information processing device are not limited to those described above. It should also be noted that the configuration of the information processing system according to this embodiment is not necessarily limited to the above, and that any configuration that a person skilled in the art could conceive of may be applied based on the technical level at the time of use.

[0057] (2.2. Application example to microscope system) The information processing system described above may be realized as, for example, a microscope system. Next, a configuration example of a microscope system in which the information processing system according to this embodiment is realized as a microscope system will be described with reference to FIG.

[0058] As shown in FIG. 3, the microscope system according to this embodiment includes a microscope 101 and a data processing unit 107.

[0059] The microscope 101 includes a stage 102 , an optical system 103 , a light source 104 , a stage driving unit 105 , a light source driving unit 106 , and a fluorescence signal acquiring unit 112 .

[0060] The stage 102 has a mounting surface on which the fluorescently stained specimen 30 can be placed, and is movable in a direction parallel to the mounting surface (xy plane direction) and a direction perpendicular to the mounting surface (z-axis direction) by driving a stage driving unit 105. The fluorescently stained specimen 30 has a thickness in the Z direction of, for example, several μm to several tens of μm, and is sandwiched between a slide glass SG and a cover glass (not shown) and fixed by a predetermined fixing method.

[0061] An optical system 103 is disposed above the stage 102. The optical system 103 includes an objective lens 103A, an imaging lens 103B, a dichroic mirror 103C, an emission filter 103D, and an excitation filter 103E. The light source 104 is, for example, a light bulb such as a mercury lamp or an LED (Light Emitting Diode), and is driven by a light source drive unit 106 to irradiate excitation light onto the fluorescent labels attached to the fluorescently stained specimen 30.

[0062] When obtaining a fluorescent image of the fluorescent-stained specimen 30, the excitation filter 103E generates excitation light by transmitting only light of an excitation wavelength that excites the fluorescent dye, out of the light emitted from the light source 104. The dichroic mirror 103C reflects the excitation light that is transmitted through the excitation filter and guides it to the objective lens 103A. The objective lens 103A focuses the excitation light onto the fluorescent-stained specimen 30. The objective lens 103A and the imaging lens 103B then magnify the image of the fluorescent-stained specimen 30 to a predetermined magnification and form the magnified image on the imaging plane of the fluorescent signal acquisition unit 112.

[0063] When excitation light is irradiated onto the fluorescently stained specimen 30, the staining agent bound to each tissue in the fluorescently stained specimen 30 emits fluorescence. This fluorescence passes through the objective lens 103A, passes through the dichroic mirror 103C, and reaches the imaging lens 103B via the emission filter 103D. The emission filter 103D absorbs the light that has been magnified by the objective lens 103A and passed through the excitation filter 103E, and transmits only a portion of the emitted light. As described above, the image of the emitted light from which the external light has been removed is magnified by the imaging lens 103B and formed on the fluorescence signal acquisition unit 112.

[0064] The data processing unit 107 is configured to drive the light source 104, acquire a fluorescent image of the fluorescently stained specimen 30 using the fluorescent signal acquisition unit 112, and perform various processes using the acquired fluorescent image. More specifically, the data processing unit 107 can function as part or all of the information acquisition unit 111, storage unit 120, processing unit 130, display unit 140, control unit 150, operation unit 160, or database 200 of the information processing device 100 described with reference to FIG. 1 . For example, the data processing unit 107 functions as the control unit 150 of the information processing device 100, thereby controlling the driving of the stage driving unit 105 and the light source driving unit 106 and controlling the acquisition of a spectrum by the fluorescent signal acquisition unit 112.

[0065] The above describes an example of the configuration of a microscope system in which the information processing system according to this embodiment is realized as a microscope system. Note that the above configuration described with reference to Fig. 3 is merely an example, and the configuration of the microscope system according to this embodiment is not limited to this example. For example, the microscope system does not necessarily have to include all of the components shown in Fig. 3, and may include components not shown in Fig. 3.

[0066] (2.3. Processing Flow) An example of the configuration of the information processing system according to this embodiment has been described above. Next, an example of the flow of various processes performed by the information processing device 100 will be described with reference to FIG.

[0067] In step S1000, the user determines the fluorescent reagent 10 and specimen 20 to be used in the analysis. In step S1004, the user stains the specimen 20 with the fluorescent reagent 10 to prepare a fluorescently stained specimen 30.

[0068] In step S1008, the fluorescent signal acquisition unit 112 of the information processing device 100 acquires image information (e.g., a fluorescent stained image) and part of the specimen information (e.g., morphological information) by capturing an image of the fluorescent stained specimen 30. In step S1012, the information acquisition unit 111 acquires reagent information (e.g., a fluorescent spectrum) and part of the specimen information (e.g., an autofluorescence spectrum) from the database 200 based on the reagent identification information 11 attached to the fluorescent reagent 10 used to generate the fluorescent stained specimen 30 and the specimen identification information 21 attached to the specimen 20.

[0069] In step S1016, the separation processing unit 131 acquires an inference model from the database 200 or the model generation unit 133. In step S1020, the separation processing unit 131 inputs the image information, reagent information, and specimen information into the inference model, and acquires image information (two-dimensional images) for each fluorescent reagent 10 (or fluorescent dye) as the output, that is, the fluorescence separation result.

[0070] In step S1024, the image generating unit 132 displays the image information acquired by the separation processing unit 131. This completes the series of processing flows according to this embodiment.

[0071] Note that the steps in the flowchart of Fig. 4 do not necessarily have to be processed in chronological order according to the order described. That is, the steps in the flowchart may be processed in an order different from the order described, or may be processed in parallel. Furthermore, the information processing device 100 may also execute processes not shown in Fig. 4. For example, the separation processing unit 131 may perform segmentation based on acquired image information, or may analyze the fixation state of the specimen 20.

[0072] (2.4. Fluorescence Separation Processing) Next, the fluorescence separation process using the inference model according to this embodiment will be described in detail with reference to the drawings.

[0073] As described above, the fluorescence separation process according to this embodiment uses an inference model constructed by machine learning to improve the accuracy of the fluorescence separation. Specifically, in addition to the fluorescent stained image obtained from the fluorescently stained specimen 30, the reference spectra of each molecule contained in the specimen 20, and the reference spectra of each fluorescent reagent 10, morphological information of the specimen 20 is also input to the inference model, thereby performing fluorescence separation process that takes into account the shapes and types of cells and tissues that make up the specimen 20.

[0074] As the inference model, for example, a machine learning model using a multi-layer neural network such as a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN) can be used.

[0075] 5 is a schematic diagram for explaining an example of the flow of fluorescence separation processing using the inference model of this embodiment. As shown in FIG. 5, in this embodiment, in addition to the fluorescent stained image and the reference spectrum, morphological information is given as input to the inference model 134.

[0076] The fluorescent stained image may be a two-dimensional image (spectral data) before fluorescence separation. As described above, the reference spectrum may be the autofluorescence spectrum (e.g., catalog values) of each molecule contained in the specimen 20 and the fluorescence spectrum (e.g., catalog values) of each fluorescent reagent 10.

[0077] As described above, the morphological information may be, for example, a bright-field image obtained by imaging the unstained specimen 20, the stained fluorescently stained specimen 30, or a specimen similar thereto or a fluorescently stained specimen using the fluorescent signal acquisition unit 112. This bright-field image may be, for example, a map showing the expression level of a target (hereinafter referred to as a target expression map), regardless of whether it is stained, unstained, or fluorescent.

[0078] When a fluorescent stained image, a reference spectrum, and morphological information are given to the input layer of the inference model 134, a two-dimensional image for each fluorescence is output from the output layer as a result of the fluorescence separation process.

[0079] (2.5. Training the inference model) 6 is a diagram illustrating training of an inference model according to this embodiment. As shown in FIG. 6, in training of the inference model 134, a fluorescent stained image (spectral spectrum data), a reference spectrum and morphological image (target expression map) of the same specimen 20 (or fluorescent stained specimen 30), and a two-dimensional image (fluorescence separation result) for each color as a ground truth image of the fluorescent stained specimen 30 are input to the model generation unit 133 as teacher data (also referred to as training data or learning data). The model generation unit 133 trains the inference model 134 by learning and updating the parameters of each layer in the inference model 134 through machine learning based on the input teacher data. This updates the inference model 134 so that the accuracy of fluorescence separation is improved.

[0080] (2.6. Actions and Effects) As described above, in this embodiment, the fluorescence separation process is performed using an inference model that inputs morphological information (a target expression map, not limited to fluorescence) of the specimen 20 (or the fluorescently stained specimen 30) in addition to the fluorescent stained image (spectral spectrum data) before fluorescence separation and the reference spectrum, making it possible to obtain more accurate fluorescence separation results (two-dimensional images for each color) with clarification of autofluorescence regions and reduced noise. As a result, it becomes possible to accurately obtain the desired pathological information.

[0081] In this embodiment, an example is given in which the separation processing unit 131 performs fluorescence separation processing using an inference model 134 that inputs morphological information (a target expression map not limited to fluorescence) of the specimen 20 (or fluorescently stained specimen 30), but this is not limited to this, and it is also possible to configure the separation processing unit 131 to perform fluorescence separation processing using, for example, LSM or the like that utilizes morphological information.

[0082] 3. Second embodiment Next, an information processing device and an information processing system according to a second embodiment of the present disclosure will be described in detail with reference to the drawings.

[0083] The information processing system according to this embodiment may have, for example, the same configuration as the information processing system according to the above-described first embodiment. However, in this embodiment, the separation processing unit 131, the model generation unit 133, and the inference model 134 are replaced with a separation processing unit 231, a model generation unit 233, and an inference model 234, respectively, and therefore the fluorescence separation processing according to this embodiment is replaced with processing details described below.

[0084] (3.1. Fluorescence Separation Processing) The inference model 234 according to this embodiment is constructed as an inference model for generating morphological information of the specimen 20 (or fluorescently stained specimen 30), unlike the inference model 134 for performing fluorescence separation processing.

[0085] 7 is a schematic diagram showing an example of the flow of fluorescence separation processing using an inference model according to this embodiment. As shown in FIG. 7, in this embodiment, to generate morphological information (e.g., a target expression map) of the specimen 20 (or a fluorescently stained specimen 30), a fluorescently stained image (spectral spectrum data) before fluorescence separation or a fluorescently stained image before fluorescence separation, a bright-field image (e.g., HE, DAB (immunostaining)), an image of a specimen identical to or similar to the unstained specimen 20 obtained by capturing an image using, for example, the fluorescent signal acquisition unit 112 (hereinafter referred to as an unstained image), and staining information (e.g., a combination of a fluorescent dye and an antibody) for the same tissue block are input to an inference model 234 of a model generation unit 233. As a result, the inference model 234 outputs morphological information for each combination of a fluorescent dye and an antibody as a binary mask (step S2000).

[0086] The morphological information generated in this manner is input to the separation processing unit 231. As in the first embodiment, the fluorescent stained image before fluorescence separation and the reference spectrum are also input to the separation processing unit 231. The separation processing unit 231 performs fluorescence separation on the input fluorescent stained image based on an algorithm such as the least squares method (LSM), the weighted least squares method (WLSM), or the constrained least squares method (CLSM) using the morphological information and reference spectrum that have also been input, thereby generating a two-dimensional image for each color (step S2004).

[0087] (Separation processing unit 231) 8 is a block diagram showing a more specific example of the configuration of the separation processing unit according to this embodiment. As shown in FIG. 8, the separation processing unit 231 includes a fluorescence separation unit 2311 and a spectrum extraction unit 2312.

[0088] The fluorescence separation unit 2311 includes, for example, a first fluorescence separation unit 2311a and a second fluorescence separation unit 2311b, and performs fluorescence separation on a molecule-by-molecule basis of the fluorescence spectrum of the fluorescent stained image of the stained sample (hereinafter also simply referred to as the stained sample) input from the fluorescence signal storage unit 122.

[0089] The spectrum extraction unit 2312 is configured to optimize the autofluorescence reference spectrum so as to obtain more accurate fluorescence separation results, and adjusts the autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 to obtain more accurate fluorescence separation results, based on the fluorescence separation results obtained by the fluorescence separation unit 2311.

[0090] More specifically, the first fluorescence separation unit 2311a separates the fluorescence spectrum of the input stained sample into spectra for each molecule by performing a fluorescence separation process using the fluorescence reference spectrum included in the reagent information and the autofluorescence reference spectrum included in the specimen information, which are input from the information storage unit 121, and the input morphological information. Note that the fluorescence separation process may use, for example, the least squares method (LSM) or the weighted least squares method (WLSM).

[0091] The spectrum extraction unit 2312 performs a spectrum extraction process on the autofluorescence reference spectrum input from the information storage unit 121 using the fluorescence separation result input from the first fluorescence separation unit 2311a, and adjusts the autofluorescence reference spectrum based on the result, thereby optimizing the autofluorescence reference spectrum to obtain a more accurate fluorescence separation result. Note that the spectrum extraction process may use, for example, non-negative matrix factorization (hereinafter also referred to as "NMF: Non-negative Matrix Factorization"), singular value decomposition (SVD), or the like.

[0092] The second fluorescence separation unit 2311b separates the fluorescence spectrum into spectra for each molecule by performing a fluorescence separation process on the input fluorescence spectrum of the stained sample using the adjusted autofluorescence reference spectrum and morphological information input from the spectrum extraction unit 2312. Note that, similar to the first fluorescence separation unit 2311a, the fluorescence separation process may use, for example, the least squares method (LSM) or the weighted least squares method (WLSM).

[0093] Although FIG. 8 illustrates an example in which the autofluorescence reference spectrum is adjusted once, the present invention is not limited to this. The fluorescence separation result obtained by the second fluorescence separation unit 2311b may be input to the spectrum extraction unit 2312, and the spectrum extraction unit 2312 may repeat the process of adjusting the autofluorescence reference spectrum again one or more times to obtain the final fluorescence separation result.

[0094] (3.2. Fluorescence separation processing using the least squares method) Next, we will explain the fluorescence separation process using the least squares method. The least squares method calculates the color mixing ratio by fitting the input fluorescence spectrum of a stained sample to a reference spectrum. The color mixing ratio is an index that indicates the degree to which each substance is mixed. The following formula (1) represents the residual obtained by subtracting the reference spectrum (St, the fluorescence reference spectrum and the autofluorescence reference spectrum) mixed at a color mixing ratio a from the fluorescence spectrum (Signal). Note that "Signal (1 × number of channels)" in formula (1) indicates that there are as many fluorescence spectra (Signal) as there are wavelength channels (for example, Signal is a matrix representing the fluorescence spectrum). Furthermore, "St (number of substances × number of channels)" indicates that there are as many reference spectra as there are wavelength channels for each substance (fluorescent substance and autofluorescent substance) (for example, St is a matrix representing the reference spectrum). Furthermore, "a (1 × number of substances)" indicates that a color mixing ratio a is set for each substance (fluorescent substance and autofluorescent substance) (for example, a is a matrix representing the color mixing ratio of each reference spectrum in the fluorescence spectrum).

[0095]

number

[0096] Then, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b calculates the color mixing ratio a of each substance that minimizes the sum of squares of the residual equation (1). The sum of squares of the residual is minimized when the result of partial differentiation of the color mixing ratio a in equation (1) representing the residual is 0. Therefore, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b calculates the color mixing ratio a of each substance that minimizes the sum of squares of the residual by solving the following equation (2). Note that "St'" in equation (2) represents the transposed matrix of the reference spectrum St. Furthermore, "inv(St*St')" represents the inverse matrix of St*St'.

[0097]

number

[0098] Here, specific examples of the values ​​of the above formula (1) are shown in the following formulas (3) to (5). The examples of formulas (3) to (5) show a case where, in a fluorescence spectrum (Signal), reference spectra (St) of three substances (the number of substances is three) are mixed at different color mixing ratios a.

[0099]

number

[0100]

number

[0101]

number

[0102] A specific example of the calculation result of the above formula (2) using the values ​​of formulas (3) and (5) is shown in the following formula (6). As shown in formula (6), it can be seen that the calculation result is correctly "a = (3 2 1)" (i.e., the same value as the above formula (4)).

[0103]

number

[0104] As described above, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b performs fluorescence separation processing using reference spectra (autofluorescence reference spectrum and fluorescence reference spectrum), thereby outputting a unique spectrum as the separation result (separation results do not differ for each excitation wavelength). This allows the operator to more easily obtain the correct spectrum. Furthermore, because the reference spectrum (autofluorescence reference spectrum) related to the autofluorescence used in separation is automatically acquired and the fluorescence separation processing is performed, the operator does not need to extract a spectrum corresponding to the autofluorescence from an appropriate space in the unstained section.

[0105] As described above, the first fluorescence separation unit 2311a / second fluorescence separation unit 2311b may extract spectra for each fluorescent substance from the fluorescence spectrum by performing calculations related to the weighted least squares method instead of the least squares method. In the weighted least squares method, weighting is performed to emphasize errors at low signal levels, taking advantage of the fact that noise in the measured fluorescence spectrum (Signal) has a Poisson distribution. However, the upper limit value above which weighting is not performed in the weighted least squares method is the Offset value. The Offset value is determined by the characteristics of the sensor used for measurement and requires separate optimization when an imaging element is used as the sensor. When the weighted least squares method is performed, the reference spectrum St in the above equations (1) and (2) is replaced with St_ expressed by the following equation (7). Note that the following equation (7) means that St_ is calculated by dividing each element (each component) of St, expressed as a matrix, by the corresponding element (each component) of the "Signal + Offset value," also expressed as a matrix (in other words, element division).

[0106]

number

[0107] Here, when the Offset value is 1 and the values ​​of the reference spectrum St and the fluorescence spectrum Signal are expressed by the above equations (3) and (5), respectively, a specific example of St_ expressed by the above equation (7) is shown in the following equation (8).

[0108]

number

[0109] A specific example of the calculation result of the color mixing ratio a in this case is shown in the following equation (9): As shown in equation (9), it can be seen that "a=(3 2 1)" is correctly calculated as the calculation result.

[0110]

number

[0111] By performing the above-described fluorescence separation process using the least squares method based on the attribute information of each pixel identified from the morphological information (e.g., information about which region the pixel belongs to in the fluorescently stained specimen 30), so that there is a correlation between pixels with similar attribute information, it is possible to obtain more accurate fluorescence separation results.

[0112] (3.3. Training the inference model) FIG. 9 is a diagram illustrating training of the inference model according to this embodiment. As shown in FIG. 9, in training the inference model 234 for generating morphological information, a fluorescent stained image (spectral spectrum data) before fluorescence separation or a fluorescent stained image before fluorescence separation, a bright-field image (HE, DAB (immunostaining), etc.) of the same tissue block, an unstained image of a specimen identical to or similar to the unstained specimen 20, staining information (e.g., a combination of a fluorescent dye and an antibody), and morphological information (binary mask) as a ground truth image of the specimen 20 (or the fluorescently stained specimen 30) are input to the model generation unit 233 as training data. The model generation unit 233 trains the inference model 234 by learning and updating the parameters of each layer in the inference model 234 by machine learning based on the input training data. This updates the inference model 234 so that more accurate morphological information (binary mask of the target expression map) can be generated.

[0113] When a fluorescent stained image (spectral spectrum data) before fluorescence separation is input into the inference model 234 to obtain morphological information, the morphology of cells, tissues, etc. in the specimen 20 may not appear in the morphological information under conditions such as when the specimen 20 is fluorescently stained using multiple different types of fluorescent reagents 10. In such cases, a fluorescence separation process using LSM or the like may be performed on the fluorescent stained image before inputting it into the inference model 234, and the inference model 234 may be trained using the fluorescence separation results.

[0114] (3.4. Actions and Effects) As described above, according to this embodiment, morphological information is generated using a machine learning inference model 234 that inputs a fluorescent stained image (spectral data) before fluorescence separation and staining information such as antibody dye combinations and outputs morphological information, making it possible to perform fluorescence separation processing using morphological information in addition to the reference spectrum in the subsequent fluorescence separation processing (S2004). This makes it possible to prevent, for example, the problem of fluorescence separation results differing for each pixel within an area showing a single cell.

[0115] Other configurations, operations, and effects may be the same as those of the first embodiment described above, and therefore detailed description thereof will be omitted here. Also, in this embodiment, the fluorescence separation processing performed by the separation processing unit 231 is exemplified as fluorescence separation processing using an LSM or the like, but this is not limited thereto, and for example, it is also possible to perform fluorescence separation processing using an inference model 134, as in the first embodiment.

[0116] 4. Third Embodiment Generally, machine learning image recognition techniques include classification (classifying an image as either a cat or a dog), detection (finding an object using a bounding box), and segmentation (acquiring and labeling an area in units of one pixel). When generating morphological information in the form of a binary mask from an input image, as in the second embodiment described above, it is necessary to employ segmentation among the above techniques. Therefore, in the third embodiment, several examples of the procedure for constructing the inference model 234 according to the second embodiment using segmentation will be described.

[0117] (4.1. Example of the first procedure) The first procedure example illustrates a case where an inference model 234 is constructed in one stage. Fig. 10 is a schematic diagram for explaining a method for constructing an inference model according to the first procedure example of this embodiment. As shown in Fig. 10, in the first procedure example, a sample image and a correct answer image are first input to the model generation unit 233.

[0118] Here, the specimen image may be, for example, a fluorescently stained image. This specimen image may be stained or unstained. If the specimen image is stained, various staining methods such as HE staining and fluorescent antibody staining may be used.

[0119] The ground truth image may be, for example, morphological information (a binary mask of the target expression map), in which regions such as tissues, cells, and nuclei, or regions such as combinations of antibodies and fluorescent dyes, are represented by a binary mask.

[0120] The model generation unit 233 learns information about tissues, cells, nuclei, etc. in the specimen image, as well as information about the combination of antibodies and fluorescence, by, for example, performing segmentation in an RNN to acquire and label regions in pixel units, thereby constructing an inference model 234 for outputting morphological information as a product.

[0121] Such a one-stage construction method has the advantage that it is possible to obtain morphological information as output from a single inference model 234.

[0122] (4.2. Second procedure example) The second procedure example illustrates a case where an inference model 234 is constructed in two stages. Fig. 11 is a schematic diagram for explaining a method for constructing an inference model according to the second procedure example of this embodiment. As shown in Fig. 11, the second procedure example executes step S3000 in which a specimen image and a supervised image are input to the model generation unit 233, and an inference model 234A is constructed as a product thereof, and step S3004 in which an input image group consisting of individual images of tissues, cells, nuclei, etc., and supervised label information are input to the model generation unit 233, and an inference model 234B is constructed as a product thereof.

[0123] In step S3000, the model generation unit 233 acquires regions of tissues, cells, nuclei, etc. in the specimen image by, for example, performing segmentation in an RNN to acquire and label regions in units of one pixel, thereby constructing an inference model 234A for outputting morphological information as a product.

[0124] Then, in step S3004, the model generation unit 233 learns information about the combination of antibody and fluorescence, for example, by performing classification on each of the regions acquired in step S3000. As a result, an inference model 234B for outputting morphological information is constructed as a product.

[0125] This two-stage construction method has the advantage that if information regarding the combination of antibody and fluorescence needs to be changed, it can be accommodated by simply replacing the second-stage inference model 234B. In other words, compared to the one-stage construction method according to the first example procedure, it has the advantage of making it easier to change the inference model.

[0126] 5. Fourth Embodiment In the second embodiment described above, a case where a spectrum for each fluorescent substance is extracted from a fluorescence spectrum by performing a fluorescence separation process using an autofluorescence reference spectrum (and a fluorescence reference spectrum) is illustrated. In contrast, in the fourth embodiment, a case where a fluorescence spectrum for each fluorescent substance is extracted directly from a stained section is illustrated.

[0127] 12 is a block diagram showing a schematic configuration example of a separation processing unit according to this embodiment. In the information processing device 100 according to this embodiment, the separation processing unit 131 is replaced with a separation processing unit 232 shown in FIG.

[0128] As shown in FIG. 12, the separation processing unit 232 includes a color separation unit 2321, a spectrum extraction unit 2322, and a data set creation unit 2323.

[0129] The color separation unit 2321 separates the fluorescence spectrum of the stained section (also called the stained sample) input from the fluorescence signal storage unit 122 into colors for each fluorescent substance.

[0130] The spectrum extraction unit 2322 is configured to improve the autofluorescence spectrum so that more accurate color separation results can be obtained, and adjusts the autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 so that more accurate color separation results can be obtained.

[0131] The data set creation unit 2323 creates a data set of the autofluorescence reference spectrum from the spectrum extraction results input from the spectrum extraction unit 2322.

[0132] More specifically, the spectrum extraction unit 2322 performs spectrum extraction processing using nonnegative matrix factorization (NMF), singular value decomposition (SVD), or the like on the autofluorescence reference spectrum input from the information storage unit 121, and inputs the results to the dataset creation unit 2323. Note that in the spectrum extraction processing according to this embodiment, for example, an autofluorescence reference spectrum is extracted for each cell tissue and / or each type using a tissue microarray (TMA).

[0133] Here, principal component analysis (hereinafter referred to as "PCA") is generally used as a method for extracting an autofluorescence spectrum from an unstained section, but PCA is not necessarily suitable when concatenated autofluorescence spectra are used for processing, as in this embodiment. Therefore, the spectrum extraction unit 1322 according to this embodiment extracts an autofluorescence reference spectrum from an unstained section by performing nonnegative matrix factorization (NMF) instead of PCA.

[0134] FIG. 13 is a diagram illustrating an overview of NMF. As shown in FIG. 13, NMF decomposes a nonnegative N-by-M (N×M) matrix A into a nonnegative N-by-k (N×k) matrix W and a nonnegative k-by-M (k×M) matrix H. Matrices W and H are determined so that the root mean square residual D between matrix A and the product (W*H) of matrix W and matrix H is minimized. In this embodiment, matrix A corresponds to the spectrum before the autofluorescence reference spectrum is extracted (N is the number of pixels, and M is the number of wavelength channels), and matrix H corresponds to the extracted autofluorescence reference spectrum (k is the number of autofluorescence reference spectra (in other words, the number of autofluorescent materials), and M is the number of wavelength channels). Here, the root mean square residual D is expressed by the following equation (10). Note that "norm(D, 'fro')" refers to the Frobenius norm of the root mean square residual D.

[0135]

number

[0136] NMF uses an iterative method for factorization, starting with random initial values ​​for matrices W and H. While the value of k (the number of autofluorescence reference spectra) is required in NMF, the initial values ​​of matrices W and H are not required and can be set as an option, and once the initial values ​​of matrices W and H are set, the solution becomes constant. On the other hand, if the initial values ​​of matrices W and H are not set, these initial values ​​are set randomly, and the solution does not become constant.

[0137] The specimen 20 varies in nature depending on the type of tissue used, the type of disease targeted, the attributes of the subject, or the lifestyle of the subject, etc., and also has a different autofluorescence spectrum. Therefore, as described above, the information processing apparatus 100 according to the second embodiment can realize a more accurate fluorescence separation process by actually measuring the autofluorescence reference spectrum for each specimen 20.

[0138] Note that the matrix A, which is the input of NMF, is a matrix consisting of the same number of rows as the number of pixels N (= Hpix × Vpix) of the specimen image and the same number of columns as the number of wavelength channels M, as described above. Therefore, when the number of pixels of the specimen image is large or the number of wavelength channels M is large, the matrix A becomes a very large matrix, increasing the calculation cost of NMF and lengthening the processing time.

[0139] In such a case, for example, as shown in FIG. 14, by clustering into the specified number of classes N (<Hpix × Vpix) with the number of pixels N (= Hpix × Vpix) of the specimen image, it is possible to suppress the elongation of the processing time due to the enlargement of the matrix A.

[0140] In clustering, for example, among the specimen images, spectra similar in the wavelength direction or intensity direction are classified into the same class. As a result, an image with a smaller number of pixels than the specimen image is generated, and thus it is possible to reduce the scale of the matrix A' with this image as the input.

[0141] The dataset creation unit 2323 creates a dataset (hereinafter also referred to as an autofluorescence dataset) necessary for the color separation process by the color separation unit 2321 from the autofluorescence reference spectra for each cell tissue and / or type input from the spectrum extraction unit 2322, and inputs the created autofluorescence dataset to the color separation unit 2321.

[0142] The color separation unit 2321 separates the fluorescence spectrum into spectra for each molecule by performing color separation processing on the fluorescence spectrum of the stained sample input from the fluorescence signal storage unit 122 using the fluorescence reference spectrum and autofluorescence reference spectrum input from the information storage unit 121 and the autofluorescence dataset input from the dataset creation unit 2323. Note that NMF or SVD can be used for the color separation processing.

[0143] The NMF performed by the color separation unit 2321 according to this embodiment can be, for example, the NMF (see FIG. 13, etc.) used when extracting an autofluorescence spectrum from an unstained section as described in the first embodiment, modified as follows:

[0144] That is, in this embodiment, matrix A corresponds to multiple specimen images acquired from a stained section (where N is the number of pixels and M is the number of wavelength channels), matrix H corresponds to the fluorescence spectrum of each extracted fluorescent substance (where k is the number of fluorescence spectra (in other words, the number of fluorescent substances) and M is the number of wavelength channels), and matrix W corresponds to the image of each fluorescent substance after fluorescence separation. Matrix D is the root mean square residual.

[0145] In this embodiment, the initial value of the NMF may be, for example, random. However, if the results vary depending on the number of times the NMF is applied, it is necessary to set an initial value to prevent this.

[0146] When fluorescence separation processing is performed using an algorithm such as NMF that rearranges the order of corresponding spectra depending on the calculation algorithm, or an algorithm that requires rearranging the order of spectra to speed up processing or improve the convergence of results, the fluorescent dyes to which each fluorescence spectrum obtained as matrix H corresponds can be identified, for example, by calculating the Pearson product-moment correlation coefficient (or cosine similarity) for each of all combinations.

[0147] Furthermore, when the default function (NMF) of MATLAB (registered trademark) is used, the order is changed even if an initial value is given. This can be fixed by using an autocorrelation function, but even if the order is changed using the default function, it is possible to find the correct combination of substance and fluorescence spectrum by using Pearson's product-moment correlation coefficient (or cosine similarity) as described above.

[0148] As described above, by configuring the system to solve NMF using the specimen image acquired from the stained section as matrix A, it is possible to extract the fluorescence spectrum of each fluorescent substance directly from the stained section without the need for procedures such as photographing unstained sections or generating autofluorescence reference spectra. This makes it possible to significantly reduce the time and labor costs required for fluorescence separation processing.

[0149] Furthermore, in this embodiment, since the fluorescence spectrum for each fluorescent substance is extracted from a specimen image obtained from the same stained section, it is possible to obtain more accurate fluorescence separation results compared to, for example, using an autofluorescence spectrum obtained from an unstained section that is different from the stained section.

[0150] The other configurations, operations, and effects may be the same as those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0151] In this embodiment, when extracting the fluorescence spectrum for each fluorescent substance, a concatenated fluorescence spectrum obtained by concatenating the fluorescence spectra for each fluorescent substance may be used. When using the concatenated fluorescence spectrum, the extraction unit of the separation processing unit 131 concatenates the multiple fluorescence spectra acquired by the fluorescence signal acquisition unit 112, and then performs processing to extract the fluorescence spectrum for each fluorescent substance from the concatenated fluorescence spectrum thus generated.

[0152] 6. Fifth Embodiment In the fourth embodiment described above, the following method can be mentioned as a method for improving the quantitative accuracy of the concentration of the dye.

[0153] Fig. 15 is a flowchart illustrating the flow of NMF according to the fifth embodiment. Fig. 16 is a diagram illustrating the processing flow in the first loop of NMF shown in Fig. 15.

[0154] As shown in Fig. 15, in NMF according to this embodiment, first, variable i is reset to zero (step S401). Variable i indicates the number of times factorization in NMF has been repeated. Therefore, matrix H0 shown in Fig. 16(a) corresponds to the initial value of matrix H. Note that in this example, for clarity, the position of the dye fluorescence spectrum in matrix H is the bottom row, but this is not limited to this and various changes are possible, such as the top row or an intermediate row.

[0155] Next, in the NMF according to this embodiment, as in the normal NMF, a non-negative N-row, M-column (N×M) matrix A is converted into a non-negative N-row, k-column (N×k) matrix W i Dividing by gives a non-negative k-by-M (k×M) matrix H i+1 (Step S402) As a result, for example, in the first loop, the matrix H1 shown in FIG.

[0156] Next, the matrix H obtained in step S402 i+1 The rows of the fluorescent dye spectra in H are replaced with the initial values ​​of the fluorescent dye spectra, i.e., the rows of the dye fluorescent spectra in matrix H0 (step S403). That is, in this embodiment, the fluorescent dye spectra in matrix H are fixed to their initial values. For example, in the first loop, as shown in FIG. 16(c), it is possible to fix the dye fluorescent spectra by replacing the bottom row in matrix H1 with the bottom row in matrix H0.

[0157] Next, in the NMF according to this embodiment, the matrix H obtained in step S403 is i+1 By dividing matrix A by matrix W i+1 is calculated (step S404).

[0158] Then, in the NMF according to this embodiment, as in the normal NMF, it is determined whether or not the root mean square residual D satisfies a predetermined branching condition (step S405). If it does (YES in step S405), the finally obtained matrix H i+1 and W i+1 On the other hand, if the predetermined branching condition is not satisfied (NO in step S405), the variable i is incremented by 1 (step S406), and then the process returns to step S402, where the next loop is executed.

[0159] As described above, the first method enables spectral extraction and color separation of multiply stained pathological section images (specimen images) without the need to capture an unstained sample of the same tissue section for autofluorescence spectrum extraction. It enables direct color separation of the stained sample using NMF while ensuring the quantification of the stained fluorescence, i.e., while maintaining the spectrum of the stained fluorescence. This makes it possible to achieve more accurate color separation than, for example, using a separate specimen. It also makes it possible to reduce the effort required to capture a separate specimen.

[0160] The method for minimizing the mean square residual D is D=|A-WH| 2 Possible methods include using a recurrence formula to minimize the σ, the quasi-Newton method (also known as the DFP (Davidon-Fletcher-Powell) method), the BFGS (Broyden-Fletcher-Goldfarb-Shanno) method, etc. In these cases, the following methods can be used to fix the dye fluorescence spectrum to its initial value.

[0161] (6.1 Fixation of dye fluorescence spectrum in minimizing mean square residual D using recurrence formula) D=|A-WH| 2 In the method of minimizing the mean square residual D using a recurrence formula that minimizes, a loop process is executed that repeats steps consisting of multiplication type update formulas as shown in the following formulas (11) and (12). Note that in formulas (11) and (12), A=(a i,j ) N×Mand H=(h i,j ) k×M and W=(w i,j ) N×k Also, t h, t w is the transpose of the submatrices h and w, respectively.

number

number

[0162] In such a loop process, in order to fix the dye fluorescence spectrum to the initial value, a method can be used in which a step of executing the following equation (13) is inserted between the step of executing equation (11) and the step of executing equation (12). Note that equation (13) is the step of executing the updated w i,j k+1 The submatrix corresponding to the dye fluorescence spectrum in is the submatrix w i,j(part) k indicates that the file will be overwritten.

number

[0163] (6.2 Fixation of dye fluorescence spectrum in minimizing root mean square residual D using DFP method, BFGS method, etc.) In addition, in the method of minimizing the mean square residual D using the DFP method or BFGS method, if the mean square residual D to be minimized is D(x), and x is the coordinate (x at the kth update), k =(a1,a2,...,an) k ), D(x) is minimized by the following steps, where B denotes the Hessian matrix. x k+1 =x k -αB k -1 D'(x k ) to update the coordinates new coordinate x k+1Displacement to gradient at ·y k =D'(x k+1 )-D'(x k ) to the inverse Hessian matrix B k+1 -1 Update

[0164] Various methods can be applied to update the Hessian matrix Bk+1, such as the DFP method shown in the following equation (14) or the BFGF method shown in the following equation (15).

number

number

[0165] In methods for minimizing the root mean square residual D using the DFP method, BFGS method, or the like, there are several methods for fixing arbitrary coordinates, i.e., fixing the dye fluorescence spectrum to an initial value. For example, the dye fluorescence spectrum can be fixed to an initial value by performing the following process (1) or process (2) when updating the coordinates. (1)-αB k -1 D'(x k )=0, that is, the partial differential D'(x k ) with zero (2) After updating the coordinates, x k+1 After calculating the coordinate x k+1 Force part of x k (or part of it)

[0166] <7. Hardware configuration example> Next, a hardware configuration example of the information processing device 100 according to each embodiment and modification will be described with reference to Fig. 17. Fig. 17 is a block diagram showing a hardware configuration example of the information processing device 100. Various processes performed by the information processing device 100 are realized by cooperation between software and the hardware described below.

[0167] 17, the information processing device 100 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing device 100 also includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing device 100 may include a processing circuit such as a DSP or an ASIC instead of or in addition to the CPU 901.

[0168] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 100 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and calculation parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. The CPU 901 may embody at least the processing unit 130 and the control unit 150 of the information processing device 100, for example.

[0169] The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a that includes a CPU bus, etc. The host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 904. Note that the host bus 904a, bridge 904, and external bus 904b do not necessarily need to be configured separately, and these functions may be implemented on a single bus.

[0170] The input device 906 is realized by a device into which the operator inputs information, such as a mouse, keyboard, touch panel, button, microphone, switch, or lever. The input device 906 may also be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or PDA that supports operation of the information processing device 100. The input device 906 may also include, for example, an input control circuit that generates an input signal based on information input by the operator using the above-mentioned input means and outputs the signal to the CPU 901. By operating the input device 906, the operator can input various data to the information processing device 100 and instruct processing operations. The input device 906 may embody, for example, at least the operation unit 160 of the information processing device 100.

[0171] The output device 907 is formed by a device capable of visually or audibly notifying the operator of acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, and lamps, audio output devices such as speakers and headphones, and printer devices. The output device 907 may embody at least the display unit 140 of the information processing device 100, for example.

[0172] The storage device 908 is a device for storing data. The storage device 908 is realized by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 908 stores programs executed by the CPU 901, various data, and various data acquired from the outside. The storage device 908 may embody at least the storage unit 120 of the information processing device 100, for example.

[0173] The drive 909 is a reader / writer for a storage medium, and is built into or externally attached to the information processing device 100. The drive 909 reads information recorded on a removable storage medium, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903. The drive 909 can also write information to the removable storage medium.

[0174] The connection port 911 is an interface connected to an external device, and is a connection port for connecting to an external device that can transmit data via, for example, a USB (Universal Serial Bus) or the like.

[0175] The communication device 913 is, for example, a communication interface formed by a communication device or the like for connecting to the network 920. The communication device 913 is, for example, a communication card for a wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 913 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 913 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP.

[0176] In this embodiment, the sensor 915 includes a sensor capable of acquiring a spectrum (e.g., an image sensor, etc.), but may also include other sensors (e.g., an acceleration sensor, a gyro sensor, a geomagnetic sensor, a pressure sensor, a sound sensor, a distance sensor, etc.) The sensor 915 may embody at least the fluorescent light signal acquisition unit 112 of the information processing device 100, for example.

[0177] The network 920 is a wired or wireless transmission path for information transmitted from devices connected to the network 920. For example, the network 920 may include public networks such as the Internet, telephone networks, and satellite communication networks, as well as various LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The network 920 may also include dedicated network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0178] The above describes an example of a hardware configuration capable of realizing the functions of the information processing device 100. Each of the above components may be realized using general-purpose components, or may be realized by hardware specialized for the function of each component. Therefore, the hardware configuration to be used can be changed as appropriate depending on the technical level at the time of implementing the present disclosure.

[0179] It is possible to create a computer program for implementing each function of the information processing device 100 as described above and install it on a PC or the like. It is also possible to provide a computer-readable recording medium on which such a computer program is stored. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network without using a recording medium.

[0180] <8.Notes> As explained above, because actual specimen information and reagent information generally differ from catalog values ​​and literature values, it is more preferable for specimen information and reagent information to be measured and managed independently within the information processing system according to the present disclosure. As a remark, the fact that actual specimen information and reagent information differ from catalog values ​​and literature values ​​will be explained with reference to Figures 18 and 19.

[0181] Fig. 18 shows a comparison of the measured spectral information of PE (Phycoerythrin), a fluorescent component, with the catalog values. Fig. 19 shows a comparison of the measured spectral information of BV421 (Brilliant Violet 421), a fluorescent component, with the catalog values. The measured values ​​shown are the results of measurements using samples prepared from these fluorescent components and mounting agents.

[0182] 18 and 19, although the peak positions in the spectral information are nearly identical between the measured values ​​and the catalog values, the shapes of the spectra on the longer wavelength side than the peak wavelength are different. Therefore, using catalog values ​​as spectral information reduces the accuracy of separating the fluorescent and autofluorescent signals.

[0183] In addition, it can be said that, in terms of accuracy, it is generally more preferable for not only spectral information but also various information contained in specimen information and reagent information to be measured independently within the information processing system according to the present disclosure.

[0184] <9. System Configuration Variations> The information processing system according to the embodiment described above (see FIG. 1) can also be configured as a server-client system. FIG. 20 is a block diagram showing an example of the schematic configuration of an information processing system configured as a server-client system.

[0185] 20, the information processing system according to this modification includes a client terminal 100A, a server device 100B, and a database 200, which are communicably connected to each other via a predetermined network 300. Various networks can be applied to the network 300, such as a WAN (Wide Area Network) (including the Internet), a LAN (Local Area Network), a public line network, or a mobile communication network.

[0186] The client terminal 100A is a terminal device used by a doctor, a researcher, or the like, and includes, for example, at least the acquisition unit 110, the display unit 140, the control unit 150, and the operation unit 160 in the configuration shown in FIG.

[0187] On the other hand, the server device 100B is not limited to a single server, but may be composed of multiple servers, or may be a cloud server. This server device 100B may include, for example, at least one of the information storage unit 121, the fluorescence signal storage unit 122, the fluorescence separation result storage unit 123, the separation processing unit 131, the image generation unit 132, and the model generation unit 133 in the configuration shown in Fig. 1. Of these components, those not included in the server device 100B may be provided in the client terminal 100A.

[0188] The number of client terminals 100A connected to the same server device 100B is not limited to one, and may be multiple, In this case, the multiple client terminals 100A may be installed in different hospitals.

[0189] Such a system configuration not only makes it possible to provide users such as doctors and researchers with a system with higher computational capabilities, but also makes it possible to store more information in the database 200. This suggests that it will be easier to expand the system to perform machine learning and the like on the big data stored in the database 200.

[0190] However, the configuration is not limited to the above, and various modifications are possible, such as a configuration in which only the database 200 is shared by a plurality of information processing devices 100 via the network 300.

[0191] <10. Application Example 1> Furthermore, in the above-described embodiment, the application of the technology according to the present disclosure to so-called multiple flow cytometry (MFI), which acquires a two-dimensional fluorescent image of a fluorescently stained specimen 30, which is a multiply stained tissue section, has been described. However, the present disclosure is not limited to this, and the technology according to the present disclosure can also be applied to so-called imaging cytometers, which acquire two-dimensional fluorescent images of microparticles such as multiply stained cells.

[0192] <11. Application Example 2> Furthermore, the technology disclosed herein can be applied to various products. For example, the technology disclosed herein may be applied to a pathology diagnostic system or a support system for such a system (hereinafter referred to as a "diagnosis support system") in which a doctor or other doctor diagnoses lesions by observing cells or tissues collected from a patient. This diagnosis support system may be a WSI (Whole Slide Imaging) system that diagnoses lesions or supports the diagnosis based on images acquired using digital pathology technology.

[0193] 21 is a diagram showing an example of a schematic configuration of a diagnosis support system 5500 to which the technology according to the present disclosure is applied. As shown in FIG. 21, the diagnosis support system 5500 includes one or more pathology systems 5510. It may further include a medical information system 5530 and a derivation device 5540.

[0194] Each of the one or more pathology systems 5510 is a system used mainly by pathologists and is installed in, for example, a research institute or a hospital. Each pathology system 5510 may be installed in a different hospital, and is connected to the medical information system 5530 and the derivation device 5540 via various networks such as a WAN (Wide Area Network) (including the Internet), a LAN (Local Area Network), a public line network, or a mobile communication network.

[0195] Each pathology system 5510 includes a microscope 5511 , a server 5512 , a display control device 5513 , and a display device 5514 .

[0196] The microscope 5511 functions as an optical microscope, captures an image of an object to be observed placed on a glass slide, and obtains a pathological image, which is a digital image. The object to be observed may be, for example, tissue or cells collected from a patient, or may be a piece of organ flesh, saliva, blood, etc.

[0197] The server 5512 stores and saves the pathological images acquired by the microscope 5511 in a storage unit (not shown). Furthermore, when the server 5512 receives a viewing request from the display control device 5513, it searches for the pathological image in the storage unit (not shown) and sends the searched pathological image to the display control device 5513.

[0198] The display control device 5513 sends a request to view a pathology image received from a user to the server 5512. Then, the display control device 5513 displays the pathology image received from the server 5512 on a display device 5514 using a liquid crystal, an EL (Electro-Luminescence), a CRT (Cathode Ray Tube), or the like. The display device 5514 may be compatible with 4K or 8K, and may be a single device or multiple devices.

[0199] When the object to be observed is a solid object such as a piece of organ flesh, the object to be observed may be, for example, a stained thin section. The thin section may be prepared by slicing a block piece cut from a specimen such as an organ. Furthermore, when slicing, the block piece may be fixed with paraffin or the like.

[0200] Thin sections may be stained using various staining methods, such as general staining that reveals tissue morphology, such as HE (Hematoxylin-Eosin) staining, or immunostaining that reveals the immune status of tissue, such as IHC (Immunohistochemistry) staining. In this case, one thin section may be stained using multiple different reagents, or two or more thin sections (also called adjacent thin sections) cut consecutively from the same block may be stained using different reagents.

[0201] The microscope 5511 may include a low-resolution imaging unit for capturing images at low resolution and a high-resolution imaging unit for capturing images at high resolution. The low-resolution imaging unit and the high-resolution imaging unit may be different optical systems or may be the same optical system. If they are the same optical system, the microscope 5511 may change its resolution depending on the object to be captured.

[0202] A glass slide containing an observation object is placed on a stage located within the field of view of the microscope 5511. The microscope 5511 first acquires an overall image within the field of view using a low-resolution imaging unit and identifies the area of ​​the observation object from the acquired overall image. Next, the microscope 5511 divides the area where the observation object exists into multiple divided areas of a predetermined size and sequentially images each divided area using a high-resolution imaging unit to acquire a high-resolution image of each divided area. To switch between the target divided areas, the stage may be moved, the imaging optical system may be moved, or both may be moved. Furthermore, each divided area may overlap with adjacent divided areas to prevent areas from being missed due to unintended sliding of the glass slide. Furthermore, the overall image may include identification information for associating the overall image with the patient. This identification information may be, for example, a character string or a QR code (registered trademark), etc.

[0203] High-resolution images acquired by the microscope 5511 are input to the server 5512. The server 5512 divides each high-resolution image into smaller partial images (hereinafter referred to as tile images). For example, the server 5512 divides one high-resolution image into a total of 100 tile images, each 10×10 in size. At this time, if adjacent divided areas overlap, the server 5512 may perform stitching processing on adjacent high-resolution images using a technique such as template matching. In this case, the server 5512 may generate tile images by dividing the entire high-resolution image that has been stitched together by the stitching processing. However, the generation of tile images from high-resolution images may occur before the above-mentioned stitching processing.

[0204] The server 5512 may also generate smaller sized tile images by further dividing the tile image. This generation of tile images may be repeated until a tile image of the size set as the minimum unit is generated.

[0205] After generating the minimum tile images in this manner, the server 5512 performs a tile compositing process on all tile images, combining a predetermined number of adjacent tile images to generate a single tile image. This tile compositing process can be repeated until a single tile image is finally generated. This process generates a group of tile images in a pyramid structure, with each layer consisting of one or more tile images. In this pyramid structure, a tile image in a given layer has the same number of pixels as a tile image in a different layer, but their resolutions are different. For example, when four tile images (2x2) are combined to generate a single tile image in the upper layer, the resolution of the tile image in the upper layer is half the resolution of the tile image in the lower layer used in the compositing.

[0206] By constructing a group of tile images with such a pyramid structure, it becomes possible to change the level of detail of the object of observation displayed on the display device depending on the layer to which the tile image to be displayed belongs. For example, when the tile image of the lowest layer is used, a narrow area of ​​the object of observation is displayed in detail, and as the tile image of the upper layer is used, a wider area of ​​the object of observation is displayed in a coarser manner.

[0207] The generated group of tile images in a pyramid structure is stored in a storage unit (not shown) together with, for example, identification information (referred to as tile identification information) that allows each tile image to be uniquely identified. When the server 5512 receives a request to acquire a tile image including tile identification information from another device (for example, the display control device 5513 or the derivation device 5540), the server 5512 transmits the tile image corresponding to the tile identification information to the other device.

[0208] Note that tile images, which are pathological images, may be generated for each imaging condition, such as focal length or staining condition. When tile images are generated for each imaging condition, a specific pathological image may be displayed alongside other pathological images corresponding to imaging conditions different from the specific imaging condition, but of the same region as the specific pathological image. The specific imaging condition may be specified by the viewer. Furthermore, when multiple imaging conditions are specified by the viewer, pathological images of the same region corresponding to each imaging condition may be displayed alongside each other.

[0209] Furthermore, the server 5512 may store the group of pyramid-structured tile images in a storage device other than the server 5512, such as a cloud server. Furthermore, part or all of the above-described tile image generation process may be executed by a cloud server or the like.

[0210] In response to an input operation from the user, the display control device 5513 extracts a desired tile image from the group of tile images in a pyramid structure and outputs it to the display device 5514. This processing allows the user to feel as if they are observing the object while changing the observation magnification. In other words, the display control device 5513 functions as a virtual microscope. The virtual observation magnification here actually corresponds to the resolution.

[0211] Any method may be used to capture high-resolution images. High-resolution images may be acquired by capturing images of divided areas while repeatedly stopping and moving the stage, or by capturing images of divided areas while moving the stage at a predetermined speed to capture high-resolution images in strips. The process of generating tile images from high-resolution images is not essential; an image with gradually changing resolution may be generated by gradually changing the resolution of the entire high-resolution image that has been stitched together by a stitching process. Even in this case, it is possible to present the user with images in stages, ranging from low-resolution images of a wide area to high-resolution images of a narrow area.

[0212] The medical information system 5530 is a so-called electronic medical record system, and stores information related to diagnosis, such as information for identifying a patient, information about the patient's disease, test information and image information used in the diagnosis, diagnostic results, and prescribed medications. For example, a pathological image obtained by capturing an image of an object to be observed of a patient may be temporarily stored via a server 5512 and then displayed on a display device 5514 by a display control device 5513. A pathologist using the pathology system 5510 makes a pathological diagnosis based on the pathological image displayed on the display device 5514. The results of the pathological diagnosis made by the pathologist are stored in the medical information system 5530.

[0213] The derivation device 5540 may perform analysis including fluorescence separation processing on the pathology image. A learning model created by machine learning may be used for this analysis. The derivation device 5540 may derive a classification result of a specific region, a tissue identification result, or the like as the analysis result. Furthermore, the derivation device 5540 may derive identification results such as cell information, number, position, and brightness information, as well as scoring information for these. The information derived by the derivation device 5540 may be displayed on the display device 5514 of the pathology system 5510 as diagnostic support information.

[0214] The derivation device 5540 may be a server system configured with one or more servers (including a cloud server). The derivation device 5540 may be configured to be incorporated into, for example, the display control device 5513 or the server 5512 in the pathology system 5510. That is, various analyses of the pathology image may be performed in the pathology system 5510.

[0215] Of the above-described configurations, the technology according to the present disclosure may be suitably applied to the entire diagnosis support system 5500. Specifically, the acquisition unit 110 may correspond to the microscope 5511, the display control device 5513 may correspond to the control unit 150, the display device 5514 may correspond to the display unit 140, the remaining components of the pathology system 5510 and the derivation device 5540 may correspond to the remaining components of the information processing device 100, and the medical information system 5530 may correspond to the database 200. In this way, applying the technology according to the present disclosure to the diagnosis support system 5500 can provide an effect such as enabling doctors and researchers to more accurately diagnose and analyze lesions.

[0216] The configuration described above can be applied not only to diagnostic support systems but also to general biological microscopes such as confocal microscopes, fluorescence microscopes, and video microscopes. The observation object may be a biological sample such as cultured cells, fertilized eggs, or sperm; a biological material such as a cell sheet or three-dimensional cell tissue; or a living organism such as a zebrafish or mouse. The observation object is not limited to a glass slide; it can also be observed in a state stored in a well plate, a petri dish, or the like.

[0217] Furthermore, a moving image may be generated from still images of an object of observation acquired using a microscope. For example, a moving image may be generated from still images captured continuously for a predetermined period of time, or an image sequence may be generated from still images captured at predetermined intervals. By generating a moving image from still images in this way, it becomes possible to use machine learning to analyze the dynamic characteristics of an object of observation, such as the pulsation, elongation, and migration of cancer cells, nerve cells, cardiac muscle tissue, sperm, etc., or the division process of cultured cells or fertilized eggs.

[0218] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0219] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0220] The following configurations also fall within the technical scope of the present disclosure. (1) 1. An information processing device comprising: a separation unit that separates a fluorescent signal derived from a fluorescent reagent from a fluorescent image of a biological sample containing cells, based on a reference spectrum derived from the biological sample or a fluorescent reagent, and morphological information of the cells. (2) The information processing device according to (1), wherein the separation unit further separates the fluorescent signal derived from the biological sample from the fluorescent image. (3) The information processing device according to (1), wherein the morphological information includes distribution information of the target in the biological sample. (4) The information processing device according to (3), wherein the target is an antigen in the biological sample, and the distribution information includes a distribution of expression levels of the antigen. (5) The information processing device according to (4), wherein the morphological information includes a binary mask image indicating a distribution of expression levels of the antigens. (6) The information processing device according to any one of (1) to (5), wherein the fluorescent reagent includes an antibody labeled with a fluorescent dye. (7) The information processing device according to any one of (1) to (6), further comprising an image generating unit that generates a corrected fluorescent image based on the separated fluorescent signals. (8) The information processing device according to any one of (1) to (7) above, further comprising an extraction unit that optimizes the reference spectrum derived from the biological sample or the fluorescent reagent. (9) The information processing device according to (8), wherein the separation unit separates the fluorescent signals of the fluorescent image by a least squares method, a weighted least squares method, or a restricted least squares method using the reference spectrum and the morphological information. (10) The information processing device according to (8), wherein the separation unit separates the fluorescent signal of the fluorescent image by inputting the fluorescent image, the reference spectrum, and the morphological information into a first image generation model. (11) The information processing device described in (10), wherein the first image generation model is a trained model trained using color separation information obtained by separating the fluorescent signal of the fluorescent image as training data. (12) The information processing device according to any one of (8) to (11), wherein the separation unit further separates the fluorescent signals of the fluorescent image based on a bright-field image and an unstained image of the biological sample. (13) The information processing device described in (12), wherein the separation unit separates the fluorescent signal of the fluorescent image by inputting the fluorescent image, the bright field image, the unstained image, and staining information into a second inference model. (14) the fluorescent reagent comprises an antibody labeled with a fluorescent dye; The information processing device according to (13), wherein the staining information includes information on a combination of the antibody and the fluorescent dye of the fluorescent reagent. (15) The information processing device described in (13) or (14), wherein the second inference model is a trained model trained using the morphological information generated as a binary mask image as training data. (16) The information processing device according to (15), wherein the morphological information learned as the training data includes region information of the biological specimen. (17) the morphological information learned as the training data includes region information of the biological specimen obtained by segmentation; The information processing device according to (16), wherein the region information of the biological sample includes region information about at least one of tissues, cells, and nuclei. (18) The information processing device according to (17), wherein the region information of the biological sample further includes the staining information. (19) the second inference model further includes a third inference model; The third inference model is a trained model that has been trained using the staining information identified by classification as training data for each of the tissue or cell region information. The information processing device according to (18) above. (20) an information processing device that acquires a fluorescence image of a biological sample containing cells and a reference spectrum derived from the biological sample or a fluorescent reagent; a program for causing the information processing device to execute a process of separating a fluorescent signal derived from the fluorescent reagent from the fluorescent image based on the fluorescent image, the reference spectrum, and morphological information of the cell; An information processing system comprising: [Explanation of symbols]

[0221] 10 Fluorescent Reagents 11 Reagent Identification Information 20 specimens 21 Specimen identification information 30 Fluorescent stained specimens 100 Information processing device 110 Acquisition Department 111 Information Acquisition Department 112 Fluorescence signal acquisition unit 120 Preservation Department 121 Information Storage Department 122 Fluorescence signal storage unit 123 Fluorescence Separation Result Storage Unit 130 Processing section 131, 231 Separation processing section 132 Image Generation Unit 133, 233 Model generation section 134, 234, 234A inference model 140 Display section 150 control section 160 Operation section 200 databases 2311 Fluorescence Separation Unit 2311a First fluorescence separation unit 2311b Second fluorescence separation unit 2312 Spectral Extraction Unit

Claims

1. 1. An information processing device comprising: a separation unit that separates a fluorescent signal derived from a fluorescent reagent from a fluorescent image of a biological sample containing cells, based on a reference spectrum derived from the biological sample or a fluorescent reagent, and morphological information of the cells.

2. The information processing device according to claim 1 , wherein the separation unit further separates a fluorescent signal derived from the biological sample from the fluorescent image.

3. The information processing device according to claim 1 , wherein the morphological information includes distribution information of targets in the biological sample.

4. The information processing device according to claim 3 , wherein the target is an antigen in the biological sample, and the distribution information includes a distribution of expression levels of the antigen.

5. The information processing device according to claim 4 , wherein the morphological information includes a binary mask image indicating a distribution of expression levels of the antigens.

6. The information processing device according to claim 1 , wherein the fluorescent reagent includes an antibody labeled with a fluorescent dye.

7. The information processing device according to claim 1 , further comprising an image generating unit that generates a corrected fluorescent image based on the separated fluorescent signals.

8. The information processing device according to claim 1 , further comprising an extraction unit that optimizes the reference spectrum derived from the biological sample or the fluorescent reagent.

9. The information processing device according to claim 8 , wherein the separation unit separates the fluorescent signals of the fluorescent image by a least squares method, a weighted least squares method, or a constrained least squares method using the reference spectrum and the morphological information.

10. The information processing apparatus according to claim 8 , wherein the separating unit separates the fluorescent signal of the fluorescent image by inputting the fluorescent image, the reference spectrum, and the morphological information into a first image generation model.

11. The information processing device according to claim 10 , wherein the first image generation model is a trained model trained using color separation information obtained by separating the fluorescent signals of the fluorescent image as training data.

12. The information processing device according to claim 8 , wherein the separation unit further separates the fluorescent signal of the fluorescent image based on a bright-field image and an unstained image of the biological sample.

13. The information processing device according to claim 12 , wherein the separation unit separates the fluorescent signal of the fluorescent image by inputting the fluorescent image, the bright-field image, the unstained image, and staining information into a second inference model.

14. the fluorescent reagent comprises an antibody labeled with a fluorescent dye; The information processing device according to claim 13 , wherein the staining information includes information on a combination of the antibody and the fluorescent dye of the fluorescent reagent.

15. The information processing device according to claim 13 , wherein the second inference model is a trained model trained using the morphological information generated as a binary mask image as training data.

16. The information processing apparatus according to claim 15 , wherein the morphological information learned as the training data includes region information of the biological specimen.

17. the morphological information learned as the training data includes region information of the biological specimen obtained by segmentation; The information processing device according to claim 16 , wherein the region information of the biological sample includes region information about at least one of tissues, cells, and nuclei.

18. The information processing device according to claim 17 , wherein the region information of the biological specimen further includes the staining information.

19. the second inference model further includes a third inference model; The third inference model is a trained model that has been trained using the staining information identified by classification as training data for each of the tissue or cell region information. The information processing device according to claim 18.

20. an information processing device that acquires a fluorescence image of a biological sample containing cells and a reference spectrum derived from the biological sample or a fluorescent reagent; a program for causing the information processing device to execute a process of separating a fluorescent signal derived from the fluorescent reagent from the fluorescent image based on the fluorescent image, the reference spectrum, and morphological information of the cell; An information processing system comprising:

Images