Information processing device and information processing system

Abstract

To make it possible to perform fluorescence separation more appropriately.SOLUTION: An information processing device includes: a fluorescence signal acquiring unit that, after a fluorescence-labeled cell, which is a cell labeled by a fluorescent material is irradiated with a plurality of excitation light beams of mutually different wavelengths, acquires a plurality of fluorescence spectra corresponding to each of the plurality of excitation light beams; a connecting unit that connects at least a portion of the plurality of fluorescence spectra in the wavelength direction to generate a connected fluorescence spectrum; and a splitting unit that splits the connected fluorescence spectra into spectra for each of the fluorescence materials by using a connected autofluorescence reference spectrum in which spectra of autofluorescence materials in the cell are connected in the wavelength direction, and reference spectra including the connected fluorescence reference spectra in which spectra of fluorescence materials in the fluorescence-labeled cell are connected in the wavelength direction.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing system. [Background technology]

[0002] In recent years, advances in immunostaining have led to advances in fluorescence and multi-labeling due to the development of cancer immunotherapy, etc. For example, a method is being performed in which an autofluorescence spectrum is extracted from an unstained section of the same tissue block, and then the autofluorescence spectrum is used to separate the fluorescence of a stained section.

[0003] Furthermore, for example, Patent Document 1 listed below discloses a technology for approximating a fluorescence spectrum obtained by irradiating excitation light onto a microparticle multiply labeled with multiple fluorescent dyes, by a linear sum of single-stain spectra obtained from microparticles individually labeled with each fluorescent dye. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-18108 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, these techniques and methods may not be able to perform fluorescence separation appropriately. For example, when an autofluorescence spectrum is extracted from an unstained section of the same tissue block and then the autofluorescence spectrum is used to perform fluorescence separation of a stained section, the operator is required to extract the autofluorescence spectrum from an appropriate space in the unstained section, so the accuracy of the fluorescence separation depends on the operator's work. In addition, since the fluorescence separation is performed for each excitation wavelength, the separation result is output for each excitation wavelength, and the spectrum obtained as the separation result is not uniquely determined.

[0006] Therefore, the present disclosure has been made in consideration of the above circumstances, and provides a new and improved information processing device and information processing system capable of more appropriately performing fluorescence separation. [Means for solving the problem]

[0007] According to an embodiment of the present disclosure, the present invention includes a fluorescence signal acquisition unit that irradiates a fluorescently labeled cell, which is a cell labeled with a fluorescent substance, with a plurality of excitation light beams having different wavelengths and acquires a plurality of fluorescence spectra corresponding to each of the plurality of excitation light beams; a connection unit that generates a connected fluorescence spectrum by connecting at least a portion of the plurality of fluorescence spectra in the wavelength direction; and a separation unit that separates the connected fluorescence spectrum into spectra for each of the fluorescent substances using a reference spectrum including a connected autofluorescence reference spectrum in which spectra of autofluorescent substances in the cell are connected in the wavelength direction, and a connected fluorescence reference spectrum in which spectra of fluorescent substances in the fluorescently labeled cell are connected in the wavelength direction. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment. [Diagram 2] 4 is a specific example of a fluorescence spectrum acquired by a fluorescence signal acquisition unit. [Diagram 3] 13A to 13C are diagrams illustrating a method for generating a connected fluorescence spectrum by a connecting portion. [Figure 4] FIG. 1 shows the fluorescence spectra of AF546 and AF555 when the wavelength resolution is set to 8 nm. [Diagram 5] FIG. 1 shows the fluorescence spectra of AF546 and AF555 when the wavelength resolution is set to 1 nm. [Figure 6] FIG. 4 is a diagram showing an example of a concatenated fluorescence spectrum generated from the fluorescence spectra shown in A to D in FIG. [Figure 7] 4 is a block diagram showing a more specific configuration example of the separation processing unit according to the first embodiment. FIG. [Figure 8]FIG. 13 shows an example of a concatenated autofluorescence reference spectrum. [Figure 9] FIG. 13 shows an example of a coupled fluorescence reference spectrum. [Figure 10] 1 is a block diagram showing an example of the configuration of a microscope system in the case where the information processing system according to the first embodiment is realized as a microscope system. [Figure 11] 5 is a flowchart showing an example of a process flow of fluorescence separation performed by the information processing device according to the first embodiment. [Figure 12] FIG. 11 is a block diagram showing a more specific example configuration of a separation processing unit according to the second embodiment. [Figure 13] FIG. 1 is a diagram illustrating an overview of nonnegative matrix factorization. [Figure 14] FIG. 1 is a diagram for explaining an overview of clustering. [Figure 15] 13 is a flowchart showing an example of a process flow for fluorescence separation performed by an information processing device according to a second embodiment. [Figure 16] FIG. 13 is a diagram for explaining a method for calculating the number of fluorescent molecules (or the number of antibodies) in one image sensor [pixel] in a modified example. [Figure 17] FIG. 13 is a block diagram showing an example of a schematic configuration of a separation processing unit according to a third embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a specimen image to be input to matrix A in the third embodiment (excitation wavelength: 392 nm). [Figure 19] FIG. 13 is a diagram showing an example of a specimen image to be input to matrix A in the third embodiment (excitation wavelength: 470 nm). [Figure 20] FIG. 13 is a diagram showing an example of a specimen image to be input to matrix A in the third embodiment (excitation wavelength: 515 nm). [Figure 21] FIG. 13 is a diagram showing an example of a specimen image to be input to matrix A in the third embodiment (excitation wavelength: 549 nm). [Figure 22] FIG. 13 is a diagram showing an example of a specimen image to be input to a matrix A in the third embodiment (excitation wavelength: 628 nm). [Diagram 23]FIG. 23 is a diagram showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment (part 1). [Figure 24] FIG. 23 is a diagram showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment (part 2). [Diagram 25] FIG. 23 is a diagram showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment (part 3). [Figure 26] FIG. 23 is a diagram showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment (part 4). [Figure 27] FIG. 23 is a diagram (part 5) showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment. [Figure 28] FIG. 23 is a diagram (part 6) showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment. [Figure 29] FIG. 23 is a diagram (part 7) showing an example of a fluorescence separation image acquired as a matrix W by NMF when the specimen images shown in FIGS. 18 to 22 are input in the third embodiment. [Diagram 30] 13 is a flowchart for explaining the flow of NMF according to the fourth embodiment. [Diagram 31] FIG. 31 is a diagram for explaining the process flow in the first loop of NMF shown in FIG. 30. [Diagram 32] 11 is a graph showing an example of initial values ​​of a dye fluorescence spectrum. [Diagram 33] 13 is a graph showing an example of a dye fluorescence spectrum after performing NMF according to the fourth embodiment. [Diagram 34] FIG. 13 is a diagram showing an example of a spectrum of a fluorescent substance extracted by a method according to a fourth embodiment that does not use an unstained sample. [Diagram 35] FIG. 13 is a diagram showing an example of a spectrum of a fluorescent substance extracted when an unstained sample is used. [Diagram 36] FIG. 13 is a diagram illustrating an example of a measurement system of an information processing system according to a sixth embodiment. [Figure 37] 20 is a flowchart showing an example of the operation of a processing unit according to the sixth embodiment. [Figure 38] FIG. 38 is a diagram for explaining the processing executed by the processing unit in each step in FIG. 37 (part 1). [Figure 39] FIG. 38 is a diagram for explaining the processing executed by the processing unit in each step in FIG. 37 (part 2). [Diagram 40] FIG. 38 is a diagram for explaining the processing executed by the processing unit in each step in FIG. 37 (part 3). [Diagram 41] 23 is a flowchart showing an example of the operation of a processing unit according to Modification 1 of the sixth embodiment. [Diagram 42] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device according to each embodiment and modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] The explanation will be given in the following order. 1. First embodiment 1.1. Configuration Example 1.2. Processing flow example 2. Second embodiment 2.1. Processing flow example 2.2. Why PCA is not a suitable method for extracting concatenated autofluorescence reference spectra from unstained sections 2.3.Application Examples 3. Variations 4. Third embodiment 5. Fourth embodiment 5.1 Fixation method of dye fluorescence spectrum in minimizing mean square residual D using recurrence formula 5.2 Fixation method of dye fluorescence spectrum in minimizing the mean square residual D using DFP method, BFGS method, etc. 6. Fifth embodiment 6.1. Overview of Processing by the Processing Unit 6.2. Example of measurement system configuration 6.3. Example of operation 6.4.1. Variation 1 6.4.2. Variation 2 6.5.Effects 7. Hardware configuration example 8. Conclusion

[0011] <1. First embodiment> First, a first embodiment according to the present disclosure will be described.

[0012] (1.1. Configuration Example) An example of the configuration of an information processing system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the information processing system according to this embodiment includes an information processing device 100 and a database 200, and a fluorescent reagent 10, a specimen 20, and a fluorescently stained specimen 30 exist as inputs to the information processing system.

[0013] (Fluorescent Reagent 10) The fluorescent reagent 10 is a chemical used to stain the specimen 20. The fluorescent reagent 10 is, for example, a fluorescent antibody (including a primary antibody used for direct labeling, or a secondary antibody used for indirect labeling), a fluorescent probe, or a nuclear staining reagent, but the type of the fluorescent reagent 10 is not limited to these. The fluorescent reagent 10 is managed by being assigned identification information (hereinafter referred to as "reagent identification information 11") that can identify the fluorescent reagent 10 (or the manufacturing lot of the fluorescent reagent 10). The reagent identification information 11 is, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited to this. Even if the fluorescent reagent 10 is the same product, its properties differ for each manufacturing lot depending on the manufacturing method, the state of the cells from which the antibody was obtained, and the like. For example, the fluorescent reagent 10 has a different spectrum, quantum yield, or fluorescent labeling rate for each manufacturing lot. Therefore, in the information processing system according to this embodiment, the fluorescent reagent 10 is managed for each manufacturing lot by being assigned the reagent identification information 11. This allows the information processing device 100 to perform fluorescence separation while taking into consideration slight differences in properties that appear for each manufacturing lot.

[0014] (specimen 20) The specimen 20 is prepared from a specimen or tissue sample taken from a human body for the purpose of pathological diagnosis or the like. The specimen 20 may be a tissue slice, a cell, or a microparticle, and the type of tissue (e.g., an organ, etc.) used, the type of disease to be treated, the attributes of the subject (e.g., age, sex, blood type, race, etc.), or the lifestyle of the subject (e.g., diet, exercise, smoking, etc.) of the specimen 20 are not particularly limited. The tissue slice may include, for example, a slice of a tissue slice to be stained (hereinafter simply referred to as a slice) before staining, a slice adjacent to a stained slice, a slice different from the stained slice in the same block (sampled from the same location as the stained slice), a slice in a different block (sampled from a different location from the stained slice) in the same tissue, a slice taken from a different patient, etc. The specimens 20 are managed with identification information (hereinafter referred to as "specimen identification information 21") that can identify each specimen 20. The specimen identification information 21, like the reagent identification information 11, is, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited thereto. The properties of the specimen 20 differ depending on the type of tissue used, the type of disease being treated, the attributes of the subject, or the lifestyle of the subject. For example, the measurement channel or spectrum of the specimen 20 differs depending on the type of tissue used, etc. In the information processing system according to this embodiment, the specimens 20 are managed individually by being assigned specimen identification information 21. This allows the information processing device 100 to perform fluorescence separation while taking into consideration slight differences in properties that appear for each specimen 20.

[0015] (Fluorescent stained specimen 30) The fluorescent stained specimen 30 is produced by staining the specimen 20 with a fluorescent reagent 10. In this embodiment, the fluorescent stained specimen 30 is produced on the assumption that the specimen 20 is stained with one or more fluorescent reagents 10, and the number of fluorescent reagents 10 used for staining is not particularly limited. The staining method is determined by the respective combinations of the specimen 20 and the fluorescent reagents 10, and is not particularly limited.

[0016] (Information processing device 100) 1, the information processing device 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160. The information processing device 100 may be, for example, a fluorescent microscope, but is not limited thereto and may include various devices. For example, the information processing device 100 may be a PC (Personal Computer), etc.

[0017] (Acquisition part 110) The acquiring section 110 is configured to acquire information used for various processes of the information processing device 100. As shown in FIG.

[0018] (Information acquisition unit 111) The information acquisition unit 111 is configured to acquire information on the fluorescent reagent 10 (hereinafter referred to as "reagent information") and information on the specimen 20 (hereinafter referred to as "specimen information"). More specifically, the information acquisition unit 111 acquires the reagent identification information 11 attached to the fluorescent reagent 10 used to generate the fluorescent stained specimen 30, and the specimen identification information 21 attached to the specimen 20. For example, the information acquisition unit 111 acquires the reagent identification information 11 and the specimen identification information 21 using a barcode reader or the like. Then, the information acquisition unit 111 acquires the reagent information based on the reagent identification information 11 and the specimen information based on the specimen identification information 21 from the database 200. The information acquisition unit 111 stores the acquired information in the information storage unit 121, which will be described later.

[0019] In this embodiment, the specimen information includes a concatenated autofluorescence reference spectrum in which the spectra of the autofluorescent substances in the specimen 20 are concatenated in the wavelength direction, and the reagent information includes a concatenated fluorescence reference spectrum in which the spectra of the fluorescent substances in the fluorescently stained specimen 30 are concatenated in the wavelength direction. The concatenated autofluorescence reference spectrum and the concatenated fluorescence reference spectrum are collectively referred to as the "reference spectrum."

[0020] (Fluorescence signal acquisition unit 112) The fluorescent signal acquisition unit 112 is configured to acquire a plurality of fluorescent signals corresponding to the plurality of excitation lights when a fluorescent stained specimen 30 (created by staining a specimen 20 with a fluorescent reagent 10) is irradiated with a plurality of excitation lights having mutually different wavelengths. More specifically, the fluorescent signal acquisition unit 112 receives light and outputs a detection signal according to the amount of received light, thereby acquiring a fluorescent spectrum of the fluorescent stained specimen 30 based on the detection signal. Here, the content of the excitation light (including excitation wavelength, intensity, etc.) is determined based on reagent information, etc. (in other words, information, etc. related to the fluorescent reagent 10). Note that the fluorescent signal referred to here is not particularly limited as long as it is a signal derived from fluorescence, and may be, for example, a fluorescent spectrum.

[0021] 2A to 2D are specific examples of the fluorescence spectrum acquired by the fluorescence signal acquisition unit 112. In FIG. 2A to 2D, the fluorescence stained specimen 30 contains four kinds of fluorescent substances, namely, DAPI, CK / AF488, PgR / AF594, and ER / AF647, and specific examples of the fluorescence spectrum acquired when the specimen is irradiated with excitation light having excitation wavelengths of 392 [nm] (FIG. 2A), 470 [nm] (FIG. 2B), 549 [nm] (FIG. 2C), and 628 [nm] (FIG. 2D) are shown. Note that the fluorescence wavelength is shifted to the longer wavelength side than the excitation wavelength due to the release of energy for fluorescence emission (Stokes shift). In addition, the fluorescent substances contained in the fluorescence stained specimen 30 and the excitation wavelength of the irradiated excitation light are not limited to the above. The fluorescence signal acquisition unit 112 stores the acquired fluorescence spectrum in the fluorescence signal storage unit 122 described later.

[0022] (Preservation section 120) The storage unit 120 is configured to store information used for various processes of the information processing device 100 or information output by various processes. As shown in FIG. 1, the storage unit 120 includes an information storage unit 121 and a fluorescent signal storage unit 122.

[0023] (Information storage section 121) The information storage unit 121 is configured to store the reagent information and specimen information acquired by the information acquisition unit 111 .

[0024] (Fluorescence signal storage unit 122) The fluorescent signal storage unit 122 is configured to store the fluorescent signal of the fluorescent stained specimen 30 acquired by the fluorescent signal acquisition unit 112 .

[0025] (Processing unit 130) The processing unit 130 is configured to perform various processes including a fluorescence separation process. As shown in FIG.

[0026] (Connection part 131) The connecting unit 131 is configured to generate a connected fluorescence spectrum by connecting at least a part of the multiple fluorescence spectra acquired by the fluorescence signal acquiring unit 112 in the wavelength direction. For example, the connecting unit 131 extracts data of a predetermined width from each of the four fluorescence spectra (A to D in FIG. 3) acquired by the fluorescence signal acquiring unit 112 so as to include the maximum value of the fluorescence intensity in each of the four fluorescence spectra (A to D in FIG. 3) acquired by the fluorescence signal acquiring unit 112. The width of the wavelength band from which the connecting unit 131 extracts data may be determined based on the reagent information, the excitation wavelength, the fluorescence wavelength, or the like, and may be different for each fluorescent substance (in other words, the width of the wavelength band from which the connecting unit 131 extracts data may be different for each of the fluorescence spectra shown in A to D in FIG. 3). Then, as shown in E in FIG. 3, the connecting unit 131 generates one connected fluorescence spectrum by connecting the extracted data to each other in the wavelength direction. Note that since the connected fluorescence spectrum is composed of data extracted from multiple fluorescence spectra, the wavelengths are not continuous at the boundaries of each connected data.

[0027] At this time, the connection unit 131 performs the above-mentioned connection after aligning the intensities of the excitation light corresponding to each of the multiple fluorescence spectra based on the intensity of the excitation light (in other words, after correcting the multiple fluorescence spectra). More specifically, the connection unit 131 performs the above-mentioned connection after aligning the intensities of the excitation light corresponding to each of the multiple fluorescence spectra by dividing each fluorescence spectrum by the excitation power density, which is the intensity of the excitation light. In this way, the fluorescence spectrum when the excitation light of the same intensity is irradiated is obtained. In addition, when the intensity of the irradiated excitation light differs, the intensity of the spectrum absorbed by the fluorescent stained specimen 30 (hereinafter referred to as the "absorption spectrum") also differs depending on the intensity. Therefore, by aligning the intensities of the excitation light corresponding to each of the multiple fluorescence spectra as described above, the absorption spectrum can be appropriately evaluated.

[0028] The intensity of the excitation light in this description may be the excitation power or the excitation power density, as described above. The excitation power or the excitation power density may be the power or the power density obtained by actually measuring the excitation light emitted from the light source 104, or may be the power or the power density obtained from the driving voltage applied to the light source 104. The intensity of the excitation light in this description may be a value obtained by correcting the above-mentioned excitation power density with the absorption rate of the slice to be observed for each excitation light, or the amplification rate of the detection signal in the detection system (such as the fluorescent signal acquisition unit 112) that detects the fluorescence emitted from the slice. That is, the intensity of the excitation light in this description may be the power density of the excitation light that actually contributed to the excitation of the fluorescent substance, or a value obtained by correcting the power density with the amplification rate of the detection system, etc. By taking the absorption rate or the amplification rate into consideration, it becomes possible to appropriately correct the intensity of the excitation light that changes depending on the change in the machine state, the environment, etc., and therefore it becomes possible to generate a concatenated fluorescence spectrum that enables color separation with higher accuracy.

[0029] In addition, the correction value based on the intensity of the excitation light for each fluorescence spectrum (also called the intensity correction value) is not limited to a value for aligning the intensity of the excitation light corresponding to each of the multiple fluorescence spectra, and may be modified in various ways. For example, the signal intensity of a fluorescence spectrum having an intensity peak on the long wavelength side tends to be lower than the signal intensity of a fluorescence spectrum having an intensity peak on the short wavelength side. Therefore, when the concatenated fluorescence spectrum includes both a fluorescence spectrum having an intensity peak on the long wavelength side and a fluorescence spectrum having an intensity peak on the short wavelength side, the fluorescence spectrum having an intensity peak on the long wavelength side may hardly be taken into account, and only the fluorescence spectrum having an intensity peak on the short wavelength side may be extracted. In such a case, for example, by setting the intensity correction value for the fluorescence spectrum having an intensity peak on the long wavelength side to a larger value, it is also possible to improve the separation accuracy of the fluorescence spectrum having an intensity peak on the short wavelength side.

[0030] Furthermore, the connecting unit 131 may correct the wavelength resolution of each of the multiple fluorescence spectra to be connected independently of the other fluorescence spectra. For example, the fluorescence spectrum of AF546 and the fluorescence spectrum of AF555 have almost the same spectral shape and peak wavelength, and the difference is that the fluorescence spectrum of AF555 has a shoulder at the base of the higher wavelength side, whereas the fluorescence spectrum of AF546 does not. In this way, when two fluorescence spectra are close to each other, a problem occurs in that it becomes difficult to separate the colors of the two by spectrum extraction.

[0031] Such a problem may be solved by increasing the wavelength resolution of the combined fluorescence spectrum. FIG. 4 shows the fluorescence spectra of AF546 and AF555 when the wavelength resolution is 8 nm, and FIG. 5 shows the fluorescence spectra of AF546 and AF555 when the wavelength resolution is 1 nm. As shown in FIG. 4, when the wavelength resolution is 8 nm, the spectral shape and peak wavelength of AF546 and the spectral shape and peak wavelength of AF555 are almost the same. Therefore, it is practically difficult to separate them by using, for example, the least squares method. On the other hand, as shown in FIG. 5, when the wavelength resolution is 8 times the wavelength resolution shown in FIG. 4, that is, 1 nm, the spectral shape and peak wavelength of AF546 and the spectral shape and peak wavelength of AF555 can be clearly separated. This shows that even when multiple fluorescence spectra with similar spectral shapes and peak wavelengths are used, it is possible to separate them by increasing the wavelength resolution.

[0032] However, increasing the wavelength resolution increases the amount of data of the concatenated fluorescence spectrum, which increases the required memory capacity and the calculation costs for the fluorescence separation process. Therefore, the concatenating unit 131 corrects, among the multiple concatenated fluorescence spectra, those fluorescence spectra that are expected to be difficult to separate into colors to have a higher wavelength resolution, and corrects those fluorescence spectra that are expected to be easy to separate into colors to have a lower wavelength resolution. This makes it possible to improve the color separation accuracy while suppressing an increase in the amount of data.

[0033] Here, a specific example will be given to explain the method for generating a concatenated fluorescence spectrum by the connector 131. In this explanation, as in the method for generating a concatenated fluorescence spectrum described above with reference to Fig. 3, an example will be given in which four fluorescence spectra obtained by irradiating a fluorescent stained specimen 30 containing four fluorescent substances, namely, DAPI, CK / AF488, PgR / AF594, and ER / AF647, with excitation light having excitation wavelengths of 392 nm, 470 nm, 549 nm, and 628 nm, respectively, are concatenated.

[0034] Fig. 6 is a diagram showing an example of a concatenated fluorescence spectrum generated from the fluorescence spectra shown in A to D of Fig. 3. As shown in Fig. 6, the concatenating unit 131 extracts a fluorescence spectrum SP1 having an excitation wavelength of 392 nm or more and 591 nm or less from the fluorescence spectrum shown in A of Fig. 3, extracts a fluorescence spectrum SP2 having an excitation wavelength of 470 nm or more and 669 nm or less from the fluorescence spectrum shown in B of Fig. 3, extracts a fluorescence spectrum SP3 having an excitation wavelength of 549 nm or more and 748 nm or less from the fluorescence spectrum shown in C of Fig. 3, and extracts a fluorescence spectrum SP4 having an excitation wavelength of 628 nm or more and 827 nm or less from the fluorescence spectrum shown in D of Fig. 3. Next, the connection unit 131 corrects the wavelength resolution of the extracted fluorescence spectrum SP1 to 16 nm (no intensity correction), corrects the intensity of the fluorescence spectrum SP2 to 1.2 times and the wavelength resolution to 8 nm, corrects the intensity of the fluorescence spectrum SP3 to 1.5 times (no wavelength resolution correction), and corrects the intensity of the fluorescence spectrum SP4 to 4.0 times and the wavelength resolution to 4 nm.The connection unit 131 then connects the corrected fluorescence spectra SP1 to SP4 in order to generate a connected fluorescence spectrum as shown in FIG.

[0035] In addition, FIG. 6 shows a case where the connection unit 131 extracts and connects the fluorescence spectra SP1 to SP4 of a predetermined bandwidth (200 nm width in FIG. 6) from the excitation wavelength when each fluorescence spectrum was acquired, but the bandwidth of the fluorescence spectrum extracted by the connection unit 131 does not need to be the same for each fluorescence spectrum and may be different. In other words, the region extracted by the connection unit 131 from each fluorescence spectrum only needs to be a region including the peak wavelength of each fluorescence spectrum, and the wavelength band and bandwidth may be changed as appropriate. In this case, the shift of the spectral wavelength due to the Stokes shift may be taken into consideration. In this way, by narrowing down the wavelength band to be extracted, it is possible to reduce the amount of data, and therefore it is possible to perform the fluorescence separation process at a higher speed.

[0036] (Separation processing unit 132) The separation processing unit 132 is configured to separate the combined fluorescence spectrum for each molecule. Fig. 7 is a block diagram showing a more specific configuration example of the separation processing unit according to this embodiment. As shown in Fig. 7, the separation processing unit 132 includes a color separation unit 1321 and a spectrum extraction unit 1322.

[0037] The color separation section 1321 includes, for example, a first color separation section 1321a and a second color separation section 1321b, and separates the combined fluorescence spectrum of the stained slice (also called a stained sample) input from the connection section 131 into colors for each molecule.

[0038] The spectrum extraction unit 1322 is configured to improve the combined autofluorescence reference spectrum so as to obtain a more accurate color separation result, and adjusts the combined autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 based on the color separation result by the color separation unit 1321 so as to obtain a more accurate color separation result.

[0039] More specifically, the first color separation unit 1321a separates the concatenated fluorescence spectrum into spectra for each molecule by performing a color separation process on the concatenated fluorescence spectrum of the stained sample input from the connection unit 131 using the concatenated fluorescence reference spectrum included in the reagent information and the concatenated autofluorescence reference spectrum included in the specimen information input from the information storage unit 121. Note that the color separation process may use, for example, the least squares method (LSM) or the weighted least squares method (WLSM).

[0040] The spectrum extraction unit 1322 executes a spectrum extraction process using the color separation result input from the first color separation unit 1321 for the combined autofluorescence reference spectrum input from the information storage unit 121, and adjusts the combined autofluorescence reference spectrum based on the result, thereby improving the combined autofluorescence reference spectrum to obtain a more accurate color separation result. Note that the spectrum extraction process may use, for example, nonnegative matrix factorization (NMF), singular value decomposition (SVD), or the like.

[0041] The second color separation unit 1321b separates the concatenated fluorescence spectrum into spectra for each molecule by performing color separation processing on the concatenated fluorescence spectrum of the stained sample input from the connection unit 131 using the adjusted concatenated autofluorescence reference spectrum input from the spectrum extraction unit 1322. Note that, for example, the least squares method (LSM) or the weighted least squares method (WLSM) may be used for the color separation processing, as in the case of the first color separation unit 1321a.

[0042] Although FIG. 7 illustrates an example in which the combined autofluorescence reference spectrum is adjusted once, the present invention is not limited to this. The color separation result by the second color separation unit 1321b may be input to the spectrum extraction unit 1322, and the spectrum extraction unit 1322 may repeat the process of adjusting the combined autofluorescence reference spectrum again one or more times to obtain the final color separation result.

[0043] FIG. 8 shows a specific example of the concatenated autofluorescence reference spectrum when the autofluorescent substances are Hemoglobin, Archidonic Acid, Catalase, Collagen, FAD, NADPH, and ProLongDiamond. FIG. 9 shows a specific example of the concatenated fluorescence reference spectrum when the fluorescent substances are CK, ER, PgR, and DAPI. The concatenated fluorescence reference spectrum and the concatenated autofluorescence reference spectrum can be generated in the same manner as the concatenated fluorescence spectrum by the concatenating unit 131 (not necessarily limited to this). More specifically, the concatenated fluorescence reference spectrum and the concatenated autofluorescence reference spectrum can be generated by concatenating in the wavelength direction data of a predetermined wavelength bandwidth in a plurality of spectra acquired by a plurality of excitation lights having the same excitation wavelength as that when the concatenated fluorescence spectrum is generated. At this time, it is assumed (not necessarily limited to this) that the intensity of the excitation light corresponding to each of the plurality of spectra is made uniform based on the intensity of the excitation light (for example, excitation power density). The method of generating the concatenated fluorescence reference spectrum and the concatenated autofluorescence reference spectrum is not necessarily limited to the above. For example, the combined fluorescence reference spectrum and the combined autofluorescence reference spectrum may be generated based on theoretical values ​​or catalog values ​​of the spectrum of each substance.

[0044] Next, a calculation related to the least squares method will be described. The least squares method calculates the color mixing ratio by fitting the concatenated fluorescence spectrum generated by the connection unit 131 to the reference spectrum. The color mixing ratio is an index indicating the degree to which each substance is mixed. The following formula (1) is a formula expressing a residual obtained by subtracting a reference spectrum (St. Concatenated fluorescence reference spectrum and concatenated autofluorescence reference spectrum) mixed at a color mixing ratio a from the concatenated fluorescence spectrum (Signal). Note that "Signal (1 x number of channels)" in formula (1) indicates that there are concatenated fluorescence spectra (Signal) as many times as the number of wavelength channels (for example, Signal is a matrix representing the concatenated fluorescence spectrum). Also, "St (number of substances x number of channels)" indicates that there are reference spectra for each substance (fluorescent substance and autofluorescent substance) as many times as the number of wavelength channels (for example, St is a matrix representing the reference spectrum). In addition, “a (1 × number of substances)” indicates that a color mixing ratio a is set for each substance (fluorescent substance and autofluorescent substance) (for example, a is a matrix representing the color mixing ratio of each reference spectrum in the concatenated fluorescence spectrum).

[0045]

number

[0046] Then, the first color separation unit 1321a / second color separation unit 1321b calculates the color mixing ratio a of each material that minimizes the sum of squares of the residual equation (1). The sum of squares of the residual is minimized when the result of partial differentiation of the color mixing ratio a for the equation (1) expressing the residual is 0, so the first color separation unit 1321a / second color separation unit 1321b calculates the color mixing ratio a of each material that minimizes the sum of squares of the residual by solving the following equation (2). Note that "St'" in equation (2) indicates the transposed matrix of the reference spectrum St. Also, "inv(St*St')" indicates the inverse matrix of St*St'.

[0047]

number

[0048] Here, specific examples of the values ​​of the above formula (1) are shown in the following formulas (3) to (5). The examples of formulas (3) to (5) show a case where the reference spectra (St) of three substances (the number of substances is three) are mixed at different color mixing rates a in the concatenated fluorescence spectrum (Signal).

[0049]

number

[0050]

number

[0051]

number

[0052] A specific example of the calculation result of the above formula (2) using the values ​​of formulas (3) and (5) is shown in the following formula (6). As shown in formula (6), it can be seen that the calculation result is correctly calculated as "a = (3 2 1)" (i.e., the same value as the above formula (4)).

[0053]

number

[0054] As described above, the first color separation unit 1321a / second color separation unit 1321b performs fluorescence separation processing using reference spectra (concatenated autofluorescence reference spectrum and concatenated fluorescence reference spectrum) linked in the wavelength direction, and can output a unique spectrum as the separation result (separation results are not different for each excitation wavelength). Therefore, the operator can more easily obtain a correct spectrum. In addition, the reference spectrum (concatenated autofluorescence reference spectrum) related to the autofluorescence used in the separation is automatically acquired and the fluorescence separation processing is performed, eliminating the need for the operator to extract a spectrum corresponding to the autofluorescence from an appropriate space in the unstained section.

[0055] As described above, the first color separation unit 1321a / second color separation unit 1321b may extract a spectrum for each fluorescent substance from the concatenated fluorescence spectrum by performing calculations related to the weighted least squares method instead of the least squares method. In the weighted least squares method, weighting is performed to emphasize errors at low signal levels by utilizing the fact that noise in the concatenated fluorescence spectrum (Signal), which is a measured value, has a Poisson distribution. However, the upper limit value at which weighting is not performed in the weighted least squares method is set as the Offset value. The Offset value is determined by the characteristics of the sensor used for the measurement, and when an image sensor is used as the sensor, it needs to be optimized separately. When the weighted least squares method is performed, the reference spectrum St in the above formulas (1) and (2) is replaced by St_ represented by the following formula (7). Note that the following formula (7) means that St_ is calculated by dividing each element (each component) of St expressed in a matrix by each corresponding element (each component) in the "Signal+Offset value" also expressed in a matrix (in other words, element division).

[0056]

number

[0057] Here, when the Offset value is 1 and the values ​​of the reference spectrum St and the concatenated fluorescence spectrum Signal are expressed by the above equations (3) and (5), respectively, a specific example of St_ expressed by the above equation (7) is shown in the following equation (8).

[0058]

number

[0059] A specific example of the calculation result of the color mixing rate a in this case is shown in the following formula (9): As shown in formula (9), it can be seen that "a=(3 2 1)" is correctly calculated as the calculation result.

[0060]

number

[0061] (Image generation unit 133) The image generating unit 133 is configured to generate image information based on the separation result of the combined fluorescence spectrum by the separation processing unit 132. For example, the image generating unit 133 can generate image information using a fluorescence spectrum corresponding to one or more fluorescent molecules, or generate image information using an autofluorescence spectrum corresponding to one or more autofluorescence molecules. The number and combination of fluorescent molecules or autofluorescence molecules used by the image generating unit 133 to generate image information are not particularly limited. In addition, when various processes (e.g., segmentation or calculation of S / N value, etc.) are performed using the fluorescence spectrum or autofluorescence spectrum after separation, the image generating unit 133 may generate image information indicating the results of those processes.

[0062] (Display section 140) The display unit 140 is configured to present the image information generated by the image generation unit 133 to the practitioner by displaying it on a display. The type of display used as the display unit 140 is not particularly limited. Although not described in detail in this embodiment, the image information generated by the image generation unit 133 may be presented to the practitioner by being projected by a projector or printed by a printer (in other words, the method of outputting the image information is not particularly limited).

[0063] (Control unit 150) The control unit 150 is a functional configuration that comprehensively controls all the processes performed by the information processing device 100. For example, the control unit 150 controls the start and end of various processes as described above (for example, a process of adjusting the placement position of the fluorescent stained specimen 30, a process of irradiating the fluorescent stained specimen 30 with excitation light, a process of acquiring a spectrum, a process of generating a concatenated fluorescent spectrum, a fluorescence separation process, a process of generating image information, a process of displaying image information, etc.) based on an operation input by an operator performed via the operation unit 160. The contents of the control by the control unit 150 are not particularly limited. For example, the control unit 150 may control processes (for example, processes related to an OS (Operating System)) that are generally performed in a general-purpose computer, a PC, a tablet PC, etc.

[0064] (Operation unit 160) The operation unit 160 is configured to receive operation input from the operator. More specifically, the operation unit 160 includes various input means such as a keyboard, a mouse, a button, a touch panel, or a microphone, and the operator can perform various inputs to the information processing device 100 by operating these input means. Information regarding the operation input performed via the operation unit 160 is provided to the control unit 150.

[0065] (Database 200) The database 200 is a device that manages reagent information, specimen information, etc. More specifically, the database 200 manages the reagent identification information 11 and the reagent information, and the specimen identification information 21 and the specimen information, by linking them together. This allows the information acquisition unit 111 to acquire the reagent information based on the reagent identification information 11 of the fluorescent reagent 10, and the specimen information based on the specimen identification information 21 of the specimen 20, from the database 200.

[0066] The reagent information managed by the database 200 is assumed to be information including, but is not necessarily limited to, a measurement channel and a concatenated fluorescence reference spectrum specific to the fluorescent substance possessed by the fluorescent reagent 10. The term "measurement channel" is a concept indicating the fluorescent substance contained in the fluorescent reagent 10, and in the example of FIG. 9, the concept indicates CK, ER, PgR, and DAPI. Since the number of fluorescent substances varies depending on the fluorescent reagent 10, the measurement channel is linked to each fluorescent reagent 10 and managed as reagent information. Furthermore, the concatenated fluorescence reference spectrum included in the reagent information is, as described above, the fluorescence spectra of each fluorescent substance contained in the measurement channel concatenated in the wavelength direction.

[0067] Moreover, it is assumed that the specimen information managed by the database 200 includes (but is not necessarily limited to) a measurement channel specific to an autofluorescent substance contained in the specimen 20 and a concatenated autofluorescence reference spectrum. The term "measurement channel" is a concept indicating an autofluorescent substance contained in the specimen 20, and in the example of FIG. 8, it is a concept indicating Hemoglobin, Archidonic Acid, Catalase, Collagen, FAD, NADPH, and ProLongDiamond. Since the number of autofluorescent substances varies depending on the specimen 20, the measurement channel is linked to each specimen 20 and managed as specimen information. Furthermore, the concatenated autofluorescence reference spectrum included in the specimen information is, as described above, an autofluorescence spectrum concatenated in the wavelength direction for each autofluorescent substance contained in the measurement channel. Note that the information managed by the database 200 is not necessarily limited to the above.

[0068] The configuration example of the information processing system according to the present embodiment has been described above. Note that the above configuration described with reference to Fig. 1 is merely an example, and the configuration of the information processing system according to the present embodiment is not limited to this example. For example, the information processing device 100 does not necessarily have to include all of the configuration shown in Fig. 1, and may include configurations not shown in Fig. 1.

[0069] Here, the information processing system according to the present embodiment may include an imaging device (including, for example, a scanner) that acquires a fluorescence spectrum, and an information processing device that performs processing using the fluorescence spectrum. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 may be realized by the imaging device, and the other components may be realized by the information processing device. In addition, the information processing system according to the present embodiment may include an imaging device that acquires a fluorescence spectrum, and software used for processing using the fluorescence spectrum. In other words, the information processing system may not be provided with a physical component (for example, a memory, a processor, etc.) that stores or executes the software. In this case, the fluorescence signal acquisition unit 112 shown in FIG. 1 may be realized by the imaging device, and the other components may be realized by the information processing device in which the software is executed. The software is provided to the information processing device via a network (for example, from a website or a cloud server, etc.), or is provided to the information processing device via any storage medium (for example, a disk, etc.). In addition, the information processing device in which the software is executed may be various servers (for example, a cloud server, etc.), a general-purpose computer, a PC, a tablet PC, etc. Note that the method in which the software is provided to the information processing device and the type of the information processing device are not limited to the above. Furthermore, please note that the configuration of the information processing system according to this embodiment is not necessarily limited to the above, and any configuration that a person skilled in the art can conceive of based on the technical level at the time of use may be applied.

[0070] The information processing system described above may be realized, for example, as a microscope system. Next, a configuration example of a microscope system in which the information processing system according to the present embodiment is realized as a microscope system will be described with reference to FIG.

[0071] As shown in FIG. 10, the microscope system according to this embodiment includes a microscope 101 and a data processing unit 107.

[0072] The microscope 101 includes a stage 102 , an optical system 103 , a light source 104 , a stage driving unit 105 , a light source driving unit 106 , and a fluorescent signal acquiring unit 112 .

[0073] The stage 102 has a mounting surface on which the fluorescently stained specimen 30 can be placed, and is movable in a direction parallel to the mounting surface (xy plane direction) and a direction perpendicular to the mounting surface (z-axis direction) by driving a stage driving unit 105. The fluorescently stained specimen 30 has a thickness in the Z direction, for example, of several μm to several tens of μm, and is sandwiched between a slide glass SG and a cover glass (not shown) and fixed by a predetermined fixing method.

[0074] An optical system 103 is disposed above the stage 102. The optical system 103 includes an objective lens 103A, an imaging lens 103B, a dichroic mirror 103C, an emission filter 103D, and an excitation filter 103E. The light source 104 is, for example, a light bulb such as a mercury lamp or an LED (Light Emitting Diode), and is driven by a light source drive unit 106 to irradiate excitation light onto the fluorescent label attached to the fluorescent stained specimen 30.

[0075] When obtaining a fluorescent image of the fluorescent stained specimen 30, the excitation filter 103E generates excitation light by transmitting only light of an excitation wavelength that excites the fluorescent dye from the light emitted from the light source 104. The dichroic mirror 103C reflects the excitation light that is transmitted through the excitation filter and incident thereon, and guides it to the objective lens 103A. The objective lens 103A focuses the excitation light on the fluorescent stained specimen 30. The objective lens 103A and the imaging lens 103B then magnify the image of the fluorescent stained specimen 30 to a predetermined magnification, and form the magnified image on the imaging plane of the fluorescent signal acquisition unit 112.

[0076] When the fluorescent stained specimen 30 is irradiated with excitation light, the staining agent bound to each tissue of the fluorescent stained specimen 30 emits fluorescence. This fluorescence passes through the dichroic mirror 103C via the objective lens 103A, and reaches the imaging lens 103B via the emission filter 103D. The emission filter 103D absorbs the light that has been magnified by the objective lens 103A and transmitted through the excitation filter 103E, and transmits only a portion of the colored light. The image of the colored light from which the external light has been removed is magnified by the imaging lens 103B as described above, and is imaged on the fluorescent signal acquisition unit 112.

[0077] The data processing unit 107 is configured to drive the light source 104, acquire a fluorescent image of the fluorescent stained specimen 30 using the fluorescent signal acquisition unit 112, and perform various processes using the acquired image. More specifically, the data processing unit 107 can function as a part or all of the information acquisition unit 111, the storage unit 120, the processing unit 130, the display unit 140, the control unit 150, the operation unit 160, or the database 200 of the information processing device 100 described with reference to FIG. 1. For example, the data processing unit 107 functions as the control unit 150 of the information processing device 100 to control the driving of the stage driving unit 105 and the light source driving unit 106, and to control the acquisition of a spectrum by the fluorescent signal acquisition unit 112. In addition, the data processing unit 107 functions as the processing unit 130 of the information processing device 100 to generate a concatenated fluorescent spectrum, separate the concatenated fluorescent spectrum into molecules, and generate image information based on the separation results.

[0078] The above describes an example of the configuration of a microscope system in the case where the information processing system according to the present embodiment is realized as a microscope system. Note that the above configuration described with reference to Fig. 10 is merely an example, and the configuration of the microscope system according to the present embodiment is not limited to this example. For example, the microscope system does not necessarily have to include all of the configurations shown in Fig. 10, and may include configurations not shown in Fig. 10.

[0079] (1.2. Processing flow example) An example of the configuration of the information processing system according to the present embodiment has been described above. Next, an example of a series of processing flows associated with fluorescence separation by the information processing device 100 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of a series of processing flows associated with fluorescence separation by the information processing device 100.

[0080] In step S1000, the fluorescence signal acquisition section 112 of the information processing device 100 acquires a fluorescence spectrum. More specifically, a plurality of excitation lights having different excitation wavelengths are irradiated onto the fluorescent stained specimen 30, and the fluorescence signal acquisition section 112 acquires a plurality of fluorescence spectra corresponding to each excitation light. The fluorescence signal acquisition section 112 then stores the acquired fluorescence spectra in the fluorescence signal storage section 122.

[0081] In step S1004, the linking unit 131 generates a linked fluorescence spectrum by linking in the wavelength direction at least a portion of the multiple fluorescence spectra stored in the fluorescence signal storage unit 122. More specifically, the linking unit 131 extracts data of a predetermined width from each of the multiple fluorescence spectra so as to include the maximum value of the fluorescence intensity in each of the multiple fluorescence spectra, and links the data together in the wavelength direction to generate one linked fluorescence spectrum.

[0082] In step S1008, the separation processing unit 132 separates the concatenated fluorescence spectrum into molecules (performs fluorescence separation). More specifically, the separation processing unit 132 separates the concatenated fluorescence spectrum into molecules by executing the process described with reference to FIG.

[0083] In the subsequent processing, for example, the image generating unit 133 generates image information using the fluorescence spectrum corresponding to one or more fluorescent molecules (or the autofluorescence spectrum corresponding to the autofluorescent molecule) after separation, and the display unit 140 presents the image information on a display to the operator.

[0084] <2. Second embodiment> The first embodiment of the present disclosure has been described above. Next, the second embodiment of the present disclosure will be described.

[0085] The information processing device 100 according to the first embodiment performs the fluorescence separation process using a combined autofluorescence reference spectrum (and a combined fluorescence reference spectrum) that is prepared in advance, whereas the information processing device 100 according to the second embodiment performs the fluorescence separation process using an actually measured combined autofluorescence reference spectrum.

[0086] More specifically, the spectrum extraction unit 1322 of the separation processing unit 132 according to the second embodiment extracts a concatenated autofluorescence reference spectrum for each autofluorescent substance from at least a portion of a plurality of autofluorescence spectra acquired by irradiating a plurality of excitation lights having different excitation wavelengths on an object identical to or similar to the specimen 20 in the wavelength direction. Then, the spectrum extraction unit 1322 performs a fluorescence separation process using the extracted concatenated autofluorescence reference spectrum and the concatenated fluorescence reference spectrum (similar to that in the first embodiment) as reference spectra.

[0087] Fig. 12 is a block diagram showing a more specific configuration example of the separation processing unit according to this embodiment. As shown in Fig. 12, the separation processing unit 132 according to this embodiment has a configuration similar to that of the separation processing unit 132 described in the first embodiment with reference to Fig. 7.

[0088] In such a configuration, the spectrum extraction unit 1322 receives the combined fluorescence spectrum (also called the combined autofluorescence spectrum) of the unstained section (also called the unstained sample) input from the connection unit 131, instead of the combined autofluorescence reference spectrum included in the specimen information.

[0089] The spectrum extraction unit 1322 executes a spectrum extraction process using the color separation result input from the first color separation unit 1321 for the combined autofluorescence spectrum of the unstained sample input from the connection unit 131, and adjusts the combined autofluorescence reference spectrum based on the result, thereby improving the combined autofluorescence reference spectrum to obtain a more accurate color separation result. For example, nonnegative matrix factorization (NMF) or singular value decomposition (SVD) may be used for the spectrum extraction process, as in the first embodiment. Furthermore, other operations may be similar to those of the separation processing unit 132 according to the first embodiment, and therefore detailed description thereof will be omitted.

[0090] Note that either an unstained section or a stained section can be used as the same or similar section as the specimen 20 used to extract the combined autofluorescence reference spectrum. For example, when an unstained section is used, a section before staining used as the stained section, a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same location as the stained section), or a section in a different block of the same tissue (sampled from a different location from the stained section) can be used.

[0091] Furthermore, when stained sections are used, it is possible to obtain color separation results for each molecule directly from the combined fluorescence spectrum without extracting a combined autofluorescence reference spectrum by performing fluorescence separation processing using the method according to the third embodiment described below.

[0092] Here, as a method for extracting an autofluorescence spectrum from an unstained section, a principal component analysis (hereinafter referred to as "PCA") is generally used, but PCA is not suitable when autofluorescence spectra linked in the wavelength direction are used for processing as in this embodiment. Therefore, the spectrum extraction unit 1322 according to this embodiment extracts a linked autofluorescence reference spectrum from an unstained section by performing non-negative matrix factorization (hereinafter referred to as "NMF") instead of PCA. The reason why PCA is not suitable as a method for extracting a linked autofluorescence reference spectrum from an unstained section will be described in detail later.

[0093] FIG. 13 is a diagram for explaining an overview of NMF. As shown in FIG. 13, NMF decomposes a non-negative matrix A with N rows and M columns (N×M) into a non-negative matrix W with N rows and k columns (N×k) and a matrix H with k rows and M columns (k×M). Matrices W and H are determined so that a mean square residual D between matrix A and the product (W*H) of matrix W and matrix H is minimized. In this embodiment, matrix A corresponds to the spectrum before the concatenated autofluorescence reference spectrum is extracted (N is the number of pixels, and M is the number of wavelength channels), and matrix H corresponds to the extracted concatenated autofluorescence reference spectrum (k is the number of concatenated autofluorescence reference spectra (in other words, the number of autofluorescent materials), and M is the number of wavelength channels). Here, the mean square residual D is expressed by the following formula (10). Note that "norm(D,'fro')" refers to the Frobenius norm of the mean square residual D.

[0094]

number

[0095] In NMF, factorization starts with an iterative method beginning with random initial values for matrix W and matrix H. In NMF, the value of k (the number of concatenated autofluorescence reference spectra) is essential, but the initial values of matrix W and matrix H are not essential and can be set as options. When the initial values of matrix W and matrix H are set, the solution becomes constant. On the other hand, when the initial values of matrix W and matrix H are not set, these initial values are randomly set and the solution does not become constant.

[0096] Specimen 20 differs in nature depending on the type of tissue used, the type of disease targeted, the attributes of the subject, or the lifestyle habits of the subject, etc., and also has different autofluorescence spectra. Therefore, as described above, the information processing apparatus 100 according to the second embodiment can realize a more accurate fluorescence separation process by actually measuring the concatenated autofluorescence reference spectra for each specimen 20.

[0097] Note that matrix A, which is the input to NMF, is a matrix consisting of the same number of rows as the number of pixels N (= Hpix × Vpix) of the specimen image and the same number of columns as the number of wavelength channels M, as described above. Therefore, when the number of pixels of the specimen image is large or the number of wavelength channels M is large, matrix A becomes a very large matrix, increasing the calculation cost of NMF and lengthening the processing time.

[0098] In such a case, for example, as shown in FIG. 14, by clustering into the number of classes N (< Hpix × Vpix) with the number of pixels N (= Hpix × Vpix) of the specimen image specified, it is possible to suppress the elongation of the processing time due to the enlargement of matrix A.

[0099] In clustering, for example, among the specimen images, spectra similar in the wavelength direction or intensity direction are classified into the same class. As a result, an image with a smaller number of pixels than the specimen image is generated, and it becomes possible to reduce the scale of matrix A' with this image as the input.

[0100] (2.1. Example of processing flow) Next, an example of a series of processing flows associated with fluorescence separation by the information processing device 100 according to the second embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of a series of processing flows associated with fluorescence separation by the information processing device 100 according to the second embodiment.

[0101] In steps S1100 and S1104, similar to the example processing flow in the first embodiment (steps S1000 and S1004 in FIG. 11), the fluorescence signal acquisition unit 112 acquires multiple fluorescence spectra corresponding to excitation light with different excitation wavelengths, and the connection unit 131 connects at least a portion of the multiple fluorescence spectra in the wavelength direction to generate a connected fluorescence spectrum.

[0102] In step S1108, the spectrum extraction unit 1322 extracts a concatenated autofluorescence reference spectrum by performing NMF using at least a portion of a plurality of autofluorescence spectra acquired by irradiating an unstained section with a plurality of excitation light beams having different excitation wavelengths, concatenated in the wavelength direction.

[0103] In step S1112, the color separation unit 1321 performs fluorescence separation processing using the extracted combined autofluorescence reference spectrum and combined fluorescence reference spectrum (similar to those in the first embodiment) as reference spectra.

[0104] In the subsequent processing, as in the first embodiment, for example, the image generating unit 133 generates image information using the fluorescence spectrum corresponding to one or more fluorescent molecules (or the autofluorescence spectrum corresponding to the autofluorescent molecule) after separation, and the display unit 140 displays the image information on a display to present it to the operator.

[0105] 2.2. Why PCA is not a suitable method for extracting concatenated autofluorescence reference spectra from unstained sections The above describes an example of a series of processing flows associated with fluorescence separation by the information processing device 100 according to the second embodiment. Next, the reason why PCA is not suitable as a method for extracting a combined autofluorescence reference spectrum from an unstained section will be described in detail.

[0106] First, if the fluorescence spectrum for one excitation wavelength of a pixel i in the pixel number n is ai and the resolution is m, the fluorescence spectrum ai is expressed by the following formula (11) (m-th order vector).

[0107]

number

[0108] Fluorescence spectra for other excitation wavelengths are similarly expressed as m-th order vectors bi, ci, and di (here, it is assumed as an example that there are four excitation wavelengths). Then, the integration of these vectors for all pixels (pixel 1 to pixel n) is expressed by the following formula (12) (matrix P with n rows and 4m columns). Because the number of pixels is remarkably (considerably) larger than the wavelength resolution, the rank of matrix P expressed by formula (12) is a maximum of 4m, and there are a maximum of 4m eigenvalues ​​and eigenvectors.

[0109]

number

[0110] Here, a real matrix A with n rows and m columns (n×m) and rank k can be subjected to singular value decomposition (SVD) as shown in the following formula (13). U and V in formula (13) each represent a singular matrix and form a normal orthogonal system (i.e., t U=U -1 And UU -1 = 1). In addition, if the real matrix A is a square matrix with mutually distinct eigenvalues, U and V are eigenvectors.

[0111]

number

[0112] By performing eigenvalue decomposition (ED) or singular value decomposition (SVD) on real matrix A and calculating singular (eigen) vectors, the factors of independent real matrix A can be analyzed. When real matrix A is a square matrix and has different eigenvalues, t The eigenvalues ​​of AA are the squares of the eigenvalues ​​of A, and the eigenvectors of AA are equal to the eigenvectors of A (see equation (14) below). In equation (14), A=VD t V).

[0113]

number

[0114] Although the spectrum obtained in this embodiment is not a square matrix, it is considered that the spectrum is determined by a linear combination of the elements that make up the autofluorescence, and therefore it is considered possible to convolute the spectrum to a square matrix by overlapping or linear transformation. t Since the eigenvalue of AA=0 is the least squares solution of A, t By finding the eigenvectors of AA, the independent components (eigenvectors) in the spectrum can be calculated. In addition, the following formulas (15) and (16) are self-evident because singular value decomposition holds. However, if the rank is not satisfied, L and R become a subset of the eigenvectors and cannot represent all points.

[0115]

number

[0116]

number

[0117] PCA is equivalent to finding the eigenvalues ​​and eigenvectors of the variance-covariance matrix of a data matrix. As shown in the following formula (17), the variance-covariance matrix is ​​the product of the transpose of the matrix in which the mean is subtracted from the data matrix. This is the product of the transpose of the data matrix minus the product of the mean of each column.

[0118]

number

[0119]

number

[0120] As shown in the following equations (19) to (23), t The eigenvectors of BB are the eigenvectors of B, and B can be constructed. t BB and t The difference between AA is t aa (the matrix of the product of the average values ​​of the columns of A). Therefore, the singular value decomposition of A finds the eigenvectors that make up the points, whereas PCA calculates the eigenvectors that represent the degree of variability of the points ( t The eigenvectors of BB and t AA's eigenvectors are not equal).

[0121]

number

[0122]

number

[0123]

number

[0124]

number

[0125]

number

[0126] In this case, in the case of a matrix such as the above formula (12) in which the fluorescence spectra for each excitation wavelength are integrated, there is no effect in singular value decomposition if they are mutually independent, but in PCA, a term of the product of averages occurs, which affects the eigenvectors. Therefore, in order to perform PCA, it is necessary to perform analysis on each data set. For the above reasons, PCA is not suitable when spectra linked in the wavelength direction are used for processing as in this embodiment.

[0127] (2.3. Application Examples) The above describes in detail why PCA is not suitable as a method for extracting a combined autofluorescence reference spectrum from an unstained section. Next, an application example according to the second embodiment will be described.

[0128] As described above, the spectrum extraction unit 1322 of the separation processing unit 132 according to the second embodiment extracts a concatenated autofluorescence reference spectrum by performing NMF using at least a portion of a plurality of autofluorescence spectra acquired by irradiating an unstained section with a plurality of excitation lights having mutually different excitation wavelengths and concatenating them in the wavelength direction. At this time, the spectrum extraction unit 1322 according to the application example may extract the concatenated autofluorescence reference spectrum by setting an initial value for NMF (the initial value of matrix H in FIG. 13) using the autofluorescence spectrum acquired in advance by the first embodiment or the like (more specifically, by setting at least a portion of the autofluorescence spectrum concatenated in the wavelength direction as the initial value for NMF). This allows a spectrum obtained as a separation result to be uniquely determined, enabling fluorescence separation with higher accuracy.

[0129] <3. Modifications> The second embodiment of the present disclosure has been described above. Next, a modified example of the present disclosure will be described.

[0130] The information obtained by the above-described fluorescence separation process is the brightness (or fluorescence intensity) of the image information, so that the operator may not be able to perform a sufficient quantitative analysis. More specifically, since the operator cannot obtain information such as the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules, it is difficult to compare the number of fluorescent molecules between multiple fluorescent substances or to compare data captured under different conditions.

[0131] This modified example was created in view of the above, and the spectrum extraction unit 1322 according to the modified example extracts a spectrum for each fluorescent substance from the combined fluorescence spectrum using a reference spectrum including a combined autofluorescence reference spectrum and a combined fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules. More specifically, the spectrum extraction unit 1322 according to the modified example calculates a combined autofluorescence reference spectrum and a combined fluorescence reference spectrum for each fluorescent molecule or each antibody by dividing each of the combined autofluorescence reference spectrum and combined fluorescence reference spectrum used in the above embodiment by the number of fluorescent molecules or the number of antibodies in one [pixel] of the image sensor, and uses these to perform a calculation related to the least squares method (or weighted least squares method) to extract a spectrum for each fluorescent substance from the combined fluorescence spectrum. In this way, the separation processing unit 132 according to the modified example can calculate the number of fluorescent molecules or the number of antibodies in the fluorescent stained specimen 30 as a result of the fluorescence separation process.

[0132] Here, a method for calculating the number of fluorescent molecules (or the number of antibodies) in one image sensor [pixel] will be described with reference to FIG. 16. As shown in FIG. 16, when the image sensor and the sample are arranged via an objective lens, the size of the bottom surface of the sample corresponding to one image sensor [pixel] is assumed to be 13 / 20 [μm] × 13 / 20 [μm]. If the thickness of the sample is assumed to be 10 [μm], the volume [m 3] is expressed as 13 / 20[μm] × 13 / 20[μm] × 10[μm] (Note that the volume [L] is 13 / 20[μm] × 13 / 20[μm] × 10[μm] × 10 3 (represented by:

[0133] If the concentration of the antibody (or, of course, the number of fluorescent molecules) contained in the sample is uniform and is 300 [nM], the number of antibodies in one [pixel] of the imaging element is expressed by the following equation (24).

[0134]

number

[0135] As described above, the number of fluorescent molecules or the number of antibodies in the fluorescently stained specimen 30 is calculated as a result of the fluorescence separation process, allowing the practitioner to compare the numbers of fluorescent molecules between multiple fluorescent substances or to compare data captured under different conditions. Furthermore, while luminance (or fluorescence intensity) is a continuous value, the number of fluorescent molecules or the number of antibodies is a discrete value, so the information processing device 100 according to the modified example can reduce the amount of data by outputting image information based on the number of fluorescent molecules or the number of antibodies.

[0136] The other configurations, operations, and effects may be similar to those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0137] <4. Third embodiment> In the above-described first and second embodiments, a case is illustrated in which a spectrum for each fluorescent substance is extracted from the combined fluorescence spectrum by performing a fluorescence separation process using the combined autofluorescence reference spectrum (and the combined fluorescence reference spectrum). In contrast, in the third embodiment, a case is illustrated in which a fluorescence spectrum for each fluorescent substance is extracted directly from a stained section.

[0138] 17 is a block diagram showing a schematic configuration example of a separation processing unit according to the present embodiment. In the information processing device 100 according to the present embodiment, the separation processing unit 132 is replaced with a separation processing unit 232 shown in FIG.

[0139] As shown in FIG. 17, the separation processing unit 232 includes a color separation unit 2321, a spectrum extraction unit 2322, and a data set creation unit 2323.

[0140] The color separation unit 2321 separates the combined fluorescence spectrum of the stained slices (also called stained samples) input from the connection unit 131 into colors for each molecule.

[0141] The spectrum extraction unit 2322 is configured to improve the autofluorescence spectrum so as to obtain more accurate color separation results, and adjusts the combined autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 so as to obtain more accurate color separation results.

[0142] The data set creation unit 2323 creates a data set of the autofluorescence reference spectrum from the spectrum extraction result input from the spectrum extraction unit 2322.

[0143] More specifically, the spectrum extraction unit 2322 executes a spectrum extraction process using nonnegative matrix factorization (NMF), singular value decomposition (SVD), or the like on the concatenated autofluorescence reference spectrum input from the information storage unit 121, and inputs the result to the dataset creation unit 2323. Note that in the spectrum extraction process according to this embodiment, for example, an autofluorescence reference spectrum is extracted for each cell tissue and / or for each type using a tissue microarray (TMA).

[0144] The dataset creation unit 2323 creates a dataset (hereinafter also referred to as an autofluorescence dataset) required for color separation processing by the color separation unit 2321 from the autofluorescence reference spectra for each cell tissue and / or type input from the spectrum extraction unit 2322, and inputs the created autofluorescence dataset to the color separation unit 2321.

[0145] The color separation unit 2321 separates the concatenated fluorescence spectrum of the stained sample input from the connection unit 131 into spectra for each molecule by performing color separation processing using the concatenated fluorescence reference spectrum and concatenated autofluorescence reference spectrum input from the information storage unit 121, and the autofluorescence dataset input from the dataset creation unit 2323. Note that NMF or SVD can be used for the color separation processing.

[0146] The NMF performed by the color separation unit 2321 in this embodiment can be, for example, the NMF (see FIG. 13, etc.) used when extracting an autofluorescence spectrum from an unstained section described in the second embodiment, modified as follows:

[0147] That is, in this embodiment, matrix A corresponds to multiple specimen images acquired from a stained section (where N is the number of pixels and M is the number of wavelength channels), matrix H corresponds to the fluorescence spectrum for each extracted fluorescent substance (where k is the number of fluorescence spectra (in other words, the number of fluorescent substances) and M is the number of wavelength channels), and matrix W corresponds to the image of each fluorescent substance after fluorescence separation. Matrix D is the root mean square residual.

[0148] In this embodiment, the initial value of NMF may be random, as in the second embodiment. However, if the results differ depending on the number of times NMF is applied, it is necessary to set an initial value to prevent this.

[0149] 18 to 22 are diagrams showing examples of specimen images to be input to matrix A in this embodiment, and Figs. 23 to 29 are diagrams showing examples of fluorescence separation images acquired as matrix W by NMF when the specimen images shown in Figs. 18 to 22 are input. For the sake of simplicity of explanation, Figs. 18 to 22 each show a case where a specimen 20 is stained with a single fluorescent reagent 10. As the initial values ​​of NMF, the fluorescence spectra of a total of eight fluorescent dyes, namely, Arachidonic Acid, Catalase, Collagen, FAD, Hemoglobin, NADPH, ProLongDiamond, and CK, are given.

[0150] When the NMF is solved using matrix A as the sample images acquired at each of the five excitation wavelengths (number of wavelength channels M=5) as shown in Figures 18 to 22, seven fluorescence separation images as shown in Figures 23 to 29 are obtained as matrix W, and the respective fluorescence spectra are obtained as matrix H.

[0151] When fluorescence separation processing is performed using an algorithm such as NMF that rearranges the order of corresponding spectra using a calculation algorithm, or an algorithm that requires rearranging the order of spectra to speed up processing or improve the convergence of results, the fluorescent dyes to which each of the fluorescence spectra obtained as matrix H corresponds can be identified, for example, by calculating the Pearson product-moment correlation coefficient (or cosine similarity) for each of all combinations.

[0152] In addition, when the default function (NMF) of MATLAB (registered trademark) is used, the order is changed and output even if an initial value is given. This can be fixed by an autocorrelation function, but even if the order is changed using the default function, it is possible to obtain the correct combination of the substance and the fluorescence spectrum by using the Pearson product moment correlation coefficient (or cosine similarity) as described above.

[0153] As described above, by configuring the system to solve the NMF with the specimen image acquired from the stained section as the matrix A, it becomes possible to extract the fluorescence spectrum for each fluorescent substance directly from the stained section without the need for procedures such as photographing unstained sections and generating a combined autofluorescence reference spectrum, etc. This makes it possible to significantly reduce the time and labor costs required for the fluorescence separation process.

[0154] Furthermore, in this embodiment, since the fluorescence spectrum for each fluorescent substance is extracted from a specimen image obtained from the same stained section, it is possible to obtain more accurate fluorescence separation results compared to, for example, using an autofluorescence spectrum obtained from an unstained section that is different from the stained section.

[0155] The other configurations, operations, and effects may be similar to those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0156] In this embodiment, when extracting the fluorescence spectrum for each fluorescent substance, a concatenated fluorescence spectrum may or may not be used. That is, in this embodiment, the concatenating unit 131 may or may not generate a concatenated fluorescence spectrum. When a concatenated fluorescence spectrum is not generated, the extracting unit of the separation processing unit 132 executes a process of extracting a fluorescence spectrum for each fluorescent substance from the multiple fluorescence spectra acquired by the fluorescence signal acquiring unit 112.

[0157] <5. Fourth embodiment> In the above-mentioned third embodiment, the following method can be given as a method for improving the quantitative accuracy of the concentration and the like of the dyed dye.

[0158] Fig. 30 is a flowchart for explaining the flow of NMF according to the fourth embodiment. Fig. 31 is a diagram for explaining the process flow in the first loop of NMF shown in Fig. 30.

[0159] As shown in Fig. 30, in NMF according to this embodiment, first, variable i is reset to zero (step S401). Variable i indicates the number of times factorization in NMF has been repeated. Therefore, matrix H0 shown in Fig. 31(a) corresponds to the initial value of matrix H. Note that in this example, for clarification, the position of the dye fluorescence spectrum in matrix H is the bottom row, but this is not limited to this and can be changed in various ways, such as to the top row or an intermediate row.

[0160] Next, in the NMF according to this embodiment, as in the case of normal NMF, a non-negative N row, M column (N×M) matrix A is converted into a non-negative N row, k column (N×k) matrix W i Dividing by , we obtain a non-negative k-by-M (k×M) matrix H i+1 (Step S402). As a result, in the first loop, for example, a matrix H1 as shown in FIG.

[0161] Next, the matrix H obtained in step S402 i+1 is replaced with the initial value of the fluorescent dye spectrum, i.e., the row of the dye fluorescent spectrum in matrix H0 (step S403). That is, in this embodiment, the fluorescent dye spectrum in matrix H is fixed to the initial value. For example, in the first loop, as shown in (c) of Fig. 31, it is possible to fix the dye fluorescent spectrum by replacing the bottom row in matrix H1 with the bottom row in matrix H0.

[0162] Next, in the NMF according to this embodiment, the matrix H i+1 By dividing matrix A by matrix W i+1 is calculated (step S404).

[0163] Then, in the NMF according to this embodiment, as in the normal NMF, it is determined whether or not the mean square residual D satisfies a predetermined branching condition (step S405). If it does (YES in step S405), the finally obtained matrix H i+1 and W i+1is regarded as a solution, and NMF is terminated. On the other hand, if the predetermined branching condition is not satisfied (NO in step S405), the variable i is incremented by 1 (step S406), and then the process returns to step S402, and the next loop is executed.

[0164] Fig. 32 is a graph showing an example of the initial value of the dye fluorescence spectrum. Fig. 33 is a graph showing an example of the dye fluorescence spectrum after performing NMF according to this embodiment. As shown in Figs. 32 and 33, it can be seen that the dye fluorescence spectrum maintains a spectrum equivalent to the initial value even when NMF according to this embodiment is performed.

[0165] Moreover, Fig. 34 is a diagram showing an example of a spectrum of a fluorescent substance extracted by a method according to this embodiment that does not use an unstained sample, and Fig. 35 is a diagram showing an example of a spectrum of a fluorescent substance extracted when an unstained sample is used. Note that Figs. 34 and 35 show an example in which CD8 is used as the marker antibody and Alexa Fluor 680 is used as the fluorescent dye. As shown in Figs. 34 and 35, according to this embodiment, it is possible to extract a spectrum of a fluorescent substance with the same accuracy as when an unstained sample is used.

[0166] As described above, in the first method, in the spectrum extraction and color separation of a multi-stained pathological section image (specimen image), it is possible to directly separate the stained sample using NMF while ensuring the quantification of the stained fluorescence, i.e., while maintaining the spectrum of the stained fluorescence, without the need to capture an unstained sample of the same tissue section for extracting the autofluorescence spectrum. This makes it possible to achieve more accurate color separation than, for example, when a separate specimen is used. It also makes it possible to reduce the effort of capturing an image of a separate specimen.

[0167] The method for minimizing the mean square residual D is D=|A-WH| 2Possible methods include a method using a recurrence formula that minimizes the above, a quasi-Newton method (also called the DFP (Davidon-Fletcher-Powell) method), a BFGS (Broyden-Fletcher-Goldfarb-Shanno) method, etc. In these cases, the following methods can be used to fix the dye fluorescence spectrum to its initial value.

[0168] 5.1 Fixation method of dye fluorescence spectrum in minimizing mean square residual D using recurrence formula D=|A-WH| 2 In the method of minimizing the mean square residual D using a recurrence formula that minimizes, a loop process is executed that repeats steps consisting of multiplication type update formulas as shown in the following formulas (25) and (26). Note that in formulas (25) and (26), A=(a i,j ) N×M and H=(h i,j ) k×M and W = (w i,j ) N×k Also, t h, t w is the transpose of the submatrices h and w, respectively.

number

number

[0169] In such a loop process, in order to fix the dye fluorescence spectrum to the initial value, a method can be used in which a step of executing the following formula (27) is inserted between the step of executing formula (25) and the step of executing formula (26). Note that formula (27) is the updated w i,j k+1 The submatrix corresponding to the dye fluorescence spectrum in is the submatrix w i,j(part) k indicates that the file will be overwritten with

number

[0170] 5.2 Fixation method of dye fluorescence spectrum in minimizing the mean square residual D using DFP method, BFGS method, etc. In addition, in the method of minimizing the mean square residual D using the DFP method, BFGS method, etc., if the mean square residual D to be minimized is D(x), and x is the coordinate (x at the kth update), k =(a1,a2,...,an) k ), D(x) is minimized by the following steps, where B is the Hessian matrix: x k+1 =x k -αB k -1 D'(x k ) to update the coordinates new coordinate x k+1 Displacement to gradient at y k =D'(x k+1 )-D'(x k ) to the inverse Hessian matrix B k+1 -1 Update

[0171] To update the Hessian matrix Bk+1, various methods can be applied, such as the DFP method shown in the following equation (28) or the BFGF method shown in the following equation (29).

number

number

[0172] In the method of minimizing the root mean square residual D using the DFP method, the BFGS method, etc., there are several methods for fixing arbitrary coordinates, i.e., fixing the dye fluorescence spectrum to an initial value. For example, the dye fluorescence spectrum can be fixed to an initial value by performing the following process (1) or process (2) when updating the coordinates. (1)-αB k-1 D'(x k )=0, that is, partial differential D'(x k ) with zero (2) After updating the coordinates, x k+1 After calculating, the obtained coordinate x k+1 Force part of x k (or part of it)

[0173] <6. Fifth embodiment> Next, a sixth embodiment of the present disclosure will be described in detail with reference to the drawings.

[0174] Methods for separating and analyzing data into its constituent elements and their coefficients are widely used in fields such as machine learning. There are various methods for decomposing data into elements (basis or spectrum; in this disclosure, these are referred to as spectra) and coefficients, such as eigenvalue decomposition, singular value decomposition, and nonnegative factorization (NMF) described in the above embodiment. In particular, NMF for nonnegative data has a high similarity between the obtained solution and the actual spectrum (e.g., the absorption spectrum or fluorescence spectrum of a material), which is advantageous in interpreting data, since both the spectrum and the coefficients are nonnegative values.

[0175] As described above, NMF expresses the data matrix A as the sum of the product of the spectrum S (corresponding to matrix H in FIG. 13) and the coefficient C (corresponding to matrix W in FIG. 13) and the error f (corresponding to the mean square residual D in FIG. 13), where the error f = |AS × C| 2 This method performs matrix factorization under non-negative constraints so that is minimized, and has the advantage of being able to easily approximate data with a minimal spectrum (low-rank approximation). In this NMF, a calculation method using a recurrence formula has been established, and by repeatedly calculating the following formula (30), a combination of (S, C) that minimizes the error f can be obtained.

number

[0176] Here, Xij and Yij are values ​​expressed by the following equation (31).

number

[0177] Also, in equation (31), the matrix t C, matrix t S is the transpose of matrix C and matrix S, respectively.

[0178] Here, consider a case where matrix A is p data (corresponding to the number of pixels N in FIG. 13) with w element components (corresponding to the number of wavelength channels M in FIG. 13) and this is approximated by a spectrum with n element components, w. In this case, matrix A can be expressed by the following formula (32).

number

[0179] In formula (32), it is necessary to refer to the matrix A(p,w) when calculating Xij and Yij given by formula (31) above. Therefore, it becomes necessary to perform calculations for all points p for each iterative calculation.

[0180] Performing calculations for all points p in each iteration is unlikely to be a problem if the data to be analyzed is small, but in the case of large-scale data with a very large number of data points, it can cause the calculation time to be long. Also, if it is not possible to expand all p pieces of data in memory (for example, RAM 903 in FIG. 42 described later), there may be a problem that access to an external storage device (for example, storage device 908 in FIG. 42) occurs frequently, making the processing time longer.

[0181] On the other hand, instead of the data matrix A to be decomposed, the inventors use its Gram matrix t We discovered that it is also possible to obtain spectrum S by non-negative decomposition of AA.

[0182] In this embodiment, the data matrix A is expressed as a Gram matrixt Convert this Gram matrix to AA. t The spectrum S, which is the solution, is obtained by decomposing AA into non-negative values. t By converting to AA, the matrix to be processed can be made a square matrix. For example, if the data matrix A has a very large number of pixels N compared to the number of wavelength channels M, it can be converted to an M × M Gram matrix. t Since the data is converted to AA, it is possible to significantly reduce the number of data points and shorten the calculation time, and also to significantly reduce the amount of memory required for calculation. As a result, it is possible to achieve more efficient analysis.

[0183] (6.1. Overview of Processing by the Processing Unit) The information processing device according to this embodiment has a configuration similar to that of the information processing device 100 (see FIG. 1) according to the above-described embodiment, in which a processing unit 130 (for example, separation processing unit 132) executes the following operations.

[0184] First, in the process of subjecting the data matrix A to non-negative value factorization or singular value decomposition into A=S×C, the processing unit 130 according to this embodiment calculates the Gram matrix of the matrix A in advance. t Calculate AA and calculate the Gram matrix t A.A. t The spectrum S is obtained by performing non-negative decomposition into AA=S×E.

[0185] Secondly, the processing unit 130 according to this embodiment calculates the Gram matrix t In the process of calculating AA, the subset A(p,w)=A1(p1-pn1,w)+A2(pn1+1-pm,w)+...+Ao(pm+1-p,w) is used, and each Gram matrix t The Gram matrix AqAq (q is an integer between 1 and n) is folded as shown in the following equation (33). t Ask for AA.

number

[0186] Third, the spectrum S obtained by non-negative value decomposition of the Gram matrix is ​​used to solve A=S×C to obtain the coefficient C.

[0187] (6.2. Measurement system configuration example) Next, a configuration example of the measurement system in the information processing device 100 according to this embodiment will be described. FIG. 36 is a diagram showing an example of the measurement system of the information processing system according to this embodiment. FIG. 36 shows an example of a measurement system for capturing a wide field of view of a fluorescently stained specimen 30 (or a specimen 20 that is an unstained specimen), such as WSI (Whole Slide Imaging). However, the measurement system according to this embodiment is not limited to the measurement system shown in FIG. 36, and may be modified in various ways as long as it is capable of acquiring image data (hereinafter referred to as wide-field image data) of a sufficient resolution of the entire photographed region or the region of interest, such as a measurement system that captures the entire photographed region or a necessary region therein (also referred to as a region of interest) at once, or a measurement system that acquires an image of the entire photographed region or the region of interest by line scanning.

[0188] As shown in FIG. 36, the measurement system according to this embodiment includes, for example, an information processing device 100, an XY stage 501, an excitation light source 510, a beam splitter 511, an objective lens 512, a spectroscope 513, and a photodetector 514.

[0189] The XY stage 501 is a stage on which the fluorescent stained specimen 30 (or specimen 20) to be analyzed is placed, and may be, for example, a stage that is movable in a plane (XY plane) parallel to the surface on which the fluorescent stained specimen 30 (or specimen 20) is placed.

[0190] The excitation light source 510 is a light source for exciting the fluorescent stained specimen 30 (or the specimen 20), and for example, emits a plurality of excitation light beams having mutually different wavelengths along a predetermined optical axis.

[0191] The beam splitter 511 is composed of, for example, a dichroic mirror, and reflects the excitation light from the excitation light source 510 and transmits the fluorescence from the fluorescently stained specimen 30 (or specimen 20).

[0192] The objective lens 512 irradiates the excitation light reflected by the beam splitter 511 onto the fluorescent stained specimen 30 (or specimen 20) on the XY stage 501.

[0193] The spectroscope 513 is configured using one or more prisms, lenses, etc., and splits the fluorescence emitted from the fluorescent stained specimen 30 (or specimen 20) and transmitted through the objective lens 512 and beam splitter 511 into beams in predetermined directions.

[0194] The photodetector 514 detects the light intensity for each wavelength of the fluorescence dispersed by the spectroscope 513, and inputs the thus obtained fluorescence signal (fluorescence spectrum and / or autofluorescence spectrum) to the fluorescence signal acquisition unit 112 of the information processing device 100.

[0195] In the above configuration, when the entire photographing area exceeds the area that can be photographed in one shot (hereinafter referred to as the field of view), such as WSI, the XY stage 501 is moved for each photographing to move the field of view, and photographing of each field of view is performed sequentially. Then, wide-field image data of the entire photographing area is generated by tiling the image data (hereinafter referred to as the field of view image data) obtained by photographing each field of view. The generated wide-field image data is stored, for example, in the fluorescent signal storage unit 122. Note that the tiling of the field of view image data may be performed in the acquisition unit 110 of the information processing device 100, in the storage unit 120, or in the processing unit 130.

[0196] Then, the processing unit 130 according to this embodiment performs the above-mentioned processing on the obtained wide-field image data to obtain the coefficient C, i.e., a fluorescence separation image for each fluorescent molecule (or an autofluorescence separation image for each autofluorescent molecule).

[0197] (6.3. Example of operation) Next, an example of the operation of the information processing device 100 according to the present embodiment will be described. Note that in the following description, attention will be focused on the operation of the processing unit 130.

[0198] Fig. 37 is a flowchart showing an example of the operation of the processing unit according to this embodiment, and Figs. 38 to 40 are diagrams for explaining the processing executed by the processing unit in each step in Fig. 37.

[0199] As shown in Figure 37, the processing unit 130 in this embodiment first generates wide-field image data of the entire shooting area (for example, see wide-field image data A in Figure 38) by tiling the field of view image data obtained by shooting each field of view (step S2001).

[0200] Next, the processing unit 130 acquires unit image data (for example, unit image data Aq in FIG. 38 (q is an integer greater than or equal to 1 and less than or equal to n) which is a part of the wide-field image data A (step S2002). The unit image data Aq may be changed to various types of image data of a narrower area than the wide-field image data A, such as image data corresponding to one field of view or image data of a preset size. Note that image data of a preset size may include image data of a size determined by the amount of data that the information processing device 100 can process at one time.

[0201] Next, as illustrated in FIG. 38, the processing unit 130 applies the transposed matrix t By multiplying A1, the Gram matrix of the unit image data A1 is obtained. t A1A1 is generated (step S2003).

[0202] Next, the processing unit 130 calculates the Gram matrix for all the unit image data A1 to An. t A1A1~ t It is determined whether or not the generation of AnAn is completed (step S2004), and the Gram matrices for all the unit image data A1 to An are generated. t A1A1~ t Steps S2002 to S2004 are repeatedly executed until the generation of AnAn is completed (NO in step S2004).

[0203] On the other hand, the Gram matrix for all the unit image data A1 to An t A1A1~ t When the generation of AnAn is completed (YES in step S2004), the processing unit 130 calculates the obtained Gram matrix by using, for example, the least squares method (or the weighted least squares method). t A1A1~ t An initial value of the coefficient C is calculated from AnAn (step S2005).

[0204] Next, the processing unit 130 calculates the generated Gram matrix t A1A1~ t Adding AnAn gives the Gram matrix for wide-field image data A. t AA is calculated (step S2006). Specifically, as described above, the subset A(p,w)=A1(p1-pn1,w)+A2(pn1+1-pm,w)+...+Ao(pm+1-p,w) is used, and the Gram matrices t By convolving AqAq (q is an integer between 1 and n) as shown in formula (33), the Gram matrix t Ask for AA.

[0205] Next, the processing unit 130 calculates the calculated Gram matrix as shown in FIG. t A.A. t A spectrum S is obtained by performing non-negative value decomposition into AA=S×E (step S2007). Note that the matrix E corresponds to a separated image obtained by separating the wide-field image data A into fluorescent images.

[0206] Then, the processing unit 130 calculates the Gram matrix as shown in FIG. t The spectrum S obtained by NMF for AA is used to solve A=S×C by the least squares method (or the weighted least squares method) to obtain the coefficient C, i.e., the fluorescence separation image for each fluorescent molecule (or the autofluorescence separation image for each autofluorescent molecule) (step S2008), and then this operation is terminated.

[0207] In addition, in the NMF in step S2007, a specific spectrum may be fixed and non-negative factorization of the data may be performed.

[0208] (6.4.1. Variation 1) In addition, in FIG. 37 to FIG. 40, the case where the entire photographed region is the processing target region is illustrated, but the present invention is not limited thereto, and the processing target region can be a region (region of interest) narrower than the entire photographed region. The region of interest may be, for example, a region in which an analysis target is projected, such as a region in which the fluorescent stained specimen 30 (or specimen 20) exists in the wide-field image data A. In addition, for example, morphological information of the fluorescent stained specimen 30 or specimen 20 (for example, cells, tissues, etc.) may be used to set the region of interest. The morphological information may be a bright-field image, an unstained image, and staining information of the same tissue block, and may be, for example, an expression map of a target in the specimen 20. In addition, the morphological information may be information generated using a technique such as segmentation (acquiring and labeling a region in units of one pixel) in machine learning image recognition technology.

[0209] Fig. 41 is a flowchart showing an example of the operation of the processing unit according to the first modified example of this embodiment. As shown in Fig. 41, the processing unit 130 according to this first modified example first generates wide-field image data A of the entire photographed area by tiling the field-of-view image data obtained by photographing each field of view (step S2101). In this first modified example, the resolution of the wide-field image data A may be lower than the resolution of the image data to be processed (for example, high-resolution image data to be described later).

[0210] Next, the processing unit 130 sets a monitoring region to be processed in the wide-field image data A (step S2102). The setting of this region of interest may be performed based on, for example, morphological information, as described above. However, the setting of the region of interest may be performed automatically by the processing unit 130 based on the morphological information, or may be performed manually by the user.

[0211] Next, the processing unit 130 requests, for example, the control unit 150 to acquire high-resolution image data of the region of interest (step S2103). In response, the control unit 150 acquires high-resolution image data of the region of interest by controlling the above-mentioned measurement system (see FIG. 36), the acquisition unit 110, and the storage unit 120. Note that the region of interest may be a range larger than one field of view.

[0212] Next, the processing unit 130 executes operations similar to those in steps S2002 to S2004 in FIG. 37 to obtain the Gram matrix of each of the unit image data Aq acquired from the high-resolution image data of the region of interest. t AqAq is generated (steps S2104 to S2106).

[0213] Next, the processing unit 130 calculates the obtained Gram matrix by using the least squares method (or the weighted least squares method) in the same manner as in step S2005 of FIG. t A1A1~ t An initial value of the coefficient C is calculated from AnAn (step S2107).

[0214] Next, the processing unit 130 performs the same process as in steps S2006 to S2008 of FIG. 37 on the generated Gram matrix t A1A1~ t Adding AnAn gives the Gram matrix for wide-field image data A. t AA is calculated (step S2108), and the calculated Gram matrix t A.A. t The spectrum S is obtained by performing non-negative value decomposition into AA=S×E (step S2109), and the Gram matrix t The coefficient C, i.e., a fluorescence separation image for each fluorescent molecule (or an autofluorescence separation image for each autofluorescent molecule), is obtained by solving A=S×C by the least squares method (or weighted least squares method) using the spectrum S obtained by NMF for AA (step S2110), and then this operation ends. Note that in the NMF of step S2109, a specific spectrum may be fixed and nonnegative factorization of the data may be performed.

[0215] (6.4.2. Variation 2) In the operation example shown in Fig. 37 and its modified example (Fig. 41), a case is illustrated in which wide-field image data of the entire imaging area or high-resolution image data of the entire region of interest is acquired first, and then unit image data that is a part of it is acquired and processed sequentially, but this is not limited to this, and for example, all or part of it can be pipelined. Specifically, for example, the Gram matrix of each unit image data t In the process for generating AqAq (for example, steps S2001 to S2004 in FIG. 37 or steps S2103 to S106 in FIG. 41), the image data of each visual field output from the measurement system (see FIG. 36) is treated as unit image data, and the above process is performed in response to input of the unit image data, thereby generating the Gram matrix for each unit image data. t AqAq may be produced.

[0216] (6.5. Effects) The effects expected from this embodiment are described below in terms of the process of obtaining a solution for spectrum S and coefficients C of A=S×C by NMF. The first case is when NMF is performed on matrix A(p,w) (case 1), and the second case is when NMF is performed on the Gram matrix obtained from matrix A (case 2). t We will explain this by taking as an example the case where NMF of AA(w,w) is performed (Case 1).

[0217] Assuming that the calculation time for the four arithmetic operations is roughly the same and that overhead is not taken into account, the amount of calculations for the NMF loop in each of Cases 1 and 2 is estimated to be approximately 6,000 times faster in Case 2, which goes through the Gram matrix tAA, compared to Case 1, which performs NMF on matrix A.

[0218] In addition, when considering the calculation of 10 to 100 unit image data of wide-field image data such as WSI, in contrast to Case 1 where wide-field image data A is directly NMFed, the Gram matrix of the wide-field image data is calculated by convolution of the Gram matrix of each unit image data. t In case 2, where AA is calculated, it is estimated that the processing speed can be increased by approximately 60,000 to 600,000 times.

[0219] Furthermore, when each unit image data is 1024 × 1024 image data and the number of wavelength channels (M) is 100, the maximum memory capacity required to expand the data is the Gram matrix A for case 1 where the matrix A(p,w) is NMFed. t In case 2 where AA(w,w) is subjected to NMF, it is possible to reduce the memory size by approximately 1 / 10,000. In addition, when considering 10 to 100 unit image data, the memory size can be further reduced, for example, by 1 / 100,000 to 1 / 1,000,000.

[0220] The other configurations, operations, and effects may be similar to those of the above-described embodiment, and therefore detailed description thereof will be omitted here.

[0221] <7. Hardware configuration example> The above describes the modified examples of the present disclosure. Next, with reference to Fig. 42, a hardware configuration example of the information processing device 100 according to each embodiment and modified example will be described. Fig. 42 is a block diagram showing a hardware configuration example of the information processing device 100. Various processes by the information processing device 100 are realized by cooperation between software and hardware described below.

[0222] 42, the information processing device 100 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing device 100 also includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing device 100 may include a processing circuit such as a DSP or an ASIC instead of or in addition to the CPU 901.

[0223] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing device 100 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. The CPU 901 may embody at least the processing unit 130 and the control unit 150 of the information processing device 100, for example.

[0224] The CPU 901, ROM 902, and RAM 903 are connected to one another by a host bus 904a including a CPU bus, etc. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 904. Note that the host bus 904a, bridge 904, and external bus 904b do not necessarily need to be configured separately, and these functions may be implemented on a single bus.

[0225] The input device 906 is realized by a device into which information is input by the implementer, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever. The input device 906 may be, for example, a remote control device using infrared or other radio waves, or an external connection device such as a mobile phone or a PDA that supports the operation of the information processing device 100. Furthermore, the input device 906 may include, for example, an input control circuit that generates an input signal based on information input by the implementer using the above-mentioned input means and outputs it to the CPU 901. The implementer can input various data to the information processing device 100 and instruct processing operations by operating this input device 906. The input device 906 may embody at least the operation unit 160 of the information processing device 100, for example.

[0226] The output device 907 is formed by a device capable of visually or audibly notifying the operator of acquired information. Such devices include display devices such as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, and a lamp, audio output devices such as a speaker and a headphone, a printer, etc. The output device 907 may embody at least the display unit 140 of the information processing device 100, for example.

[0227] The storage device 908 is a device for storing data. The storage device 908 is realized by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. This storage device 908 stores the programs and various data executed by the CPU 901, and various data acquired from the outside. The storage device 908 may embody at least the storage unit 120 of the information processing device 100, for example.

[0228] The drive 909 is a reader / writer for a storage medium, and is built into or externally attached to the information processing device 100. The drive 909 reads information recorded on a removable storage medium, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903. The drive 909 can also write information to the removable storage medium.

[0229] The connection port 911 is an interface connected to an external device, and is a connection port for connecting to an external device capable of transmitting data via, for example, a USB (Universal Serial Bus) or the like.

[0230] The communication device 913 is, for example, a communication interface formed by a communication device for connecting to the network 920. The communication device 913 is, for example, a communication card for a wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 913 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 913 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP.

[0231] In this embodiment, the sensor 915 includes a sensor capable of acquiring a spectrum (e.g., an image sensor, etc.), but may also include other sensors (e.g., an acceleration sensor, a gyro sensor, a geomagnetic sensor, a pressure sensor, a sound sensor, a distance measurement sensor, etc.) The sensor 915 may embody at least the fluorescent light signal acquisition unit 112 of the information processing device 100, for example.

[0232] The network 920 is a wired or wireless transmission path for information transmitted from devices connected to the network 920. For example, the network 920 may include public line networks such as the Internet, telephone lines, and satellite communication networks, as well as various types of LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The network 920 may also include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0233] The above shows an example of a hardware configuration capable of realizing the functions of the information processing device 100. Each of the above components may be realized using a general-purpose member, or may be realized by hardware specialized for the function of each component. Therefore, the hardware configuration to be used can be changed as appropriate depending on the technical level at the time of implementing the present disclosure.

[0234] It is possible to create a computer program for implementing each function of the information processing device 100 as described above and install it in a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium includes, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, and the like. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0235] <8. Conclusion> As described above, the information processing device 100 according to the first embodiment of the present disclosure irradiates the fluorescent stained specimen 30 with a plurality of excitation lights having different wavelengths, acquires a plurality of fluorescence spectra corresponding to the plurality of excitation lights, corrects the plurality of fluorescence spectra based on the intensity of the excitation lights, and generates a concatenated fluorescence spectrum by concatenating at least a portion of the plurality of fluorescence spectra in the wavelength direction. Then, the information processing device 100 extracts a spectrum for each fluorescent substance from a reference spectrum including a concatenated autofluorescence reference spectrum in which the spectra of autofluorescent substances are concatenated in the wavelength direction, and a concatenated fluorescence reference spectrum in which the spectra of fluorescent substances are concatenated in the wavelength direction. Then, the information processing device 100 separates the concatenated fluorescence spectrum for each molecule using the extracted spectrum for each fluorescent substance.

[0236] In this way, the information processing device 100 performs fluorescence separation processing using reference spectra concatenated in the wavelength direction, and can output a unique spectrum as the separation result (separation results are not different for each excitation wavelength). Therefore, the practitioner can more easily obtain a correct spectrum. In addition, the reference spectrum (concatenated autofluorescence reference spectrum) related to the autofluorescence used in separation is automatically acquired, and the fluorescence separation processing is performed, eliminating the need for the practitioner to extract a spectrum corresponding to the autofluorescence from an appropriate space in the unstained section.

[0237] Moreover, the information processing device 100 according to the second embodiment of the present disclosure performs fluorescence separation processing using a combined autofluorescence reference spectrum actually measured for each specimen 20. This enables the information processing device 100 to realize more accurate fluorescence separation processing.

[0238] Furthermore, the information processing device 100 according to the modified example of the present disclosure separates the combined fluorescence spectrum into molecules using a reference spectrum including a combined autofluorescence reference spectrum and a combined fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules, thereby enabling the information processing device 100 to calculate the number of fluorescent molecules or antibodies in the fluorescent stained specimen 30 as a result of the fluorescence separation process.

[0239] Furthermore, the information processing device 100 according to the third embodiment of the present disclosure solves the NMF with the specimen image acquired from the stained section as the matrix A. This makes it possible to extract the fluorescence spectrum for each fluorescent substance directly from the stained section while significantly reducing the time and labor costs required for the fluorescence separation process. In addition, in the third embodiment of the present disclosure, since the fluorescence spectrum for each fluorescent substance is extracted from the specimen image acquired from the same stained section, it is possible to acquire a more accurate fluorescence separation result compared to, for example, the case of using an autofluorescence spectrum acquired from an unstained section different from the stained section.

[0240] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0241] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0242] Note that the following configurations also fall within the technical scope of the present disclosure. (1) a fluorescence signal acquisition unit that irradiates a plurality of excitation lights having different wavelengths onto a fluorescence-labeled cell, the cell being labeled with a fluorescent substance, and acquires a plurality of fluorescence spectra corresponding to the plurality of excitation lights; a connection unit that generates a connected fluorescence spectrum by connecting at least a portion of the plurality of fluorescence spectra in a wavelength direction; a separation unit that separates the combined fluorescence spectrum into spectra for each of the fluorescent substances using a reference spectrum including a combined autofluorescence reference spectrum in which spectra of autofluorescent substances in the cell are combined in the wavelength direction and a combined fluorescence reference spectrum in which spectra of fluorescent substances in the fluorescent-labeled cell are combined in the wavelength direction; Equipped with Information processing device. (2) an extracting unit that updates the combined autofluorescence reference spectrum using the spectrum for each of the fluorescent substances separated by the separating unit; The information processing device according to (1). (3) the extraction unit extracts the concatenated autofluorescence reference spectrum from a concatenation, in a wavelength direction, of at least a portion of a plurality of autofluorescence spectra acquired by irradiating an object identical to or similar to the cell with the plurality of excitation lights; The information processing device according to (2). (4) the extraction unit extracts the concatenated autofluorescence reference spectrum by performing non-negative matrix factorization using at least a portion of a plurality of autofluorescence spectra obtained by irradiating an object identical to or similar to the cell with the plurality of excitation lights in a wavelength direction; The information processing device according to (3). (5) the extraction unit extracts the concatenated autofluorescence reference spectrum by setting an initial value in the non-negative matrix factorization using a previously acquired autofluorescence spectrum; The information processing device according to (4). (6) The separation unit separates the reference spectrum into spectra for each of the fluorescent substances by using either a least squares method or a weighted least squares method. The information processing device according to any one of (1) to (5) above. (7) the separation unit separates the spectrum into spectra for each of the fluorescent substances by calculating matrix a that represents the color mixing ratio when a sum of squares of values ​​represented by the following formula (34) is minimized, where Signal is a matrix that represents the concatenated fluorescence spectrum, St is a matrix that represents the reference spectrum, and a is a matrix that represents a color mixing ratio of each of the reference spectra in the concatenated fluorescence spectrum: The information processing device according to (6).

number

number

[0243] 10 Fluorescent Reagents 11 Reagent Identification Information 20 specimens 21 Specimen Identification Information 30 Fluorescent stained specimens 100 Information processing device 110 Acquisition Department 111 Information Acquisition Department 112 Fluorescence signal acquisition unit 120 Preservation Department 121 Information Storage Department 122 Fluorescence signal storage unit 130 Processing section 131 Connecting part 132 Separation Processing Unit 133 Image Generation Unit 140 Display section 150 Control section 160 Operation section 200 Databases

Claims

1. a fluorescence signal acquisition unit that irradiates a plurality of excitation lights having different wavelengths onto a fluorescence-labeled cell, the cell being labeled with a fluorescent substance, and acquires a plurality of fluorescence spectra corresponding to the plurality of excitation lights; a connection unit that generates a connected fluorescence spectrum by connecting at least a portion of the plurality of fluorescence spectra in a wavelength direction; a separation unit that separates the combined fluorescence spectrum into spectra for each of the fluorescent substances using a reference spectrum including a combined autofluorescence reference spectrum in which spectra of autofluorescent substances in the cell are combined in the wavelength direction and a combined fluorescence reference spectrum in which spectra of fluorescent substances in the fluorescent-labeled cell are combined in the wavelength direction; Equipped with Information processing device.

2. an extracting unit that updates the combined autofluorescence reference spectrum using the spectrum for each of the fluorescent substances separated by the separating unit; The information processing device according to claim 1 .

3. the extraction unit extracts the concatenated autofluorescence reference spectrum from a concatenation, in a wavelength direction, of at least a portion of a plurality of autofluorescence spectra acquired by irradiating an object identical to or similar to the cell with the plurality of excitation lights; The information processing device according to claim 2 .

4. the extraction unit extracts the concatenated autofluorescence reference spectrum by performing non-negative matrix factorization using at least a portion of a plurality of autofluorescence spectra obtained by irradiating an object identical to or similar to the cell with the plurality of excitation lights in a wavelength direction; The information processing device according to claim 3 .

5. the extraction unit extracts the concatenated autofluorescence reference spectrum by setting an initial value in the non-negative matrix factorization using a previously acquired autofluorescence spectrum; The information processing device according to claim 4.

6. The separation unit separates the reference spectrum into spectra for each of the fluorescent substances by using either a least squares method or a weighted least squares method. The information processing device according to claim 1 .

7. the separation unit separates the spectrum into spectra for each of the fluorescent substances by calculating matrix a that represents the color mixing rate when a sum of squares of values ​​represented by the following formula (1) is minimized, where Signal is a matrix that represents the concatenated fluorescence spectrum, St is a matrix that represents the reference spectrum, and a is a matrix that represents a color mixing rate of each of the reference spectra in the concatenated fluorescence spectrum: The information processing device according to claim 6. [0010]

8. When the weighted least squares method is used, the separation unit sets an upper limit value at which no weighting is performed as an Offset value, and replaces a matrix S representing the reference spectrum in Equation (1) with a matrix S represented by the following Equation (2): The information processing device according to claim 7. [0025]

9. the separation unit separates the combined fluorescence spectrum into spectra for each of the fluorescent substances by using the reference spectrum including the combined autofluorescence reference spectrum and the combined fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules, or the reference spectrum including the combined autofluorescence reference spectrum and the combined fluorescence reference spectrum for each of the fluorescent molecules or each of the antibodies. The information processing device according to claim 1 .

10. The separation unit separates the concatenated fluorescence spectrum into spectra for the respective fluorescent substances by performing nonnegative matrix factorization on the concatenated fluorescence spectrum. The information processing device according to claim 1 .

11. the separation unit calculates a product-moment correlation coefficient between the spectrum extracted by the nonnegative matrix factorization and an initial value used in the nonnegative matrix factorization, thereby identifying a correspondence relationship between the fluorescent substance and the extracted spectrum. The information processing device according to claim 10.

12. The connection unit corrects the plurality of fluorescence spectra and connects at least a portion of the plurality of corrected fluorescence spectra in the wavelength direction. The information processing device according to claim 1 .

13. The coupling unit corrects the intensities of the plurality of fluorescence spectra. The information processing device according to claim 12.

14. The coupling unit corrects intensities of the plurality of fluorescence spectra by dividing the plurality of fluorescence spectra by an excitation power density. The information processing device according to claim 13.

15. The connection unit corrects the wavelength resolution of at least one of the plurality of fluorescence spectra to a wavelength resolution different from the wavelength resolutions of the other fluorescence spectra. The information processing device according to claim 12.

16. the connecting unit extracts a fluorescence spectrum of a wavelength band including an intensity peak from each of the plurality of fluorescence spectra, and generates the connected fluorescence spectrum by connecting the extracted fluorescence spectra. The information processing device according to claim 1 .

17. The coupled fluorescence spectrum is discontinuously coupled in the wavelength direction between the plurality of fluorescence spectra. The information processing device according to claim 1 .

18. the fluorescent signal acquisition unit acquires first image data including the plurality of fluorescent spectra obtained by imaging the fluorescent labeled cells; the separation unit separates the first image data into spectra for each of the fluorescent substances by performing non-negative matrix factorization on a first Gram matrix of the first image data. The information processing device according to claim 1 .

19. the separation unit calculates the first Gram matrix by convolving a second Gram matrix of each of a plurality of second image data obtained by dividing the first image data. The information processing device according to claim 18.

20. the fluorescence signal acquisition unit acquires first image data by capturing an image of the unstained cell irradiated with the excitation light; the extraction unit extracts a spectrum for each of the autofluorescent substances from the first image data by performing non-negative matrix factorization on a first Gram matrix of the first image data, and updates the concatenated autofluorescence reference spectrum using the extracted spectrum for each of the autofluorescent substances. The information processing device according to claim 2 .

21. An information processing system including: a light source that irradiates a fluorescent-labeled cell, which is a cell labeled with a fluorescent substance, with a plurality of excitation lights having mutually different wavelengths; an imaging device that acquires a plurality of fluorescence spectra corresponding to the plurality of excitation lights; and software used in processing using the plurality of fluorescence spectra, The software is executed on an information processing device, generating a concatenated fluorescence spectrum by concatenating at least a portion of the plurality of fluorescence spectra in a wavelength direction; separating the combined fluorescence spectrum into spectra for each of the fluorescent substances using a reference spectrum including a combined autofluorescence reference spectrum in which spectra of autofluorescent substances in the cell are combined in the wavelength direction and a combined fluorescence reference spectrum in which spectra of fluorescent substances in the fluorescent-labeled cell are combined in the wavelength direction; To achieve this, Information processing system.

Images