Information processing equipment and microscope systems

Abstract

The present invention makes it possible to perform fluorescence separation more appropriately. An information processing device according to one embodiment is provided with: a fluorescence signal acquisition unit (112) which acquires a plurality of fluorescence spectra respectively corresponding to a plurality of excitation lights having mutually different wavelengths and irradiating a fluorescent staining sample (30) prepared by staining a sample (20) with a fluorescent reagent (10); a linking unit (131) which links at least some of the plurality of fluorescence spectra in a wavelength direction to generate a linked fluorescence spectrum; a separation processing unit (132) which separates the linked fluorescence spectrum into a spectrum of each fluorescent material using a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum, in which the spectra of the autofluorescence material in the sample are linked in the wavelength direction, and in which the spectra of the fluorescent material in the fluorescent staining sample are linked in the wavelength direction; and an extraction unit (132) which updates the linked autofluorescence reference spectrum using the spectrum of each fluorescent material separated by the separation processing unit.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and a microscope system. Background Art

[0002] In recent years, multiple labeling with fluorescence and immunostaining has been progressing with the development of cancer immunotherapy, etc. For example, a method has been performed to extract autofluorescence spectra from non-stained sections of the same tissue block and then perform fluorescence separation on stained sections using the autofluorescence spectra.

[0003] Furthermore, for example, Patent Document 1 below discloses a technique of approximating a fluorescence spectrum obtained by irradiating microparticles multi-labeled with a plurality of fluorescent pigments with excitation light to a linear sum of single-staining spectra obtained from microparticles individually labeled with each fluorescent pigment.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2012-18108 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] However, according to these techniques or methods, there are cases where fluorescence separation cannot be performed appropriately. For example, in the case where an autofluorescence spectrum is extracted from a non-stained section of the same tissue block and then fluorescence separation of the stained section is performed using the autofluorescence spectrum, the physician is required to extract the autofluorescence spectrum from an appropriate space in the non-stained section, and therefore, the accuracy of fluorescence separation depends on the work done by the physician. In addition, since fluorescence separation is performed for each excitation wavelength, the separation result is output for each excitation wavelength, so that the spectrum obtained as a separation result is not uniquely determined.

[0009] Therefore, the present disclosure has been made in view of the above circumstances, and provides a novel and improved information processing apparatus and microscope system capable of more appropriately performing fluorescence separation.

[0010] Solution to the problem

[0011] According to an embodiment of the present disclosure, an information processing device includes: a fluorescence signal acquisition unit, which acquires multiple fluorescence spectra corresponding to each of a plurality of excitation lights having different wavelengths and irradiated to a fluorescently stained sample, wherein the fluorescently stained sample is generated by staining the sample with a fluorescent reagent; a linking unit, which generates a linked fluorescence spectrum by linking at least a portion of the multiple fluorescence spectra to each other in a wavelength direction; a separation unit, which separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum, in which the spectra of the autofluorescence substances in the sample are linked to each other in the wavelength direction, and in which the spectra of the fluorescent substances in the fluorescently stained sample are linked to each other in the wavelength direction; and an extraction unit, which updates the linked autofluorescence reference spectrum using the spectrum of each fluorescent substance separated by the separation unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing a configuration example of the information processing system according to the first embodiment.

[0013] Figure 2 is a diagram showing a specific example of a fluorescence spectrum acquired by the fluorescence signal acquisition unit.

[0014] Figure 3 is a diagram for describing a method of generating linked fluorescence spectra by linking units.

[0015] Figure 4 is a graph showing the fluorescence spectra of AF546 and AF555 at a wavelength resolution of 8 nm.

[0016] Figure 5 is a graph showing the fluorescence spectra of AF546 and AF555 at a wavelength resolution of 1 nm.

[0017] Figure 6 It is shown from Figure 3 The fluorescence spectra shown in A to D are diagrams of examples of linked fluorescence spectra generated.

[0018] Figure 7 is a block diagram showing a more specific configuration example of the separation processing unit according to the embodiment of the first embodiment.

[0019] Figure 8 is a diagram showing a specific example of linked autofluorescence reference spectra.

[0020] Fig. 9 is a diagram showing a specific example of linked fluorescence reference spectra.

[0021] Fig.10: is a block diagram showing a configuration example of a microscope system in a case where the information processing system according to the first embodiment is implemented as a microscope system.

[0022] Fig.11 : is a flowchart showing an example of the flow of processing of fluorescence separation performed by the information processing apparatus according to the first embodiment.

[0023] Fig.12 is a block diagram showing a more specific configuration example of the separation processing unit according to the second embodiment.

[0024] Fig.13 is a diagram for describing an overview of non-negative matrix factorization.

[0025] Fig.14 is a diagram for describing an overview of clustering.

[0026] Fig.15 : is a flowchart showing an example of the flow of processing of fluorescence separation performed by the information processing apparatus according to the second embodiment.

[0027] Fig.16 : is a diagram for describing a method of calculating the number of fluorescent molecules (or the number of antibodies) in the imaging element 1 [pixel] in the modification.

[0028] Fig.17 is a block diagram showing a schematic configuration example of a separation processing unit according to the third embodiment.

[0029] Fig.18 : is a diagram showing an example of a sample image (excitation wavelength: 392 nm) input to the matrix A in the third embodiment.

[0030] Fig.19 : is a diagram showing an example of a sample image (excitation wavelength: 470 nm) input to the matrix A in the third embodiment.

[0031] Fig. 20 : is a diagram showing an example of a sample image (excitation wavelength: 515 nm) input to the matrix A in the third embodiment.

[0032] Fig.21 : is a diagram showing an example of a sample image (excitation wavelength: 549 nm) input to the matrix A in the third embodiment.

[0033] Fig. 22 : is a diagram showing an example of a sample image input to the matrix A in the third embodiment (excitation wavelength: 628 nm).

[0034] Fig.23 It is shown that in the third embodiment (part 1) input Figures 18 to 22FIG. 5 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0035] Fig.24 It is shown that in the third embodiment (part 2) input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0036] Fig.25 It is shown that in the third embodiment (Part 3) input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0037] Fig.26 It is shown that in the third embodiment (part 4) input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0038] Fig. 27 It is shown that in the third embodiment (part 5) input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0039] Fig.28 It is shown that in the third embodiment (part 6) input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0040] Fig.29 It is shown that in the third embodiment (part 7) input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as a matrix W in the case of the sample image shown.

[0041] Fig.30 is a flowchart for describing the NMF process according to the fourth embodiment.

[0042] Fig.31 Is used to describe Fig.30 Schematic diagram of the process flow in the first cycle of NMF shown.

[0043] Fig.32 is a diagram showing an example of the initial value of the dye fluorescence spectrum.

[0044] Fig.33 : is a diagram showing an example of a dye fluorescence spectrum after performing NMF according to the fourth embodiment.

[0045] Fig.34: is a diagram showing an example of a spectrum of a fluorescent substance extracted by a method not using a non-stained sample according to the fourth embodiment.

[0046] Fig.35 is a diagram showing an example of a spectrum of a fluorescent substance extracted in the case of using a non-stained sample.

[0047] Fig.36 is a diagram showing an example of a measurement system of the information processing system according to the sixth embodiment.

[0048] Fig.37 is a flowchart showing an operation example of the processing unit according to the sixth embodiment.

[0049] Fig.38 is used to describe Fig.37 Illustration of the processing performed by the processing unit in each step in (Part 1).

[0050] Fig.39 is used to describe Fig.37 Illustration of the processing performed by the processing unit in each step in (Part 2).

[0051] Fig.40 is used to describe Fig.37 (Part 3) is a diagram of the processing performed by the processing unit in each step.

[0052] Fig.41 is a flowchart illustrating an operation example of a processing unit according to a first modification example of the sixth embodiment.

[0053] Fig.42 is a block diagram showing a hardware configuration example of an information processing apparatus according to each of the embodiments and modifications. DETAILED DESCRIPTION

[0054] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and overlapping descriptions will be omitted.

[0055] Note that the description will be given in the following order.

[0056] 1. First Embodiment

[0057] 1.1. Configuration Example

[0058] 1.2. Processing flow example

[0059] 2. Second Embodiment

[0060] 2.1. Processing flow example

[0061] 2.2. Reasons why PCA is not suitable as a method for extracting linked autofluorescence reference spectra from non-stained sections

[0062] 2.3 Application Examples

[0063] 3. Modification Example

[0064] 4. Third embodiment

[0065] 5. Fourth embodiment

[0066] 5.1. Fixed method for dye fluorescence spectra in minimizing the mean square residual D using a recursive formula

[0067] 5.2. Fixed method of dyeing fluorescence spectra in minimizing the mean square residual D using DFP method, BFGS method, etc.

[0068] 6. Fifth embodiment

[0069] 6.1. Processing overview of processing units

[0070] 6.2. Example of measurement system configuration

[0071] 6.3. Operation examples

[0072] 6.4.1. First Modification

[0073] 6.4.2. Second Modification

[0074] 6.5. Effect

[0075] 7. Hardware Configuration Example

[0076] 8. Conclusion

[0077] <1. First Embodiment>

[0078] First, a first embodiment of the present disclosure will be described.

[0079] (1.1. Configuration example)

[0080] Will refer to Figure 1 A configuration example of an information processing system according to this embodiment is described. Figure 1 As shown, the information processing system according to the present embodiment includes an information processing device 100 and a database 200, and has a fluorescent reagent 10, a sample 20, and a fluorescently stained sample 30 as inputs of the information processing system.

[0081] (Fluorescent reagent 10)

[0082] The fluorescent reagent 10 is a chemical used to stain the sample 20. The fluorescent reagent 10 is, for example, a fluorescent antibody (including a primary antibody for direct labeling or a secondary antibody for indirect labeling), a fluorescent probe, a nuclear staining reagent, etc., but the type of the fluorescent reagent 10 is not limited thereto. In addition, the fluorescent reagent 10 is managed by attaching identification information (hereinafter referred to as "reagent identification information 11") that can identify the fluorescent reagent 10 (or the production batch of the fluorescent reagent 10) to the fluorescent reagent 10. The reagent identification information 11 is, for example, barcode information, etc. (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited thereto. The properties of the fluorescent reagent 10 are different for each production batch according to the production method, the state of the cell from which the antibody is obtained, etc., even if the products are the same as each other. For example, in the fluorescent reagent 10, the spectrum, quantum yield, fluorescent labeling rate, etc. are different for each production batch. Therefore, in the information processing system according to the present embodiment, by attaching the reagent identification information 11 to the fluorescent reagent 10, the fluorescent reagent 10 is managed for each production batch. Therefore, the information processing apparatus 100 can perform fluorescence separation in consideration of minute differences in properties occurring in each production lot.

[0083] (Sample 20)

[0084] For the purpose of pathological diagnosis, etc., the sample 20 is prepared from a clinical sample or a tissue sample collected from a human body. The sample 20 may be a tissue slice, a cell, or a particle, and regarding the sample 20, there is no particular limitation on the type of tissue used (e.g., an organ, etc.), the type of target disease, the attributes of the target person (e.g., age, sex, blood type, etc.), or the lifestyle of the target person (e.g., eating habits, exercise habits, smoking habits, etc.). Note that the tissue slice may include, for example, a slice before staining of the tissue slice to be stained (hereinafter also referred to as a slice), a slice adjacent to the stained slice, a slice different from the stained slice in the same block (sampled from the same position as the stained slice), a slice in a different block in the same tissue (sampled from a different position from the stained slice), a slice collected from a different patient, etc. In addition, the sample 20 is managed by attaching identification information (hereinafter referred to as "sample identification information 21") that can identify each sample 20 to the sample 20. The sample identification information 21 is, for example, barcode information, etc. (one-dimensional barcode information, two-dimensional barcode information, etc.), similar to the reagent identification information 11, but not limited thereto. The properties of the sample 20 differ depending on the type of tissue used, the type of target disease, the properties of the target person, the lifestyle of the target person, etc. For example, in the sample 20, the measurement channel, the spectrum, etc. differ depending on the type of tissue used, etc. Therefore, in the information processing system according to the present embodiment, the sample 20 is managed individually by attaching the sample identification information 21 to the sample 20. Therefore, the information processing device 100 can perform fluorescence separation in consideration of slight differences in the properties occurring in each sample 20.

[0085] (Fluorescent staining sample 30)

[0086] The fluorescently stained sample 30 is produced by staining the sample 20 with the fluorescent reagent 10. In the present embodiment, it is assumed that the fluorescently stained sample 30 is produced by staining the sample 20 with one or more fluorescent reagents 10, but the number of fluorescent reagents 10 used for staining is not particularly limited. In addition, the staining method is determined by the combination of the sample 20 and the fluorescent reagent 10, etc., and is not particularly limited.

[0087] (Information Processing Device 100)

[0088] like Figure 1 As shown, the information processing device 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160. The information processing device 100 may be, for example, a fluorescence microscope, etc., but is not necessarily limited thereto, and may include various devices. For example, the information processing device 100 may be a personal computer (PC), etc.

[0089] (Acquisition unit 110)

[0090] The acquisition unit 110 is a configuration that acquires information used for various processes of the information processing device 100. Figure 1 As shown, the acquisition unit 110 includes an information acquisition unit 111 and a fluorescence signal acquisition unit 112 .

[0091] (Information Acquisition Unit 111)

[0092] The information acquisition unit 111 is a configuration that acquires information about the fluorescent reagent 10 (hereinafter referred to as "reagent information") or information about the sample 20 (hereinafter referred to as "sample information"). More specifically, the information acquisition unit 111 acquires the reagent identification information 11 attached to the fluorescent reagent 10 used to generate the fluorescent staining sample 30 and the sample identification information 21 attached to the sample 20. For example, the information acquisition unit 111 acquires the reagent identification information 11 and the sample identification information 21 using a barcode reader or the like. Then, the information acquisition unit 111 acquires the reagent information based on the reagent identification information 11 and the sample information based on the sample identification information 21 from the database 200, respectively. The information acquisition unit 111 stores the acquired information in the information storage unit 121 as described below.

[0093] Here, in the present embodiment, it is assumed that the sample information includes linked autofluorescence reference spectra in which the spectra of the autofluorescent substances in the sample 20 are linked to each other in the wavelength direction, and the reagent information includes linked fluorescence reference spectra in which the spectra of the fluorescent substances in the fluorescent-stained sample 30 are linked to each other in the wavelength direction. Note that the linked autofluorescence reference spectra and the linked fluorescence reference spectra are collectively referred to as “reference spectra”.

[0094] (Fluorescence signal acquisition unit 112)

[0095] The fluorescence signal acquisition unit 112 is a configuration for acquiring a plurality of fluorescence signals when a fluorescent staining sample 30 (a sample produced by staining a sample 20 with a fluorescent reagent 10) is irradiated with a plurality of excitation lights, each fluorescence signal corresponding to a plurality of excitation lights having different wavelengths. More specifically, the fluorescence signal acquisition unit 112 receives light and outputs a detection signal according to the amount of light received to acquire the fluorescence spectrum of the fluorescent staining sample 30 based on the detection signal. Here, the content of the excitation light (including the excitation wavelength, intensity, etc.) is determined based on reagent information, etc. (in other words, information about the fluorescent reagent 10, etc.). Note that the fluorescence signal mentioned here is not particularly limited as long as it is a signal derived from fluorescence, and may be, for example, a fluorescence spectrum.

[0096] Figure 2 A to D of FIG. 1 are specific examples of the fluorescence spectrum acquired by the fluorescence signal acquisition unit 112. Figure 2 In A to D of FIG. 1 , it is shown that the fluorescent staining sample 30 includes four types of fluorescent substances (eg, DAPI, CK / AF488, PgR / AF594, and ER / AF647) and is respectively 392 [nm] ( Figure 2 A), 470[nm]( Figure 2 B), 549[nm]( Figure 2 C) and 628[nm]( Figure 2 D) is a specific example of a fluorescence spectrum obtained when the excitation light of the excitation wavelength is irradiated. It should be noted that the energy released for fluorescence emission causes the fluorescence wavelength to shift to the wavelength side longer than the excitation wavelength (Stokes shift). In addition, the fluorescent substance included in the fluorescent staining sample 30 and the excitation wavelength of the irradiated excitation light are not limited to those described above. The fluorescence signal acquisition unit 112 stores the acquired fluorescence spectrum in the fluorescence signal storage unit 122, as described later.

[0097] (Storage unit 120)

[0098] The storage unit 120 is a configuration that stores information used for various processes of the information processing device 100 or information output through various processes. Figure 1As shown, the storage unit 120 includes an information storage unit 121 and a fluorescent signal storage unit 122 .

[0099] (Information storage unit 121)

[0100] The information storage unit 121 is a configuration that stores reagent information and sample information acquired by the information acquisition unit 111 .

[0101] (Fluorescent signal storage unit 122)

[0102] The fluorescence signal storage unit 122 is a configuration that stores the fluorescence signal of the fluorescence-stained sample 30 acquired by the fluorescence signal acquisition unit 112 .

[0103] (Processing unit 130)

[0104] The processing unit 130 is a configuration that performs various processes including fluorescence separation processing. Figure 1 As shown, the processing unit 130 includes a linking unit 131 , a separation processing unit 132 , and an image generating unit 133 .

[0105] (Link unit 131).

[0106] The linking unit 131 is a configuration that generates linked fluorescence spectra by linking at least a portion of the plurality of fluorescence spectra acquired by the fluorescence signal acquisition unit 112 to each other in the wavelength direction. For example, the linking unit 131 extracts data of a predetermined width in each fluorescence spectrum so as to include the four fluorescence spectra ( Figure 3 The width of the wavelength band in which the link unit 131 extracts data may be determined based on reagent information, excitation wavelength, fluorescence wavelength, etc., and may be different for each fluorescent substance (in other words, for Figure 3 For each fluorescence spectrum shown in A to D of FIG. 1 , the width of the wavelength band in which the link unit 131 extracts data may be different from each other). Then, as Figure 3 As shown in E, the linking unit 131 generates a linked fluorescence spectrum by linking the extracted data to each other in the wavelength direction. It should be noted that since the linked fluorescence spectrum includes data extracted from a plurality of fluorescence spectra, the wavelength is discontinuous at the boundary of each linked data.

[0107] At this time, the linking unit 131 performs the above-mentioned linking after aligning the intensities of the excitation light corresponding to each of the multiple fluorescence spectra with each other (in other words, after correcting the multiple fluorescence spectra) based on the intensity of the excitation light. More specifically, the linking unit 131 performs the above-mentioned linking after aligning the intensities of the excitation light corresponding to each of the multiple fluorescence spectra with each other by dividing each fluorescence spectrum by the excitation power density (which is the intensity of the excitation light). Therefore, the fluorescence spectrum in the case of irradiating the excitation light of the same intensity is obtained. In addition, in the case where the intensities of the irradiated excitation light are different from each other, the intensities of the spectra absorbed by the fluorescent-stained sample 30 (hereinafter referred to as "absorption spectra") are also different from each other according to the intensity of the irradiated excitation light. Therefore, as described above, the absorption spectrum can be appropriately evaluated by aligning the intensities of the excitation light corresponding to each of the multiple fluorescence spectra with each other.

[0108] As described above, the intensity of the excitation light in this specification may be the excitation power or the excitation power density. The excitation power or the excitation power density may be the power or the power density obtained by actually measuring the excitation light emitted from the light source 104, or may be the power or the power density obtained from the driving voltage applied to the light source 104. Note that the intensity of the excitation light in this specification may be a value obtained by correcting the above-mentioned excitation power density using the absorbance of each excitation light of the slice as the observation target, the amplification factor of the detection signal in the detection system (fluorescence signal acquisition unit 112, etc.) that detects the fluorescence emitted from the slice, and the like. That is, the intensity of the excitation light in this specification may be the power density of the excitation light that actually contributes to the excitation of the fluorescent substance, the value obtained by correcting the power density with the amplification factor of the detection system, and the like. By taking the absorbance, the amplification factor, and the like into consideration, the intensity of the excitation light that changes according to changes in the machine state, the environment, and the like can be appropriately corrected, and thus a linked fluorescence spectrum that can achieve more accurate color separation can be generated.

[0109] Note that the correction value based on the intensity of the excitation light of each fluorescence spectrum (also referred to as the intensity correction value) is not limited to the value used to align the intensities of the excitation light corresponding to each of the multiple fluorescence spectra with each other, and various modifications can be made. For example, the signal intensity of the fluorescence spectrum having an intensity peak on the long wavelength side is often lower than the signal intensity of the fluorescence spectrum having an intensity peak on the short wavelength side. Therefore, in the case where the linked fluorescence spectrum includes a fluorescence spectrum having an intensity peak on the long wavelength side and a fluorescence spectrum having an intensity peak on the short wavelength side, there is a case where the fluorescence spectrum having an intensity peak on the long wavelength side is hardly added and only the fluorescence spectrum having an intensity peak on the short wavelength side is extracted. In this case, for example, by setting the intensity correction value of the fluorescence spectrum having an intensity peak on the long wavelength side to a larger value, the separation accuracy of the fluorescence spectrum on the short wavelength side can be improved.

[0110] In addition, the linking unit 131 can correct the wavelength resolution of each of the multiple fluorescence spectra to be linked to each other independently of other fluorescence spectra. For example, the fluorescence spectrum of AF546 and the fluorescence spectrum of AF555 have almost the same spectral shape and peak wavelength, and the difference between the fluorescence spectrum of AF546 and the fluorescence spectrum of AF555 is that the fluorescence spectrum of AF555 has a shoulder at the bottom of the high wavelength side, while the fluorescence spectrum of AF546 does not have a shoulder. In this way, when the two fluorescence spectra are close to each other, the problem of difficulty in color separation of the two fluorescence spectra from each other by spectral extraction arises.

[0111] This problem can be solved by improving the wavelength resolution of the linked fluorescence spectra. Figure 4 is a diagram showing the fluorescence spectra of AF546 and AF555 at a wavelength resolution of 8 nm, Figure 5 is a diagram showing the fluorescence spectra of AF546 and AF555 at a wavelength resolution of 1 nm. Figure 4 As shown in FIG. 1 , when the wavelength resolution is 8 nm, the spectral shape and peak wavelength of AF546 and the spectral shape and peak wavelength of AF555 are substantially consistent with each other. Therefore, it is actually difficult to separate these spectral shapes and peak wavelengths from each other using, for example, the least square method. On the other hand, when the wavelength resolution is Figure 4 The wavelength resolution is 8 times that of 1 nm. Figure 5 As shown, the spectral shape and peak wavelength of AF546 and the spectral shape and peak wavelength of AF555 can be clearly separated from each other. This shows that even in the case of using multiple fluorescence spectra with close spectral shapes and peak wavelengths, color separation can be performed using multiple fluorescence spectra by increasing the wavelength resolution.

[0112] However, when the wavelength resolution increases, the amount of data of the linked fluorescence spectra becomes larger, so that the required storage capacity, the calculation cost in the fluorescence separation process, etc. increase. Therefore, the linking unit 131 corrects the fluorescence spectrum that is assumed to be difficult to color separate among the multiple fluorescence spectra to be linked to each other so that the wavelength resolution of the fluorescence spectrum becomes higher, and corrects the fluorescence spectrum that is assumed to be easy to color separate among the multiple fluorescence spectra so that the wavelength resolution of the fluorescence spectrum becomes lower. Therefore, the accuracy of color separation can be improved while suppressing the increase in data amount.

[0113] Here, a method for generating linked fluorescence spectra by the linking unit 131 will be described by way of a specific example. Figure 3The described method of generating linked fluorescence spectra illustrates the case of linking four fluorescence spectra obtained by irradiating a fluorescently stained sample 30 including four types of fluorescent substances (e.g., DAPI, CK / AF488, PgR / AF594, and ER / AF647) with excitation light (each excitation light having an excitation wavelength of 392 nm, 470 nm, 549 nm, and 628 nm).

[0114] Figure 6 It is shown from Figure 3 The fluorescence spectra shown in A to D are diagrams of examples of linked fluorescence spectra generated. Figure 6 As shown, the link unit 131 is Figure 3 The fluorescence spectrum shown in A is extracted from the fluorescence spectrum SP1 in the wavelength band where the excitation wavelength is greater than 392 nm and less than 591 nm. Figure 3 The fluorescence spectrum shown in B is extracted from the fluorescence spectrum SP2 in the wavelength band where the excitation wavelength is greater than 470 nm and less than 669 nm. Figure 3 The fluorescence spectrum SP3 in the wavelength band with an excitation wavelength of 549 nm to 748 nm is extracted from the fluorescence spectrum shown in C, and Figure 3 The fluorescence spectrum SP4 in the wavelength band with an excitation wavelength of 628 nm or more and 827 nm or less is extracted from the fluorescence spectrum shown in D of FIG. Next, the linking unit 131 corrects the wavelength resolution of the extracted fluorescence spectrum SP1 to 16 nm (without intensity correction), corrects the intensity of the fluorescence spectrum SP2 to 1.2 times, and corrects the wavelength resolution of the fluorescence spectrum SP2 to 8 nm, corrects the intensity of the fluorescence spectrum SP3 to 1.5 times (without wavelength resolution correction), corrects the intensity of the fluorescence spectrum SP4 to 4.0 times, and corrects the wavelength resolution of the fluorescence spectrum SP4 to 4 nm. Then, the linking unit 131 generates the fluorescence spectrum SP1 to SP4 by sequentially linking the fluorescence spectra SP1 to SP4 to each other after correction. Figure 6 Fluorescence spectra of the links are shown.

[0115] Note that Figure 6 , when each fluorescence spectrum has been acquired, the linking unit 131 has extracted and linked the fluorescence spectrum with a predetermined bandwidth ( Figure 6In the case of fluorescence spectra SP1 to SP4 (with a width of 200 nm in the middle), the bandwidths of the fluorescence spectra extracted by the link unit 131 do not need to be consistent with each other in each fluorescence spectrum, and can be different from each other. That is, the area extracted from each fluorescence spectrum by the link unit 131 only needs to be an area including the peak wavelength of each fluorescence spectrum, and the wavelength band and bandwidth of each fluorescence spectrum can be appropriately changed. At this time, the displacement of the spectrum wavelength due to the Stokes shift can be considered. In this way, by narrowing the wavelength band to be extracted, the amount of data can be reduced, so the fluorescence separation process can be performed at a higher speed.

[0116] (Separation Processing Unit 132)

[0117] The separation processing unit 132 is a configuration that separates the linked fluorescence spectrum for each molecule. Figure 7 1 is a block diagram showing a more specific configuration example of the separation processing unit according to the present embodiment. Figure 7 As shown, the separation processing unit 132 includes a color separation unit 1321 and a spectrum extraction unit 1322.

[0118] The color separation unit 1321 includes, for example, a first color separation unit 1321 a and a second color separation unit 1321 b , and performs color separation on the linked fluorescence spectrum of the stained section (also referred to as a stained sample) input from the linking unit 131 for each molecule.

[0119] The spectrum extraction unit 1322 is a configuration that improves the linked autofluorescence reference spectrum so that a more accurate color separation result can be obtained, and adjusts the linked autofluorescence reference spectrum included in the sample information input from the information storage unit 121 based on the color separation result of the color separation unit 1321 so as to obtain a more accurate color separation result.

[0120] More specifically, the first color separation unit 1321a performs a color separation process for the linked fluorescence spectrum of the stained sample input from the linking unit 131, and separates the linked fluorescence spectrum into a spectrum for each molecule by using the linked fluorescence reference spectrum included in the reagent information and the linked autofluorescence reference spectrum included in the sample information input from the information storage unit 121. Note that, for example, a least square method (LSM), a weighted least square method (WLSM), or the like can be used for the color separation process.

[0121] The spectrum extraction unit 1322 performs spectrum extraction processing on the linked autofluorescence reference spectrum input from the information storage unit 121 by using the color separation result input from the first color separation unit 1321a, and adjusts the linked autofluorescence reference spectrum based on the result of the spectrum extraction processing to improve the linked autofluorescence reference spectrum so as to obtain a more accurate color separation result. Note that, for example, non-negative matrix factorization (NMF), singular value decomposition (Singular Value Decomposition), etc. can be used for the spectrum extraction processing.

[0122] The second color separation unit 1321b performs a color separation process on the linked fluorescence spectrum of the stained sample input from the linking unit 131, and separates the linked fluorescence spectrum into a spectrum for each molecule by using the adjusted linked autofluorescence reference spectrum input from the spectrum extraction unit 1322. Note that, for example, similar to the first color separation unit 1321a, a least square method (LSM), a weighted least square method (WLSM), or the like can be used for the color separation process.

[0123] Note that Figure 7 The case in which the adjustment of the linked autofluorescence reference spectrum has been performed once has been illustrated, but the present disclosure is not limited to this, and after the color separation result of the second color separation unit 1321b is input to the spectrum extraction unit 1322 and the process for again performing the adjustment of the linked autofluorescence reference spectrum is repeated one or more times in the spectrum extraction unit 1322, the final color separation result can be obtained.

[0124] exist Figure 8 , specific examples of linked autofluorescence reference spectra are shown in the case where the autofluorescent substances are hemoglobin, Archidonic Acid, Catalase, Collagen, FAD, NADPH, and ProLong Diamond. Fig. 9, a specific example of linked fluorescence reference spectra when the fluorescent substances are CK, ER, PgR and DAPI is shown. Linked fluorescence reference spectra and linked autofluorescence reference spectra can be generated by a method similar to (but not necessarily limited to) the method for generating linked fluorescence spectra by linking unit 131. More specifically, linked fluorescence reference spectra and linked autofluorescence reference spectra can be generated by linking data with a predetermined wavelength bandwidth in a plurality of spectra acquired by a plurality of excitation lights having the same excitation wavelength as when generating linked fluorescence spectra in the wavelength direction. At this time, (but not necessarily limited to) it is assumed that the intensity of the excitation light corresponding to each of the plurality of spectra is aligned with each other based on the intensity of the excitation light (e.g., excitation power density). Note that the method for generating linked fluorescence reference spectra and linked autofluorescence reference spectra is not necessarily limited to the above method. For example, linked fluorescence reference spectra and linked autofluorescence reference spectra can be generated based on theoretical values, catalog values, etc. of the spectrum of each substance.

[0125] Next, calculations regarding the least squares method will be described. The least squares method is to calculate the color mixing rate by fitting the linked fluorescence spectrum generated by the linking unit 131 to the reference spectrum. Note that the color mixing rate is an indicator indicating the degree to which the corresponding substances are mixed with each other. The following equation (1) is an equation representing the residual obtained by subtracting the reference spectrum (St) (linked fluorescence reference spectrum and linked autofluorescence reference spectrum) mixed at the color mixing rate a from the linked fluorescence spectrum (Signal). Note that "Signal (1×Number of Channels)" in equation (1) indicates that the linked fluorescence spectrum (Signal) exists in the number of wavelength channels (for example, Signal is a matrix representing the linked fluorescence spectrum). In addition, "St (Number of Substances×Number of Channels)" indicates that the reference spectrum exists in the number of wavelength channels for each substance (fluorescent substance and autofluorescent substance) (for example, St is a matrix representing the reference spectrum). Furthermore, “a(1×number of substances)” indicates that a color mixing ratio a is provided for each substance (fluorescent substance and autofluorescent substance) (for example, a is a matrix indicating the color mixing ratio of each reference spectrum in the linked fluorescence spectrum).

[0126] [Equation 1]

[0127] Signal(1×Number of channels)-a(1×Number of substances)*ST(Number of substances×Number of channels)(1)

[0128] Then, the first color separation unit 1321a or the second color separation unit 1321b calculates the color mixing rate a of each substance, wherein, at this color mixing rate a, the sum of squares of the residual equation (1) is minimized. Since for the equation (1) representing the residual, the sum of squares of the residual becomes minimized when the result of the partial differentiation with respect to the color mixing rate a is 0, the first color separation unit 1321a or the second color separation unit 1321b calculates the color mixing rate a of each substance by solving the following equation (2), wherein, at this color mixing rate a, the sum of squares of the residual becomes minimized. Note that "St" in equation (2) represents the transposed matrix of the reference spectrum St. In addition, "inv(St*St') represents the inverse matrix of St*St'.

[0129] [Equation 2]

[0130]

[0131] Here, a specific example of each value of the above equation (1) is represented by the following equations (3) to (5). In the examples of equations (3) to (5), a case where reference spectra (St) of three types of substances (the number of substances is 3) are mixed with each other at different color mixing rates a in the linked fluorescence spectrum (Signal) is represented.

[0132] [Equation 3]

[0133]

[0134] [Equation 4]

[0135] a=(3 2 1) (4)

[0136] [Equation 5]

[0137] Signal=a*St=(170.1 351 410 215 78) (5)

[0138] Then, a specific example of the calculation result of the above equation (2) by each value of equations (3) and (5) is represented by the following equation (6). As can be seen from equation (6), "a = (321)" (i.e., the same value as the above equation (4)) is correctly calculated as the calculation result.

[0139] [Equation 6]

[0140] a=Signal*St′*inv(St*St′)=(3 2 1) (6)

[0141] As described above, the first color separation unit 1321a or the second color separation unit 1321b can output a unique spectrum as a separation result (the separation result is not different for each excitation wavelength) by performing fluorescence separation processing using reference spectra linked in the wavelength direction (linked autofluorescence reference spectrum and linked fluorescence reference spectrum). Therefore, the physician can more easily obtain the correct spectrum. In addition, the reference spectrum for the autofluorescence used for separation (linked autofluorescence reference spectrum) is automatically acquired, and the fluorescence separation processing is performed, so that the physician does not need to extract the spectrum corresponding to the autofluorescence from the appropriate space of the non-stained slice.

[0142] Note that, as described above, the first color separation unit 1321a or the second color separation unit 1321b can extract the spectrum of each fluorescent substance from the linked fluorescence spectrum by performing a calculation on the weighted least squares method instead of the least squares method. In the weighted least squares method, the noise of the linked fluorescence spectrum (signal) as the measured value has a Poisson distribution, and a weight is assigned so as to attach importance to the error of the low signal level. However, the upper limit value that is not weighted by the weighted least squares method is set to the displacement value (Offset value). The displacement value is determined by the characteristics of the sensor used for measurement, and needs to be optimized separately when the imaging element is used as a sensor. In the case of performing the weighted least squares method, the reference spectrum St in the above equations (1) and (2) is replaced by St_ represented by the following equation (7). Note that the following equation (7) means that St_ is calculated by dividing each element (each component) of St represented by the matrix by (in other words, element division) each corresponding element (each component) in "Signal+Offset value" also represented by the matrix.

[0143] [Equation 7]

[0144]

[0145] Here, when the displacement value is 1 and the values ​​of the reference spectrum St and the linked fluorescence spectrum signal are respectively expressed by the above equations (3) and (5), a specific example of St_ expressed by the above equation (7) is expressed by the following equation (8).

[0146] [Equation 8]

[0147]

[0148] Then, a specific example of the calculation result of the color mixing ratio a in this case is represented by the following equation (9): As can be seen from equation (9), "a=(321)" is correctly calculated as the calculation result.

[0149] [Equation 9]

[0150] a=Signal*St_′*inv(St*St_′)=(3 2 1) (9)

[0151] (Image Generation Unit 133)

[0152] The image generation unit 133 is a configuration that generates image information based on the separation result of the linked fluorescence spectrum by the separation processing unit 132. For example, the image generation unit 133 can generate image information using a fluorescence spectrum corresponding to one or more fluorescent molecules, or generate image information using an autofluorescence spectrum corresponding to one or more autofluorescence molecules. Note that the number or combination of fluorescent molecules or autofluorescence molecules used by the image generation unit 133 to generate image information is not particularly limited. In addition, in the case of performing various processing (e.g., segmentation, calculation of a signal-to-noise ratio value, etc.) using the separated fluorescence spectrum or autofluorescence spectrum, the image generation unit 133 can generate image information indicating the results of those processing.

[0153] (Display unit 140)

[0154] The display unit 140 is a configuration that displays the image information generated by the image generation unit 133 on a display to present the image information to the physician. Note that the type of display used as the display unit 140 is not particularly limited. In addition, although not described in detail in the present embodiment, the image information generated by the image generation unit 133 may be projected by a projector or may be printed by a printer to present to the physician (in other words, the method of outputting the image information is not particularly limited).

[0155] (Control Unit 150)

[0156] The control unit 150 is a functional configuration that comprehensively controls the general processing performed by the information processing device 100. For example, the control unit 150 controls the start, end, etc. of various processing as described above (for example, adjustment processing of the placement position of the fluorescent staining sample 30, irradiation processing of the fluorescent staining sample 30 with excitation light, spectrum acquisition processing, linked fluorescence spectrum generation processing, fluorescence separation processing, image information generation processing, image information display processing, etc.) based on the operation input performed by the physician via the operation unit 160. Note that the control content of the control unit 150 is not particularly limited. For example, the control unit 150 can control processing that is usually performed on a general-purpose computer, PC, tablet computer, etc. (for example, processing related to the operating system (OS)).

[0157] (Operation unit 160)

[0158] The operation unit 160 is a configuration for receiving operation input from a physician. More specifically, the operation unit 160 includes various input devices, such as a keyboard, a mouse, a button, a touch panel, a microphone, etc., and the physician can perform various inputs to the information processing device 100 by operating these input devices. Information about the operation input performed via the operation unit 160 is provided to the control unit 150.

[0159] (Database 200)

[0160] The database 200 is a device for managing reagent information, sample information, etc. More specifically, the database 200 manages the reagent identification information 11 and the reagent information, and the sample identification information 21 and the sample information in association with each other. Therefore, the information acquisition unit 111 can acquire the reagent information based on the reagent identification information 11 of the fluorescent reagent 10 from the database 200, and acquire the sample information based on the sample identification information 21 of the sample 20.

[0161] The reagent information managed by the database 200 is assumed to include information of linked fluorescence reference spectrum unique to the fluorescent substance possessed by the fluorescent reagent 10 and a measurement channel (but not necessarily limited thereto). The "measurement channel" is a concept indicating the fluorescent substance included in the fluorescent reagent 10 and is a term indicating Fig. 9 . Since the number of fluorescent substances varies according to the fluorescent reagent 10, the measurement channel is managed in association with each fluorescent reagent 10 as reagent information. In addition, as described above, the linked fluorescence reference spectrum included in the reagent information is a spectrum generated by linking the fluorescence spectrum of each fluorescent substance included in the measurement channel to each other in the wavelength direction.

[0162] In addition, the sample information managed by the database 200 is assumed to include information of linked autofluorescence reference spectrum and measurement channel specific to the autofluorescence substance possessed by the sample 20 (but not necessarily limited thereto). The “measurement channel” is a concept indicating the autofluorescence substance included in the sample 20, and is a concept indicating the autofluorescence reference spectrum and the measurement channel specific to the autofluorescence substance possessed by the sample 20. Figure 8 The concepts of hemoglobin, Archidonic Acid, Catalase, Collagen, FAD, NADPH, and ProLong Diamond in the example of . Since the number of autofluorescent substances varies depending on the sample 20, the measurement channel is managed in association with each sample 20 as sample information. In addition, as described above, the linked autofluorescence reference spectrum included in the sample information is a spectrum generated by linking the autofluorescence spectra of each autofluorescent substance included in the measurement channel to each other in the wavelength direction. Note that the information managed by the database 200 is not necessarily limited to those described above.

[0163] The configuration example of the information processing system according to the present embodiment has been described above. Figure 1 The configuration described is merely an example, and the configuration of the information processing system according to the present embodiment is not limited to such an example. For example, the information processing device 100 may not necessarily include Figure 1 All configurations shown, or may include Figure 1 Configuration not shown.

[0164] Here, the information processing system according to the present embodiment may include an imaging device (for example, including a scanner, etc.) that acquires a fluorescence spectrum and an information processing device that performs processing using the fluorescence spectrum. In this case, Figure 1 The fluorescence signal acquisition unit 112 shown may be implemented by an imaging device, and other configurations may be implemented by an information processing device. In addition, the information processing system according to the present embodiment may include an imaging device for acquiring a fluorescence spectrum and software for processing using the fluorescence spectrum. In other words, the information processing system may not have a physical configuration (e.g., a memory, a processor, etc.) for storing or executing software. In this case, Figure 1 The fluorescent signal acquisition unit 112 shown can be implemented by an imaging device, and other configurations can be implemented by an information processing device on which software is executed. Then, the software can be provided to the information processing device via a network (for example, from a website, a cloud server, etc.), or the software can be provided to the information processing device via any storage medium (for example, a compact disc, etc.). In addition, the information processing device on which the software is executed can be various servers (for example, a cloud server, etc.), general-purpose computers, PCs, tablet computers, etc. Note that the method of providing software to the information processing device and the type of information processing device are not limited to those described above. In addition, it should be noted that the configuration of the information processing system according to the present embodiment is not necessarily limited to the above configuration, and the so-called configuration that can be thought of by a person skilled in the art can be applied based on the technical level at the time of use.

[0165] The above-mentioned information processing system can be implemented as a microscope system, for example. Fig.10 A configuration example of a microscope system in a case where the information processing system according to the present embodiment is implemented as a microscope system is described.

[0166] like Fig.10 As shown, the microscope system according to this embodiment includes a microscope 101 and a data processing unit 107 .

[0167] The microscope 101 includes a stage 102 , an optical system 103 , a light source 104 , a stage driving unit 105 , a light source driving unit 106 , and a fluorescent signal acquiring unit 112 .

[0168] The stage 102 has a placement surface on which the fluorescent dye sample 30 can be placed, and can be moved in a direction parallel to the placement surface (xy plane direction) and a direction perpendicular to the placement surface (z axis direction) by driving the stage driving unit 105. The fluorescent dye sample 30 has a thickness of, for example, several micrometers to several tens of micrometers in the Z direction, and is sandwiched between the slide glass SG and a cover glass (not shown), and is fixed by a predetermined fixing method.

[0169] The optical system 103 is disposed above the stage 102. The optical system 103 includes an objective lens 103A, an image forming lens 103B, a dichroic mirror 103C, an emission filter 103D, and an excitation filter 103E. The light source 104 is, for example, a bulb (e.g., a mercury lamp, etc.), a light emitting diode (LED), etc., and irradiates the fluorescent label attached to the fluorescent dye sample 30 with excitation light by the drive of the light source drive unit 106.

[0170] In the case of obtaining a fluorescent image of the fluorescent dye sample 30, the excitation filter 103E generates excitation light by transmitting only light of an excitation wavelength that excites the fluorescent pigment in the light emitted from the light source 104. The dichroic mirror 103C reflects the excitation light that has passed through the excitation filter and then is incident on the dichroic mirror 103C, and guides the excitation light to the objective lens 103A. The objective lens 103A focuses the excitation light on the fluorescent dye sample 30. Then, the objective lens 103A and the image forming lens 103B magnify the image of the fluorescent dye sample 30 to a predetermined magnification, and form the magnified image on the image forming surface of the fluorescent signal acquisition unit 112.

[0171] When the fluorescent dye sample 30 is irradiated with excitation light, the dye of each tissue bound to the fluorescent dye sample 30 emits fluorescence. The fluorescence is transmitted through the dichroic mirror 103C via the objective lens 103A, and reaches the image forming lens 103B via the emission filter 103D. The emission filter 103D absorbs the light that has been magnified by the above-mentioned objective lens 103A and has passed through the excitation filter 103E, and transmits only a part of the color light. As described above, the image of the color light that loses the external light is magnified by the image forming lens 103B and formed on the fluorescence signal acquisition unit 112.

[0172] The data processing unit 107 is a configuration that drives the light source 104, acquires a fluorescent image of the fluorescently stained sample 30 using the fluorescent signal acquisition unit 112, and performs various processes using the fluorescent image. More specifically, the data processing unit 107 can be used as a reference. Figure 1The information processing device 100 described herein may include some or all of the information acquisition unit 111, the storage unit 120, the processing unit 130, the display unit 140, the control unit 150, and the operation unit 160 or the database 200. For example, the data processing unit 107 is used as the control unit 150 of the information processing device 100 to control the driving of the stage driving unit 105 and the light source driving unit 106, or to control the spectrum acquisition of the fluorescence signal acquisition unit 112. In addition, the data processing unit 107 is used as the processing unit 130 of the information processing device 100 to generate linked fluorescence spectra, separate the linked fluorescence spectra of each molecule, or generate image information based on the separation result.

[0173] The configuration example of the microscope system in the case where the information processing system according to the present embodiment is implemented as a microscope system has been described above. Fig.10 The configuration described is merely an example, and the configuration of the microscope system according to the present embodiment is not limited to such an example. For example, the microscope system may not necessarily include Fig.10 All configurations shown, or may include Fig.10 Configuration not shown.

[0174] (1.2. Processing flow example)

[0175] The configuration example of the information processing system according to the present embodiment has been described above. Fig.11 An example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100 is described. Fig.11 : is a flowchart showing an example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100 .

[0176] In step S1000, the fluorescence signal acquisition unit 112 of the information processing device 100 acquires a fluorescence spectrum. More specifically, the fluorescent dye sample 30 is irradiated with a plurality of excitation lights having different excitation wavelengths, and the fluorescence signal acquisition unit 112 acquires a plurality of fluorescence spectra corresponding to each excitation light. Then, the fluorescence signal acquisition unit 112 stores the acquired fluorescence spectrum in the fluorescence signal storage unit 122.

[0177] In step S1004, the linking unit 131 generates a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra stored in the fluorescence signal storage unit 122 to each other in the wavelength direction. More specifically, the linking unit 131 generates one linked fluorescence spectrum by extracting data of a predetermined width in each fluorescence spectrum to include the maximum value of the fluorescence intensity of each of the plurality of fluorescence spectra and linking the data to each other in the wavelength direction.

[0178] In step S1008, the separation processing unit 132 separates the linked fluorescence spectrum for each molecule (performs fluorescence separation). More specifically, the separation processing unit 132 performs a reference Figure 7 The processing described is used to separate the linked fluorescence spectra for each molecule.

[0179] In subsequent processing, for example, the image generation unit 133 generates image information using separated fluorescence spectra corresponding to one or more fluorescent molecules (or separated autofluorescence spectra corresponding to autofluorescent molecules), and the display unit 140 displays the image information on a display to present the image information to a physician.

[0180] <2. Second Embodiment>

[0181] The first embodiment of the present disclosure has been described above. Next, the second embodiment of the present disclosure will be described.

[0182] The information processing apparatus 100 according to the first embodiment has performed fluorescence separation processing using previously prepared linked autofluorescence reference spectra (and linked fluorescence reference spectra). On the other hand, the information processing apparatus 100 according to the second embodiment performs fluorescence separation processing using actually measured linked autofluorescence reference spectra.

[0183] More specifically, the spectrum extraction unit 1322 of the separation processing unit 132 according to the second embodiment extracts a linked autofluorescence reference spectrum of each autofluorescent substance from the linked autofluorescence spectrum, the linked autofluorescence spectrum being generated by linking at least a portion of a plurality of autofluorescence spectra to each other in the wavelength direction, the plurality of autofluorescence spectra being obtained by irradiating a slice with a plurality of excitation lights having different excitation wavelengths, and the slice being the same as or similar to the sample 20. Then, the spectrum extraction unit 1322 performs fluorescence separation processing using the extracted linked autofluorescence reference spectrum and the linked fluorescence reference spectrum (the linked fluorescence reference spectrum is similar to the linked fluorescence reference spectrum of the first embodiment) as reference spectra.

[0184] Fig.12 1 is a block diagram showing a more specific configuration example of the separation processing unit according to the present embodiment. Fig.12 As shown, the separation processing unit 132 according to this embodiment has a similar structure to that of the first embodiment. Figure 7 The configuration of the separation processing unit 132 is described.

[0185] In this configuration, instead of the linked autofluorescence reference spectrum included in the sample information, the linked fluorescence spectrum (also referred to as a linked autofluorescence spectrum) of the non-stained section (also referred to as a non-stained sample) input from the linking unit 131 is input to the spectrum extraction unit 1322.

[0186] The spectrum extraction unit 1322 performs spectrum extraction processing by using the color separation result of the linked autofluorescence spectrum of the non-stained sample input from the link unit 131 input from the first color separation unit 1321a, and adjusts the linked autofluorescence reference spectrum based on the result of the spectrum extraction processing, thereby improving the linked autofluorescence reference spectrum to obtain a more accurate color separation result. For example, similar to the first embodiment, non-negative matrix factorization (NMF), singular value decomposition (SVD), etc. can be used for spectrum extraction processing. In addition, other operations can be similar to the operations of the separation processing unit 132 according to the first embodiment, so their detailed description will be omitted.

[0187] Note that any of the non-stained slice and the stained slice can be used for the same or similar slice as the sample 20 used to extract the linked autofluorescence reference spectrum. For example, in the case of using a non-stained slice, a slice before staining can be used as a stained slice, a slice adjacent to a stained slice, a slice different from a stained slice in the same block (sampled from the same position as the stained slice), a slice in a different block in the same tissue (sampled from a different position from the stained slice), etc. can be used.

[0188] Furthermore, in the case of using a stained section, by performing fluorescence separation processing by the method according to the third embodiment described later, it is also possible to obtain the color separation result for each molecule directly from the linked fluorescence spectrum without extracting a linked autofluorescence reference spectrum.

[0189] Here, principal component analysis (hereinafter referred to as "PCA") can generally be used as a method for extracting autofluorescence spectra from non-stained sections, but PCA is not suitable for the case where autofluorescence spectra linked to each other in the wavelength direction are used for processing as in the present embodiment. Therefore, the spectrum extraction unit 1322 according to the present embodiment extracts linked autofluorescence reference spectra from non-stained sections by performing non-negative matrix factorization (hereinafter referred to as NMF) instead of PCA. Note that the reason why PCA is not suitable as a method for extracting linked autofluorescence reference spectra from non-stained sections will be described in detail later.

[0190] Fig.13 is a diagram for describing the overview of NMF. Fig.13As shown, NMF decomposes a non - negative N - row M - column (N×M) matrix A into a non - negative N - row k - column (N×k) matrix W and a non - negative k - row M - column (k×M) matrix H. The matrices W and H are determined such that the mean - square residual D between the matrix A and the product of the matrices W and H (W*H) is minimized. In this embodiment, the matrix A corresponds to the spectrum before extracting the autofluorescence reference spectrum of the link (N is the number of pixels, M is the number of wavelength channels), and the matrix H corresponds to the extracted autofluorescence reference spectrum of the link (k is the number of autofluorescence reference spectra of the link (in other words, the number of autofluorescent substances), M is the number of wavelength channels). Here, the mean - square residual D is represented by the following equation (10). Note that "norm(D,'fro')" refers to the Frobenius norm of the mean - square residual D.

[0191] [Equation 10]

[0192]

[0193] The factorization of NMF uses an iterative method starting from random initial values of the matrices W and H. In NMF, the value of k (the number of autofluorescence reference spectra of the link) is mandatory, but the initial values of the matrices W and H can be set as options rather than being mandatory, and when the initial values of the matrices W and H are set, the solution is constant. On the other hand, in the case where the initial values of the matrices W and H are not set, these initial values are set randomly, and the solution is not constant.

[0194] Depending on the type of tissue used, the type of target disease, the attributes of the target person, the lifestyle of the target person, etc., the nature of the sample 20 is different, and the autofluorescence spectrum of the sample 20 is also different. Therefore, as described above, the information processing device 100 according to the second embodiment can achieve more accurate fluorescence separation processing by actually measuring the autofluorescence reference spectrum of each sample 20.

[0195] Note that, as described above, the matrix A as the input of NMF is a matrix having the same number of rows as the number of pixels N (=Hpix×Vpix) of the sample image and the same number of columns as the number of wavelength channels M. Therefore, in the case where the number of pixels of the sample image is large or in the case where the number of wavelength channels M is large, the matrix A becomes a very large matrix, increasing the computational cost of NMF and lengthening the processing time.

[0196] In this case, for example, as Fig.14 shown, by clustering the number of pixels N (=Hpix×Vpix) of the sample image into a specified number N (<Hpix×Vpix) of categories, the redundancy of the processing time due to the enlargement of the matrix A can be suppressed.

[0197] In clustering, for example, similar spectra in the wavelength direction and the intensity direction in the sample image are classified into the same category. Therefore, an image having a smaller number of pixels than the sample image is generated, and thus the scale of the matrix A' can be reduced using the image as input.

[0198] (2.1. Processing flow example)

[0199] Next, we will refer to Fig.15 An example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100 according to the second embodiment is described. Fig.15 : is a flowchart showing an example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100 according to the second embodiment.

[0200] In step S1100 and step S1104, a flow example similar to that of the processing in the first embodiment ( Fig.11 The fluorescence signal acquisition unit 112 acquires a plurality of fluorescence spectra corresponding to excitation light having different excitation wavelengths, and the linking unit 131 generates a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra to each other in a wavelength direction.

[0201] In step S1108, the spectral extraction unit 1322 extracts a linked autofluorescence reference spectrum by performing NMF using the linked autofluorescence spectra, wherein the linked autofluorescence spectrum is generated by linking at least a portion of a plurality of autofluorescence spectra to each other in a wavelength direction, and the plurality of autofluorescence spectra are obtained by irradiating a non-stained slice with a plurality of excitation lights having different excitation wavelengths.

[0202] In step S1112 , the color separation unit 1321 performs fluorescence separation processing using the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum (the linked fluorescence reference spectrum is similar to the fluorescence reference spectrum of the first embodiment) extracted as described above as reference spectrums.

[0203] In subsequent processing, similar to the first embodiment, for example, the image generation unit 133 generates image information using separated fluorescence spectra corresponding to one or more fluorescent molecules (or separated autofluorescence spectra corresponding to autofluorescent molecules), and the display unit 140 displays the image information on a display to present the image information to a physician.

[0204] (2.2. Reasons why PCA is not suitable as a method for extracting linked autofluorescence reference spectra from non-stained sections)

[0205] An example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100 according to the second embodiment has been described above. Next, details of the reason why PCA is not suitable as a method of extracting linked autofluorescence reference spectra from non-stained sections will be described.

[0206] First, when the fluorescence spectrum of a certain pixel i among the number of pixels n is ai for one excitation wavelength and the resolution is m, the fluorescence spectrum ai is expressed by the following equation (11) (m-order vector).

[0207] [Equation 11]

[0208] ai=(ai1,ai2,~aim) (11)

[0209] Similarly, the fluorescence spectra of other excitation wavelengths are also expressed as bi, ci, and di as m-order vectors (here, the case where the number of types of excitation wavelengths is assumed to be 4 is taken as an example). Then, the matrix in which these vectors are integrated with each other for all pixels (pixel 1 to pixel n) is expressed by the following equation (12) (n-row and 4m-column matrix P). Since the number of pixels is significantly (substantially) larger than the wavelength resolution, the rank of the matrix P expressed by equation (12) is at most 4m, and there are up to 4m eigenvalues ​​and eigenvectors.

[0210] [Equation 12]

[0211]

[0212] Here, a singular value decomposition (SVD) represented by the following equation (13) may be performed on a real matrix A of n rows and m columns (n×m) of rank k. U and V in equation (13) both represent singular matrices and form a normal matrix system (i.e., t U=U- 1 and UU- 1 =1). In addition, in the case where the real matrix A is a square matrix with different eigenvalues, U and V are eigenvectors.

[0213] [Equation 13]

[0214] A=UD t V (13)

[0215] The independent factors of a real matrix A can be analyzed by performing eigenvalue decomposition (ED) or singular value decomposition (SVD) on the real matrix A to calculate the singular (eigen) vectors. In the case where the real matrix A is a square matrix and has different eigenvalues, t The eigenvalue of AA is the square of the eigenvalue of A, and t The eigenvector of AA is equal to the eigenvector of A (see the following equation (14), where A=VD t V).

[0216] [Equation 14]

[0217] t AA= t (VD t V(VD t V)=V t D t VVD t V=V(DD)V (14)

[0218] The spectrum obtained in this embodiment is not a square matrix, but because the spectrum can be considered to be determined by a linear combination of elements configuring autofluorescence, it is considered that the spectrum can be convolved into a square matrix by replication or linear transformation. Even in the presence of errors, t The eigenvalue of AA = 0 is also the least squares solution of A. Therefore, by obtaining t The eigenvector of AA can calculate the independent components (eigenvectors) in the spectrum. In addition, the following equations (15) and (16) are obvious because the singular value decomposition is established. However, in the case of not satisfying the rank, L40R becomes a subset of the eigenvector, making it impossible to represent all points.

[0219] [Equation 15]

[0220] Anm=LnrRrm (15)

[0221] [Equation 16]

[0222]

[0223] PCA is equivalent to obtaining the eigenvalues ​​and eigenvectors of the variance-covariance matrix of the data matrix. As represented by the following equation (17), the variance-covariance matrix is ​​the product between the matrix obtained by subtracting the mean from the data matrix and the transposed matrix. This is (the product of the data matrix and the transposed - the product of the mean of each column).

[0224] [Equation 17]

[0225]

[0226] [Equation 18]

[0227]

[0228] As expressed in the following equations (19) to (23), t The eigenvector of BB can be transformed into the eigenvector of B to construct B, and t BB and t The difference between AA is taa (a matrix of products of the mean values ​​of the columns of A). Therefore, in the singular value decomposition of A, the eigenvectors of the collocation points are obtained, while in PCA, the eigenvectors representing the degree of change of the points are calculated ( t The eigenvectors and t The eigenvectors of AA are not equal to each other).

[0229] [Equation 19]

[0230] t BBij=∑(aki-ai)akj-aj)=∑(akiakj-aiaj)= t AAij-aiaj (19)

[0231] [Equation 20]

[0232] t AAij=∑akiakj (20)

[0233] [Equation 21]

[0234] a=(a1,a2,~am) (21)

[0235] [Equation 22]

[0236] aiaj= t aa (22)

[0237] [Equation 23]

[0238] t BB= t AA- t aa (23)

[0239] At this time, in the case of a matrix such as the above equation (12), in which the fluorescence spectra of each excitation wavelength are integrated with each other, in the singular value decomposition, if they are independent of each other, they are not affected, but in PCA, a product term of the mean value appears, so the eigenvector is affected. Therefore, in order to perform PCA, it is necessary to analyze each data set. As described above, in the case of processing using spectra linked to each other in the wavelength direction as in the present embodiment, PCA is not appropriate.

[0240] (2.3. Application examples)

[0241] The details of the reason why PCA is not suitable as a method for extracting a linked autofluorescence reference spectrum from a non-stained section have been described above. Next, an application example according to the second embodiment will be described.

[0242] As described above, the spectrum extraction unit 1322 of the separation processing unit 132 according to the second embodiment extracts a linked autofluorescence reference spectrum by performing NMF using linked autofluorescence spectra, wherein the linked autofluorescence spectrum is generated by linking at least a portion of a plurality of autofluorescence spectra to each other in the wavelength direction, and a plurality of autofluorescence spectra are acquired by irradiating a non-stained section with a plurality of excitation lights having different excitation wavelengths. At this time, the spectrum extraction unit 1322 according to the application example can set the initial value ( Fig.13 A linked autofluorescence reference spectrum is extracted by setting the linked autofluorescence spectrum generated by linking at least part of the autofluorescence spectra to each other in the wavelength direction as the initial value in NMF). Therefore, the spectrum obtained as a separation result is uniquely determined, and more accurate fluorescence separation can be performed.

[0243] <3. Modifications>

[0244] The second embodiment of the present disclosure has been described above. Next, a modification example of the present disclosure will be described.

[0245] Since the information obtained by the above-mentioned fluorescence separation process is the brightness (or fluorescence intensity) in the image information, there are cases where the physician cannot fully perform quantitative analysis. More specifically, since the physician cannot obtain information such as the number of fluorescent molecules, the number of antibodies bound to fluorescent molecules, etc., it is difficult for the physician to compare the number of fluorescent molecules in a plurality of fluorescent substances, or to compare data imaged under different conditions from each other.

[0246] In view of the above-mentioned problems, this modification is made, and the spectrum extraction unit 1322 according to this modification extracts the spectrum of each fluorescent substance from the linked fluorescence spectrum using a reference spectrum, wherein the reference spectrum includes a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules. More specifically, the spectrum extraction unit 1322 according to this modification calculates the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum of each fluorescent molecule or each antibody by dividing each linked autofluorescence reference spectrum and the linked fluorescence reference spectrum used in the above-mentioned embodiment by the number of fluorescent molecules or the number of antibodies in the imaging element 1 [pixel], and uses the calculated linked autofluorescence reference spectrum and the linked fluorescence reference spectrum to perform calculations regarding the least squares method (or weighted least squares method), thereby extracting the spectrum of each fluorescent substance from the linked fluorescence spectrum. Therefore, the separation processing unit 132 according to this modification can calculate the number of fluorescent molecules or the number of antibodies in the fluorescent staining sample 30 as a result of the fluorescence separation processing.

[0247] Here, reference will be made to Fig.16 Describe the method for calculating the number of fluorescent molecules (or the number of antibodies) in imaging element 1 [pixel]. Fig.16 As shown in the figure, in the case where the imaging element and the sample are arranged with the objective lens interposed therebetween, it is assumed that the size of the bottom surface of the sample corresponding to 1 [pixel] of the imaging element is 13 / 20 [μm] × 13 / 20 [μm]. Then, when it is assumed that the thickness of the sample is 10 [μm], the volume of this rectangular parallelepiped [m 3 ] is expressed as 13 / 20[μm]×13 / 20[μm]×10[μm] (Note that volume [L] is expressed as 13 / 20[μm]×13 / 20[μm]×10[μm]×10 3 express).

[0248] Then, when it is assumed that the concentration of the antibody included in the sample (which may be the number of fluorescent molecules, of course) is uniform and is 300 [nM], the number of antibodies in the imaging element 1 [pixel] is expressed by the following equation (24).

[0249] [Equation 24]

[0250]

[0251] As described above, as a result of the fluorescence separation process, the number of fluorescent molecules or the number of antibodies in the fluorescent staining sample 30 is calculated, so that the physician can compare the number of fluorescent molecules in a plurality of fluorescent substances, or compare the data imaged under different conditions from each other. In addition, although the brightness (or fluorescence intensity) is a continuous value, the number of fluorescent molecules or the number of antibodies is a discrete value. Therefore, the information processing device 100 according to this modification can reduce the amount of data by outputting image information based on the number of fluorescent molecules or the number of antibodies.

[0252] Other configurations, operations, and effects may be similar to those in the above-described embodiments, and thus detailed descriptions thereof will be omitted here.

[0253] <4. Third embodiment>

[0254] In the first and second embodiments described above, the case where the spectrum of each fluorescent substance is extracted from the linked fluorescence spectrum by performing fluorescence separation processing using the linked autofluorescence reference spectrum (and the linked fluorescence reference spectrum) has been exemplified. On the other hand, in the third embodiment, the case where the fluorescence spectrum of each fluorescent substance is extracted directly from the stained section will be exemplified.

[0255] Fig.17 1 is a block diagram showing a schematic configuration example of a separation processing unit according to the present embodiment. In the information processing device 100 according to the present embodiment, the separation processing unit 132 is Fig.17 The separation processing unit 232 is shown instead.

[0256] like Fig.17 As shown, the separation processing unit 232 includes a color separation unit 2321 , a spectrum extraction unit 2322 and a data set creation unit 2323 .

[0257] The color separation unit 2321 performs color separation on the linked fluorescence spectra of the stained slice (also referred to as a stained sample) input from the linking unit 131 for each molecule.

[0258] The spectrum extraction unit 2322 is a configuration that improves the autofluorescence spectrum so that a more accurate color separation result can be obtained, and adjusts the linked autofluorescence reference spectrum included in the sample information input from the information storage unit 121 so as to obtain a more accurate color separation result.

[0259] The data set creation unit 2323 creates a data set of autofluorescence reference spectra based on the spectrum extraction result input from the spectrum extraction unit 2322 .

[0260] More specifically, the spectrum extraction unit 2322 performs spectrum extraction processing on the linked autofluorescence reference spectrum input from the information storage unit 121 using non-negative matrix factorization (NMF), singular value decomposition (SVD), etc., and inputs the result of the spectrum extraction processing to the data set creation unit 2323. Note that in the spectrum extraction processing according to the present embodiment, the autofluorescence reference spectrum of each cell tissue and / or each type is extracted using, for example, a tissue microarray (TMA).

[0261] The dataset creation unit 2323 creates a dataset (hereinafter also referred to as an autofluorescence dataset) required for the color separation processing by the color separation unit 2321 based on the autofluorescence reference spectrum of each cell tissue and / or each type input from the spectral extraction unit 2322, and inputs the created autofluorescence dataset into the color separation unit 2321.

[0262] The color separation unit 2321 performs a color separation process by using the linked fluorescence reference spectrum and the linked autofluorescence reference spectrum input from the information storage unit 121 and the autofluorescence data set for the linked fluorescence spectrum of the stained sample input from the linking unit 131 input from the data set creation unit 2323, separating the linked fluorescence spectrum into a spectrum for each molecule. Note that NMF or SVD can be used to perform the color separation process.

[0263] As the NMF performed by the color separation unit 2321 according to the present embodiment, for example, the NMF (see FIG. 2 ) obtained when extracting the autofluorescence spectrum from the non-stained section as described in the second embodiment can be used. Fig.13etc.) with the following changed NMF.

[0264] That is, in this embodiment, the matrix A corresponds to a plurality of sample images acquired from the stained sections (N is the number of pixels, M is the number of wavelength channels), the matrix H corresponds to the fluorescence spectrum of each extracted fluorescent substance (k is the number of fluorescence spectra (in other words, the number of fluorescent substances) and M is the number of wavelength channels), and the matrix W corresponds to the image of each fluorescent substance after fluorescence separation. Note that the matrix D is the mean square residual.

[0265] In addition, in this embodiment, similar to the second embodiment, the initial value of NMF can be random. However, in the case where the result of each execution of NMF is different, it is necessary to set the initial value to prevent the result from changing.

[0266] Figures 18 to 22 is a diagram showing an example of a sample image input to the matrix A in the present embodiment, and Figures 23 to 29 It is shown in the input Figures 18 to 22 FIG. 1 is a diagram showing an example of a fluorescence separation image obtained by NMF as the matrix W in the case of the sample image shown in FIG. Figures 18 to 22 In each of the above, in order to simplify the description, the case where the sample 20 has been stained with a single fluorescent reagent 10 is shown. In addition, it is assumed that the fluorescence spectra of a total of eight fluorescent pigments, namely Archidonic Acid, Catalase, Collagen, FAD, Hemoglobin, NADPH, ProLong Diamond and CK, are given as the initial value of NMF.

[0267] When used as Figures 18 to 22 When the sample images obtained at each of the five excitation wavelengths (the number of wavelength channels (M) = 5) are used as the matrix A to solve the NMF, the following is obtained: Figures 23 to 29 The seven fluorescence separation images shown are taken as matrix W, and the individual fluorescence spectra are acquired as matrix H.

[0268] Note that in the case where fluorescence separation processing has been performed using an algorithm that changes the order of corresponding spectra according to a calculation algorithm or an algorithm that needs to change the order of spectra to speed up processing or improve the convergence of the results (e.g., NMF), it is possible to specify which fluorescent pigment each fluorescence spectrum obtained as the matrix H corresponds to by, for example, obtaining the Pearson product-moment correlation coefficient (or cosine similarity) for each of all combinations.

[0269] In addition, when the default function (NMF) of MATLAB (registered trademark) has been used, even if the initial value is given, the order is changed and the output is performed. This can be fixed by the self function, but even if the order is changed due to the use of the default function, the correct combination of the substance and the fluorescence spectrum can be obtained by using the Pearson product-moment correlation coefficient (or cosine similarity) as described above.

[0270] As described above, by using a configuration in which NMF is solved using a sample image acquired from a stained section as the matrix A, the fluorescence spectrum of each fluorescent substance can be directly extracted from the stained section without requiring processes such as imaging of a non-stained section, generation of a linked autofluorescence reference spectrum, etc. Therefore, the time and work cost required for fluorescence separation processing can be significantly reduced.

[0271] Furthermore, in the present embodiment, the fluorescence spectrum of each fluorescent substance is extracted from the sample image obtained from the same stained section, and therefore a more accurate fluorescence separation result can be obtained compared to, for example, the case of using an autofluorescence spectrum obtained from a non-stained section different from the stained section.

[0272] Other configurations, operations, and effects may be similar to those in the above-described embodiments, and thus detailed descriptions thereof will be omitted here.

[0273] Note that in the present embodiment, when extracting the fluorescence spectrum of each fluorescent substance, linked fluorescence spectrum may be used or may not be linked. That is, in the present embodiment, the linking unit 131 may generate or may not generate linked fluorescence spectrum. In the case where the linking unit 131 does not generate linked fluorescence spectrum, the extraction unit of the separation processing unit 132 performs processing for extracting the fluorescence spectrum of each fluorescent substance from the plurality of fluorescence spectra acquired by the fluorescence signal acquisition unit 112.

[0274] <5. Fourth embodiment>

[0275] In the above-mentioned third embodiment, the following method can be mentioned as a method of enhancing quantitative properties (for example, regarding the concentration of the coloring pigment, etc.).

[0276] Fig.30 is a flowchart for describing the NMF process according to the fourth embodiment. Fig.31 Is used to describe Fig.30 Schematic diagram of the process flow in the first cycle of NMF shown.

[0277] like Fig.30 As shown, in the NMF according to the present embodiment, first, the variable i is reset to zero (step S401). The variable i represents the number of repetitions of factorization in the NMF. Therefore, Fig.31The matrix H0 shown in (a) corresponds to the initial value of the matrix H. Note that in this example, for clarity, the position of the dye fluorescence spectrum in the matrix H is the bottom row, but is not limited thereto and may be variously changed to the top row, the middle row, etc.

[0278] Next, in the NMF according to the present embodiment, similar to the normal NMF, the non-negative N-row, M-column (N×M) matrix A is divided by the non-negative N-row, k-column (N×k) matrix W i , obtain a non-negative k-row M-column (k×M) matrix H i+1 (Step S402). Therefore, for example, in the first cycle, the following is obtained: Fig.31 The matrix H1 shown in (b).

[0279] Next, the matrix H obtained in step S402 is i+1 A row of fluorescent staining spectra in matrix H0 is replaced with the initial value of the fluorescent staining spectrum, that is, a row of dyed fluorescent spectra in matrix H0 (step S403). That is, in this embodiment, the fluorescent staining spectra in matrix H0 are fixed to the initial value. For example, in the first cycle, the dyed fluorescent spectra can be fixed by replacing the bottom row in matrix H1 with the bottom row in matrix H0, such as Fig.31 as shown in (c).

[0280] Next, in the NMF according to the present embodiment, by dividing the matrix A by the matrix H obtained in step S403 i+1 To obtain the matrix W i+1 (Step S404).

[0281] Thereafter, in the NMF according to the present embodiment, similarly to the normal NMF, it is determined whether the mean square residual D satisfies a predetermined branching condition (step S405), and in the case where the predetermined branching condition is satisfied (yes in step S405), the NMF is performed with the finally obtained matrix H i+1 and W i+1 On the other hand, in the case where the predetermined branch condition is not satisfied (No in step S405), the variable i is incremented by 1 (step S406), and then the process returns to step S402, and the next loop is executed.

[0282] Fig.32 is a diagram showing an example of the initial value of the dye fluorescence spectrum. Fig.33 is a diagram showing an example of a dye fluorescence spectrum after performing NMF according to the present embodiment. Fig.32 and 33 As shown, it can be seen that even in the case of performing NMF according to the present embodiment, the fluorescence spectrum of the dye is maintained as a spectrum equivalent to the initial value.

[0283] also, Fig.34 is a diagram showing an example of a spectrum of a fluorescent substance extracted by a method not using a non-stained sample according to the present embodiment, Fig.35 is a diagram showing an example of a spectrum of a fluorescent substance extracted in the case of using a non-stained sample. Fig.34 and 35 In the example, CD8 has been used as a labeling antibody and Alexa Fluor 680 has been used as a fluorescent dye. Fig.34 and 35 As shown, according to this embodiment, the spectrum of the fluorescent substance can be extracted with the same accuracy as in the case of using a non-stained sample.

[0284] As described above, in the first method, in the spectrum extraction and color separation of a multi-stained pathological section image (sample image), the stained sample can be directly color-separated using NMF while ensuring the quantitative properties of the staining fluorescence, that is, while maintaining the spectrum of the staining fluorescence, without imaging the same tissue section non-stained sample for autofluorescence spectrum extraction. Therefore, for example, accurate color separation can be achieved compared to the case of using other samples. In addition, the time and labor for imaging other samples, etc. can be reduced.

[0285] Note that using the minimized D = |A-WH| 2 The method of using the recursive formula of , the method using the quasi-Newton method (Davidon-Fletcher-Powell (DFP) method), the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method, etc. can be considered as a method of minimizing the mean square residual D. In these cases, the following method can be considered as a method of fixing the dye fluorescence spectrum to the initial value.

[0286] 5.1. Method for fixing the dye fluorescence spectrum in minimizing the mean square residual D using a recursive formula

[0287] When using the minimized D = |A-WH| 2 In the method of minimizing the mean square residual D by the recursive formula of , a loop process is performed, which repeats the steps including the multiplication type update formula represented by the following equations (25) and (26). Note that in equations (25) and (26), A = (a i,j ) N×M , H=(h i,j ) k×M , W=(w i,j ) N×k .also, t h and t w are the transposed matrices of sub-matrices h and w respectively.

[0288] [Equation 25]

[0289]

[0290] [Equation 26]

[0291]

[0292] In such a loop process, in order to fix the dye fluorescence spectrum to the initial value, a method of inserting the step of executing the following equation (27) between the step of executing equation (25) and the step of executing equation (26) can be used. Note that equation (27) represents the updated w corresponding to i,j k+1 The submatrix of the dye fluorescence spectra in is folded by the submatrix w i,j(part) k Overwrite, this submatrix is ​​the initial value of the stain fluorescence spectrum.

[0293] [Equation 27]

[0294] w i,j k+1 ←w i,j(part) k (27)

[0295] 5.2. Methods for fixing dye fluorescence spectra in minimizing the mean square residual D using the DFP method, BFGS method, etc.

[0296] In addition, in the method of minimizing the mean square residual D using the DFP method, the BFGS method, etc., when the mean square residual D of the minimization target is D(x), and x is the coordinate (at the kth update, x k =(a1, a2, ...an) k ), minimize D(x) by the following steps. In the following steps, B represents the Hessian matrix.

[0297] - via x k+1 =x k -αB k -1 D'(x k ) Update coordinates

[0298] - New coordinate x k+1 The gradient shift at

[0299] -From y k =D'(x k+1 )-D'(x k ) Update the Hessian inverse matrix B k+1 -1

[0300] Various methods (for example, the DFP method represented by the following equation (28), the BFGF method represented by the following equation (29), etc.) can be applied to the update of the Hessian matrix Bk+1.

[0301] [Equation 28]

[0302]

[0303] [Equation 29]

[0304]

[0305] In this method of minimizing the mean square residual D using the DFP method, the BFGS method, etc., there are several methods as methods of fixing arbitrary coordinates, that is, methods of fixing the dye fluorescence spectrum to the initial value. For example, the dye fluorescence spectrum can be fixed to the initial value by performing the following process (1) or process (2) at the time of updating the coordinates.

[0306] (1)-αB k -1 D'(x k )=0, that is, the partial differential D'(x k ) is replaced by zero

[0307] (2) Calculate x after updating the coordinates k+1 After that, force the obtained coordinate x k+1 Replace part of with x k (or x k part of

[0308] <6. Fifth embodiment>

[0309] Next, a sixth embodiment of the present disclosure will be described in detail with reference to the drawings.

[0310] Methods for separating and analyzing certain data into elements and coefficients of the elements that configure the data are widely used, including machine learning. As methods for decomposing data into elements (basis or spectrum) (they are referred to as spectrum in the present disclosure) and coefficients, there are various methods, for example, eigenvalue decomposition, singular value decomposition, non-negative matrix factorization (NMF), etc. described in the above embodiments. It can be said that, in particular, NMF for non-negative data has a high similarity between the obtained solution and the actual spectrum (for example, the absorption spectrum of a material, the fluorescence spectrum, etc.), and is advantageous in interpreting the data because both the spectrum and the coefficients are non-negative values.

[0311] As mentioned above, NMF is obtained by using the spectrum S (corresponding to Fig.13 The matrix H) and coefficients C (corresponding to Fig.13 The product of the matrix W) and the error f (corresponding to Fig.13The data matrix A is represented by the sum of the mean squared residuals D) and matrix factorization is performed under the non-negative constraint so that the error (f = |A - S × C| 2 ) becomes the smallest method, and has the characteristic of being easy to approximate data (low-rank approximation) by the minimum spectrum. In this NMF, a calculation method using a recursive formula is established, and a combination of S and C that minimizes the error f can be obtained by repeatedly calculating the following equation (30).

[0312] [Equation 30]

[0313]

[0314] Note that Xij and Yij are values ​​respectively expressed by the following equation (31).

[0315] [Equation 31]

[0316]

[0317] In addition, in equation (31), the matrix t C and the matrix t S is the transposed matrix of matrix C and matrix S respectively.

[0318] Here, consider a matrix A with the number w (corresponding to Fig.13 The number of wavelength channels M) p data (corresponding to Fig.13 The case where the number of pixels is N) and these data are approximated by n spectra of the number of elemental components w. In this case, the matrix A can be expressed by the following equation (32).

[0319] [Equation 32]

[0320] A(p,w)=S(n,w)×C(p,n) (32)

[0321] In equation (32), when calculating Xij and Yij given by equation (31) above, the reference matrix A(p, w) is required. Therefore, each iterative calculation needs to be calculated for all points p.

[0322] If the data to be analyzed is small-scale data, there is little problem in performing calculations for all points p for each iteration, but in the case of large-scale data with a very large number of data points, this becomes a factor that increases the calculation time. In addition, in the case where the data cannot be stored in a memory (for example, the memory described later), Fig.42 In the case where all p data is expanded in the RAM 903 in the memory, frequent access to the external storage device (for example, Fig.42 The problem of storage device 908) makes the processing time more redundant.

[0323] On the other hand, the inventors have found that the Gram matrix of the data matrix A can be t AA performs a non-negative decomposition instead of the data matrix A to be decomposed to obtain the spectrum S.

[0324] Therefore, in this embodiment, by converting the data matrix A into a Gram matrix t AA and Gram matrix t AA performs non-negative decomposition to obtain the spectrum S as the solution. By converting the data matrix A into a Gram matrix t AA, can make the processing symmetric matrix into a square matrix. Thus, for example, a data matrix A in which the number of pixels N is very large relative to the number of wavelength channels M is converted into an M×M Ram matrix t AA, and thus the number of data points can be significantly reduced to shorten the calculation time and significantly reduce the amount of storage required for calculation. As a result, highly efficient analysis can be achieved.

[0325] (6.1. Processing overview of processing unit)

[0326] The information processing device according to the present embodiment has, for example, a configuration similar to the information processing device 100 according to the above-described embodiment (see Figure 1 ) configuration, and the processing unit 130 (e.g., the separation processing unit 132) performs the following operations.

[0327] First, according to the present embodiment, the processing unit 130 performs non-negative factorization or singular value decomposition on the data matrix A into A=S×C by pre-calculating the Gram matrix of the matrix A. t AA will calculate the Gram matrix t AA performs non-negative decomposition into t AA=S×E to obtain spectrum S.

[0328] Secondly, when calculating the Gram matrix t During the AA process, the processing unit 130 according to the present embodiment convolves each Gram matrix as in the following equation (33) by using a subset where A(p,w)=A1(p1-pn1,w)+A2(pn1+1-pm,w)+...+Ao(pm+1-p,w) t AqAq (q is an integer greater than or equal to 1 and less than or equal to n) to obtain the Gram matrix t AA.

[0329] [Equation 33]

[0330] t AA= t A1A1+ t A2A2+...+t AnAn (33)

[0331] Third, the coefficient C is obtained by solving A=S×C using the spectrum S obtained by non-negative decomposition with respect to the above-mentioned Gram matrix.

[0332] (6.2. Example of measurement system configuration)

[0333] Next, a configuration example of a measurement system in the information processing apparatus 100 according to the present embodiment will be described. Fig.36 is a diagram showing an example of a measurement system of the information processing system according to the present embodiment. Fig.36 , an example of a measurement system when imaging a wide field of view of a fluorescently stained sample 30 (or a sample 20 that is a non-stained sample), such as whole slide imaging (WSI), is shown. However, the measurement system according to the present embodiment is not limited to Fig.36 The measuring system shown in the figure can be subjected to various modifications as long as it is a measuring system capable of acquiring image data of sufficient resolution for the entire imaging area or the area of ​​interest (hereinafter referred to as wide-field image data), for example, a measuring system that images the entire imaging area or a necessary area (also referred to as the area of ​​interest) at one time, a measuring system that acquires an image of the entire imaging area or the area of ​​interest by line scanning, etc.

[0334] like Fig.36 As shown, the measurement system according to the present embodiment includes, for example, an information processing device 100 , an XY stage 501 , an excitation light source 510 , a beam splitter 511 , an objective lens 512 , a spectroscope 513 , and a photodetector 514 .

[0335] The XY stage 501 is a stage on which the fluorescently stained sample 30 (or sample 20) as an analysis target is placed, and may be, for example, a stage movable on a plane (XY plane) parallel to the placement surface of the fluorescently stained sample 30 (or sample 20).

[0336] The excitation light source 510 is a light source for exciting the fluorescent-stained sample 30 (or the sample 20 ), and emits a plurality of excitation lights having different wavelengths along a predetermined optical axis, for example.

[0337] The beam splitter 511 includes, for example, a dichroic mirror, etc., reflects the excitation light from the excitation light source 510, and transmits the fluorescence from the fluorescent-stained sample 30 (or sample 20).

[0338] The objective lens 512 irradiates the fluorescent-stained sample 30 (or sample 20 ) on the XY stage 501 with the excitation light reflected by the beam splitter 511 .

[0339] The beam splitter 513 is configured by using one or more prisms, lenses, etc., and scatters the fluorescence emitted from the fluorescent-stained sample 30 (or the sample 20 ) and transmitted through the objective lens 512 and the beam splitter 511 in a predetermined direction.

[0340] The photodetector 514 detects the light intensity of each wavelength of the fluorescence scattered by the spectroscope 513 , and inputs a fluorescence signal (fluorescence spectrum and / or autofluorescence spectrum) obtained by the detection to the fluorescence signal acquisition unit 112 of the information processing apparatus 100 .

[0341] In the configuration as described above, in the case where the entire imaging area exceeds the area that can be imaged at one time (hereinafter referred to as the field of view), for example, WSI, the field of view is moved by moving the XY stage 501 for each imaging, and imaging of each field of view s is performed in sequence. Then, by tiling the image data obtained by imaging each field of view (hereinafter referred to as the field of view image data), wide field of view image data of the entire imaging area is generated. The generated wide field of view image data is stored in, for example, the fluorescence signal storage unit 122. Note that the tiling of the field of view image data can be performed by the acquisition unit 110 of the information processing device 100, can be performed by the storage unit 120 of the information processing device 100, or can be performed by the processing unit 130 of the information processing device 100.

[0342] Then, the processing unit 130 according to the present embodiment acquires the coefficient C, that is, the fluorescence separation image of each fluorescent molecule (or the autofluorescence separation image of each autofluorescent molecule) by performing the above-mentioned processing on the obtained wide-field image data.

[0343] (6.3. Operation examples)

[0344] Next, an operation example of the information processing device 100 according to the present embodiment will be described. Note that the following description will focus on the operation of the processing unit 130.

[0345] Fig.37 is a flowchart showing an example of the operation of the processing unit according to the present embodiment. Figures 38 to 40 is used to describe Fig.37 An illustration of the processing performed by the processing unit in each step of FIG.

[0346] like Fig.37 As shown, first, the processing unit 130 according to this embodiment generates wide field of view image data of the entire imaging area by tiling the field of view image data obtained by imaging each field of view (for example, see Fig.38 Wide field of view image data A) (step S2001).

[0347] Next, the processing unit 130 acquires unit image data (eg, Fig.38 The unit image data Aq (q is an integer greater than or equal to 1 and less than or equal to n) in (step S2002). The unit image data Aq can be changed variously as long as it is image data of an area smaller than the wide field of view image data A, for example, image data corresponding to one field of view, image data of a preset size, etc. Note that the image data of the preset size may include image data of a size determined by the amount of data that can be processed at one time by the information processing device 100.

[0348] Next, if Fig.38 As shown, the processing unit 130 multiplies the data matrix (for the sake of clarity, the data matrix is ​​referred to as A1) of the acquired unit image data Aq (for the sake of clarity in the following description, the unit image data A1) by the transposed matrix t A1 generates the Gram matrix of the unit image data A1 t A1A1 (step S2003).

[0349] Next, the processing unit 130 determines whether the Gram matrix of all the unit image data A1 to An has been completed. t A1A1 to t The generation of AnAn (step S2004), and steps S2002 to S2004 are repeated until the Gram matrices of all unit image data A1 to An are completed. t A1A1 to t Generation of AnAn (No in step S2004).

[0350] On the other hand, when the Gram matrix for all unit image data A1 to An is completed t A1A1 to t When AnAn is generated (Yes in step S2004), the processing unit 130 obtains the Gram matrix by using, for example, the least squares method (or the weighted least squares method). t A1A1 to t AnAn calculates the initial value of the coefficient C (step S2005).

[0351] Next, the processing unit 130 generates the Gram matrix t A1A1 to t AnAn is added to calculate the Gram matrix of the wide field of view image data A tAA (step S2006). Specifically, as described above, the processing unit 130 convolves each Gram matrix as in the above equation (33) by using a subset of A(p,w)=A1(p1-pn1,w)+A2(pn1+1-pm,w)+...+Ao(pm+1-p,w). t AqAq (q is an integer greater than or equal to 1 and less than or equal to n) to obtain the Gram matrix t AA.

[0352] Next, if Fig.39 As shown, the processing unit 130 calculates the Gram matrix t AA non-negative decomposition into t AA=S×E to obtain spectrum S (step S2007 ). Note that matrix E corresponds to the separated image fluorescence-separated from the wide-field image data A.

[0353] Afterwards, if Fig.40 As shown, the processing unit 130 uses the least squares method (or weighted least squares method) to calculate the Gram matrix by NMF. t The spectrum S obtained by AA is solved by A=S×C to obtain the coefficient C, ie, the fluorescence separation image of each fluorescent molecule (or the autofluorescence separation image of each autofluorescent molecule) (step S2008), and the current operation is terminated thereafter.

[0354] Note that in the NMF of step S2007, non-negative factorization of data may be performed with a fixed specific spectrum.

[0355] (6.4.1. First Modification)

[0356] Note that already Figures 37 to 40 , the case where the entire imaging area is set as the processing target area is illustrated, but the processing target area is not limited to this, and can also be set to an area narrower than the entire imaging area (region of interest). The region of interest can be, for example, an area where the analysis target is projected, for example, an area where the fluorescently stained sample 30 (or sample 20) exists in the wide-field image data A, etc. In addition, for example, morphological information of the fluorescently stained sample 30 or sample 20 (e.g., cells, tissues, etc.) can be used to set the region of interest. Note that the morphological information can be a bright field image, a non-stained image, and staining information of the same tissue block, or can be, for example, an expression map of a target in the sample 20. In addition, the morphological information can be information generated using techniques such as segmentation (acquiring and marking regions in units of one pixel) in image recognition technology using machine learning.

[0357] Fig.41 1 is a flowchart showing an example of the operation of the processing unit according to the first modified example of the present embodiment. Fig.41As shown, first, the processing unit 130 according to the first modification generates wide field of view image data of the entire imaging area by tiling the field of view image data obtained by imaging each field of view (step S2101). In the first modification, the resolution of the wide field of view image data A may be lower than the resolution of the image data as the processing target (e.g., high-resolution image data described later).

[0358] Next, the processing unit 130 sets a monitoring area as a processing target area in the wide field of view image data A (step S2102). As described above, the setting of the region of interest can be performed based on, for example, morphological information, etc. However, the setting of the region of interest can be automatically performed by the processing unit 130 based on morphological information, etc., or can be manually performed by the user.

[0359] Next, the processing unit 130 requests the control unit 150 to obtain high-resolution image data of the region of interest (step S2103). In response to such a request, the control unit 150 controls the above-mentioned measurement system (see Fig.36 ), the acquisition unit 110 and the storage unit 120 to acquire high-resolution image data of the region of interest. Note that the region of interest may be a range wider than a field of view.

[0360] Next, the processing unit 130 performs a process similar to, for example, Fig.37 The operations of steps S2002 to S2004 generate a Gram matrix of each unit image data Aq acquired from the high-resolution image data of the region of interest. t AqAq (steps S2104 to S2106).

[0361] Next, similar to e.g. Fig.37 In step S2005, the processing unit 130 uses the least squares method (or weighted least squares method) to obtain the Gram matrix t A1A1 to t AnAn calculates the initial value of the coefficient C (step S2107).

[0362] Next, the processing unit 130 generates the Gram matrix t A1A1 to t AnAn is added to calculate the Gram matrix of the wide field of view image data A t AA (step S2108), by calculating the Gram matrix t AA performs non-negative decomposition as t AA = S × E to obtain the spectrum S (step S2109), and obtain the coefficient C, that is, similar to, for example Fig.37 Steps S2006 to S2008 are performed by using the Gram matrixt The spectrum S obtained by NMF of AA is solved by least square method (or weighted least square method) to obtain A=S×C (step S2110), and the fluorescence separation image of each fluorescent molecule (or the autofluorescence separation image of each autofluorescent molecule) is obtained, and then the current operation is terminated. Note that in the NMF of step S2109, a fixed specific spectrum can be used to perform non-negative factorization of data.

[0363] (6.4.2. Second Modification)

[0364] Note that Fig.37 The operation example shown and its modified example ( Fig.41 ), an example has been given of first acquiring wide-field image data of the entire imaging area or high-resolution image data of the entire region of interest, and then acquiring and sequentially processing unit image data as a part of the wide-field image data or the high-resolution image data, but the present disclosure is not limited thereto, and all or part of the wide-field image data or the high-resolution image data may also be executed in a pipeline processing manner. Specifically, for example, with respect to the Gram matrix until each unit image data is generated t AqAq treatment (e.g. Fig.37 Steps S2001 to S2004 in or Fig.41 Steps S2103 to S106 in FIG. 100 can be performed by using the measurement system (see Fig.36 ) outputs the image data of each field of view as unit image data, and performs the above processing in response to the input of the unit image data to generate a Gram matrix of each unit image data t AqAq.

[0365] (6.5. Effect)

[0366] Regarding the effects expected by the present embodiment, hereinafter, regarding the process until a solution of the spectrum S and coefficient C of A=S×C is obtained by NMF, a case where NMF of the matrix A(p, w) has been performed (case 1) and a Gram matrix obtained from the matrix A has been performed will be described by way of example. t Case of NMF of AA(w,w) (Case 1).

[0367] Under the assumption that the computation times of the four arithmetic operations are almost equal to each other and under the assumption that no overhead is considered, when the computation amount of the NMF loop is calculated for each of Case 1 and Case 2, it is estimated that the computation amount of the NMF loop in Case 2 via the Gram matrix is ​​0.0447 W / m 2 in comparison with Case 1 where the matrix A is subjected to NMF. t AA's processing speed can be increased by approximately 6,000 times.

[0368] Furthermore, in the case where 10 to 100 unit image data are calculated using wide field of view image data (e.g., WSI), compared with the case 1 where the wide field of view image data A is subjected to NMF as it is, the Gram matrix of the wide field of view image data is calculated by convolving the Gram matrix of each unit image data. t In case 2 of AA, it is estimated that the processing speed can be increased by about 60,000 to 600,000 times.

[0369] Furthermore, in the case where each unit image data is 1024×1024 image data and the number of wavelength channels (M) is 100 points, compared with Case 1 where the matrix A(p, w) is subjected to NMF, in the case where the Gram matrix of the matrix A is t In case 2 where AA(w, w) is subjected to NMF, the maximum amount of storage required for the extended data can be reduced to about 1 / 10000. In addition, in the case where ten to one hundred unit image data are already considered, the amount of storage can be further reduced, for example, the amount of storage can be reduced to 1 / 100000 to 1 / 1000000.

[0370] Other configurations, operations, and effects may be similar to those in the above-described embodiments, and thus detailed descriptions thereof will be omitted here.

[0371] <7. Hardware Configuration Example>

[0372] The modified examples of the present disclosure have been described above. Fig.42 A hardware configuration example of the information processing apparatus 100 according to each of the embodiment and the modification example is described. Fig.42 is a block diagram showing a hardware configuration example of the information processing apparatus 100. Various processes of the information processing apparatus 100 are realized by cooperation between software and hardware described below.

[0373] like Fig.42 As shown, the information processing device 100 includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, a random access memory (RAM) 903, and a host bus 904a. In addition, the information processing device 100 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. Instead of the CPU 901 or together with the CPU 901, the information processing device 100 may have a processing circuit such as a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc.

[0374] The CPU 901 functions as an arithmetic processing device and a control device, and generally controls operations in the information processing device 100 according to various programs. In addition, the CPU 901 may be a microprocessor. The ROM 902 stores programs, operation parameters, and the like used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change appropriately in the execution, and the like. For example, the CPU 901 may embody at least the processing unit 130 and the control unit 150 of the information processing device 100.

[0375] The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a host bus 904a including a CPU bus, etc. The host bus 904a is connected to an external bus 904b such as a peripheral component interconnect / interface (PCI) bus, etc., via a bridge 904. Note that the host bus 904a, the bridge 904, and the external bus 904b do not necessarily need to be configured separately, and the functions of the host bus 904a, the bridge 904, and the external bus 904b may be mounted on a single bus.

[0376] The input device 906 is implemented by, for example, a device such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, a lever, etc., to which the physician inputs information. In addition, the input device 906 may be, for example, a remote control device using infrared or other radio waves, or may be an external connection device corresponding to the operation of the information processing device 100, for example, a mobile phone, a personal digital assistant (PDA), etc. In addition, the input device 906 may include, for example, an input control circuit, which generates an input signal based on the information input by the physician using the above-mentioned input device, and outputs the generated input signal to the CPU 901. The physician may input various data to the information processing device 100, or instruct the information processing device 100 to perform a processing operation by operating the input device 906. For example, the input device 906 may include at least the operation unit 160 of the information processing device 100.

[0377] The output device 907 is a device that can visually or auditorily notify the physician of the acquired information. Such a device includes a display device (e.g., a cathode ray tube (CRT) display device, a liquid crystal display device, a plasma display device, an electroluminescent (EL) display device, a lamp, etc.), a sound output device (e.g., a speaker, a headset, etc.), a printer device, etc. For example, the output device 907 may at least embody the display unit 140 of the information processing device 100.

[0378] The storage device 908 is a device for storing data. The storage device 908 is implemented, for example, by a magnetic storage unit device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage device 908 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded in the storage medium, etc. The storage device 908 stores a program or various data executed by the CPU 901, various data acquired from the outside, etc. For example, the storage device 908 may embody at least the storage unit 120 of the information processing device 100.

[0379] The drive 909 is a reader / writer of a storage medium, and is embedded in or externally mounted on the information processing apparatus 100. The drive 909 reads information recorded in a mounted removable storage medium (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc.), and outputs the read information to the RAM 903. Furthermore, the drive 909 can write information to the removable storage medium.

[0380] The connection port 911 is an interface connected to an external device, and is a connection port for an external device capable of transmitting data through, for example, a universal serial bus (USB) or the like.

[0381] The communication device 913 is, for example, a communication interface including a communication device for connecting to the network 920, etc. The communication device 913 is, for example, a communication card for a wired or wireless local area network (LAN), long term evolution (LTE), Bluetooth (registered trademark), or wireless USB (WUSB), etc. In addition, the communication device 913 may be a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), a modem for various communications, etc. The communication device 913 may send a signal to the Internet or another communication device, or receive a signal from the Internet or another communication device, etc., according to a predetermined protocol, for example, a transmission control protocol / Internet protocol (TCP / IP), etc.

[0382] In this embodiment, the sensor 915 includes a sensor capable of acquiring a spectrum (e.g., an imaging element, etc.), and may include other sensors (e.g., an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, a pressure sensor, a sound sensor, a distance measurement sensor, etc.). For example, the sensor 915 may at least embody the fluorescence signal acquisition unit 112 of the information processing device 100.

[0383] Note that the network 920 is a wired or wireless transmission path for information transmitted from a device connected to the network 920. For example, the network 920 may include a public network (e.g., the Internet, a telephone network, a satellite communication network, etc.), various local area networks including Ethernet (registered trademark), a wide area network (WAN), etc. In addition, the network 920 may include a dedicated line network such as an Internet Protocol-Virtual Private Network (IP-VPN), etc.

[0384] The above has described an example of a hardware configuration that can realize the functions of the information processing device 100. Each of the above components can be implemented using a general-purpose component, or can be implemented by hardware dedicated to the functions of each component. Therefore, when executing the present disclosure, the hardware configuration to be used can be appropriately changed according to the technical level.

[0385] Note that a computer program for realizing each function of the information processing device 100 as described above can be created and installed in a personal computer (PC) or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium includes, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above-mentioned computer program can be distributed via, for example, a network without using a recording medium.

[0386] <8. Conclusion>

[0387] As described above, the information processing device 100 according to the first embodiment of the present disclosure irradiates the fluorescent dye sample 30 with multiple excitation lights having different wavelengths, obtains multiple fluorescence spectra corresponding to each of the multiple excitation lights, corrects the multiple fluorescence spectra based on the intensity of the excitation light, and links at least a portion of the multiple fluorescence spectra to each other in the wavelength direction to generate a linked fluorescence spectrum. Then, the information processing device 100 extracts the spectrum of each fluorescent substance from a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum, wherein in the linked autofluorescence reference spectrum, the spectra of the autofluorescence substances are linked to each other in the wavelength direction, and in the linked fluorescence reference spectrum, the spectra of the fluorescent substances are linked to each other in the wavelength direction. Then, the information processing device 100 uses the extracted spectrum of each fluorescent substance to separate the linked fluorescence spectrum of each molecule.

[0388] In this way, the information processing device 100 can output a unique spectrum as a separation result (the separation result is not different for each excitation wavelength) by performing fluorescence separation processing using the reference spectrum linked in the wavelength direction. Therefore, the physician can more easily obtain the correct spectrum. In addition, the reference spectrum (linked autofluorescence reference spectrum) for the autofluorescence used for separation is automatically acquired, and the fluorescence separation processing is performed, so that the physician does not need to extract the spectrum corresponding to the autofluorescence from the appropriate space of the non-stained slice.

[0389] Furthermore, the information processing apparatus 100 according to the second embodiment of the present disclosure performs the fluorescence separation process using the linked autofluorescence reference spectrum actually measured for each sample 20. Therefore, the information processing apparatus 100 can realize a more accurate fluorescence separation process.

[0390] In addition, the information processing device 100 according to the modified example of the present disclosure uses a reference spectrum to separate the linked fluorescence spectrum of each fluorescent substance, wherein the reference spectrum includes a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules. Therefore, as a result of the fluorescence separation process, the information processing device 100 can calculate the number of fluorescent molecules or the number of antibodies in the fluorescent staining sample 30.

[0391] In addition, the information processing device 100 according to the third embodiment of the present disclosure uses the sample image obtained from the stained section as the matrix A to solve the NMF. Therefore, the fluorescence spectrum of each fluorescent substance can be directly extracted from the stained section, while significantly reducing the time and work cost required for the fluorescence separation process. In addition, in the third embodiment of the present disclosure, the fluorescence spectrum of each fluorescent substance is extracted from the sample image obtained from the same stained section, so that a more accurate fluorescence separation result can be obtained compared with, for example, using the autofluorescence spectrum obtained from a non-stained section different from the stained section.

[0392] In the above, the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to these examples. It is obvious to those skilled in the art in the field of the present disclosure that various modifications or changes can be conceived within the scope of the technical ideas described in the claims, and it is naturally understood that these modifications or changes also fall within the technical scope of the present disclosure.

[0393] In addition, the effects described in this specification are merely illustrative or exemplary, and not restrictive. That is, in addition to or in place of the above effects, the technology according to the present disclosure can achieve other effects that are obvious to those skilled in the art from the description of this specification.

[0394] Note that the following configurations also fall within the technical scope of the present disclosure.

[0395] (1) An information processing device comprising:

[0396] a fluorescence signal acquisition unit that acquires a plurality of fluorescence spectra corresponding to each of a plurality of excitation lights having different wavelengths and irradiated to a fluorescence-stained sample produced by staining the sample with a fluorescent agent;

[0397] a linking unit that generates a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra to each other in a wavelength direction;

[0398] a separation unit that separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum in which the spectra of the autofluorescence substances in the sample are linked to each other in a wavelength direction and a linked fluorescence reference spectrum in which the spectra of the fluorescent substances in the fluorescent-stained sample are linked to each other in a wavelength direction; and

[0399] The extracting unit updates the linked autofluorescence reference spectrum using the spectrum for each fluorescent substance separated by the separating unit.

[0400] (2) The information processing device according to (1) above, wherein:

[0401] The extraction unit extracts the linked autofluorescence reference spectrum from linked autofluorescence spectra generated by linking at least a portion of a plurality of autofluorescence spectra to each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating a slice with the plurality of excitation lights, and the slice is the same as or similar to the sample.

[0402] (3) The information processing device according to (2) above, wherein:

[0403] The extraction unit extracts the linked autofluorescence reference spectrum by performing non-negative matrix factorization using the linked autofluorescence spectra generated by linking at least a portion of a plurality of autofluorescence spectra to each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating the slice with the plurality of excitation lights, and the slice is the same as or similar to the sample.

[0404] (4) The information processing device according to (3) above, wherein:

[0405] The extraction unit extracts the linked autofluorescence reference spectrum by setting an initial value in the non-negative matrix factorization using a pre-acquired autofluorescence spectrum.

[0406] (5) The information processing device according to any one of (1) to (4) above, wherein:

[0407] The separation unit separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using any one of a least square method or a weighted least square method using the reference spectrum.

[0408] (6) The information processing device according to (5) above, wherein:

[0409] The separation unit separates the linked fluorescence spectrum into spectra for each fluorescent substance by setting the matrix representing the linked fluorescence spectrum to Signal, setting the matrix representing the reference spectrum to St, setting the matrix representing the color mixing ratio of each reference spectrum in the linked fluorescence spectrum to a, and calculating the matrix a representing the color mixing ratio when the sum of squares of values ​​represented by the following equation (34) becomes minimum:

[0410] [Equation 34]

[0411] Signal-a*St (34)

[0412] (7) The information processing device according to (6) above, wherein:

[0413] In the case of using the weighted least squares method, the separation unit sets the upper limit value where weighting is not performed as the Offset value, and replaces the matrix St representing the reference spectrum in equation (34) with the matrix St_ represented by the following equation (35):

[0414] [Equation 35]

[0415]

[0416] (8) The information processing device according to any one of (1) to (7) above, wherein:

[0417] The separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules.

[0418] (9) The information processing device according to (8) above, wherein:

[0419] The separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum of each fluorescent molecule or each antibody.

[0420] (10) The information processing device according to any one of (1) to (9) above, wherein:

[0421] The separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance by performing non-negative matrix factorization on the linked fluorescence spectrum.

[0422] (11) The information processing device according to (10) above, wherein:

[0423] The separation unit calculates a product-moment correlation coefficient with an initial value adopted by the non-negative matrix factorization for the spectrum extracted by the non-negative matrix factorization, and specifies a correspondence relationship between the fluorescent substance and the extracted spectrum.

[0424] (12) The information processing device according to any one of (1) to (11) above, wherein:

[0425] The linking unit corrects the plurality of fluorescence spectra and links at least a portion of the corrected plurality of fluorescence spectra to each other in a wavelength direction.

[0426] (13) The information processing device according to (12) above, wherein:

[0427] The linking unit corrects the intensities of the plurality of fluorescence spectra.

[0428] (14) The information processing device according to (13) above, wherein:

[0429] The linking unit corrects the intensities of the plurality of fluorescence spectra by dividing the plurality of fluorescence spectra by excitation power density.

[0430] (15) The information processing device according to any one of (12) to (14) above, wherein:

[0431] The linking unit corrects a wavelength resolution of at least one of the plurality of fluorescence spectra to a wavelength resolution different from wavelength resolutions of the other fluorescence spectra.

[0432] (16) The information processing device according to any one of (1) to (15) above, wherein:

[0433] The linking unit extracts a fluorescence spectrum in a wavelength band including an intensity peak from each of the plurality of fluorescence spectra, and generates the linked fluorescence spectrum by linking the extracted fluorescence spectra to each other.

[0434] (17) The information processing device according to any one of (1) to (16) above, wherein:

[0435] The linked fluorescence spectra are linked discontinuously in the wavelength direction among the plurality of fluorescence spectra.

[0436] (18) The information processing device according to any one of (1) to (17) above, wherein:

[0437] The fluorescence signal acquisition unit acquires first image data obtained by imaging the fluorescence-stained sample and including the plurality of fluorescence spectra, and

[0438] The separation unit separates the first image data into a spectrum for each fluorescent substance by performing non-negative matrix factorization on a first Gram matrix of the first image data.

[0439] (19) The information processing device according to (18) above, wherein:

[0440] The separation unit calculates the first Gram matrix by convolving a second Gram matrix of each of a plurality of second image data obtained by dividing the first image data.

[0441] (20) The information processing device according to any one of (1) to (17) above, wherein:

[0442] The fluorescence signal acquisition unit acquires first image data by imaging a non-stained sample irradiated by the excitation light, and

[0443] The extraction unit extracts a spectrum of each autofluorescence substance from the first image data by performing non-negative matrix factorization on a first Gram matrix of the first image data, and updates the linked autofluorescence reference spectrum using the extracted spectrum of each autofluorescence substance.

[0444] (21) The information processing device according to (20) above, wherein:

[0445] The extracting unit calculates a first Gram matrix by convolving a second Gram matrix of each of a plurality of second image data obtained by dividing the first image data.

[0446] (22) A microscope system comprising: a light source that irradiates a fluorescently stained sample with a plurality of excitation lights having different wavelengths, the fluorescently stained sample being produced by staining the sample with a fluorescent agent; an imaging device that acquires a plurality of fluorescence spectra corresponding to each of the plurality of excitation lights; and software for processing using the plurality of fluorescence spectra, wherein

[0447] executing the software on an information processing device, and

[0448] accomplish:

[0449] generating a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra to each other in a wavelength direction;

[0450] The linked fluorescence spectrum is separated into spectra of each fluorescent substance using a reference spectrum comprising a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum, wherein,

[0451] In the linked autofluorescence reference spectrum, the spectra of the autofluorescent substances in the sample are linked to each other in the wavelength direction, and in the linked fluorescence reference spectrum, the spectra of the fluorescent substances in the fluorescently stained sample are linked to each other in the wavelength direction; and

[0452] The linked autofluorescence reference spectrum is updated using the separated spectrum of each fluorescent substance.

[0453] Reference numerals list

[0454] 10 Fluorescence reagents

[0455] 11 Reagent identification information

[0456] 20 samples

[0457] 21 Sample identification information

[0458] 30 fluorescent staining samples

[0459] 100 Information processing equipment

[0460] 110 Get Unit

[0461] 111 Information Acquisition Unit

[0462] 112 Fluorescence signal acquisition unit

[0463] 120 storage units

[0464] 121 Information storage unit

[0465] 122 fluorescence signal storage units

[0466] 130 processing units

[0467] 131 Link Units

[0468] 132 Separation Processing Unit

[0469] 133 Image Generation Unit

[0470] 140 display units

[0471] 150 control unit

[0472] 160 operation units

[0473] 200 databases.

Claims

1. An information processing device, comprising: a fluorescence signal acquisition unit that acquires a plurality of fluorescence spectra corresponding to each of a plurality of excitation lights having different wavelengths and irradiated to a fluorescence-stained sample produced by staining the sample with a fluorescent agent; a linking unit that generates a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra to each other in a wavelength direction; a separation unit that separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum in which the spectra of the autofluorescence substances in the sample are linked to each other in a wavelength direction and a linked fluorescence reference spectrum in which the spectra of the fluorescent substances in the fluorescent-stained sample are linked to each other in a wavelength direction; and an extracting unit that updates the linked autofluorescence reference spectrum using the spectrum for each fluorescent substance separated by the separating unit, wherein the extracting unit extracts the linked autofluorescence reference spectrum from linked autofluorescence spectra generated by linking at least a part of a plurality of autofluorescence spectra to each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating a slice with the plurality of excitation lights, and the slice is identical to the sample, The linked fluorescence spectra are linked discontinuously in the wavelength direction of the plurality of fluorescence spectra.

2. The information processing device according to claim 1, wherein: The extraction unit extracts the linked autofluorescence reference spectrum by performing non-negative matrix factorization using the linked autofluorescence spectra generated by linking at least a portion of a plurality of autofluorescence spectra to each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating the slice with the plurality of excitation lights, and the slice is identical to the sample.

3. The information processing device according to claim 2, wherein: The extraction unit extracts the linked autofluorescence reference spectrum by setting an initial value in the non-negative matrix factorization using a pre-acquired autofluorescence spectrum.

4. The information processing device according to claim 1, wherein: The separation unit separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using any one of a least square method or a weighted least square method using the reference spectrum.

5. The information processing device according to claim 4, wherein: The separation unit separates the linked fluorescence spectrum into spectra for each fluorescent substance by setting the matrix representing the linked fluorescence spectrum to Signal, setting the matrix representing the reference spectrum to St, setting the matrix representing the color mixing ratio of each reference spectrum in the linked fluorescence spectrum to a, and calculating the matrix a representing the color mixing ratio when the sum of squares of values ​​represented by the following equation (1) becomes minimum: [Equation 1] Signal-a*St (1).

6. The information processing device according to claim 5, wherein: In the case of using the weighted least squares method, the separation unit sets the upper limit value for which weighting is not performed as Offsetvalue, and replaces the matrix St representing the reference spectrum in equation (1) with the matrix St_ represented by the following equation (2): [Equation 2] 7. The information processing device according to claim 1, wherein: The separation unit separates the linked fluorescence spectrum into the spectrum of each fluorescent substance using a reference spectrum including the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules, or using a reference spectrum including the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum of each fluorescent molecule or each antibody.

8. The information processing device according to claim 1, wherein: The separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance by performing non-negative matrix factorization on the linked fluorescence spectrum.

9. The information processing device according to claim 8, wherein: The separation unit calculates a product-moment correlation coefficient with an initial value adopted by the non-negative matrix factorization for the spectrum extracted by the non-negative matrix factorization, and specifies a correspondence relationship between the fluorescent substance and the extracted spectrum.

10. The information processing device according to claim 1, wherein: The linking unit corrects the plurality of fluorescence spectra and links at least a portion of the corrected plurality of fluorescence spectra to each other in a wavelength direction.

11. The information processing device according to claim 10, wherein: The linking unit corrects the intensities of the plurality of fluorescence spectra.

12. The information processing device according to claim 11, wherein: The linking unit corrects the intensities of the plurality of fluorescence spectra by dividing the plurality of fluorescence spectra by excitation power density.

13. The information processing device according to claim 10, wherein: The linking unit corrects a wavelength resolution of at least one of the plurality of fluorescence spectra to a wavelength resolution different from wavelength resolutions of the other fluorescence spectra.

14. The information processing device according to claim 1, wherein: The linking unit extracts a fluorescence spectrum in a wavelength band including an intensity peak from each of the plurality of fluorescence spectra, and generates the linked fluorescence spectrum by linking the extracted fluorescence spectra to each other.

15. The information processing device according to claim 1, wherein: The fluorescence signal acquisition unit acquires first image data obtained by imaging the fluorescence-stained sample and including the plurality of fluorescence spectra, and The separation unit separates the first image data into a spectrum for each fluorescent substance by performing non-negative matrix factorization on a first Gram matrix of the first image data.

16. The information processing device according to claim 15, wherein: The separation unit calculates the first Gram matrix by convolving a second Gram matrix of each of a plurality of second image data obtained by dividing the first image data.

17. The information processing device according to claim 1, wherein: The fluorescence signal acquisition unit acquires first image data by imaging a non-stained sample irradiated by the excitation light, and The extraction unit extracts a spectrum of each autofluorescence substance from the first image data by performing non-negative matrix factorization on a first Gram matrix of the first image data, and updates the linked autofluorescence reference spectrum using the extracted spectrum of each autofluorescence substance.

18. A microscope system comprising: a light source for irradiating a fluorescently stained sample with a plurality of excitation lights having different wavelengths, the fluorescently stained sample being produced by staining the sample with a fluorescent agent; an imaging device for acquiring a plurality of fluorescence spectra corresponding to each of the plurality of excitation lights; and software for processing using the plurality of fluorescence spectra, wherein executing the software on an information processing device, and accomplish: generating a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra to each other in a wavelength direction; separating the linked fluorescence spectrum into a spectrum of each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum in which the spectra of the autofluorescence substances in the sample are linked to each other in a wavelength direction and a linked fluorescence reference spectrum in which the spectra of the fluorescent substances in the fluorescently stained sample are linked to each other in a wavelength direction; and The linked autofluorescence reference spectrum is updated using the spectrum of each separated fluorescent substance, wherein the linked autofluorescence reference spectrum is extracted from linked autofluorescence spectra generated by linking at least a portion of a plurality of autofluorescence spectra to each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating a slice with the plurality of excitation lights, and the slice is the same as the sample.

Images