Information processing device and microscope for separating fluorescence of fluorescent reagent from autofluorescence of sample

Abstract

An information processing device is provided, comprising: an image acquisition unit (112); an information acquisition unit (111); and an analysis unit (131). The image acquisition unit (112) acquires image information about a sample (30) colored with a fluorescent reagent. The information acquisition unit (111) acquires information about the sample (11) and information about the fluorescent reagent (10). The analysis unit (131) separates a first fluorescent signal and a second fluorescent signal from the image information based on the information about the sample and the information about the fluorescent reagent.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, an information processing system, and a program. Background Art

[0002] In the past, in fluorescent immunostaining, antigens were visualized by using fluorescently labeled secondary antibodies. When this method was used, the type of visualized antigens was limited by the type of antibody used as the primary antibody.

[0003] Application uses the fluorescence amplification method of enzyme-labeled secondary antibody to solve the above problem. When using this method, although the type of visible antigen is limited by the type of available enzyme, however, by repeating the antigen-antibody reaction, fluorescent dye is deposited in the sample and the antibody is separated and a large amount of antigens are visualized, as in the case of using conventional staining immunity. However, as disclosed in NPL 1, when using the fluorescence amplification method, the time required for operation may increase with the increase of the number of repetitions of the cycle from antigen-antibody reaction to antibody separation. Further, during each repeated cycle, the sample may deteriorate.

[0004] A method of applying a reagent labeled with a direct fluorescent dye to a primary antibody or fluorescent probe has been developed to solve the above problem. This method allows visualization of antigens or target molecules of all types of fluorescent dyes.

[0005] [Citation List]

[0006] [Non-patent literature]

[0007] NPL 1: Zhang W. et al. “Fully automated 5-plex fluorescent immunohistochemistry with tyramide signal amplification and same species antibodies,” [Online], May 15, 2017, Laboratory Investigation, Internet<URL:http: / / www.nature.com / articles / labinvest201737> Summary of the invention

[0008] [Technical issues]

[0009] However, when using the above method, only limited types of fluorescent dyes can be coupled to the primary antibody and the fluorescent probe. Therefore, in some cases, the fluorescent signal to be measured may be hidden in the autofluorescence signal of the sample. Therefore, it is difficult to detect the fluorescent signal.

[0010] The present disclosure is made in view of the above circumstances. The purpose of the present disclosure is to provide a novel and improved information processing device, information processing method, information processing system, and program that can more appropriately separate at least one of a fluorescence signal to be measured or an autofluorescence signal of a sample from image information about the sample.

[0011] [Solution to the problem]

[0012] According to one aspect of the present disclosure, an information processing device is provided, which includes an image acquisition unit, an information acquisition unit, and an analysis unit. The image acquisition unit acquires image information about a sample colored by a fluorescent reagent. The information acquisition unit acquires information about the sample and information about the fluorescent reagent. The analysis unit separates a first fluorescent signal and a second fluorescent signal from the image information based on the information about the sample and the information about the fluorescent reagent.

[0013] [Advantageous Effects of the Invention]

[0014] As described above, the present disclosure can more appropriately separate at least one of a fluorescence signal to be measured or an autofluorescence signal of a sample from image information about the sample.

[0015] It should be noted that the advantageous effects described above are not necessarily limiting. In addition to or in place of the advantageous effects described above, any advantageous effects described in this specification or other advantageous effects envisioned from this specification may be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] [ Figure 1 ] Figure 1 2 is a diagram showing removal of an autofluorescence signal originating from the sample 20 .

[0017] [ Figure 2 ] Figure 2 is another diagram illustrating removal of the autofluorescence signal originating from the sample 20 .

[0018] [ Figure 3 ] Figure 3 is a block diagram illustrating an exemplary configuration of an information processing system according to an embodiment of the present disclosure.

[0019] [ Figure 4 ] Figure 4 are flowcharts illustrating various exemplary processes performed by the information processing apparatus 100 .

[0020] [ Figure 5 ] Figure 5 is another flowchart illustrating various exemplary processes performed by the information processing device 100 .

[0021] [ Figure 6 ] Figure 6 is a diagram showing a practical example of removing an autofluorescence signal originating from the sample 20 .

[0022] [ Figure 7 ] Figure 7 is another diagram showing a practical embodiment of removing the autofluorescence signal originating from the sample 20 .

[0023] [ Fig. 8A ] Fig. 8A is another diagram showing a practical embodiment of removing the autofluorescence signal originating from the sample 20 .

[0024] [ Figure 8B ] Figure 8B is another diagram showing a practical embodiment of removing the autofluorescence signal originating from the sample 20 .

[0025] [ Fig.9A ] Fig.9A is another diagram showing a practical embodiment of removing the autofluorescence signal originating from the sample 20 .

[0026] [ Fig. 9B ] Fig. 9B is another diagram showing a practical embodiment of removing the autofluorescence signal originating from the sample 20 .

[0027] [ Fig.10 ] Fig.10 1 is a diagram showing an example of using image information when tonsil tissue is used as the sample 20 and using an autofluorescence signal obtained from the tonsil tissue as the autofluorescence signal used in color separation calculation when a fluorescence signal is extracted from the image information.

[0028] [ Fig.11 ] Fig.11 is a diagram showing an example of using image information when tonsil tissue is used as the sample 20 and using an autofluorescence signal obtained from lung tissue (a tissue different from tonsil tissue) as an autofluorescence signal used in color separation calculation when extracting a fluorescence signal from the image information.

[0029] [ Fig.12 ] Fig.12 is a diagram showing the color separation result when the labeled antibody CD4 labeled with the fluorescent dye Alexa Fluor 488 is used as the fluorescent reagent 10 .

[0030] [ Fig.13 ] Fig.13 is a diagram showing the color separation result when the labeled antibody PDL-1 labeled with the fluorescent dye Alexa Fluor 594 is used as the fluorescent reagent 10 .

[0031] [ Fig.14 ] Fig.14 is a diagram showing a practical example of analysis of a sample 20 in a fixed state.

[0032] [ Fig.15 ] Fig.15 is another diagram showing a practical embodiment of the analysis of the sample 20 in a fixed state.

[0033] [ Fig.16 ] Fig.16 is another diagram showing a practical embodiment of the analysis of the sample 20 in a fixed state.

[0034] [ Fig.17 ] Fig.17 is another diagram showing a practical embodiment of the analysis of the sample 20 in a fixed state.

[0035] [ Fig.18 ] Fig.18 is a block diagram showing an exemplary hardware configuration of the information processing apparatus 100 .

[0036] [ Fig.19 ] Fig.19 is a diagram showing how actual sample information and reagent information differ from catalog and literature values.

[0037] [ Fig. 20 ] Fig. 20 is another diagram showing how actual sample information and reagent information differ from catalog and literature values.

[0038] [ Fig.21 ] Fig.21 is a block diagram describing an exemplary configuration of an information processing system according to a modification of the present embodiment.

[0039] [ Fig. 22 ] Fig. 22 is a block diagram depicting an embodiment of a schematic configuration of a diagnostic system. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present disclosure are now described in detail with reference to the accompanying drawings. In this specification and the accompanying drawings, elements having substantially the same functions are denoted by the same reference numerals and will not be redundantly described.

[0041] The description will be given in the following order.

[0042] 1. Overview

[0043] 2. Implementation Method

[0044] 3. Practical Examples

[0045] 4. Example Hardware Configuration

[0046] 5. Notes

[0047] 6. System configuration changes

[0048] 7. Conclusion

[0049] 8. Exemplary Applications

[0050] <1. Overview>

[0051] First, the present disclosure is summarized below.

[0052] In conventional staining immunoassays, for example, antigens labeled by primary antibodies are visualized by depositing color dyes (color absorbing dyes) into the sample using enzyme-labeled secondary antibodies. When this method is used to visualize a large number of antigens, the types of enzymes that can be used are limited. Further, the types of visualized antigens are also limited depending on the resolution of the dye spectrum.

[0053] On the other hand, in the past, in fluorescent immunostaining, antigens were visualized by using fluorescently labeled secondary antibodies. When this method is used, the type of visualized antigens is limited depending on the type of antibody used as the primary antibody.

[0054] In order to address the above problems, the fluorescence amplification method is performed by using an enzyme-labeled secondary antibody. When using this method, a large amount of antigens are visualized by repeating the antigen-antibody reaction, depositing fluorescent dyes in the sample, and separating antibodies, but the type of the visualized antigen is limited by the type of the available enzyme, as in the case of conventional fluorescent immunostaining. However, as disclosed in NPL 1, when using the fluorescence amplification method, the time required for operation may increase with the increase of the number of repetitions of the cycle from antigen-antibody reaction to antibody separation. Further, during each repetition cycle, the sample may deteriorate.

[0055] In order to solve the above problems, a method has been developed to apply a reagent labeled with a direct fluorescent dye to a primary antibody or a fluorescent probe. This method allows the visualization of antigens or target molecules of all types of fluorescent dyes. However, this method only allows limited types of fluorescent dyes to be coupled to primary antibodies and fluorescent probes. Therefore, in some cases, the fluorescent signal to be measured may be hidden in the autofluorescence signal of the sample. Therefore, it is difficult to detect the fluorescent signal.

[0056] Specifically, in the case where tissue samples are used, differences in autofluorescence components occur between internal organs, and differences in autofluorescence intensity occur due to the fixed state of the tissue samples, making it more difficult to distinguish the fluorescence signal to be measured from the autofluorescence signal of the sample.

[0057] In conventional measurements of tissue samples after fluorescent labeling, in order to remove the autofluorescence signal originating from the sample, for example, the fluorescence signal of an unlabeled sample having the same or similar origin as the former sample is measured, and then, the measured fluorescence signal of the unlabeled sample is subtracted from the fluorescence signal obtained from the tissue sample. However, for example, the spectrum of autofluorescence and the spatial distribution of the spectrum are different depending on the state of the sample, the method of sample storage, and the method of sample preparation. Therefore, in the case where this method is used, it is necessary to measure the fluorescence signal of the unlabeled sample in each measurement and perform a subtraction process by using the fluorescence signal. Therefore, a huge burden is imposed on practitioners.

[0058] In view of the above situation, the inventors of the present disclosure created the technology according to the present disclosure. The inventors of the present disclosure proposed the technology according to the present disclosure based on the discovery that the spectrum of the autofluorescence component included in the autofluorescence spectrum remains unchanged, while the overall autofluorescence spectrum of the sample changes due to the method of fixing the sample and the fluorescence signal composition ratio of each component is extremely important.

[0059] More specifically, based on the information about the sample and the information about the fluorescent reagent, the information processing apparatus 100 according to the present disclosure separates the autofluorescence signal of the sample and the fluorescent signal of the fluorescent reagent from the image information about the sample stained by at least one fluorescent reagent. Further, in the case where the sample is stained by two or more fluorescent reagents, the information processing apparatus 100 separates the fluorescent signal of each fluorescent reagent from the image information based on the information about the sample and the information about the fluorescent reagent.

[0060] Here, "information about a sample" (hereinafter referred to as "sample information") refers to information including measurement channels and spectrum information unique to the autofluorescence components included in the sample. "Measurement channel" refers to a concept representing one or more elements included in an autofluorescence signal. More specifically, an autofluorescence spectrum is a mixture of fluorescence spectra possessed by one or more autofluorescence components. For example, Figure 1 Alexa Fluor 488 (in Figure 1 Alexa Fluor 555 (in Figure 1 , denoted as "Alexa 555"), and Alexa Fluor 647 (in Figure 1 The fluorescence spectrum includes autofluorescence when Alexa 647 (indicated as "Alexa 647") is used as a fluorescent reagent. NADH (nicotinamide adenine dinucleotide reduced form), FAD (flavin adenine dinucleotide), and porphyrin are included as autofluorescence components. Figure 1The autofluorescence spectrum includes a mixture of fluorescence spectra possessed by these autofluorescence components. Figure 1 In the embodiment of the present invention, the measurement channel according to the present invention is a concept indicating NADH, FAD, and porphyrin included in the autofluorescence spectrum, and a total of three measurement channels are used (that is, the measurement channel according to the present invention is information indicating the autofluorescence components included in the sample). Since the number of autofluorescence components varies from one sample to another, the measurement channel is associated with each sample and managed as sample information. Further, the "spectral information" included in the sample information refers to information about the autofluorescence spectrum possessed by each autofluorescence component included in each sample. It should be noted that the content of the sample information is not limited to the above-mentioned cases. The sample information will be described in detail later.

[0061] "Information about fluorescent reagents" (hereinafter referred to as "reagent information") refers to information including spectral information about fluorescent components included in the fluorescent reagents. The "spectral information" included in the reagent information refers to information about the fluorescence spectrum possessed by each fluorescent component included in each fluorescent reagent. It should be noted that the content of the reagent information is not limited to the above-mentioned cases. The reagent information will be described in detail later.

[0062] The information processing device 100 according to the present disclosure can extract the fluorescence signal of the fluorescent reagent by estimating the autofluorescence signal from the sample according to the measurement channel and spectral information included in the sample information and removing the autofluorescence signal from the image information. For example, the information processing device 100 can extract the fluorescence signal provided by NADH, FAD, and porphyrin as autofluorescence components by removing the autofluorescence signal from the image information. Figure 2 Fluorescent signals provided by Alexa Fluor 488, Alexa Fluor 555, and Alexa Fluor 647 as indicated in FIG.

[0063] As described above, for the purpose of performing subtraction processing on the autofluorescence signal, the information processing device 100 according to the present disclosure does not need to measure the fluorescence signal of the unlabeled sample. This reduces the burden on practitioners. Further, the information processing device 100 can separate the autofluorescence signal and the fluorescence signal with higher accuracy by using the sample information and the reagent information instead of using any one of the sample information and the reagent information.

[0064] Further, in order to obtain the fluorescence signal to be measured, a method of amplifying the fluorescence signal until the autofluorescence signal originating from the sample is negligible is generally and widely used. However, when this method is used, it is difficult to obtain the autofluorescence signal originating from the sample. At the same time, the information processing device 100 according to the present disclosure is capable of extracting the autofluorescence signal originating from the sample from the image information. Therefore, based on the distribution of the autofluorescence signal and the fluorescence signal in the image information, the information processing device 100 is capable of performing segmentation (or regional segmentation) of regions for identifying objects (e.g., cells, intracellular structures (cytoplasm, cell membrane, or cell nucleus), or tissues (tumor sites, non-tumor sites, connective tissues, blood vessels, vascular walls, lymphatic vessels, fibrotic structures, or necrosis)) included in the image information.

[0065] Furthermore, since the overall autofluorescence signal of the sample changes depending on the fixation state of the sample (e.g., fixation method), the information processing apparatus 100 can analyze (estimate) the fixation state of the sample based on the extracted autofluorescence signal. More specifically, the information processing apparatus 100 can analyze the fixation state of the sample based on the autofluorescence signal composition ratio of each of the two or more autofluorescence emission components.

[0066] <2. Implementation Method>

[0067] The present disclosure has been summarized above. Embodiments of the present disclosure will now be described.

[0068] (2.1. Example Configuration)

[0069] First, refer to Figure 3 An exemplary configuration of an information processing system according to the present embodiment is described. Figure 3 As shown in FIG. 1 , the information processing system according to the present embodiment includes an information processing apparatus 100 and a database 200. There are a fluorescent reagent 10, a sample 20, and a fluorescently colored sample 30 as inputs to the information processing system.

[0070] (Fluorescent reagent 10)

[0071] Fluorescent reagent 10 is a chemical used to color sample 20. For example, fluorescent reagent 10 is a fluorescent antibody (including a primary antibody for direct labeling or a secondary antibody for indirect labeling), a fluorescent probe, or a nuclear staining reagent. However, fluorescent reagent 10 is not limited to this type. Further, for the purpose of management, identification information (hereinafter referred to as "reagent identification information 11") that can identify fluorescent reagent 10 (and the production batch of fluorescent reagent 10) is attached to fluorescent reagent 10. For example, reagent identification information 11 is barcode information (for example, one-dimensional barcode information and two-dimensional barcode information). However, reagent identification information 11 is not limited to this information. For example, even if the fluorescent reagent 10 of the same product (product of the same type) is produced, its properties are different for different production batches according to the state of cells when producing methods and obtaining antibodies. For example, with respect to spectral information, quantum yield, or fluorescent labeling rate (also referred to as F / P value or fluorescein / protein value and indicating the number of fluorescent molecules used for antibody labeling), fluorescent reagent 10 is different from one production batch to another production batch. Therefore, in the information processing system according to the present embodiment, in order to manage the fluorescent reagent 109 based on each production batch (that is, to manage the reagent information about each fluorescent reagent 10 based on each production batch), the reagent identification information 11 is attached to the fluorescent reagent 10. This enables the information processing device 100 to separate the fluorescent signal and the autofluorescence signal by additionally considering the slight property difference between the production batches. It should be noted that managing the fluorescent reagent 10 based on each production batch is only an embodiment. Alternatively, the fluorescent reagent 10 can be managed in units smaller than the production batch.

[0072] (Sample 20)

[0073] For example, for the purpose of pathological diagnosis or clinical examination, the sample 20 is made from a specimen sample or tissue sample taken from a human body. The sample 20 is not limited to a specific type of sample tissue (e.g., viscera or cells), the disease of the object, the attributes of the object (e.g., age, gender, blood type, or race), or the lifestyle of the object (e.g., eating habits, exercise habits, or smoking habits). Further, for the purpose of sample management, identification information (hereinafter referred to as "sample identification information 21") that can identify each sample 20 is attached to the sample 20. As in the case of the reagent identification information 11, for example, the sample identification information 21 is barcode information (e.g., one-dimensional barcode information and two-dimensional barcode information). However, the sample identification information 21 is not limited to this information. For example, the properties of the sample 20 change due to the type of sample tissue, the type of disease of the object, the attributes of the object, or the lifestyle of the object. For example, with respect to the sample 20, the measurement channel or sample information is different due to, for example, the type of sample tissue. Accordingly, in the information processing system according to the present embodiment, in order to manage the sample 20 individually, the sample identification information 21 is attached to the sample 20. This enables the information processing apparatus 100 to separate the fluorescence signal and the autofluorescence signal by additionally taking into account minute differences in properties between the individual samples 20 .

[0074] (Fluorescent coloring sample 30)

[0075] The fluorescent colored sample 30 is produced by allowing the fluorescent reagent 10 to color the sample 20. The present embodiment assumes that the fluorescent colored sample 30 is produced by allowing at least one fluorescent reagent 10 to color the sample 20. However, the number of fluorescent reagents 10 used for coloring is not particularly limited. Further, for example, the method of coloring is determined according to the combination of the sample 20 and the fluorescent reagent 10, and is not particularly limited. The fluorescent colored sample 30 is input to the information processing device 100 and then imaged.

[0076] (Information Processing Device 100)

[0077] like Figure 3 As shown in FIG. 1 , the information processing apparatus 100 includes an acquisition section 110 , a storage section 120 , a processing section 130 , a display section 140 , a control section 150 , and an operation section 160 .

[0078] (Acquisition Unit 110)

[0079] The acquisition unit 110 acquires information used in various processes of the information processing device 100. Figure 3 As shown in FIG. 1 , the acquisition unit 110 includes an information acquisition unit 111 and an image acquisition unit 112 .

[0080] (Information Acquisition Unit 111)

[0081] The information acquisition section 111 acquires reagent information and sample information. More specifically, the information acquisition section 111 acquires reagent identification information 11 attached to the fluorescent reagent 10 used to make the fluorescent colored sample 30 and sample identification information 21 attached to the sample 20. For example, the information acquisition device 111 uses a barcode reader to acquire the reagent identification information 11 and the sample identification information 21. Then, the information acquisition section 111 acquires the reagent information from the database 200 based on the reagent identification information 11 and acquires the sample information from the database 200 based on the sample identification information 21. The information acquisition section 111 stores the acquired information in the information storage section 121 described later.

[0082] (Image Acquisition Unit 112)

[0083] The image acquisition section 112 acquires image information about the fluorescently stained sample 30 (the sample 20 stained by at least one fluorescent reagent 10). More specifically, the image acquisition section 112 includes a predetermined imaging element (e.g., a CCD or CMOS sensor), and acquires image information by imaging the fluorescently stained sample 30 using the imaging element. Here, it should be noted that "image information" refers to a concept including not only an image of the fluorescently stained sample 30, but also measurements or other values ​​that are not visualized as an image. For example, the image information may include information about the wavelength of the spectrum of fluorescence radiated from the fluorescently stained sample 30 (hereinafter referred to as a fluorescence spectrum). The image acquisition section 112 stores the image information in the image information storage section 122 described later.

[0084] (Storage Unit 120)

[0085] The storage unit 120 stores (memorizes) information used in various processes of the information processing device 100 or output through various processes. Figure 3 As shown in FIG. 1 , the storage unit 120 includes an information storage unit 121 , an image information storage unit 122 , and an analysis result storage unit 123 .

[0086] (Information Storage Unit 121)

[0087] The information storage section 121 stores the reagent information and sample information acquired by the information acquisition section 111. It should be noted that after terminating the analysis process performed by the analysis section 131 described later and the image information generation process (image information reconstruction process) performed by the image generation section 132 described later, the information storage section 121 can increase its available capacity by deleting the reagent information and sample information used for the processing.

[0088] (Image Information Storage Unit 122)

[0089] The image information storage section 122 stores image information about the fluorescent-stained sample 30 acquired by the image acquisition section 112. As in the case of the information storage section 121, it should be noted that after the analysis process performed by the analysis section 131 and the image information generation process (image information reconstruction process) performed by the image generation section 132 are terminated, the image information storage section 122 can enhance its available capacity by deleting the image information used for the processing.

[0090] (Analysis Result Storage Unit 123)

[0091] The analysis result storage unit 123 stores the results of the analysis process performed by the analysis unit 131 described later. For example, the analysis result storage unit 123 stores the fluorescence signal of the fluorescent reagent separated by the analysis unit 131 or the autofluorescence signal of the sample 20. Further, for example, the analysis result storage unit 123 provides the results of the analysis process to the database 200 to provide improved analysis accuracy through separated machine learning. It should be noted that after providing the results of the analysis process to the database 200, the analysis result storage unit 123 can increase its available capacity by appropriately deleting the results of the analysis process stored therein.

[0092] (Processing Unit 130)

[0093] The processing unit 130 has a function of performing various processes by using image information, reagent information, and sample information. Figure 3 As shown in FIG. 1 , the processing unit 130 includes an analyzing unit 131 and an image generating unit 132 .

[0094] (Analysis Department 131)

[0095] The analysis unit 131 performs various analysis processes by using the image information, the sample information, and the reagent information. For example, the analysis unit 131 performs a process of separating the autofluorescence signal of the sample 20 and the fluorescence signal of the fluorescent reagent 10 from the image information based on the sample information and the reagent information.

[0096] More specifically, the analysis section 131 identifies one or more elements of the autofluorescence signal according to the measurement channel included in the sample information. For example, the analysis section 131 identifies one or more autofluorescence components of the autofluorescence signal. Then, the analysis section 131 predicts the autofluorescence signal included in the image information by using the spectral information about the autofluorescence component included in the sample information. Subsequently, based on the spectral information about the fluorescence component of the fluorescent reagent 10 included in the reagent information and based on the predicted autofluorescence signal, the analysis section 131 separates the autofluorescence signal and the fluorescence signal from the image information.

[0097] Here, in the case where the sample 20 is colored by two or more fluorescent reagents 10, the analyzing section 131 separates the fluorescence signal of each of the two or more fluorescent reagents 10 from the image information (or from the fluorescence signal separated together with the autofluorescence signal) based on the sample information and the reagent information. For example, the analyzing section 131 uses the spectrum information about the fluorescence component of each fluorescent reagent 10 included in the reagent information to separate the fluorescence signal of each fluorescent reagent 10 from the entire fluorescence signal after separation together with the autofluorescence signal.

[0098] Further, in the case where the autofluorescence signal includes two or more autofluorescence components, the analyzing section 131 separates the autofluorescence signal of each autofluorescence component from the image information (or from the autofluorescence signal separated together with the fluorescence signal) according to the sample information and the reagent information. For example, the analyzing section 131 uses the spectrum information about each autofluorescence component included in the sample information to separate the autofluorescence signal of each autofluorescence component from the entire autofluorescence signal separated together with the fluorescence signal.

[0099] After separating the fluorescence signal and the autofluorescence signal, the analysis section 131 performs various processes using these signals. For example, the analysis section 131 can perform subtraction processing (also referred to as "background subtraction processing") on the image information about different samples 20 using the separated autofluorescence signal to extract the fluorescence signal from the image information about the different samples 20. For example, in the case where there are a plurality of samples 20 that are the same or similar in terms of the tissue used as the sample 20, the type of the disease of the subject, the attribute of the subject, or the lifestyle of the subject, the autofluorescence signals of the samples 20 are likely to be similar to each other. For example, here, similar samples include tissue slices before staining of tissue slices to be stained (hereinafter referred to as slices), slices adjacent to the stained slices, slices different from the stained slices in the same block (sampled from the same place as the position of the stained slices), slices in different blocks of the same tissue (sampled from different places from the stained slices), slices collected from different patients, and the like. Therefore, in the case where the autofluorescence signal is extracted from a specific sample 20, the analysis section 131 can extract the fluorescence signal from the image information about the different samples 20 by removing the autofluorescence signal from the image information about the different samples 20. Further, when calculating the S / N value by using the image information about the different samples 20 , the analyzing section 131 can improve the S / N value by using the background after removing the autofluorescence signal.

[0100] Moreover, the analysis unit 131 can not only perform background subtraction processing but also perform various other processing by using the separated fluorescence signal or autofluorescence signal. For example, the analysis unit 131 can analyze the fixed state of the sample 20 by using the above-mentioned signal and perform segmentation (or regional segmentation) of the region for identifying the object (e.g., cell, intracellular structure (cytoplasm, cell membrane, or cell nucleus), or tissue (tumor site, non-tumor site, connective tissue, blood vessel, blood vessel wall, lymphatic vessel, fibrotic structure, or necrosis)) included in the image information. The analysis and segmentation of the fixed state of the sample 20 will be described in detail later.

[0101] (Image Generating Unit 132)

[0102] The image generating unit 132 generates (reconstructs) image information based on the fluorescence signal or autofluorescence signal separated by the analyzing unit 131. For example, the image generating unit 132 can generate image information including only the fluorescence signal and image information including only the autofluorescence signal. In this case, if the fluorescence signal includes multiple fluorescence components or the autofluorescence signal includes multiple autofluorescence components, the image generating unit 132 can generate image information for the individual components. Further, in the case where the analyzing unit 131 performs various processes (e.g., analysis of the fixed state of the sample 20, segmentation, or calculation of the S / N value) by using the separated fluorescence signal or autofluorescence signal, the image generating unit 132 can generate image information indicating the result of such processing. According to the present configuration, the distribution information of the fluorescent reagent 10 labeled to the target molecule or the like (i.e., the secondary diffusion of fluorescence, or the intensity, wavelength and positional relationship) is visualized, and in particular, in the tissue image analysis area where the information of the target substance is relatively complex, the visibility of the doctor or researcher as a user can be improved.

[0103] The image generation section 132 can be controlled to distinguish the fluorescence signal from the autofluorescence signal based on the fluorescence signal or the autofluorescence signal separated by the analysis section 131, thereby generating image information. Specifically, the image generation section 132 can generate image information by controlling to improve the brightness of the fluorescence spectrum of the fluorescent reagent 10 labeled to the target molecule, etc., extract only the fluorescence spectrum labeled with the fluorescent reagent 10 and fade the fluorescent reagent 10, extract the fluorescence spectrum of each of two or more fluorescent reagents 10 from the sample 20 labeled with two or more fluorescent reagents 10 and fade the fluorescent reagents 10 with different colors, extract only the autofluorescence spectrum of the sample 20 and apply division or subtraction, improve the dynamic range, etc. This allows the user to clearly distinguish the color information derived from the fluorescent reagent coupled to the intended target substance.

[0104] (Display unit 140)

[0105] The display section 140 presents the image information generated by the image generation section 132 to the user by displaying the image information on the display. It should be noted that the type of display used as the display section 140 is not particularly limited. Further, although the details are not described in conjunction with the present embodiment, the image information generated by the image generation section 132 can be presented to the user by allowing a projector to project the image information or by allowing a printer to print the image information (i.e., the method of outputting the image information is not particularly limited).

[0106] (Control Unit 150)

[0107] The control section 150 has a function of providing overall control of all processes performed by the information processing device 100. For example, based on the operation input by the user through the operation section 160, the control section 150 controls the start and end of the various processes described above (e.g., the imaging process of the fluorescent staining sample 30, the analysis process, the image information generation process (image information reconstruction process), and the image information display process). It should be noted that the details of the control provided by the control section 150 are not specifically limited. For example, the control section 150 can control the processing performed by a general-purpose computer, a PC, or a tablet PC (e.g., a process involving an OS (operating system)).

[0108] (Operation unit 160)

[0109] The operation section 160 receives an operation input from a user. More specifically, the operation section 160 includes various input devices such as a keyboard, a mouse, a button, a touch panel, or a microphone, and allows the user to perform various inputs on the information processing device 100 by operating these input devices. Information about the operation input through the operation section 160 is provided to the control section 150.

[0110] (Database 200)

[0111] The database 200 is a device for managing sample information, reagent information, and results of the analysis process. More specifically, the database 200 performs management by associating the sample identification information 21 with the sample information and the reagent identification information 11 with the reagent information. This enables the information acquisition section 111 to acquire sample information from the database 200 based on the sample identification information 21 about the sample 20 to be measured and acquire reagent information from the database 200 based on the reagent identification information 11 about the fluorescent reagent 10.

[0112] As described previously, the sample information managed by the database 200 includes measurement channels and spectral information unique to the autofluorescence components included in the sample 20. However, in addition to the above information, the sample information may also include target information about each sample 20, specifically, information about the type of sample tissue (e.g., viscera, cells, blood, body fluids, ascites, pleural effusion, etc.), the type of the subject's disease, the subject's attributes (e.g., age, gender, blood type, or race), or the subject's lifestyle (e.g., eating habits, exercise habits, or smoking habits), and for each sample, information including measurement channels and spectral information unique to the autofluorescence components included in the sample 20 and target information may be associated with each other. This makes it easy to track information including measurement channels and vector information unique to the autofluorescence components included in the sample 20 from the target information. For example, the analysis section becomes able to perform a separation process similar to the process performed previously from the similarity of target information between a plurality of samples 20 to reduce the measurement time. It should be noted that the "tissue used" is not particularly limited to tissue collected from a target and may include in vivo tissues and cell lines of humans and animals as well as solutions, solvents, solutes, and materials contained in the target item of measurement.

[0113] Further, as described previously, the reagent information managed by the database 200 includes spectral information about the fluorescent reagent 10. However, in addition to the above information, the reagent information may also include various other information about the fluorescent reagent 10, such as information about production batches, fluorescent components, antibodies, clones, fluorescent labeling rates, quantum yields, fading coefficients (information indicating the ease of reducing the fluorescence intensity of the fluorescent reagent 10), and absorption cross-sections (or molar absorption coefficients). Further, the spectral information and reagent information managed by the database 200 may be managed by configurations different from each other, and in particular, the information about the reagents may be a reagent database that presents the best combination of reagents to the user.

[0114] Here, for example, it is assumed that sample information and reagent information are supplied from a manufacturer (manufacturer) or sample information and reagent information are uniquely measured in an information processing system according to the present disclosure. For example, in many cases, the manufacturer of fluorescent reagent 10 does not measure and supply spectral information and fluorescent labeling rate about each production batch. Therefore, the separation accuracy of fluorescent signal and autofluorescence signal can be improved by uniquely measuring and managing the above-mentioned information in an information processing system according to the present disclosure. Further, for simplifying management, for example, database 200 can use the catalog value announced by the manufacturer (maker) or the document value described in various documents as sample information and reagent information (specifically, as reagent information). However, in many cases, actual sample information and reagent information are usually different from catalog value and document value. Therefore, preferably, sample information and reagent information are uniquely measured and managed in an information processing system according to the present disclosure described above.

[0115] Further, for example, by using the machine learning technology of the sample information, reagent information, and the result of the analysis process managed by the database 200, the accuracy of the analysis process (for example, the process of separating the fluorescence signal and the autofluorescence signal) can be improved. For example, although the agent using the machine learning technology for learning purposes is not specifically limited, however, the present embodiment is described below with reference to an exemplary case, in which the learning is performed by the analysis unit 131 of the information processing device 100. For example, the analysis unit 131 uses a neural network to generate a classifier or an estimator for machine learning by using learning data, wherein the separated fluorescence signal and the autofluorescence signal are obtained by associating the image information, sample information, and reagent information used for separation. Subsequently, in the case where the image information, sample information, and reagent information are newly acquired, the analysis unit 131 can predict and output the fluorescence signal and the autofluorescence signal included in the image information by inputting the newly acquired information into the classifier or the estimator.

[0116] Further, the analysis unit 131 can output such a method, that is, a similar separation process (a separation process using similar image information, sample information, or reagent information) performed before calculation, which is more accurate than the predicted fluorescence signal and autofluorescence signal; statistically or recursively analyzing the details of the calculated process (e.g., the information and parameters used for processing); and improving the process of separating the fluorescence signal and the autofluorescence signal according to the analysis results. It should be noted that the machine learning method is not limited to the method described above. Well-known machine learning techniques can be used. Moreover, artificial intelligence can be used to perform the process of separating the fluorescence signal and the autofluorescence signal. In addition, machine learning or other techniques can be used to improve not only the process of separating the fluorescence signal and the autofluorescence signal, but also various other processes (e.g., analysis or segmentation of the fixed state of the sample 20) using the separated fluorescence signal and the autofluorescence signal.

[0117] An exemplary configuration of the information processing system according to the present embodiment has been described above. Figure 3 The configuration described is only an example. The configuration of the information processing system according to the present embodiment is not limited to the configuration described above. For example, the information processing device 100 does not need to always include Figure 3 Further, the information processing apparatus 100 may include a database 200. The functional elements of the information processing apparatus 100 may be flexibly changed based on specifications and operations.

[0118] Further, the information processing device 100 can perform processes other than those described above. For example, as long as the reagent information includes information about the fluorescent reagent 10, such as quantum yield, fluorescent labeling rate, and absorption cross section (or molar absorption coefficient), the information processing device 100 can calculate the number of fluorescent molecules in the image information and the number of antibodies coupled to the fluorescent molecules by using the reagent information and the image information from which the autofluorescence signal is removed. More specifically, in the case where an imaging element (e.g., a CMOS sensor) performs an imaging process on a fluorescently colored sample 30, the number of fluorescent molecules in a specific pixel A is expressed by the following equation 1 or equation 2. Further, the number of absorbed photons in equation 2 is expressed by the following equation 3.

[0119] [Mathematical formula 1]

[0120]

[0121] [Mathematical formula 2]

[0122] The number of emitted photons per molecule

[0123] = Number of absorbed photons × quantum yield

[0124] …(Equation 2)

[0125] [Mathematical formula 3]

[0126] The number of absorbed photons

[0127] = Excitation photon density × absorption cross section

[0128] …(Equation 3)

[0129] The information processing device 100 calculates the number of absorbed photons by actually measuring the excitation photon density in Equation 3 and multiplying the excitation photon density by the absorption cross section. Further, the information processing device 100 calculates the number of emitted photons per molecule according to Equation 2 by multiplying the number of absorbed photons by the quantum yield in the reagent information. Then, the information processing device 100 calculates the number of fluorescent molecules in pixel A according to Equation 1 by dividing the number of photons detected by pixel A by the number of emitted photons per molecule. Subsequently, the information processing device 100 is able to calculate the number of fluorescent molecules in each pixel by performing the above calculation on all pixels.

[0130] Furthermore, the information processing apparatus 100 can convert the number of fluorescent molecules detected by the pixel A into the number of antibodies by performing the calculation expressed by Equation 4 using the fluorescent labeling rate in the reagent information. Subsequently, the information processing apparatus 100 can calculate the number of antibodies in each pixel by performing the operation expressed by Equation 4 on all pixels.

[0131] [Formula 4]

[0132]

[0133] It should be noted that when performing the process of calculating the number of fluorescent molecules and the number of antibodies, as long as the reagent information includes a fading coefficient indicating the difficulty of reducing the fluorescence intensity, the information processing device 100 can correct the brightness in the image information to the non-faded brightness. This enables the information processing device 100 to more accurately calculate the number of fluorescent molecules and the number of antibodies by considering the influence of the decayed fluorescent components. The process of calculating the number of fluorescent molecules and the number of antibodies can be implemented by the analysis unit 131 of the information processing device 100. However, the calculation process is not limited to the above process. For example, the information processing device 100 can present image information reflecting information about the calculated number of fluorescent molecules and antibodies to the user.

[0134] (2.2. Processing flow)

[0135] An exemplary configuration of the information processing system according to the present embodiment has been described above. Figure 4 and Figure 5An exemplary flow of a process executed by the information processing apparatus 100 is described.

[0136] In step S1000 , the user determines the fluorescent reagent 10 and the sample 20 to be used for analysis. In step S1004 , the user prepares the fluorescent-stained sample 30 by staining the sample 20 with the fluorescent reagent 10 .

[0137] In step S1008, the image acquisition section 112 of the information processing apparatus 100 acquires image information by imaging the fluorescently stained sample 30. In step S1012, the information acquisition section 111 acquires reagent information and sample information from the database 200 based on the reagent identification information 11 attached to the fluorescent reagent 10 used to generate the fluorescently stained sample 30 and based on the sample identification information 21 attached to the sample 20.

[0138] In step S1016, the analysis section 131 separates the autofluorescence signal of the sample and the fluorescence signal of the fluorescent reagent 10 from the image information based on the sample information and the reagent information. Here, in the case where the fluorescence signal includes signals of a plurality of fluorescent dyes ("Yes" in step S1020), the analysis section 131 separates the fluorescence signal of each fluorescent dye in step S1024. It should be noted that in the case where the fluorescence signal does not include signals of a plurality of fluorescent dyes ("No" in step S1020), the analysis section 131 does not perform the process of separating the fluorescence signal of each fluorescent dye in step S1024.

[0139] In step S1028, the image generation section 132 generates image information by using the fluorescence signal separated by the analysis section 131. For example, the image generation section 132 generates image information by removing the autofluorescence signal in the image information or generates image information indicating the fluorescence signal of each fluorescent dye. In step S1032, the display section 140 displays the image information generated by the image generation section 132. A series of processes are terminated here.

[0140] It should be noted that Figure 4 and Figure 5 The steps in the flowchart do not need to always be executed in the described chronological order. That is, the steps in the flowchart may be executed in an order other than the described order or in parallel. For example, after separating the autofluorescence signal of the sample and the fluorescence signal of the fluorescent reagent 10 from the image information in step S1016, the analysis unit 131 may immediately separate the fluorescence signal of each fluorescent dye from the image information, rather than separating the fluorescence signal of each fluorescent dye in step S1024 (further, after separating the fluorescence signal of each fluorescent dye from the image information, the analysis unit 131 may separate the autofluorescence signal of the sample from the image information). Furthermore, in addition, the information processing device 100 may execute Figure 4 or Figure 5 For example, the analysis section 131 may not only separate the signal but also perform segmentation or analyze the fixed state of the sample 20 based on the separated fluorescence signal or autofluorescence signal.

[0141] <3. Practical Example>

[0142] (3.1. Practical Example of Autofluorescence Signal Removal)

[0143] An exemplary flow of various processes performed by the information processing apparatus 100 has been described above. A practical example of autofluorescence signal removal will now be described.

[0144] Figure 6 An example of image information in the case where K562 (ie, chronic myeloid leukemia cell line) and Jurkat (ie, human T cell-derived cell line) are used as the sample 20 is shown. More specifically, Figure 6 A-1 in FIG. 1 describes image information about a fluorescently colored sample 30 colored by a fluorescent reagent 10, namely, AF488-β-tubulin, AF555-E-cadherin, AF647-CD3, and PE-Cy5-CD45. Further, Figure 6 A-2 in FIG. 1 describes image information about the unstained sample 20 (image information describing only the autofluorescence signals of the autofluorescence components (ie, NADH, FAD, and porphyrin)).

[0145] on the other hand, Figure 6 B-1 and B-2 in A-2 respectively describe image information about the fluorescently stained sample 30 and image information about the sample 20, which are obtained after allowing the analysis section 131 to separate the fluorescent signal and the autofluorescence signal and remove the autofluorescence signal from the image information. (As described above, the image information described in A-2 includes only the autofluorescence signal, and the image information described in B-2 does not include the signal due to the removal of the autofluorescence signal).

[0146] Reference Figure 7 9 , a practical example of autofluorescence signal removal will be described with respect to a case in which a formalin-fixed paraffin-embedded (FFPE: formalin-fixed paraffin-embedded) breast cancer slide is used as the sample 20 .

[0147] Figure 7 Image information about the fluorescently stained sample 30 from which the autofluorescence signal has not been removed is described. Fig. 8ASpectral information about fluorescence signals and autofluorescence signals is described. The embodiment in FIG8 describes spectral information about Alexa Fluor 488 (denoted as “Alexa 488” in the figure), background (denoted as “background” in the figure), tissue (denoted as “tissue” in the figure), and red blood cells (denoted as “blood cells” in the figure). Figure 8B Image information in which the signal of the unstained sample 20 (the background, tissue, and autofluorescence signal of red blood cells) is measured and then removed from the image information about the fluorescent stained sample 30 (i.e., the case in which the autofluorescence signal is removed by conventional methods) is described as a comparison object.

[0148] on the other hand, Fig.9A Describes the use of Alexa Fluor 488 (in Fig.9A , represented by "Alexa 488"), NADH, FAD, and porphyrin spectral information. Fig. 9B Image information in the case where the analyzing section 131 of the information processing apparatus 100 according to the present disclosure removes autofluorescence signals (autofluorescence signals about NADH, FAD, and porphyrin) from image information about the fluorescent-stained sample 30 by using sample information and reagent information is described.

[0149] Obviously, the above description is for illustrative purposes only. The technology according to the present disclosure can be applied to any fluorescent reagent 10 and any sample 20.

[0150] (3.2. Practical Example of Separating Fluorescence Signals from Each Other)

[0151] Further, as described above, by calculating the color separation of the autofluorescence signal and the fluorescence signal from the image information obtained by coloring the sample 20 using the antibody molecule and imaging the fluorescent colored sample 30, a specific coloring can be confirmed, and the antibody molecule is obtained by separately coloring multiple labeled molecules using multiple fluorescent dyes.

[0152] Therefore, in the present practical embodiment, not only the separation of the autofluorescence signal from the fluorescence signal and the separation of the autofluorescence signals from each other but also the separation of the fluorescence signals from each other are described using specific examples.

[0153] Fig.10 2 is a diagram describing an embodiment of image information in which tonsil tissue is used as a sample 20 and an autofluorescence signal obtained from the tonsil tissue is used as an autofluorescence signal used for color separation calculation when extracting a fluorescence signal from the image information. Specifically, Fig.10(A1) to (A6) describe images of autofluorescence signals separated from image information (hereinafter referred to as autofluorescence images), and (B1) to (B8) describe images of fluorescence signals originating from fluorescent reagent 10 separated from image information (hereinafter referred to as fluorescence images).

[0154] It should be noted that Fig.10 In the described embodiment, a case is indicated in which human tonsil tissue used as a sample 20 is stained using eight different fluorescent reagents 10. As the eight different fluorescent reagents 10, in the present embodiment, fluorescent dye DAPI, which is used in principle in nuclear staining, labeled antibody CD4 labeled with fluorescent dye Alexa Fluor 488, labeled antibody CD3 labeled with fluorescent dye Alexa Fluor 555, labeled antibody Ki-67 labeled with fluorescent dye Alexa Fluor 568, labeled antibody PDL-1 labeled with fluorescent dye Alexa Fluor 594, labeled antibody PD-1 labeled with fluorescent dye Alexa Fluor 647, labeled antibody CD8 labeled with fluorescent dye Alexa Fluor 680, and labeled antibody CD68 labeled with fluorescent dye Alexa Fluor 700 are used.

[0155] exist Fig.10 In (A1) to (A6) of FIG. 1 , six autofluorescence images derived from the sample 20 and separated from the image information are described. Fig.10 In (B1), a fluorescent image derived from the fluorescent dye DAPI and separated from the image information is described; in (B2), a fluorescent image derived from the fluorescent dye Alexa Fluor 488 (in Fig.10 In (B3), a fluorescence image derived from the fluorescent dye Alexa Fluor 555 (in Fig.10 In (B4), a fluorescence image derived from the fluorescent dye Alexa Fluor 568 (in Fig.10 In (B5), a fluorescence image derived from the fluorescent dye Alexa Fluor 594 (in Fig.10 In (B6), a fluorescence image derived from the fluorescent dye AlexaFluor 647 (in Fig.10In (B7), a fluorescence image derived from the fluorescent dye Alexa Fluor 680 (in Fig.10 , denoted as "AF680") and also separated from the image information; and in (B8), a fluorescence image derived from the fluorescent dye Alexa Fluor 700 (in Fig.10 , denoted as "AF700") and the fluorescence image also separated from the image information.

[0156] By reference Fig.10 (A1) to (A6) and (B1) to (B8) can recognize that, according to the present practical embodiment, even if a plurality of fluorescent reagents 10 are used to color the sample 20, the analysis unit 131 can separate the autofluorescence signal and the fluorescence signal from the image information obtained by the image acquisition unit 112 imaging the fluorescently colored sample 30 based on the spectrum information of the fluorescence component of the fluorescent reagent 10 and the autofluorescence signal included in the reagent information. Then, the analysis unit 131 can use the spectrum information of the fluorescence component of the fluorescent reagent 10 included in the reagent information to separate the fluorescence signal of the fluorescent reagent 10 from the entire fluorescence signal from which the autofluorescence signal is separated. Further, the analysis unit 131 can also use the spectrum information of the autofluorescence component included in the sample information to separate the individual autofluorescence signals of the autofluorescence component from the entire autofluorescence signal from which the fluorescence signal is separated.

[0157] For example, when the autofluorescence signal is separated from the fluorescence signal, a process of fitting the autofluorescence spectrum estimated based on the spectrum information included in the reagent information to the spectrum of the image information and subtracting the autofluorescence signal from the image information based on the coefficient in the fitting is performed.

[0158] Further, when the autofluorescence signals are separated from each other, a process is performed in which the spectrum of the fluorescent reagent 10 included in the reagent information is fitted to the spectrum of the image sensor after the autofluorescence signal is removed and the fluorescence signal of the fluorescent reagent 10 is extracted from the image information based on the coefficient during the fitting.

[0159] Further, also when the autofluorescence signals are separated from each other, a process is similarly performed in which the spectrum of the fluorescent reagent 10 included in the reagent information is fitted to the spectrum of the image information after the autofluorescence signal is removed and then the fluorescence signal of the fluorescent reagent 10 is extracted from the image information based on the coefficients during fitting.

[0160] Naturally, the above is always exemplary, and the technology according to the present disclosure can be applied to any fluorescent reagent 10 and any sample 20 .

[0161] (3.3. Practical Example of Autofluorescence Signal for Extracting Fluorescence Signal)

[0162] Subsequently, a practical embodiment for extracting an autofluorescence signal of a fluorescence signal is described. As described above, in a case where there are a plurality of samples 20 that are the same or similar in terms of tissue used as the sample 20, the kind of disease of the subject, the attribute of the subject, the lifestyle of the subject, etc., there is a high possibility that the autofluorescence signals of the samples 20 are similar to each other. This is considered to be similar not only to tissues collected from the same patient, but also to tissues collected from different patients.

[0163] Thus, the specific embodiments and advantageous effects achieved are described with respect to the case where, in the present practical embodiment, when extracting a fluorescence signal from image information, or in other words, when calculating a fluorescence signal and an autofluorescence signal for color separation, an autofluorescence signal obtained from a tissue that is the same as or similar to the sample 20 and an autofluorescence signal obtained from a tissue that is different from the sample 20 are used. It should be noted that, for example, the color separation process of extracting a fluorescence signal from image information may be a process corresponding to the “background subtraction process” described above.

[0164] In the description with reference to the actual embodiment Fig.10 , a case is illustrated in which tonsil tissue is used as the sample 20 and an autofluorescence signal obtained from the tonsil tissue is used as an autofluorescence signal used in a color separation process when extracting a fluorescence signal from an image signal. In contrast, Fig.11 An embodiment is described in which tonsil tissue is used as the sample 20 and an autofluorescence signal obtained from lung tissue (ie, tissue different from tonsil tissue) is used as image information in the case of the autofluorescence signal used in color separation calculation when extracting a fluorescence signal from an image signal.

[0165] It should be noted that Fig.10 Similar, in Fig.11 In FIG. 1 , six autofluorescence images originating from the sample 20 and separated from the image information are described in (A1) to (A6). Further, in Fig.11 In (B1), a fluorescent image derived from the fluorescent dye DAPI and separated from the image information is described; in (B2), a fluorescent image derived from the fluorescent dye Alexa Fluor 488 (in Fig.11 In (B3), a fluorescence image derived from the fluorescent dye Alexa Fluor 555 (in Fig.11In (B4), a fluorescence image derived from the fluorescent dye Alexa Fluor 568 (in Fig.11 In (B5), a fluorescence image derived from the fluorescent dye Alexa Fluor 594 (in Fig.11 In (B6), a fluorescence image derived from the fluorescent dye Alexa Fluor 647 (in Fig.11 In (B7), a fluorescence image derived from the fluorescent dye Alexa Fluor 680 (in Fig.11 , denoted as "AF680") and also separated from the image information; and in (B8), a fluorescence image derived from the fluorescent dye AlexaFluor 700 (in Fig.11 , denoted as "AF700") and the fluorescence image also separated from the image information.

[0166] from Fig.10 and Fig.11 Comparison between the two makes it possible to recognize that, compared with the case where the autofluorescence signal originated from a tissue different from the sample 20 (in the present embodiment, the lung tissue) is used, in the case where the autofluorescence signal originated from the same or similar tissue as the sample 20 (in the present embodiment, the tonsil tissue) is used for the color separation calculation, a clearer fluorescence image is obtained (reference Fig.10 (B1) to (B6) and Fig.11 (B1) to (B8) of FIG. 1 ). This indicates that the S / N ratio of the fluorescent image is improved because the autofluorescence signal with respect to the background is more accurately removed from the image information.

[0167] The following description focuses on the use of a labeled antibody CD4 labeled with the fluorescent dye Alexa Fluor 488 as a fluorescent reagent 10 ( Fig.10 (B1) and Fig.11 In the case of (B1)) and the case in which the labeled antibody PDL-1 labeled with the fluorescent dye AlexaFluor 594 is used as the fluorescent reagent 10 (reference Fig.10 (B5) and Fig.11 This is described in another case of (B5)).

[0168] It should be noted that in the present description, the background may be an area that is not colored by the fluorescent reagent 10 or a signal value of the area. Accordingly, the background may include autofluorescence signals and other noises before background subtraction processing is performed. On the other hand, after background subtraction processing is performed, the background may include autofluorescence signals that have not been completely removed, other noises, etc.

[0169] Fig.12 is a diagram showing a case where a labeled antibody CD4 labeled with a fluorescent dye Alexa Fluor 488 is used as the fluorescent reagent 10, and Fig.13 is a diagram showing a case where a labeled antibody PDL-1 labeled with a fluorescent dye Alexa Fluor 594 is used as the fluorescent reagent 10. Fig.12 and Fig.13 , (A1) describes a fluorescent image in which an autofluorescence signal originating from a tissue that is the same as or similar to sample 20 (in this embodiment, tonsil tissue) is used, and (A2) describes the pixel values ​​(grayscale values) along line X1-X2 in (A1). (B1) describes a fluorescent image in which an autofluorescence signal originating from a tissue that is different from sample 20 (in this embodiment, lung tissue) is used as the autofluorescence signal used for color separation calculation, and (B2) describes the pixel values ​​(grayscale values) along line X1-X2 in (B1). It should be noted that in Fig.12 and Fig.13 , X1 corresponds to the pixel (0) at the left end of the fluorescent image, and X2 corresponds to the pixel (1023) at the right end of the fluorescent image.

[0170] from Fig.12 and Fig.13 As is apparent from the comparison between (A1) and (B1), in the case where an autofluorescence signal originating from a tissue identical or similar to the sample 20 (in the present embodiment, tonsil tissue) is used as the autofluorescence signal used for the color separation calculation, the amount of the signal value (corresponding to the pixel value) of the background subtracted from the value of the image information is greater than in the case where an autofluorescence signal originating from a tissue different from the sample 20 (in the present embodiment, lung tissue) is used. Therefore, the S / N ratio in the fluorescence image is reduced and the obtained fluorescence image is clearer.

[0171] This is considered to be caused by the fact that the spectrum of autofluorescence originating from the same or similar tissue is closer to the autofluorescence signal of the sample 20 than the autofluorescence signal originating from a different tissue.

[0172] Thus, it is considered preferable to use an autofluorescence signal originating from the same or similar tissue as the sample 20 as the autofluorescence signal for color separation calculation. However, this does not exclude from the technical scope of the present disclosure the use of an autofluorescence signal originating from a tissue different from the sample 20 for color separation calculation.

[0173] On the other hand, for example, information on autofluorescence signals originating from tissues identical or similar to the sample 20 can be managed through the database 200. The information acquisition section 111 can acquire sample information of a sample identical or similar to the sample 20 from the database 200 based on the acquired sample identification information 21 and input the acquired sample information to the analysis section 131 through the information storage section 121. The analysis section 131 performs processes such as separation of autofluorescence signals from fluorescence signals, separation of fluorescence signals from each other, and separation of autofluorescence signals from each other based on the input sample information.

[0174] (3-4. Analysis of fixed state)

[0175] A practical embodiment of the autofluorescence signal removal has been described above. A practical embodiment of the analysis of the fixed state of the sample 20 will now be described.

[0176] As described previously, the entire autofluorescence signal of the sample 20 changes depending on the fixation state (e.g., fixation method) of the sample 20. Therefore, the information processing device 100 is capable of analyzing (evaluating) the fixation state of the sample 20 based on the autofluorescence signal component ratio of each of the two or more autofluorescence emission components. More specifically, as long as NADH (nicotinamide adenine dinucleotide reduced form), FAD (flavin adenine dinucleotide), and other autofluorescence components are involved, the fluorescence intensity changes according to fixation by formalin without substantially changing the shape of the autofluorescence spectrum. Therefore, with respect to the above-mentioned autofluorescence components, the information processing device 100 is capable of analyzing the fixation state (e.g., fixation method) of the sample 20 based on the fluorescence intensity ratio of each autofluorescence component separated from the image information.

[0177] Reference Figures 14 to 17 , the following describes changes in autofluorescence spectra in the case of fixing various autofluorescent components by formalin.

[0178] Fig.14 : is a diagram showing the autofluorescence spectra measured at the start of fixation (zero minute) and 15, 30, 45, and 60 minutes after fixation in the case where K562 (i.e., chronic myeloid leukemia cell line) is fixed by formalin having a concentration of 5%. With respect to K562 involved, Fig.14This means that as the period of time during which the fixation is provided by formalin becomes longer, the shape of the autofluorescence spectrum changes to reduce the fluorescence intensity. Note that “the shape of the autofluorescence spectrum changes” means that even if the autofluorescence spectra acquired at each time point shift in parallel, the autofluorescence spectra do not overlap with each other.

[0179] Fig.15 is a diagram showing the autofluorescence spectra measured at the beginning of fixation (zero minute) and 10, 30, 60, 90, and 120 minutes after fixation in the case of fixing NADPH by formalin having a concentration of 2%. In clear contrast to the above-mentioned K562, as far as NADPH is concerned, Fig.15 This means that even if the fixed time period provided by formalin becomes longer, the fluorescence intensity decreases but the shape of the autofluorescence spectrum does not substantially change. It should be noted that “the fluorescence intensity decreases without substantially changing the shape of the autofluorescence spectrum” means that when the fluorescence intensity of the autofluorescence spectrum acquired at each time point changes vertically (i.e., when the autofluorescence spectrum moves vertically in parallel), the autofluorescence spectra substantially overlap with each other.

[0180] Fig.16 : is a diagram showing the autofluorescence spectra measured at the start of fixation (zero minute) and 10, 30, 60, 90, and 120 minutes after fixation in the case where FAD (30 μM) is fixed by formalin having a concentration of 2%. With respect to FAD (30 μM) involved, Fig.16 This indicates that the shape of the autofluorescence spectrum and the fluorescence intensity do not change substantially even when the fixed period of time provided by formalin becomes longer.

[0181] Fig.17 : is a diagram showing the autofluorescence spectra measured at the beginning of fixation (zero minute) and 15, 30, and 60 minutes after fixation in the case where FAD (50 μM) is mixed with a cell extract and fixed by formalin having a concentration of 2%. With respect to FAD (50 μM) involved, Fig.17 This indicates that even if the fixed period of time provided by formalin becomes longer, the fluorescence intensity increases but the shape of the autofluorescence spectrum does not substantially change.

[0182] As Figures 15 to 17 As indicated in the examples in FIG. 1 , with respect to NADPH, FAD, and other autofluorescent components involved, the fluorescence intensity changes depending on the fixation state (e.g., fixation method), but the shape of the autofluorescence spectrum is not substantially changed (it should be noted that, as Fig.16 As indicated in , in some cases, the fluorescence intensity may also remain substantially unchanged). Here, when the sample information acquired by the information acquisition unit 111 from the database 200 includes Figures 15 to 17When the spectrum information described in is obtained, the analyzing section 131 can analyze (evaluate) the fixation state of the sample 20 based on the autofluorescence signal component ratio of each of the two or more autofluorescence emission components.

[0183] For example, the analysis section 131 identifies the autofluorescence components included in the sample 20 according to the measurement channel included in the sample information, and acquires spectrum information (which varies according to fixation) about each autofluorescence component as the sample information. Then, the analysis section 131 is able to analyze the fixation method (e.g., the chemical used for fixation, the concentration of the chemical, or the fixation process) and the elapsed time from fixation according to the spectrum information about each autofluorescence component acquired and according to the component ratio of each component of the autofluorescence signal separated from the image information.

[0184] Although the above description assumes that NADPH and FAD are autofluorescent components, the types of autofluorescent components applicable to the analysis in a fixed state are not limited to the above types. That is, an autofluorescent component whose shape of the autofluorescent spectrum remains substantially unchanged due to fixation can be applied to the analysis in a fixed state.

[0185] (3-5. Division)

[0186] A practical embodiment of the analysis of the fixed state of the sample 20 has been described above. The following describes segmentation (or region segmentation) for identifying a region of an object (e.g., a cell, an intracellular structure (cytoplasm, a cell membrane, or a cell nucleus), or a tissue (a tumor site, a non-tumor site, connective tissue, a blood vessel, a blood vessel wall, a lymphatic vessel, a fibrotic structure, or necrosis)) included in the image information.

[0187] In order to obtain the fluorescence signal to be measured, a method of amplifying the fluorescence signal until the autofluorescence signal originating from the sample 20 is negligible is generally and widely used. However, when this method is used, it is difficult to obtain the autofluorescence signal. Therefore, when the conventional method is used, it is difficult to identify the region of the cell included in the sample 20, the intracellular structure (cytoplasm, cell membrane, cell nucleus, etc.), or the tissue (tumor site, non-tumor site, connective tissue, blood vessel, blood vessel wall, lymphatic vessel, fibrotic structure, necrosis, etc.) by using the autofluorescence signal.

[0188] On the other hand, the analysis section 131 of the information processing device 100 can extract the autofluorescence signal from the image information about the fluorescent staining sample 30 according to the sample information and the reagent information. Therefore, according to the distribution of the autofluorescence signal and the fluorescence signal in the image information, the analysis section 131 can identify the region of the cell included in the sample 20, the intracellular structure (cytoplasm, cell membrane, cell nucleus, etc.), or the tissue (tumor site, non-tumor site, connective tissue, blood vessel, blood vessel wall, lymphatic vessel, fibrotic structure, necrosis, etc.), and perform segmentation. This enables the analysis section 131 to obtain position information about the target molecule and realize image information, such as positioning between different colorants.

[0189] In the past, a method has been proposed that performs segmentation by attaching a unique marker to each cell and each intracellular structure. Another method has been proposed that corrects the expression level of the target molecule in the cell membrane by labeling a molecule (endogenous control) that is different from the target molecule and is regularly expressed in the same region as the target molecule using different fluorescence. The autofluorescent component has a unique position. Therefore, using the positional information about the autofluorescent component obtained according to the present disclosure makes it possible to improve the accuracy of the processing while reducing the number of colorants and endogenous controls for conventional segmentation. It should be noted that the method of segmentation based on the technology disclosed in the present disclosure is not limited to the method described above.

[0190] <4. Exemplary Hardware Configuration>

[0191] The segmentation based on the technology according to the present disclosure has been described above. Fig.18 An exemplary hardware configuration of the information processing apparatus 100 is described.

[0192] Fig.18 90 is a block diagram showing an exemplary hardware configuration of the information processing apparatus 100. The information processing apparatus 100 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device (HDD) 910, a drive 911, and a communication device 912.

[0193] The CPU 901 functions as an algorithm processing device and a control device and provides overall control of operations in the information processing device 100 according to various programs. Further, the CPU 901 may be a microprocessor. For example, the ROM 902 stores programs and operating parameters used by the CPU 901. For example, the RAM 903 temporarily stores programs used when the CPU 901 executes and parameters that are appropriately changed when the execution. These components are interconnected by, for example, a host bus 904 including a CPU bus. The CPU 901, the ROM 902, and the RAM 903 cooperate to realize the functions of the processing unit 130 and the control unit 150.

[0194] The control program of the image processing apparatus according to the present technology is a program for causing the information processing apparatus to execute a function similar to the control function executed by the control section 150 of the information processing apparatus described above and is provided in a state of being stored in a recording medium such as, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory so that it is loaded into a PC or the like and used by the PC or the like. Alternatively, the control program transmitted from the outside through a network such as the Internet may also be downloaded into a PC or the like and used by the PC or the like.

[0195] Specifically, it can also be considered that the information processing system (for example, a microscope system) according to the present embodiment is composed of the information processing device 100 described above and a program configuration for enabling the information processing device 100 to perform control functions to separate fluorescence signals and / or autofluorescence signals from image information based on information about the sample 20 and information about the fluorescent reagent 10.

[0196] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, through a bridge 905. It should be noted that the host bus 904, the bridge 905, and the external bus 906 do not always need to be provided separately. Alternatively, a single bus may be used to implement the functions of the host bus 904, the bridge 905, and the external bus 906.

[0197] For example, the input device 908 includes an input control circuit used as an information input device of a user, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, a joystick, or various sensors (including an imaging element), etc., generates an input signal based on the user input and outputs the generated input signal to the CPU 901. The input device 908 implements some functions of the acquisition section 110 and the functions of the operation section 160.

[0198] For example, the output device 909 includes a display device such as a CRT (cathode ray tube) display device, a liquid crystal display (LCD) device, an OLED (organic light emitting diode) device, or a lamp. Further, the output device 909 includes an audio output device such as a speaker or earphone. The output device 909 displays various information such as video data as an image or text. At the same time, for example, the audio output device converts audio data into sound and outputs the sound. The output device 909 implements the function of the display unit 140.

[0199] The storage device 910 is a device for storing data. For example, the storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. For example, an HDD (hard disk drive) can be used as the storage device 910. In this case, the storage device 910 drives the hard disk to store the program executed by the CPU 901 and various data. The storage device 910 implements the function of the storage unit 120.

[0200] The drive 911 is a reader / writer for a storage medium, and is built in or externally attached to the information processing apparatus 100. The drive 911 reads information recorded on a removable storage medium 913 such as an inserted magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the read information to the RAM 903. Further, the drive 911 can write information to the removable storage medium 913.

[0201] For example, the communication device 912 is a communication interface including a communication device for connecting to the communication network 914. The communication device 912 realizes some functions of the acquisition section 110.

[0202] It should be noted that the hardware configuration of the information processing apparatus 100 is not limited to Fig.18 For example, in the case where communication is established through a connected external communication device, the information processing apparatus 100 does not need to include the communication apparatus 912. Further, the communication apparatus 912 is capable of establishing communication through a plurality of communication methods. Further, the communication apparatus 912 may be implemented in whole or in part through one or two or more ICs (Integrated Circuits). Fig.18 The configuration described in .

[0203] <5. Notes>

[0204] The above description indicates that sample information and reagent information are preferably measured and managed in a unique manner in the information processing system according to the present disclosure. The reason is that in many cases, the actual sample information and reagent information are different from the catalog values ​​and literature values. Therefore, reference is now made to Fig.19 and Fig. 20 , describe the differences between the actual sample information and reagent information and the catalog and literature values ​​as notes.

[0205] Fig.19 is a diagram showing a comparison result between a measured value indicating spectral information about PE (phycoerythrin) (ie, a type of fluorescent component) and a catalog value. Further, Fig. 20 is a diagram showing a comparison result between a measured value indicating spectral information about BV421 (Brilliant Violet 421) (ie, the kind of fluorescent component) and a catalog value. It should be noted that the measured value indicating the measurement result is obtained when a sample is prepared by mixing a fluorescent component with an encapsulant.

[0206] like Fig.19 and Fig. 20 As shown in , at the peak position indicated by the spectral information, the measured value and the catalog value are basically consistent with each other, but at wavelengths longer than the peak wavelength, the spectral shape is different. Therefore, using the catalog value as spectral information reduces the accuracy of separation of the fluorescence signal and the autofluorescence signal.

[0207] In terms of accuracy, it should be noted that, in general, within the information processing system according to the present disclosure, not only the spectrum information but also various information included in the spectrum information and the reagent information is preferably measured in a unique manner.

[0208] <6. Variation of system configuration>

[0209] It should be noted that the information processing system according to the above-described embodiment (refer to Figure 3 ) can also have a server-client type system configuration. Fig.21 is a block diagram depicting an illustrative configuration embodiment of an information processing system configured in a server-client type configuration.

[0210] like Fig.21 , the information processing system according to the present variation includes a client terminal 100A, a server device 100B, and a database 200 connected and communicating with each other via a predetermined network 300. Various networks such as a WAN (Wide Area Network) (including the Internet), a LAN (Local Area Network), a public network, and a mobile network can be applied to the network 300.

[0211] The client terminal 100A is a terminal device used by doctors, researchers, etc. and includes at least Figure 3 The acquisition section 110, the display section 140, the control section 150, and the operation section 160 of the configuration described in FIG.

[0212] On the other hand, the server device 100B is not limited to a single server and may be configured by a plurality of servers or may be configured by a cloud server. The server device 100B includes Figure 3 At least one of the information storage unit 121, image information storage unit 122, analysis result storage unit 123, analysis unit 131, and image generation unit 132 of the configuration described in the above. Any of the mentioned components not included in the server device 100B may be provided in the client terminal 100A.

[0213] The number of such client terminals 100A connected to the server device 100B is not limited to one, but may be plural. In this case, a plurality of client terminals 100A different from each other may be introduced into the hospital.

[0214] By adopting such a system configuration as described above, not only can a system with higher computing power be provided to users such as doctors or researchers, but also a larger amount of information can be accumulated in the database 200. This indicates that the expansion of the system that performs machine learning and the like on the big data accumulated in the database 200 is promoted.

[0215] It should be noted that the configuration described above has no limitation and various modifications are possible, for example, a similar configuration in which only the database 200 is shared by a plurality of information processing apparatuses 100 via the network 300 .

[0216] <7. Conclusion>

[0217] As described above, based on the sample information and the reagent information, the information processing apparatus 100 according to the present disclosure can separate the autofluorescence signal of the sample 20 and the fluorescence signal of the fluorescent reagent 10 from the image information about the sample 20 colored by at least one coloring reagent 10. Further, the information processing apparatus 100 can perform various processes (e.g., analysis or segmentation of the fixed state of the sample 20) by using the separated fluorescence signal or the autofluorescence signal. Moreover, the information processing apparatus 100 can improve the above-mentioned processes such as the separation process of the autofluorescence signal and the fluorescence signal by performing machine learning based on the sample information, the reagent information, and the analysis results managed by the database 200.

[0218] <8. Exemplary Applications>

[0219] The technology according to the present disclosure can be applied to various productions. For example, the technology according to the present disclosure can be applied to a pathological diagnosis system or a support system thereof (hereinafter referred to as a diagnosis support system) that allows doctors and the like to observe cells or tissues collected from patients to diagnose injuries. The diagnosis support system can be a DPI (digital pathology imaging) system for diagnosing injuries or supporting diagnosis based on images acquired using digital pathology technology.

[0220] Fig. 22 1 is a diagram illustrating an embodiment of a schematic configuration of a diagnosis support system 5500 to which the technology according to the present disclosure is applied. Fig. 22 As described in , the diagnosis support system 5500 includes one or more pathology systems 5510, a medical information system 5530, and an export device 5540.

[0221] Each of the one or more pathology systems 5510 is a system used by a pathologist in principle and is introduced into a laboratory or hospital, for example. The pathology systems 5510 may be introduced into hospitals different from each other and individually connected to the medical information system 5530 and the export device 5540 through various networks such as a WAN (Wide Area Network) (including the Internet), a LAN (Local Area Network), a public network, or a mobile network.

[0222] Each pathology system 5510 includes a microscope 5511 , a server 5512 , a display control device 5513 , and a display device 5514 .

[0223] Microscope 5511 has the function of an optical microscope and images an observation object mounted on a glass slide to obtain a pathological image in the form of a digital image. For example, the observation object may be a tissue or cell collected from a patient and may be a piece of flesh of an organ, saliva, blood, etc.

[0224] The server 5512 stores and retains the pathological image acquired by the microscope 5511 in an undescribed storage unit. Further, in the case where the server 5512 receives a read request from the display control device 5513, it searches for the pathological image from the undescribed storage unit and sends the searched pathological image to the display control device 5513.

[0225] The display control device 5513 sends a read request regarding the pathological image accepted from the user to the server 5512. Then, the display control device 5513 controls the display device 5514 using a liquid crystal unit, an EL (electroluminescence) unit, a CRT (cathode ray tube), etc. to display the pathological image accepted from the server 5512. It should be noted that the display device 5514 can be prepared as a 4K or 8K display system, and the number of such display devices 5514 is not limited to one, but may be plural.

[0226] Here, for example, in the case where the observation object is a solid such as a meat slice of an organ, the observation object may be a stained thin slice. The thin slice may be produced by thin slice cutting of a block cut from an organ sample, etc. Further, at the time of thin slice cutting, the block may be fixed by paraffin, etc.

[0227] In order to stain the thin sections, various staining methods such as general staining indicating the form of an organ such as HE (Hematoxylin-Eosin) staining or staining immunoassay indicating the immune status of an organ such as IHC (Immunohistochemistry) staining can be applied. Meanwhile, one thin section can be stained with a plurality of different reagents, or two or more thin sections (also referred to as adjacent thin sections) cut continuously from the same piece can be stained with reagents different from each other.

[0228] The microscope 5511 can include a low-resolution imaging section for imaging at low resolution and a high-resolution imaging section for imaging at high resolution. The low-resolution imaging section and the high-resolution imaging section can be different optical systems from each other or can be the same optical system. In the case of the same optical system, the microscope 5511 can have a resolution that changes in response to the imaging object.

[0229] A glass slide in which an observation object is mounted is placed on a stage positioned within the viewing angle of the microscope 5511. First, the microscope 5511 uses a low-resolution imaging unit to acquire the entire image within the viewing angle and specifies the area of ​​the observation object from the acquired entire image. Then, the microscope 5511 divides the area in which the observation object exists into a plurality of segmented areas of a predetermined size and continuously images the segmented areas using a high-resolution imaging unit to acquire high-resolution images of each segmented area. When switching the object segmented area, the stage may be moved or the imaging optical system may be moved, or both may be moved. Further, each segmented area may overlap with an adjacent segmented area to prevent imaging omission areas due to accidental displacement of the glass slide or similar reasons. Further, the entire image may include identification information for associating the entire image with the patient. For example, the identification information may be a string, a QR code (registered trademark), or the like.

[0230] The medical information system 5530 is a so-called electronic medical record system and stores information for identifying patients, disease information of patients, examination information and image information used in diagnosis, and information about diagnosis such as diagnosis results, prescription drugs, etc. For example, a pathological image obtained by imaging an observation object of a specific patient is retained once by the server 5512 and can be displayed on the display device 5514 by the display control device 5513. A pathologist using the pathological system 5510 performs pathological diagnosis based on the pathological image displayed on the display device 5514. The result of the pathological diagnosis performed by the pathologist is stored in the medical information system 5530.

[0231] The exporting device 5540 can perform analysis on the pathological image. For this analysis, a learning model generated by machine learning can be used. The exporting device 5540 can export the classification result of a specific area, the identification result of a tissue, etc. Further, the exporting device 5540 can export the identification result of cell information, quantity, position, brightness information, etc., the scoring information thereon, etc. The information exported by the exporting device 5540 can be displayed on the display device 5514 of the pathological system 5510 as diagnosis support information.

[0232] Among the constituent elements described above, the technology according to the present disclosure can be appropriately applied to the entire diagnosis support system 5500. Specifically, the acquisition section 110 corresponds to the microscope 5511; the display control device 5513 corresponds to the control section 150; the display device 5514 corresponds to the display section 140; the remaining components of the pathology system 5510 and the derivation device 5540 correspond to the remaining components of the information processing device 100; and the medical information system 5530 can correspond to the database 200. By applying the technology according to the present disclosure to the diagnosis support system 5500, the advantageous effect can be shown that it becomes possible to allow a doctor or researcher to perform diagnosis or analysis of damage more accurately.

[0233] Although the preferred embodiments according to the present disclosure have been described with reference to the attached drawings, the technical scope of the present disclosure is not limited to the above-mentioned scope. It is obvious to a person skilled in the art of the present disclosure that various changes and modifications can be made without departing from the essence and scope of the present disclosure as defined by the following claims. It should be understood that such changes and modifications are within the technical scope of the present disclosure.

[0234] Further, the advantages described in this specification are illustrative or illustrative and non-limiting. That is, in addition to or in place of the advantageous effects described above, considering the description in this specification, the technology according to the present disclosure can provide other advantages conceived by those skilled in the art.

[0235] It should be noted that the following configurations are also within the technical scope of the present disclosure.

[0236] (1) An information processing device comprising:

[0237] An image acquisition unit, which acquires image information of a sample colored by a fluorescent reagent;

[0238] an information acquisition section that acquires information about the sample and information about the fluorescent reagent; and

[0239] The analyzing unit separates the first fluorescent signal and the second fluorescent signal from the image information based on the information about the sample and the information about the fluorescent reagent.

[0240] (2) The information processing device according to (1), wherein:

[0241] The first fluorescent signal includes an autofluorescent signal of the sample; and

[0242] The second fluorescent signal includes a fluorescent signal of the fluorescent reagent.

[0243] (3) The information processing device according to (1), wherein:

[0244] The sample includes a first autofluorescence component and a second autofluorescence component;

[0245] The first fluorescence signal includes an autofluorescence signal of a first autofluorescence component; and

[0246] The second fluorescence signal includes an autofluorescence signal of a second autofluorescence component.

[0247] (4) The information processing device according to (1), wherein:

[0248] The fluorescent reagent includes a first fluorescent reagent and a second fluorescent reagent;

[0249] The first fluorescent signal includes a fluorescent signal of a first fluorescent agent; and

[0250] The second fluorescent signal includes a fluorescent signal of a second fluorescent reagent.

[0251] (5) The information processing device according to (4), wherein:

[0252] The analyzing section further separates a third fluorescent signal from the image information based on the information about the sample and the information about the fluorescent reagent; and

[0253] The third fluorescent signal includes an autofluorescence signal of the sample.

[0254] (6) The information processing device according to (3), wherein:

[0255] The analysis unit analyzes the fixed state of the sample based on a component ratio between an autofluorescence signal of the first autofluorescence component and an autofluorescence signal of the second autofluorescence component.

[0256] (7) The information processing device according to any one of (1) to (6), wherein:

[0257] The information about the sample includes an autofluorescence component included in the sample and spectrum information of the autofluorescence component.

[0258] (8) The information processing device according to (7), wherein:

[0259] The autofluorescent components included in the sample include at least one or more of NADH (nicotinamide adenine dinucleotide reduced form), FAD (flavin adenine dinucleotide), or porphyrin.

[0260] (9) The information processing device according to (7), wherein:

[0261] The information about the sample further includes information about the kind of tissue to be used, the kind of the subject's disease, the subject's attributes, and the subject's living habits.

[0262] (10) The information processing device according to any one of (1) to (9), wherein:

[0263] The information on the fluorescent reagent includes spectrum information of the fluorescent component included in the fluorescent reagent.

[0264] (11) The information processing device according to any one of (1) to (10), wherein:

[0265] The information acquisition section acquires information about the sample based on sample identification information capable of identifying the sample and acquires information about the fluorescent reagent based on reagent identification information capable of identifying the fluorescent reagent.

[0266] (12) The information processing device according to (11), wherein:

[0267] The reagent identification information can further identify the production batch of the fluorescent reagent.

[0268] (13) The information processing device according to any one of (1) to (12), further comprising:

[0269] The display unit displays image information generated based on the first fluorescent signal or the second fluorescent signal.

[0270] (14) The information processing device according to (13), wherein:

[0271] The display unit performs control to distinguish the first fluorescent signal from the second fluorescent signal and displays the generated image information.

[0272] (15) The information processing device according to (14), wherein:

[0273] The first fluorescence signal includes an autofluorescence signal of the sample;

[0274] The second fluorescent signal includes a fluorescent signal of a fluorescent agent; and

[0275] The display section performs subtraction processing on the autofluorescence signal of the sample to perform control so as to distinguish the fluorescence signal of the fluorescent reagent and display the resulting image.

[0276] (16) The information processing device according to (2), wherein:

[0277] The analyzing section performs subtraction processing on the image information of the different samples using the separated autofluorescence signals to extract the fluorescence signals from the image information of the different samples.

[0278] (17) The information processing device according to any one of (1) to (16), wherein:

[0279] The analyzing section identifies a region of an object included in the image information based on distribution of the first fluorescent signal and the second fluorescent signal in the image information.

[0280] (18) The information processing device according to any one of (1) to (17), wherein:

[0281] The analysis section performs separation according to a result of machine learning performed using the first and second fluorescent signals after separation, image information used in the separation, information about the sample, and information about the fluorescent reagent.

[0282] (19) The information processing device according to any one of (1) to (18), wherein:

[0283] At least one of the first fluorescent signal and the second fluorescent signal includes an autofluorescence signal;

[0284] The information about the fluorescent agent includes information related to the quantum yield, fluorescent labeling rate, and absorption cross section or molar absorption coefficient of the fluorescent agent; and

[0285] The analyzing section calculates at least one of the number of fluorescent molecules in the image information or the number of antibodies coupled to the fluorescent molecules, using the image information from which the autofluorescence signal has been removed and the information on the fluorescent agent.

[0286] (20) A microscope system comprising:

[0287] An information processing device including an image acquisition section that acquires image information of a sample stained with a fluorescent agent and an information acquisition section that acquires information about the sample and information about the fluorescent agent; and

[0288] A program for causing an information processing device to execute a control function for controlling to separate a first fluorescent signal and a second fluorescent signal from image information based on information about a sample and information about a fluorescent agent.

[0289] (21) An information processing device comprising:

[0290] an image acquisition section that acquires image information about a sample stained with at least one fluorescent agent; an information acquisition section that acquires information about the sample and information about the fluorescent agent; and

[0291] The analyzing unit separates the autofluorescence signal of the sample and the fluorescence signal of the fluorescent reagent from the image information based on the information about the sample and the information about the fluorescent reagent.

[0292] (22) The information processing device according to (21), wherein:

[0293] In the case where the sample is colored with two or more fluorescent reagents, the analyzing section separates the fluorescent signal of each of the two or more fluorescent reagents from the image information based on the information about the sample and the information about the fluorescent reagents.

[0294] (23) The information processing device according to (21) or (22), wherein:

[0295] In a case where the autofluorescence signal includes two or more autofluorescence emission components, the analyzing section separates the autofluorescence signal of each of the two or more autofluorescence emission components from the image information based on the information about the sample and the information about the fluorescent agent.

[0296] (24) The information processing device according to (23), wherein:

[0297] The analysis unit analyzes the immobilization state of the sample based on the autofluorescence signal component ratio of each of the two or more autofluorescence emission components.

[0298] (25) The information processing device according to any one of (21) to (24), wherein:

[0299] The information about the sample includes information indicating an autofluorescence emission component possessed by the sample and spectrum information about the autofluorescence emission component.

[0300] (26) The information processing device according to any one of (21) to (25), wherein:

[0301] The information about the fluorescent reagent includes spectrum information about the fluorescent component included in the fluorescent reagent.

[0302] (27) The information processing device according to any one of (21) to (26), wherein:

[0303] The information acquisition section acquires information about the sample based on sample identification information capable of identifying the sample and acquires information about the fluorescent reagent based on reagent identification information capable of identifying the fluorescent reagent.

[0304] (28) The information processing device according to (27), wherein:

[0305] The reagent identification information can also identify the production batch of the fluorescent reagent.

[0306] (29) The information processing device according to any one of (21) to (28), further comprising:

[0307] The display unit displays image information generated based on the fluorescent signal or the autofluorescent signal.

[0308] (30) The information processing device according to any one of (21) to (29), wherein:

[0309] The analyzing section extracts the fluorescence signal from the image information about the different samples by performing a subtraction process on the image information about the different samples using the autofluorescence signal.

[0310] (31) The information processing device according to any one of (21) to (30), wherein:

[0311] The analyzing section recognizes a region of the object included in the image information based on distribution of the autofluorescence signal and the fluorescence signal in the image information.

[0312] (32) The information processing device according to any one of (21) to (31), wherein:

[0313] The analyzing section performs separation according to a result of machine learning performed by using the separated fluorescence signal and autofluorescence signal, image information used for separation, information on the sample, and information on the fluorescent reagent.

[0314] (33) The information processing device according to any one of (21) to (32), wherein:

[0315] The information about the fluorescent agent includes information related to the quantum yield, fluorescent labeling rate, and absorption cross section or molar absorption coefficient of the fluorescent agent; and

[0316] The analyzing section calculates at least one of the number of fluorescent molecules in the image information or the number of antibodies coupled to the fluorescent molecules by using the image information without the autofluorescence signal and the information on the fluorescent agent.

[0317] (34) An information processing method executed by a computer, the information processing method comprising:

[0318] Acquiring image information of a sample stained with a fluorescent reagent;

[0319] obtaining information about the sample and information about the fluorescent reagent; and

[0320] An autofluorescence signal of the sample and a fluorescence signal of the fluorescent agent are separated from the image information according to the information about the sample and the information about the fluorescent agent.

[0321] (35) A program for causing a computer to execute a process, the process comprising:

[0322] Acquiring image information of a sample stained with a fluorescent reagent;

[0323] obtaining information about the sample and information about the fluorescent reagent;

[0324] An autofluorescence signal of the sample and a fluorescence signal of the fluorescent agent are separated from the image information according to the information about the sample and the information about the fluorescent agent.

[0325] (36) An information processing system comprising:

[0326] An image acquisition unit that acquires image information about a sample stained with a fluorescent reagent;

[0327] An information acquisition unit, which acquires information about the sample and information about the fluorescent reagent;

[0328] an analyzing section that separates an autofluorescence signal of the sample and a fluorescence signal of the fluorescent reagent from the image information based on the information about the sample and the information about the fluorescent reagent; and

[0329] The display unit displays image information generated based on the fluorescent signal or the autofluorescent signal.

[0330] [Reference Numbers List]

[0331] 10 Fluorescence reagents

[0332] 11 Reagent identification information

[0333] 20 samples

[0334] 21 Sample identification information

[0335] 30 Fluorescent Coloring Samples

[0336] 100 Information processing device

[0337] 100A Client Terminal

[0338] 100B Server Device

[0339] 110 Acquisition Department

[0340] 111 Information Acquisition Department

[0341] 112 Image acquisition unit

[0342] 120 Storage

[0343] 121 Information Storage Department

[0344] 122 Image information storage unit

[0345] 123 Analysis result storage unit

[0346] 130 Processing Department

[0347] 131 Analysis Department

[0348] 132 Image Generation Unit

[0349] 140 Display unit

[0350] 150 Control Department

[0351] 160 Operation Department

[0352] 200 Database

[0353] 300 Network

Claims

1. An information processing device, comprising: An image acquisition unit, which acquires image information of a sample colored by a fluorescent reagent; An information acquisition unit, which acquires information about the sample and information about the fluorescent reagent; as well as an analyzing unit, separating a first fluorescent signal and a second fluorescent signal from the image information according to information about the sample and information about the fluorescent reagent, The sample includes a first autofluorescence component and a second autofluorescence component; The first fluorescence signal includes an autofluorescence signal of the first autofluorescence component; and The second fluorescence signal includes an autofluorescence signal of the second autofluorescence component; The analysis unit analyzes the fixed state of the sample based on a component ratio between an autofluorescence signal of the first autofluorescence component and an autofluorescence signal of the second autofluorescence component.

2. The information processing device according to claim 1, wherein: The fluorescent reagent includes a first fluorescent reagent and a second fluorescent reagent; The first fluorescent reagent emits a third fluorescent signal; and The second fluorescent reagent emits a fourth fluorescent signal.

3. The information processing device according to claim 2, wherein: The analyzing section further separates the third fluorescent signal and the fourth fluorescent signal from the image information based on the information about the sample and the information about the fluorescent reagent.

4. The information processing device according to claim 1, wherein: The information about the sample includes an autofluorescence component included in the sample and spectrum information of the autofluorescence component.

5. The information processing device according to claim 4, wherein: The autofluorescence components included in the sample include at least one or more of nicotinamide adenine dinucleotide reduced form, flavin adenine dinucleotide and porphyrin.

6. The information processing device according to claim 4, wherein: The information about the sample further includes information about the kind of tissue to be used, the kind of disease of the subject, the attributes of the subject, and the living habits of the subject.

7. The information processing device according to claim 1, wherein: The information about the fluorescent reagent includes spectrum information of a fluorescent component included in the fluorescent reagent.

8. The information processing device according to claim 1, wherein: The information acquisition section acquires information about the sample based on sample identification information capable of identifying the sample, and acquires information about the fluorescent reagent based on reagent identification information capable of identifying the fluorescent reagent.

9. The information processing device according to claim 8, wherein: The reagent identification information can further identify the production batch of the fluorescent reagent.

10. The information processing device according to claim 1, further comprising: The display unit displays image information generated based on the first fluorescent signal or the second fluorescent signal.

11. The information processing device according to claim 10, wherein: The display section performs control to distinguish the first fluorescent signal from the second fluorescent signal and display the generated image information.

12. The information processing device according to claim 1, wherein: The analyzing section performs a subtraction process on the image information of different samples using the separated autofluorescence signal to extract the fluorescence signal from the image information of the different samples.

13. The information processing device according to claim 1, wherein: The analyzing section identifies a region of an object included in the image information based on distribution of the first fluorescent signal and the second fluorescent signal in the image information.

14. The information processing device according to claim 1, wherein: The analysis section performs separation based on a result of machine learning performed using the first and second fluorescent signals after separation, the image information used in separation, information about the sample, and information about the fluorescent reagent.

15. The information processing device according to claim 1, wherein: The information about the fluorescent agent includes the quantum yield, fluorescent labeling rate, and absorption cross section or molar absorption coefficient of the fluorescent agent; and The analyzing section calculates at least one of the number of fluorescent molecules in the image information and the number of antibodies coupled to the fluorescent molecules, using the image information from which the autofluorescence signal has been removed and information on the fluorescent agent.

16. A microscope system comprising: An information processing device including an image acquisition section that acquires image information of a sample stained with a fluorescent agent and an information acquisition section that acquires information about the sample and information about the fluorescent agent; and a program for causing the information processing device to execute a control function for controlling to separate a first fluorescent signal and a second fluorescent signal from the image information based on information about the sample and information about the fluorescent reagent, The sample includes a first autofluorescence component and a second autofluorescence component; The first fluorescence signal includes an autofluorescence signal of the first autofluorescence component; and The second fluorescence signal includes an autofluorescence signal of the second autofluorescence component; The program also analyzes the fixation state of the sample based on a component ratio between the autofluorescence signal of the first autofluorescence component and the autofluorescence signal of the second autofluorescence component.

Images