Information provision method, information provision device, and program
The method integrates bright-field and fluorescence imaging with analysis scoring to provide objective and comprehensive support information for pathological diagnosis, addressing subjective variations and improving accuracy by evaluating tissue section relationships.
Patent Information
- Application Number
- JP2020572199
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-02-15
- Filing Date
- 2020-02-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-02-06
AI Technical Summary
Conventional pathological diagnosis relies heavily on subjective judgments by pathologists, leading to variations in diagnostic results, and existing methods fail to comprehensively evaluate the relationships between regions, structures, and cell types on tissue sections, limiting the accuracy of prognosis prediction and drug efficacy assessment.
An information provision method and device that acquires digitized bright-field and fluorescence images of tissue sections, calculates an analysis score based on region area, structure presence, cell type density, and localization relationships, and presents this score as objective support information.
Provides highly accurate and objective support information for pathological judgments by integrating multiple types of information from tissue sections, enhancing the reliability and consistency of diagnostic and predictive assessments.
Smart Images

Figure 0007593112000001 
Figure 0007593112000002 
Figure 0007593112000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information providing method, an information providing device, and a program. [Background technology]
[0002] Conventionally, various judgments have been made based on information obtained from tissue sections. For example, in pathological diagnosis, pathologists stain tissue sections and observe them under a microscope to obtain morphological information and staining information such as changes in the size and shape of cell nuclei and changes in tissue patterns, and observe the presence or absence of lesions and the state of the lesions based on this information. In addition, this information is also used in predicting the prognosis of patients based on the observation of tissue sections, predicting drug efficacy in clinical practice or during the drug development process, and so on.
[0003] In recent years, it has been known that the density of specific cells or structures on tissue sections can be useful information for predicting the prognosis of a patient and for determining a subsequent treatment plan in pathological diagnosis, etc. For example, Patent Document 1 captures an image of a tissue section obtained from tumor tissue to obtain a virtual slide image, and measures the density of cells and / or blood vessels located near the tumor boundary. It has been revealed that the prognosis of a patient can be predicted based on the measured density. Patent Document 2 also counts the number of immune cells on a tissue section obtained from tumor tissue, and shows that there is a correlation between the density of immune cells present in the tumor tissue and the prognosis of the patient. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Special Publication No. 2013-537969 [Patent Document 2] Special Publication No. 2016-530505 Summary of the Invention [Problem to be solved by the invention]
[0005] In conventional pathological diagnoses, etc., it was difficult to eliminate variability in diagnostic results because the diagnosis was based on the subjective judgment of the pathologist, etc. In this regard, the invention described in Patent Document 1 uses technology for digitally analyzing microscopic images, and can be said to guarantee objectivity compared to diagnoses by pathologists, etc.
[0006] On the other hand, in pathological diagnosis, etc., the area, structure, and cell type present on the tissue section are observed comprehensively. That is, the pathologist, etc., first evaluates the area of the tumor area on the tissue section, then evaluates the presence or absence and number of specific structures such as blood vessels and lymphatic vessels present in the tissue section, and further judges the number of tumor cells and the distance between the interstitial cells and the tumor cells, and also uses the arrangement relationship between them as a judgment material. In the invention described in Patent Document 1, it is possible to predict the prognosis based on the density of cells and / or blood vessels, but it is not possible to make a comprehensive judgment based on the relationship between the area, structure, and cell type. In addition, the invention described in Patent Document 2 evaluates the density of immune cells, but does not judge the relationship between other cell types, structures, areas, etc., so there is room for improvement in the evaluation accuracy.
[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information providing method, an information providing device, and a program capable of providing objective, highly accurate support information for various judgments made based on information obtained from tissue sections. [Means for solving the problem]
[0008] In order to solve the above problem, the information providing method according to claim 1 comprises: 1. A method for providing support information for supporting a decision based on information obtained from a tissue section, comprising: an image acquisition step of acquiring a digitized bright-field image of the stained tissue section for bright-field observation; a score creation step of acquiring a plurality of types of information from the bright field image and creating an analysis score by combining the plurality of types of information; and an information presenting step of presenting the analysis score as the support information, In the score creation step teeth , various regions present in the tissue section Score based on area percentage or ratio , the presence or absence of various structures, the density of various structures, or the occurrence of various structures Number-based score , the number of each cell type, the density of each cell type, the distance between different cell types or the distance between a particular cell type and a particular structure. Away Based on Kus core In order of A calculation is performed, and the analysis score is created based on the calculated score.
[0009] The invention described in claim 2 is the information providing method described in claim 1, In the image acquisition step, a bright-field image of the entire tissue section is acquired by imaging the entire tissue section using a virtual microscope slide creation device capable of imaging the entire tissue section.
[0011] Claim 3 The invention described in claim 1 or 2 In the information providing method described in In the score generation step, information regarding the localization relationships between regions, structures and cell types present in the tissue section is obtained.
[0012] Claim 4 The invention described in claim Any one of 1 to 3 In the information providing method described in the tissue section is stained with a staining reagent in which a biological substance recognition site is bound to a fluorescent substance-holding nanoparticle having a plurality of fluorescent substances accumulated therein, so that a specific biological substance present in the tissue section can be fluorescently observed; The image acquisition step further comprises acquiring a digitized fluorescent image of the tissue section; In the score creation step, information relating to the presence of the specific biological material is further obtained from the fluorescent image, and an analysis score is created by combining the information with the multiple types of information.
[0013] Claim 5 The invention described in claim 14 In any one of the information providing methods described above, The method includes a region visualization step of staining biological material present in a specific region on the tissue section with a fluorescent substance so that the biological material can be fluorescently observed.
[0014] The information providing device according to claim 6 comprises: An information providing device that provides support information for supporting a decision based on information obtained from a tissue section, comprising: an image acquisition unit that acquires a digitized bright-field image of a tissue section stained to enable bright-field observation; a score creation unit that acquires a plurality of types of information from the bright field image and creates an analysis score by combining the plurality of types of information; an information presenting unit that presents the analysis score as the support information, The above score creation Department is calculated in the following order: a score based on the area proportion or ratio of various regions present in the tissue section; a score based on the presence or absence of various structures, the density of various structures or the number of occurrences of various structures; and a score based on the number of each cell type, the density of each cell type, the distance between different cell types, or the distance between a specific cell type and a specific structure, and the analysis score is created based on the calculated scores.
[0015] The program according to claim 7 comprises: a computer of an information providing device that provides support information for supporting a decision based on information obtained from a tissue section, an image acquisition unit for acquiring a digitized bright-field image of a tissue section stained to enable bright-field observation; A plurality of types of information are obtained from the bright field image, and an analysis score is created by combining the plurality of types of information. Score creation department, an information presenting unit that presents the analysis score as the support information; Function as 、 The score creation unit calculates, in the order of appearance, from the bright field image, a score based on the area proportion or ratio of various regions present in the tissue section, a score based on the presence or absence of various structures, the density of various structures or the number of occurrences of various structures, and a score based on the number of each cell type, the density of each cell type, the distance between different cell types or the distance between a specific cell type and a specific structure, and creates the analysis score based on the calculated scores. Effect of the Invention
[0016] According to the present invention, it is possible to provide an information providing method, an information providing device, and a program capable of providing objective, highly accurate support information for various judgments made based on information obtained from tissue sections. [Brief description of the drawings]
[0017] [Figure 1] 1 is a diagram showing a schematic configuration of a pathological diagnosis support system according to the present invention. [Diagram 2] 2 is a block diagram showing a functional configuration of the information providing device of FIG. 1. [Diagram 3] 4 is a flowchart showing an operation of the information providing device. [Figure 4] 13 is a flowchart showing a process for identifying a score calculation target. [Diagram 5] 13 is a flowchart showing a cell type identification process. [Figure 6] 13 is a flowchart showing a calculation process of a morphological score. [Figure 7] 13 is a flowchart showing a biological substance score calculation process. [Figure 8A] FIG. 1 is a diagram showing an example of a bright-field image. [Figure 8B] FIG. 13 is a diagram illustrating an example of a method for calculating an analysis score. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] First, the following terms used in the present invention will be explained.
[0019] 1.Area The term "region" refers to an area in which a certain amount of the same structure or cells are present in a group. Among the regions, the "tumor region" in this embodiment refers to a region formed by a certain amount of tumor cells, which will be described later, and the "stromal region" refers to a region formed by a certain amount of interstitial cells, which will be described later. 2. Structure A "structure" refers to a structure in which a certain number of cells are gathered and perform some physiological activity. Examples of structures include blood vessels, lymphatic vessels, and secretory glands. 3.Cell type "Cell type" refers to the type of cell that is the smallest unit that has a function and can operate by itself. Among the cell types, "tumor cells" in this embodiment refer to cells that grow autonomously and excessively against the control in the living body, and include "cancer cells" that form malignant tumors. "Stroma cells" broadly refer to cells that form the support structure of biological tissues, and include immune cells, inflammatory cells, fibroblasts, endothelial cells, etc. It is known that cancer progresses due to close interactions between cancer cells and stromal cells.
[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited thereto.
[0021] <Configuration of the pathology diagnosis support system 100> 1 shows an example of the overall configuration of a pathology diagnosis support system 100 that executes an information provision method according to this embodiment. The pathology diagnosis support system 100 is a system that acquires and analyzes microscopic images of tissue sections stained with a predetermined staining reagent, and outputs support information. Conventionally, various judgments such as prediction of efficacy of a drug administered to a living body, prediction of prognosis of an observation target, or pathological diagnosis are made based on information on a specific region, structure, or cell type present in a tissue section, the expression level of a specific biological substance present in the tissue section, or information on the localization of an active ingredient of a drug administered to a living body. Such judgments are not only made by users such as pathologists and drug development personnel, but also in recent years, these information is automatically recognized and judged using artificial intelligence. The support information according to this embodiment is provided to support the improvement of the accuracy of these judgments. Specifically, the support information can take the form of an analysis score calculated based on information such as the number and arrangement of a specific region, structure, or cell type on a tissue section, and functions as an objective and highly accurate index to support the above-mentioned various judgments.
[0022] As shown in FIG. 1, a pathology diagnosis support system 100 includes a microscopic image acquisition device 1A and an information providing device 2A connected via a communication network N so as to be capable of transmitting and receiving data. The pathology diagnosis support system 100 may be either a system in which the microscopic image acquisition device 1A and the information providing device 2A are disposed near each other, such as in the same building, or a system in which the microscopic image acquisition device 1A and the information providing device 2A are disposed at locations far away from each other. The communication network N is not particularly limited, and the connection method may be wired or wireless. When the microscopic image acquisition device 1A and the information providing device 2A are disposed near each other, for example, a LAN (Local Area Network) is used, and when the microscopic image acquisition device 1A and the information providing device 2A are disposed at remote locations, for example, a WAN (Wide Area Network) such as the Internet is used.
[0023] The microscope image acquisition device 1A is a virtual microscope slide creation device (see, for example, Published Japanese Translation of PCT International Publication No. 2002-514319) that scans a slide on a slide fixing stage of a microscope to acquire a digital image of the entire tissue section, acquires image data that allows the entire image of the tissue section on the slide to be viewed at once on a display unit, and transmits the image data to the information provision device 2A. The microscope image acquisition device 1A is configured with an illumination means, an imaging means, an image capture means, a communication I / F, etc. The illumination means is configured with a light source, a filter, etc., and illuminates light onto a tissue slice on a slide placed on a slide fixing stage. The image capture means is configured with an eyepiece, an objective lens, etc., and forms an image of transmitted light, reflected light, or fluorescence emitted from the tissue slice on the slide by the irradiated light. The image capture means is a microscope-mounted camera equipped with a CCD (Charge Coupled Device) sensor, etc., that captures an image formed on an image formation surface by the image capture means to generate digital image data of the microscope image. The communication I / F transmits the generated digital image data of the microscope image to the information providing device 2A via the communication network N. In this embodiment, the microscope image acquisition device 1A is equipped with a bright-field unit that combines an illumination means and an image capture means suitable for bright-field observation, and a fluorescent unit that combines an illumination means and an image capture means suitable for fluorescent observation, and the former can be used to obtain a bright-field image, and the latter can be used to obtain a fluorescent image.
[0024] When the microscopic image acquisition device 1A is connected to the Internet as a communications network N, the acquired microscopic images can be stored in a server device on the Internet and can be viewed on an information providing device 2A that is also connected to the Internet. In other words, even if the microscopic image acquisition device 1A and the information providing device 2A are located at locations far away from each other, the user can use the information acquired by the microscopic image acquisition device from a remote location via the information providing device 2A.
[0025] In this embodiment, a virtual microscope slide creation device is used as the microscope image acquisition device 1A, but the present invention is not limited to this, and a known camera-equipped microscope may be used. In the camera-equipped microscope, a microscopic image of a predetermined field of view can be acquired for a tissue slice on a slide placed on a slide fixing stage.
[0026] The information providing device 2A outputs support information that is generated by analyzing the microscope image transmitted from the microscope image acquiring device 1A. Fig. 2 shows an example of a functional configuration of the information providing device 2A. As shown in Fig. 2, the information providing device 2A includes a control unit 21, an operation unit 22, a display unit 23, a communication I / F 24, a storage unit 25, etc., and each unit is connected via a bus 26.
[0027] The control unit 21 is configured with a CPU (Central Processing Unit), a RAM (Random Access Memory), etc., and executes various processes in cooperation with various programs stored in the storage unit 25, and generally controls the operation of the information providing device 2A. For example, the control unit 21 realizes functions as an image acquisition unit, a score creation unit, and an information presentation unit in cooperation with the programs stored in the storage unit 25.
[0028] The operation unit 22 is configured with a keyboard equipped with letter input keys, number input keys, and various function keys, etc., and a pointing device such as a mouse, and outputs a press signal of a key pressed on the keyboard and an operation signal by the mouse to the control unit 21 as input signals.
[0029] The display unit 23 is configured to include a monitor such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display), and displays various screens according to instructions of a display signal input from the control unit 21.
[0030] The communication I / F 24 is an interface for transmitting and receiving data between external devices including the microscopic image acquisition device 1A via the communication network N, and functions as a means for inputting fluorescent images captured by the microscopic image acquisition device 1A to the information providing device 2A.
[0031] The storage unit 25 is configured with, for example, a HDD (Hard Disk Drive), a semiconductor non-volatile memory, etc. In the storage unit 25, various programs, various data, etc. are stored as described above. In addition, the information providing device 2A may be configured to include a LAN adapter, a router, etc., and to be connected to an external device via the communication network N.
[0032] <Information provision method> The information providing method according to this embodiment will be described below. The information providing method according to the present invention is a method for providing support information for supporting various decisions based on information obtained from tissue sections, such as prediction of drug efficacy, prognosis, or pathological diagnosis of a subject under observation.
[0033] The information providing method of the present invention comprises at least the steps of: 1. acquiring a digitized bright-field image of a tissue section stained with a staining reagent that can be observed in a bright-field (image acquiring step); 2. acquiring multiple types of information from the bright-field image and creating an analysis score by combining the multiple types of information (score creating step); and 3. presenting the analysis score to a user as support information (information presenting step). In addition to the above steps, the method further includes 4. a step of acquiring a digitized fluorescent image of the tissue section stained with fluorescent substance-aggregated nanoparticles (image acquisition step), and in the score creation step, an analysis score is created by combining information acquired from the fluorescent image with information acquired from the bright field image, thereby improving the accuracy of the support information. The information providing method according to this embodiment includes the above steps 1 to 4, and creates an analysis score based on information obtained from bright-field images and fluorescent images.
[0034] (1) Tissue specimen The tissue specimen generally takes the form of a specimen slide on which a tissue slice or a cell is placed, as is commonly used in cases where the expression level of a specific biological substance is evaluated by immunohistochemical staining, etc. In this embodiment, a tissue slice taken from a tumor tissue is used. The method for preparing the tissue specimen is not particularly limited, and generally, for example, tissue slices taken from a subject are fixed using formalin or the like, dehydrated with alcohol, treated with xylene, and embedded in paraffin by immersing in hot paraffin to prepare tissue samples, which are then cut into 3-4 μm slices. The tissue slices are then placed on a glass slide and dried to prepare a specimen slide.
[0035] (2) Dyeing process The staining process for staining a tissue specimen with a staining reagent will be described below.
[0036] (2.1) Specimen preparation process (2.1.1) Deparaffinization The tissue slices are immersed in a container containing xylene to remove the paraffin. The temperature is not particularly limited, but the process can be carried out at room temperature. The immersion time is preferably 3 minutes or more and 30 minutes or less. If necessary, the xylene may be replaced during the immersion process.
[0037] Next, the tissue slice is immersed in a container containing ethanol to remove the xylene. The temperature is not particularly limited, but the process can be carried out at room temperature. The immersion time is preferably 3 minutes or more and 30 minutes or less. If necessary, the ethanol may be replaced during the immersion.
[0038] The tissue slice is immersed in a container containing water to remove the ethanol. The temperature is not particularly limited, but the process can be carried out at room temperature. The immersion time is preferably 3 minutes or more and 30 minutes or less. If necessary, the water may be replaced during the immersion process.
[0039] (2.1.2) Activation treatment In order to stain the target substance in the fluorescent staining step described below, the target substance is activated according to a known method. There are no particular restrictions on the activation conditions, but the activation solution may be 0.01 M citrate buffer (pH 6.0), 1 mM EDTA solution (pH 8.0), 5% urea, 0.1 M Tris-HCl buffer, or the like. The pH condition is set to a range of 2.0 to 13.0 depending on the tissue section used, so that a signal is generated and the tissue is rough enough to evaluate the signal. Usually, the pH is set to 6.0 to 8.0, but for special tissue sections, it can be set to, for example, 3.0. Heating equipment that can be used includes an autoclave, microwave, pressure cooker, water bath, etc. The temperature is not particularly limited, but can be room temperature. The temperature is 50 to 130°C, and the time is 5 to 30 minutes.
[0040] Next, the sections after activation treatment are immersed in a container containing PBS and washed. The temperature is not particularly limited, but washing can be performed at room temperature. The immersion time is preferably 3 minutes or more and 30 minutes or less. If necessary, the PBS may be replaced during the immersion.
[0041] (2.2) Morphological staining process The morphological staining step is a step of staining cells, tissues, organs, etc., for visualizing the morphology thereof so that it can be observed in a bright field. For morphological observation of tissue specimens, the standard staining method is eosin, which stains cytoplasm, interstitium, various fibers, red blood cells, and keratinocytes red to deep red. Hematoxylin staining is also used as a standard method, which stains cell nuclei, calcified areas, cartilage tissue, bacteria, and mucus blue to light blue (the method of performing these two stains simultaneously is known as hematoxylin-eosin staining (HE staining)). The morphological staining step can be carried out according to a conventional method. For example, in the case of HE staining, the tissue specimen is immersed in Mayer's hematoxylin solution to carry out hematoxylin staining, and then the tissue specimen is washed with running water and immersed in eosin solution to carry out eosin staining. The conditions for carrying out the morphological staining step, i.e., the temperature and immersion time when immersing the tissue specimen in the staining solution, can be appropriately adjusted according to the conventional method.
[0042] (2.3) Fluorescent staining process The fluorescent staining process is a process for staining a target substance with fluorescent substance-aggregated nanoparticles for fluorescent observation. In the fluorescent staining process, in order to stain the target substance, an immunohistochemical staining is performed by placing an immunostaining agent solution on the section and allowing it to react with the target substance. The immunostaining agent solution used in the fluorescent staining process may be prepared in advance of this process. The fluorescent staining step may be performed after the morphological staining step, or may be performed before the morphological staining step.
[0043] (2.3.1) Target substance The target substance refers to a substance that is the subject of immunohistochemical staining using a fluorescent label, mainly for detection or quantification from the viewpoint of pathological diagnosis. Specifically, biological substances such as proteins (antigens) and nucleic acids present in tissue sections are assumed. Furthermore, antibodies and nucleic acids contained in pharmaceuticals (antibody drugs, nucleic acid drugs, etc.) administered to living organisms are also examples of target substances. In this embodiment, any of the biological substances exemplified below is to be stained as the target substance. Examples of proteins include proteins that are expressed in immune cells and can be used as biomarkers, such as PD-1, CTLA-4, TIM3, Foxp3, CD3, CD4, CD8, CD25, CD27, CD28, CD70, CD40, CD40L, CD80, CD86, CD160, CD57, CD226, CD112, CD155, OX40 (CD134), OX40L (CD252), ICOS (CD278), ICOSL (CD275), 4-1BB (CD137), 4-1BBL (CD137L), 2B4 (CD244), and GITR. (CD357), B7-H3 (CD276), LAG-3 (CD223), BTLA (CD272), HVEM (CD270), GITRL, Galectin-9, B7-H4, B7-H5, PD-L2, KLRG-1, E-Cadherin, N-Cadherin, R-Cadherin, and IDO, TDO, CSF-1R, HDAC, CXCR4, FLT-3, TIGIT, and the like. Examples of nucleic acids include DNA and RNA-related substances (mRNA, tRNA, rRNA, miRNA, non-coding RNA, etc.). Examples of pharmaceuticals containing an antibody as an ingredient include pharmaceuticals containing an antibody as an active ingredient, pharmaceuticals containing an antibody that recognizes a cancer cell-specific protein as a means of delivery to cancer cells, and pharmaceuticals containing an antibody that targets a factor in a signal transduction pathway related to a factor (protein) targeted by the active ingredient, in addition to pharmaceuticals containing an antibody as an active ingredient. The antibodies contained in these pharmaceuticals are preferably antibodies that specifically recognize cancer growth regulators, metastasis regulators, or cancer cell-specific proteins, and may be monoclonal or polyclonal antibodies. The class and subclass of the antibody are not particularly limited, and examples of the classes include IgA, IgG, IgE, IgD, and IgM, and examples of the subclasses include IgG1, IgG2, IgG3, IgG4, IgA1, and IgA2. In this specification, the term "antibody" includes not only full-length antibodies, but also antibody fragments such as Fab, F(ab)'2, Fv, and scFv, and derivatives such as chimeric antibodies (humanized antibodies, etc.) and multifunctional antibodies. Pharmaceuticals containing nucleic acids as ingredients are not particularly limited as long as they are DNA or RNA nucleic acids, or artificial nucleic acids such as PNA, and preferred examples include decoys, antisense, siRNA, miRNA, ribozymes, aptamers, and plasmid DNA.
[0044] (2.3.2) Fluorescent substance-integrated nanoparticles A phosphor integrated dot (PID) is a nano-sized particle that emits fluorescence when irradiated with excitation light, and is capable of emitting fluorescence of sufficient intensity to represent individual molecules of a target substance as bright dots. Fluorescent substance-integrated nanoparticles are nano-sized particles that have a structure in which a plurality of fluorescent substances (e.g., fluorescent organic dyes and quantum dots, which will be described later) are encapsulated within and / or adsorbed onto their surface, and fluorescent dye-integrated nanoparticles, quantum dot-integrated nanoparticles, etc. are used. In the phosphor-holding nanoparticles, the host material and the phosphor preferably have substituents or sites having opposite charges, and thus electrostatic interaction occurs.
[0045] (2.3.2.1) Fluorescent Substances Examples of fluorescent substances used in staining reagents for acquiring fluorescent images include fluorescent organic dyes and quantum dots (semiconductor particles), which preferably emit visible to near-infrared light with a wavelength in the range of 400 to 1000 nm when excited by ultraviolet to near-infrared light with a wavelength in the range of 200 to 700 nm.
[0046] Examples of fluorescent organic dyes include fluorescein-based dye molecules, rhodamine-based dye molecules, Alexa Fluor (Invitrogen)-based dye molecules, BODIPY (Invitrogen)-based dye molecules, cascade-based dye molecules, coumarin-based dye molecules, eosin-based dye molecules, NBD-based dye molecules, pyrene-based dye molecules, Texas Red-based dye molecules, and cyanine-based dye molecules.
[0047] Specifically, 5-carboxy-fluorescein, 6-carboxy-fluorescein, 5,6-dicarboxy-fluorescein, 6-carboxy-2',4,4',5',7,7'-hexachlorofluorescein, 6-carboxy-2',4,7,7'-tetrachlorofluorescein, 6-carboxy-4',5'-dichloro-2',7'-dimethoxyfluorescein, naphthofluorescein, 5-carboxy-rhodamine, 6-carboxy-rhodamine, 5,6-dicarboxy-rhodamine, rhodamine 6G, tetramethylrhodamine, X-rhodamine, and Alexa Fluor 350, Alexa Fluor 405, Alexa Fluor 430, Alexa Fluor 488, Alexa Fluor 500, Alexa Fluor 514, Alexa Fluor 532, Alexa Fluor 546, Alexa Fluor 555, Alexa Fluor 568, Alexa Examples of such an agent include Fluor 594, Alexa Fluor 610, Alexa Fluor 633, Alexa Fluor 635, Alexa Fluor 647, Alexa Fluor 660, Alexa Fluor 680, Alexa Fluor 700, Alexa Fluor 750, BODIPY FL, BODIPY TMR, BODIPY 493 / 503, BODIPY 530 / 550, BODIPY 558 / 568, BODIPY 564 / 570, BODIPY 576 / 589, BODIPY 581 / 591, BODIPY 630 / 650, BODIPY 650 / 665 (all manufactured by Invitrogen), methoxycoumarin, eosin, NBD, pyrene, Cy5, Cy5.5, Cy7, etc. These agents may be used alone or in combination.
[0048] As the quantum dots, any of the II-VI group compounds, the III-V group compounds, and quantum dots containing a group IV element as a component (also called "II-VI group quantum dots", "III-V group quantum dots", and "IV group quantum dots", respectively) can be used. They may be used alone or in combination of multiple types.
[0049] Specific examples include, but are not limited to, CdSe, CdS, CdTe, ZnSe, ZnS, ZnTe, InP, InN, InAs, InGaP, GaP, GaAs, Si, and Ge.
[0050] It is also possible to use quantum dots having the above quantum dots as a core and a shell thereon. Hereinafter, in the present specification, quantum dots having a shell are expressed as CdSe / ZnS when the core is CdSe and the shell is ZnS. For example, CdSe / ZnS, CdS / ZnS, InP / ZnS, InGaP / ZnS, Si / SiO2, Si / ZnS, Ge / GeO2, Ge / ZnS, etc. can be used, but are not limited to these. If necessary, the quantum dots used may be surface-treated with an organic polymer, etc. Examples include CdSe / ZnS (manufactured by Invitrogen) having a surface carboxy group, and CdSe / ZnS (manufactured by Invitrogen) having a surface amino group.
[0051] In addition to the fluorescent organic dyes and quantum dots mentioned above, examples of fluorescent substances to be accumulated in the phosphor-aggregated nanoparticles include "long-afterglow phosphors" that use Y2O3, Zn2SiO4, or the like as a host and Mn2+, Eu3+, or the like as an activator.
[0052] (2.3.2.2) Maternal Examples of organic substances among the base materials include resins generally classified as thermosetting resins, such as melamine resin, urea resin, aniline resin, guanamine resin, phenol resin, xylene resin, and furan resin; resins generally classified as thermoplastic resins, such as styrene resin, acrylic resin, acrylonitrile resin, AS resin (acrylonitrile-styrene copolymer), and ASA resin (acrylonitrile-styrene-methyl acrylate copolymer); other resins such as polylactic acid; and polysaccharides. Examples of inorganic substances among the base materials include silica and glass.
[0053] (2.3.2.3) Quantum dot-integrated nanoparticles The quantum dot-assembled nanoparticles have a structure in which the quantum dots are encapsulated within the host material and / or adsorbed onto the surface thereof. When the quantum dots are encapsulated in a host material, the quantum dots only need to be dispersed within the host material, and may or may not be chemically bonded to the host material itself.
[0054] (2.3.2.4) Fluorescent dye-integrated nanoparticles The fluorescent dye-holding nanoparticles have a structure in which the fluorescent organic dye is encapsulated within the matrix and / or adsorbed onto the surface thereof. When the fluorescent organic dye is encapsulated in a matrix, it is sufficient that the fluorescent organic dye is dispersed within the matrix, and it does not matter whether the fluorescent organic dye is chemically bonded to the matrix itself.
[0055] (2.3.2.5) Preparation of fluorescent substance-integrated nanoparticles The phosphor-holding nanoparticles can be prepared according to known methods. Specifically, for example, fluorescent-substance-encapsulated silica particles, which have silica as a base material and fluorescent substances encapsulated therein, can be produced by dropping a solution in which a fluorescent substance such as a quantum dot or a fluorescent organic dye and a silica precursor such as tetraethoxysilane are dissolved into a solution in which ethanol and ammonia are dissolved, and hydrolyzing the silica precursor.
[0056] On the other hand, phosphor-accumulated resin particles, which use a resin as a base material and have a phosphor adsorbed onto the surface of the resin particles or encapsulated within the resin particles, can be produced by first preparing a solution or dispersion of fine particles of the resin, adding a phosphor such as quantum dots or a fluorescent organic dye, and stirring it. Alternatively, phosphor-accumulated resin particles can be produced by adding a phosphor to a solution of raw resin material and then allowing a polymerization reaction to proceed. For example, when a thermosetting resin such as melamine resin is used as the base resin, the fluorescent dye-integrated resin particles can be produced by heating a reaction mixture containing the raw material of the resin (monomer or oligomer or prepolymer, for example, methylolmelamine, which is a condensation product of melamine and formaldehyde), a fluorescent organic dye, and preferably also a surfactant and a polymerization reaction accelerator (acid, etc.) and proceeding with a polymerization reaction by emulsion polymerization. When a thermoplastic resin such as a styrene copolymer is used as the base resin, the fluorescent dye-integrated resin particles can be produced by heating a reaction mixture containing the raw material of the resin, a fluorescent organic dye (monomers to which an organic fluorescent dye has been previously bonded by covalent bonds, etc. may be used as raw material monomers for the resin), and a polymerization initiator (benzoyl peroxide, azobisisobutyronitrile, etc.) and proceeding with a polymerization reaction by radical polymerization or ionic polymerization.
[0057] (2.3.2.6) Average particle size There are no particular limitations on the average particle size of the fluorescent-substance-aggregated nanoparticles used in the present embodiment, but particles with a large diameter have difficulty accessing antigens, while particles with a small diameter and low brightness value have the signal from the fluorescent-substance-aggregated nanoparticles buried in background noise (camera noise and cellular autofluorescence), so particles with a diameter of approximately 20 to 500 nm are preferable. Furthermore, the coefficient of variation (=(standard deviation / average value)×100%), which indicates the variation in particle size, is not particularly limited, but may be 20% or less, and is preferably 5 to 15%. The average particle size was determined by taking an electron micrograph using a scanning electron microscope (SEM), measuring the cross-sectional area of a sufficient number of particles, and calculating the diameter of the circle when each measured value was taken as the area of the circle. In this application, the average particle size was determined as the arithmetic average of the particle sizes of 1000 particles. The coefficient of variation was also calculated from the particle size distribution of 1000 particles.
[0058] (2.3.3) Antibodies As the primary antibody, an antibody (IgG) that specifically recognizes and binds to a protein as an antigen, which is a target substance, can be used. The primary antibody does not have to be a natural full-length antibody, but may be an antibody fragment or derivative, so long as it has the ability to specifically recognize and bind to a specific biological substance (antigen).
[0059] As the secondary antibody, an antibody (IgG) that specifically recognizes and binds to the primary antibody as an antigen can be used.
[0060] Both the primary and secondary antibodies may be polyclonal antibodies, but monoclonal antibodies are preferred from the viewpoint of quantitative stability. The type of animal (immunized animal) producing the antibody is not particularly limited, and may be selected from mice, rats, guinea pigs, rabbits, goats, sheep, etc., as in the past.
[0061] (2.3.4) Immunostaining agents The immunostaining agent is produced by dispersing in an appropriate medium a labeled antibody in which an antibody capable of binding directly or indirectly to a target substance is directly or indirectly bound to a labeling substance. In order to improve the efficiency of fluorescent labeling and minimize the time that leads to deterioration of fluorescence, it is preferable to use a complex in which the primary antibody and the fluorescent substance-holding nanoparticles are indirectly linked by a bond other than a covalent bond, such as by using an antigen-antibody reaction or an avidin-biotin reaction, but this is not limiting.
[0062] An example of an immunostaining agent in which an antibody and fluorescent nanoparticles are indirectly linked is [primary antibody against a target substance] ... [antibody against the primary antibody (secondary antibody)] ~ [fluorescent nanoparticle (phosphor-aggregated nanoparticle)]. Here, "..." indicates that they are bound by an antigen-antibody reaction, and the form of the bond indicated by "~" is not particularly limited, and examples include covalent bonds, ionic bonds, hydrogen bonds, coordinate bonds, physical adsorption, and chemical adsorption, and may be mediated by a linker molecule as necessary.
[0063] The secondary antibody-fluorescent substance-holding nanoparticle conjugate can be prepared, for example, by using a silane coupling agent, which is a compound that is widely used to bond inorganic and organic substances. This silane coupling agent is a compound that has an alkoxysilyl group that gives a silanol group by hydrolysis at one end of the molecule and a functional group such as a carboxyl group, an amino group, an epoxy group, or an aldehyde group at the other end, and bonds to inorganic substances via the oxygen atom of the silanol group. Specific examples include mercaptopropyltriethoxysilane, glycidoxypropyltriethoxysilane, aminopropyltriethoxysilane, and a silane coupling agent having a polyethylene glycol chain (for example, PEG-silaneno.SIM6492.7 manufactured by Gelest). When using a silane coupling agent, two or more types may be used in combination.
[0064] A known method can be used for the reaction procedure between the fluorescent substance-containing nanoparticles and the silane coupling agent. For example, the obtained silica nanoparticles encapsulating the fluorescent substance are dispersed in pure water, and aminopropyltriethoxysilane is added and reacted at room temperature for 12 hours. After the reaction is completed, the silica nanoparticles encapsulating the fluorescent substance and having the surface modified with aminopropyl groups can be obtained by centrifugation or filtration. Then, the amino group is reacted with the carboxyl group in the antibody, so that the antibody can be bonded to the silica nanoparticles encapsulating the fluorescent substance via an amide bond. If necessary, a condensing agent such as EDC (1-Ethyl-3-[3-Dimethylaminopropyl] carbodiimide Hydrochloride: manufactured by Pierce) can also be used.
[0065] If necessary, a linker compound having a site capable of directly binding to silica nanoparticles containing an organic molecule-modified fluorescent substance and a site capable of binding to a molecular target substance can be used. As a specific example, sulfo-SMCC (Sulfosuccinimidyl-4-[N-maleimidomethyl] cyclohexane-1-carboxylate: manufactured by Pierce) having both a site selectively reacting with an amino group and a site selectively reacting with a mercapto group can be used to bind the amino group of the silica nanoparticles containing a fluorescent substance modified with aminopropyltriethoxysilane to the mercapto group in the antibody, thereby obtaining silica nanoparticles containing a fluorescent substance bound to an antibody.
[0066] When binding a biological substance recognition site (a site capable of specifically recognizing a biological substance, such as biotin, avidin, or an antibody) to a polystyrene particle containing a fluorescent substance, the same procedure can be applied whether the fluorescent substance is a fluorescent organic dye or a quantum dot. That is, by impregnating polystyrene nanoparticles having a functional group such as an amino group with quantum dots or a fluorescent organic dye, fluorescent substance-integrated polystyrene particles having a functional group can be obtained, and then by using EDC or sulfo-SMCC, fluorescent substance-integrated polystyrene particles to which an antibody is bound can be obtained.
[0067] Another example of an immunostaining agent in which an antibody and fluorescent nanoparticles are indirectly linked is a complex consisting of three molecules linked in the following manner: [primary antibody against the target substance] ... [antibody against the primary antibody (secondary antibody)] - [biotin] / [avidin] - [fluorophore (fluorescent substance-harboring nanoparticle)] (here, "..." indicates that the molecules are linked via an antigen-antibody reaction, "-" indicates that the molecules are linked via a covalent bond which may be via a linker molecule if necessary, and " / " indicates that the molecules are linked via the avidin-biotin reaction).
[0068] The secondary antibody-biotin conjugate (biotin-modified secondary antibody) can be prepared based on a known method for binding biotin to a desired antibody (protein), for example, by using a commercially available biotin labeling reagent (kit). In addition, if a biotin-modified secondary antibody itself in which biotin is bound to a desired antibody in advance is commercially available, it may be used.
[0069] The fluorescent substance-holding nanoparticle-avidin conjugate (avidin-modified fluorescent substance) can also be prepared based on a known method for binding avidin to a fluorescent substance, for example, by using a commercially available avidin labeling reagent (kit). In this case, the avidin may be an improved type such as streptavidin or neutravidin, which has a stronger binding strength with biotin than avidin.
[0070] A specific example of a method for producing a fluorescent substance-holding nanoparticle-avidin conjugate is as follows. When the fluorescent substance-holding nanoparticles are based on a resin, the functional group of the resin and the functional group of the avidin (protein) can be bonded via a linker molecule such as PEG having functional groups at both ends of the molecule, as necessary. For example, in the case of melamine resin, a functional group such as an amino group can be used, and in the case of acrylic resin, styrene resin, etc., a monomer having a functional group (e.g., an epoxy group) in the side chain can be copolymerized to use the functional group itself or a functional group converted from the functional group (e.g., an amino group generated by reacting with ammonia water), and further, these functional groups can be used to introduce another functional group.
[0071] Furthermore, when the phosphor-holding nanoparticles have a silica base, a desired functional group can be introduced by surface modification with a silane coupling agent; for example, an amino group can be introduced by using aminopropyltrimethoxysilane. On the other hand, a thiol group can be introduced into avidin by reacting, for example, N-succinimidyl S-acetylthioacetate (SATA) with the amino group of avidin. Then, by using a crosslinker reagent having an N-hydroxysuccinimide (NHS) ester reactive with amino groups and a maleimide group reactive with thiol groups at both ends of a polyethylene glycol (PEG) chain, it is possible to link fluorescent substance-holding nanoparticles having amino groups to avidin into which a thiol group has been introduced.
[0072] The conditions for the fluorescent staining process, i.e., the temperature and immersion time when the tissue specimen is immersed in the solution of the immunostaining agent, can be appropriately adjusted so as to obtain an appropriate signal, in accordance with conventional immunohistochemical staining methods. The reaction temperature is not particularly limited, but the reaction can be carried out at room temperature. The reaction time is preferably from 30 minutes to 24 hours. Before carrying out the above-mentioned treatment, it is preferable to drop a known blocking agent such as BSA-containing PBS or a surfactant such as Tween 20.
[0073] For example, when the immunostaining agent is a complex of [primary antibody (probe)]...[secondary antibody]-[biotin] / [avidin]-[fluorescent nanoparticle (fluorescent substance-integrated nanoparticle, etc.)], the tissue specimen may first be immersed in a solution of the primary antibody (primary reaction process), then in a solution of the secondary antibody-biotin conjugate (secondary reaction process), and finally in a solution of avidin-fluorescent nanoparticles dispersed in the fluorescent nanoparticle dilution solution of the present invention (fluorescent labeling process).
[0074] (2.4) Specimen post-processing process After the morphological staining and fluorescent staining steps, the tissue specimen is preferably subjected to processes such as fixation, dehydration, clearing, and mounting so as to be suitable for observation.
[0075] Fixation and dehydration can be achieved by immersing the tissue specimen in a fixative (a cross-linking agent such as formalin, paraformaldehyde, glutaraldehyde, acetone, ethanol, or methanol). Clearance can be achieved by immersing the tissue specimen after fixation and dehydration in a clearing solution (such as xylene). Mounting can be achieved by immersing the tissue specimen after clearing in a mounting solution. The conditions for these treatments, for example the temperature and time for immersing the tissue specimen in a prescribed treatment solution, can be appropriately adjusted so as to obtain an appropriate signal in accordance with conventional immunostaining methods.
[0076] (3) Image acquisition process In the image acquisition step, first, the microscope image acquisition device 1A is set to a bright-field unit and then set to a desired magnification, and a bright-field image of the entire tissue specimen stained in the morphological staining step is observed and photographed. Next, the microscopic image acquisition device 1A is set to the fluorescence unit, and excitation light corresponding to each fluorescent substance that fluorescently labels the target substance used in the fluorescent staining process is irradiated onto the tissue specimen, and a fluorescent image of the entire tissue specimen due to the fluorescence emitted from those fluorescent substances is observed and photographed.
[0077] (4) Score creation process In the score creation process, first, multiple types of information such as area, structure, and cell type in the bright field image are obtained, and each piece of information is combined to calculate a morphological score. An example of software that can be used for image processing and scoring is "ImageJ" (open source). By using such image processing software, it is possible to semi-automatically and quickly perform processes such as discrimination of regions, structures, and cell types based on the shape of each cell and the shape of regions and structures formed by gathering cells from bright-field images in which each cell is stained to be identifiable, and calculation of scores based on information such as the number and area of regions, structures, and cell types.
[0078] Next, information on the presence of the target substance in the fluorescent image is obtained and scored to obtain a biological substance score. The information on the presence of the target substance includes the amount of the target substance (protein, nucleic acid, antibody contained in a pharmaceutical product, etc.) present in one cell and the intracellular localization of the target substance. That is, the information providing device 2A performs image processing on the fluorescent image captured regarding the target substance, extracts fluorescent labeling signals such as fluorescent bright spots corresponding to the target substance, identifies the coordinates of each bright spot, or measures the number of these, and quantifies and scores the amount and intracellular localization of the target substance. When multiple types of target substances are stained, the biological substance score can be calculated for each of them. Software such as the aforementioned "ImageJ" (open source) can also be used for image processing and expression analysis. That is, processes such as extracting bright spots of a certain wavelength (color) from a fluorescent image and counting the number of bright spots with a certain brightness or higher can be performed semi-automatically and quickly.
[0079] Finally, the morphology score calculated in the bright field image and the biological material score calculated in the fluorescent image are added together to generate a final analysis score.
[0080] (5) Area visualization process The pathology diagnosis support system 100 according to this embodiment may include a step of performing fluorescent staining to distinguish between tumor regions and stromal regions as necessary. That is, for example, by fluorescently labeling a substance present in large amounts in stromal regions and performing fluorescent observation, the user can confirm, by fluorescent observation, the boundary between stromal regions in which fluorescence is observed and tumor regions in which no fluorescence is observed or where fluorescence tends to be weak. This can support the distinction between tumor regions and stromal regions based on bright-field images. Specifically, the following staining methods are effective:
[0081] (5.1) Fluorescent staining of cytokines By fluorescently labeling cytokines present in the interstitial region, it is possible to clearly distinguish between the tumor region and the interstitial region by fluorescence observation. In other words, since cytokines are present in greater amounts in the interstitial region than in the tumor region, the boundary between the tumor region and the interstitial region can be visually confirmed by the difference in fluorescence brightness. In this process, an antibody (IgG) that specifically recognizes and binds to the cytokine as an antigen is used, and the tissue specimen is stained with an immunostaining agent in which the antibody is labeled with a fluorescent substance. Note that the fluorescent substance used for fluorescent labeling is not limited to fluorescent substance-aggregated nanoparticles, and the tumor region and the stromal region can be sufficiently distinguished even if a single fluorescent dye is used. Specifically, the staining can be performed by the same procedure as in the above "(2.3) Fluorescent staining step." Furthermore, the fluorescent staining of cytokines can be performed simultaneously with the fluorescent staining of the target substance in the above "(2.3) Fluorescent staining step." However, it is preferable to use a fluorescent substance with a different fluorescent wavelength from that of the fluorescent substance-holding nanoparticles that label the target substance. Examples of cytokines include IL-1, IL-2, IL-4, IL-6, IL-10, IL-12, IL-18, IFN-α, IFN-β, IFN-γ, TNF, and TGF-β.
[0082] (5.2) Fluorescent staining of proteins in immune cells Proteins in immune cells here refer to proteins that are specifically expressed in immune cells, and fluorescent labeling of these proteins also allows clear identification of tumor and stromal regions by fluorescent observation. That is, the fluorescent brightness of immune cells in the stromal region is significantly greater than that of immune cells in the tumor region, so the boundary between the regions can be visually confirmed from the difference in brightness. When fluorescent staining of proteins highly expressed in the stromal region is performed in the above "(2.3) Fluorescent Staining Step", this step can be substituted by fluorescent observation of the target substance, but when this is not performed, it is effective to fluorescently stain the proteins exemplified below in this step. In this process, an antibody (IgG) that specifically recognizes and binds to a protein in immune cells as an antigen is used, and the tissue specimen is stained with an immunostaining agent in which the antibody is labeled with a fluorescent substance. The fluorescent substance used for fluorescent labeling is not limited to fluorescent substance-aggregated nanoparticles, and a single fluorescent dye can be used to sufficiently distinguish between tumor and interstitial regions. The staining can be performed using the same procedure as in the above "(2.3) Fluorescent staining process." Examples of proteins in immune cells include PD-1, CTLA-4, TIM3, Foxp3, CD3, CD4, CD8, CD25, CD27, CD28, CD70, CD40, CD40L, CD80, CD86, CD160, CD57, CD226, CD112, CD155, OX40 (CD134), OX40L (CD252), ICOS (CD278), ICOSL (CD275), 4-1BB (CD137), 4-1BBL (CD137L), These include 2B4 (CD244), GITR (CD357), B7-H3 (CD276), LAG-3 (CD223), BTLA (CD272), HVEM (CD270), GITRL, Galectin-9, B7-H4, B7-H5, PD-L2, KLRG-1, E-Cadherin, N-Cadherin, R-Cadherin, and IDO, TDO, CSF-1R, HDAC, CXCR4, FLT-3, and TIGIT.
[0083] <Operation of the pathology diagnosis support system 100> The following describes the operation of acquiring and analyzing the above-described fluorescent images and bright-field images in the pathology diagnosis support system 100. Here, the description takes as an example a case in which an arbitrary target substance is selected from proteins expressed in immune cells, and the observation subject is a tissue specimen stained with a staining reagent containing fluorescent substance-aggregated nanoparticles bound to a biological substance recognition moiety that recognizes the target substance.
[0084] First, the operator stains the tissue specimen with two types of staining reagents: an HE staining reagent and a staining reagent that uses, as a fluorescent labeling material, fluorescent substance-holding nanoparticles to which a biological substance recognition moiety that recognizes a target substance is bound. Thereafter, in the microscopic image acquiring device 1A, a bright field image and a fluorescent image are acquired in the order of (a1) to (a5). (a1) An operator places tissue specimens stained with both an HE staining reagent and a staining reagent containing fluorescent substance-holding nanoparticles on a slide, and places the slide on a slide fixing stage of microscope image acquisition device 1A. (a2) Set it to the bright field unit and adjust the shooting magnification and focus. (a3) The imaging means captures an image of the entire tissue specimen to generate image data of a bright-field image, and transmits the image data to the information providing device 2A. (a4) Change the unit to a fluorescent unit. (a5) Without changing the field of view and the imaging magnification, the imaging means captures an image of the entire tissue specimen to generate image data of a fluorescent image, and transmits the image data to the information providing device 2A.
[0085] Next, the information providing device 2A executes an image acquiring step, a score creating step, and an information presenting step. 3 is a flowchart showing the process in the information providing device 2A. The process shown in FIG.
[0086] First, under the control of the control unit 21, when a bright field image is input from the microscope image acquisition device 1A via the communication I / F 24 (step S1: image acquisition step), a target for score calculation is specified (step S2). 4 shows a detailed flow of the process in step S2. The process in step S2 is executed by the control unit 21 in cooperation with a program stored in the storage unit 25.
[0087] In step S2, first, the cell type is identified (step S21). 5 shows a detailed flow of the process in step S21. The process in step S21 is executed by the control unit 21 in cooperation with a program stored in the storage unit 25.
[0088] In step S21, first, a cell nucleus is extracted (step S211). Specifically, first, the bright-field image is converted into a monochrome image, and then threshold processing is performed on the monochrome image using a predetermined threshold, and the value of each pixel is binarized. Furthermore, noise processing is performed by performing closing processing and opening processing. After the noise processing, an image from which cell nuclei have been extracted (cell nucleus image) is obtained. The noise-processed image is then subjected to a labeling process, in which a label is assigned to each of the extracted cell nuclei. The labeling process identifies objects in an image by assigning the same label (number) to connected pixels. The labeling process makes it possible to identify and label each cell nucleus in the noise-processed image. In step S211, in addition to the method of extracting cell nuclei, the cells themselves may be extracted. In this case, an image in which cells are extracted is obtained after noise processing, and a label is assigned to each cell by labeling processing.
[0089] Next, the cell features are calculated (step S212). Specifically, for all cell nuclei in the cell nucleus image extracted in step S211, cell features such as the area A of the cell nucleus, the average concentration B of the cell nucleus, the pixel brightness variation (σ value) C within the cell nucleus, the circularity D of the cell nucleus, and the flattening E of the cell nucleus are calculated from the cell nucleus image. The area A of the cell nucleus is determined by calculating the size of a pixel in advance by measuring a reference length corresponding to the cell nucleus image, and accumulating the number of pixels in each cell nucleus extracted in step 211. The average cell concentration B is determined by obtaining the luminance signal value converted into gray scale for each pixel in the cell nucleus and calculating the average value. The pixel brightness variation C is determined by calculating the standard deviation of the brightness signal value of each picture element (pixel) within the cell nucleus. The circularity D and flattening E of the cell nucleus are determined for each cell nucleus extracted in step 211 by applying certain values obtained from the cell nucleus image to the following formulas (d) and (e). (Circularity D) = 4πS / L2 … (d) (Oblateness E)=(ab) / a … (e) In formula (d), "S" represents the area of the cell (area A of the cell nucleus), and "L" represents the circumference of the cell nucleus. In formula (e), "a" represents the semimajor axis, and "b" represents the semiminor axis.
[0090] Next, threshold processing of the cell features is performed based on the calculated cell features (step S213). For example, thresholds for distinguishing between tumor cells and normal cells are set for the above-mentioned cell features A to E, and these threshold processings are performed on all cell nuclei in the cell nucleus image to identify them as tumor cells or normal cells. Threshold processing may also be performed to distinguish cell types such as immune cells, inflammatory cells, fibroblasts, and endothelial cells in the interstitial region. In addition, various cell types can be identified by setting any threshold depending on the cell type to be identified.
[0091] The above-mentioned cell feature factors A to E are merely examples, and other factors different from the above may be used as cell feature factors. For example, when comparing tumor cells and normal cells in pathological diagnosis, cell size and shape, the position, size, change in shape and nuclear division image of the cell nucleus, color of the cytoplasm, the presence or absence of mucus production, and the cell nucleus / cytoplasm area ratio are observed. Such items that are of interest in actual pathological diagnosis may be extracted as cell features and used to identify the cell type.
[0092] Next, structures such as blood vessels, lymphatic vessels, and secretory glands are identified (step S22). In step S22, for example, structures formed by a certain amount of cells are extracted from the cell nucleus image. Next, feature values such as area, average brightness, brightness variation, flatness, and circularity of each extracted structure are calculated, and blood vessels, lymphatic vessels, secretory glands, etc. are identified based on the feature values.
[0093] Next, the area is specified (step S23). In identifying the regions, a tumor region where a certain amount of tumor cells are present and a stromal region where a certain amount of stromal cells are present are identified. In step S23, for example, when a collection of cells (cell nuclei) identified as tumor cells on the cell nucleus image in step S21 occupies a predetermined area or more, the collection of tumor cells can be identified as a tumor region. Similarly, when a collection of cells (cell nuclei) identified as stromal cells in step S21 occupies a predetermined area or more, the collection of stromal cells can be identified as a stromal region. By the process of step S23, the specification of the score calculation target is completed.
[0094] It should be noted that the process of identifying the score calculation target in steps S21 to S23 above is basically performed automatically in cooperation with the control unit 21 and a program stored in the memory unit 25, but such process may involve assisted work by a user such as a pathologist. The auxiliary work by the user includes, for example, in the region specifying process of step S21, an operation in which the user selects a specific region on the input bright-field image displayed on the display unit 23 of the information providing device 2A by surrounding it with a closed figure such as a polygon or a free curve using the operation unit 22, and defines each selected point as a "tumor region," "stromal region," etc. It also includes an operation to confirm the result of the region specifying process in the bright-field image by observing the fluorescence in the fluorescence image. The process of identifying the structure in step S22 and the process of identifying the cell type in step S23 also include the action of the user selecting and defining on the bright-field image. Furthermore, the threshold processing of the cell feature amounts in step S233 includes, for example, gradually adjusting the threshold values of the cell feature amounts for the program stored in the memory unit 25, and visually confirming the identified cell types.
[0095] Furthermore, the process of identifying the score calculation target in steps S21 to S23 may be performed automatically using an image recognition technique trained by machine learning, in addition to the above-mentioned method. That is, for example, by using a method such as a neural network, a random forest, or an SVM (Support Vector Machine), the regions, structures, and cell types appearing in the bright-field image can be learned, and these items in the input bright-field image can be automatically identified.
[0096] Next, a morphology score is calculated (step S3: score generation step). A detailed flow of the process in step S3 is shown in Fig. 6. The process in step S24 is executed by the control unit 21 in cooperation with a program stored in the storage unit 25.
[0097] In step S3, first, a score based on the region is calculated (step S31). The area-based score is a score obtained from information obtained from various areas such as the tumor area and the stromal area present on the tissue section. Since the size of the tumor area is associated with the malignancy of cancer, it is effective to score, for example, the proportion of the tumor area to the total area of the imaged tissue section, the proportion of the area of the tumor area and the stromal area to the total area, the ratio of the area of the stromal area to the area of the tumor area, etc.
[0098] Next, a structure-based score is calculated (step S32). The structure-based score is a score obtained from information obtained from various structures such as blood vessels, lymphatic vessels, and secretory vessels present on a tissue section. Since it is known that blood vessel density is an evaluation index of the malignancy of cancer, it is effective to score, for example, the presence or absence, density, or number of occurrences of structures related to the malignancy of cancer on a tissue section.
[0099] Next, a score based on the cell type is calculated (step S33). The score based on cell type is a score obtained from information obtained from cell types such as tumor cells and stromal cells present on a tissue section. For example, the number of cancer cells and normal cells present on a tissue section may be scored. In addition, various information that is of interest in pathological diagnosis, such as the density of these cancer cells on the tissue section, the distance between different cell types such as the distance between cancer cells in the tumor region and immune cells in the stromal region, and the distance between a specific cell and a specific structure such as the distance between cancer cells in the tumor region and blood vessels in the stromal region, may be scored. The calculation of the morphological score is completed by the process in step S33, and the process returns to FIG.
[0100] In addition, in steps S31 to S33, weighting may be applied to each score depending on the purpose, or a subtotal of the scores may be calculated at any stage and weighting may be applied to the subtotal. However, it is preferable to calculate each score based on the region, structure, and cell type in the order of steps S31 to S33. That is, for example, by first evaluating the area of the tumor region present on the tissue slice, then evaluating the structure of blood vessels and the like present on the tissue slice, and then evaluating the number and arrangement of tumor cells, the order is the same as that when a pathologist actually observes a tissue specimen and makes a diagnosis, and it is possible to approach a clinical diagnosis method, so that the accuracy of the support information can be improved. The above-mentioned methods of calculating the region score, structure score, and cell type score are merely examples, and are not intended to limit the scope of the invention. In addition, it is desirable to score items of interest in pathological diagnosis.
[0101] On the other hand, when a fluorescent image is input from the microscopic image acquisition device 1A via the communication I / F 24 (step S4: image acquisition step), fluorescent bright spots are extracted from the fluorescent image (step S5). In step S5, first, the R component is extracted from the fluorescent image, and the image from which the R component has been extracted is subjected to Tophat transformation. Tophat transformation is a process in which the value of each pixel in the input image is subtracted from the value of the corresponding pixel in an image in which the input image has been filtered through a minimum value filter and a maximum value filter in this order. The minimum value filter replaces the value of the pixel of interest with the minimum value of pixels (e.g., 3×3 pixels) in the vicinity of the pixel of interest. The maximum value filter replaces the value of the pixel of interest with the maximum value of pixels (e.g., 3×3 pixels) in the vicinity of the pixel of interest. The Tophat transformation makes it possible to extract small protrusions (areas with higher brightness than the pixels in the vicinity) on the grayscale profile. This makes it possible to obtain a fluorescent bright spot candidate image. Next, noise is removed from the fluorescent bright spot candidate image to obtain an image from which fluorescent bright spots have been extracted (fluorescent bright spot image), and a labeling process is performed on the image after noise removal, and a label is assigned to each of the extracted fluorescent bright spots.
[0102] Next, the biological substance score is calculated (step S6: score generation step). 7 shows a detailed flow of the process in step S6. The process in step S6 is executed by the control unit 21 in cooperation with a program stored in the storage unit 25.
[0103] In step S6, first, the bright-field image and the fluorescent image are aligned based on an information source commonly detected in the bright-field image and the fluorescent image (step S61). The bright-field image used for the alignment is the cell nucleus image obtained in step S2, and the fluorescent image used for the alignment is the fluorescent spot image obtained in step S5. As an information source commonly detected in the bright-field image and the fluorescent image, information recognizable in both the bright-field image and the fluorescent image is used, and in this embodiment, eosin staining information, which is a staining material for tissue sections, is used. From the eosin staining information, image feature quantities such as contrast information, edge information, and contour information that characterize each image in the bright-field image and the fluorescent image are calculated. Contrast information, edge information, and contour information for each image are calculated from the eosin staining information of the bright-field image and the fluorescent image. The information between these images is then compared, and the common points are matched to align the bright-field image and the fluorescent image.
[0104] Next, the presence of the target substance in each cell is determined from both the aligned bright-field image (cell nucleus image obtained in step S2) and the fluorescent image (fluorescent spot image obtained in step S5) (step S62). Specifically, for example, in a state where the bright field image and the fluorescent image are superimposed, the number of fluorescent bright spots in the portion corresponding to the cell identified in step S23 is calculated. In addition to the number of fluorescent bright spots, a luminance value may be calculated for each bright spot region, and an integrated luminance value may be calculated by adding up the luminance values of all the bright spot regions in the fluorescent image.
[0105] Next, a score according to the presence state is calculated (step S63). The score according to the presence state is a score calculated according to the amount of target substance present per cell, intracellular localization, etc. When multiple target substances are fluorescently stained, the processes of steps S5 and S6 are performed for each target substance, and a biosubstance score for each target substance can be obtained. In step S63, the calculation of the biological substance score is completed.
[0106] Next, the analysis score is calculated (step S7: score generation step). The analysis score is calculated by adding up the morphology score calculated in step S3 and the biological substance score calculated in step S6. When adding up the morphology score and the biological substance score, any weighting may be applied so that the user can obtain the support information that he or she desires.
[0107] Next, the calculated analysis score is displayed on the display unit 23 (step S8: information presentation step). This allows the user to confirm the analysis score presented as support information by the information providing device 2A. Note that in step S8, the manner in which the analysis score is presented is not particularly limited, and the analysis score may be presented as a numerical value only, or may be presented with the analysis score (which may be a morphology score and a biological material score) displayed on the bright field image and the fluorescent image. When the analysis score and the image are displayed in association with each other, the relationship between the analysis score and the state of the tissue section is easily visible, which is highly convenient for the user. This completes the process in the information providing device 2A.
[0108] As described above, in the information providing method according to the present embodiment, multiple types of information are obtained from a digitized bright-field image of a tissue section, and an analysis score is created by combining these pieces of information and scoring them, and is presented to the user as support information. In actual pathological diagnosis, etc., it is performed based on not only a single piece of information (e.g., the number of tumor cells) but also multiple types of information (e.g., the number of tumor cells and the number of immune cells) obtained from the tissue section. In the information providing method according to the present embodiment, an analysis score is provided that comprehensively evaluates the tissue section by taking into account multiple types of information, as in the case of actual judgment. As a result, the analysis score functions as a highly accurate index for various judgments, and the user can perform a pathological diagnosis, etc. based on the value of the analysis score, and the user can check the accuracy of his / her own judgment by comparing the analysis score displayed on the information providing device 2A with the results of a pathological diagnosis, etc., performed by the user by observing the tissue section. Furthermore, since the images of the tissue sections are digitally analyzed and evaluated, it is possible to provide more objective information than subjective judgments made by pathologists, etc., and the information provision method according to this embodiment supports various judgments such as pathological diagnoses, thereby improving accuracy.
[0109] Furthermore, by acquiring microscopic images using a virtual microscope slide creation device, judgments can be made by observing the entire tissue section, without the need to make judgments from a representative field of view.
[0110] In addition, the analysis score can be created by combining information obtained from the region, structure, and cell type. Similar to a pathological diagnosis by an actual pathologist, the score is evaluated comprehensively based on the relationships between these, so highly accurate support information can be provided. Furthermore, by creating an analysis score using information related to the localization relationships between regions, structures, and cell types, such as the distance between cancer cells in the tumor region and immune cells in the interstitial region, or the distance between cancer cells and specific structures such as blood vessels, it is possible to provide more reliable support information. In addition, by setting the items to be scored and weighting according to the purpose, based on the principle of scoring each area, structure, and cell type, the information that the user wants can be presented as a score.
[0111] Furthermore, by staining a specific biological material with fluorescent substance-holding nanoparticles so that the material can be fluorescently observed, the analysis score can be correlated with the amount of the biological material present and used as a new index for pathological diagnosis and the like.
[0112] Furthermore, by dyeing the regions differently in the region dyeing process, the region boundaries can be clearly seen, thereby increasing the credibility of the support information.
[0113] In the above embodiment, the target substance is fluorescently stained, but the fluorescent staining step may be omitted. As described above, the present invention aims to support various judgments by pathologists and the like by automatically scoring and presenting information obtained from pathological sections. This aim can be fully achieved by obtaining a score that combines multiple types of information (information relating to area, structure, and cell type) obtained from bright field images to obtain a result of evaluating the pathological section in a manner that is close to the judgment criteria that a pathologist and the like actually uses to make a judgment. In other words, unlike Patent Documents 1 and 2, it is not necessarily necessary to fluorescently stain biological materials, and it is possible to create an analysis score using only the morphological staining process, thereby eliminating the variation between slides and the effort required for each staining.
[0114] A specific example of calculation of the analysis score will be described below with reference to FIG. 8A shows a bright-field image 200 obtained by photographing a tissue specimen, which is a tissue slice taken from a subject and stained with HE, using the bright-field unit of the microscopic image acquisition device 1 A. As shown in FIG. 8A, a tumor region 210 and a stromal region 220 are present in the bright-field image 200, and blood vessels 230 are present in the stromal region 220. The tissue sections were stained for the target substance (HER2) using fluorescent substance-aggregated nanoparticles, and although not shown in the figure, a fluorescent image of the same field of view as bright field image 200 was acquired using a fluorescent unit of the microscopic image acquisition device 1A.
[0115] Fig. 8B is a flow chart showing a procedure for calculating an analysis score in which information used for scoring is set and the weighting of the score is defined so that the support information desired by the user is output. Hereinafter, a method for calculating an analysis score according to the flow shown in Fig. 8B will be described using the tissue specimen shown in Fig. 8A as an example. First, the area of the region is scored (step S101). Here, a value N1 is calculated by scoring the ratio of the area of tumor region 210 and stromal region 220 to the area of the entire tissue specimen, and N1 is a standardized value obtained by the formula (area of tumor region 210 and stromal region 220 / area of bright-field image 200). From the tissue specimen in Figure 8A, N1 = 40 was obtained.
[0116] Next, the presence or absence of a structure is scored (step S102). Here, a value N2 is obtained by scoring the presence or absence of blood vessels on the tissue specimen, and N2=20 if blood vessels are present, and N2=0 if blood vessels are not present. In the tissue specimen of FIG. 8A, blood vessels 230 were observed in the interstitial region 220, so N2=20 was obtained.
[0117] Next, N1 and N2 are weighted based on the number of structures that appear (step S103). Here, if the number of blood vessels that appear on the tissue specimen is greater than a predetermined number, N3 = (N1 + N2) × 2 is used, and if it is less than the predetermined number, N3 = (N1 + N2) × 1 is used. In the tissue specimen of FIG. 8A, the number of blood vessels 230 is greater than the predetermined number, so N3=(N1+N2)×2=(40+20)×2=120 was obtained.
[0118] Next, the number of cell types is scored (step S104). Here, the number of tumor cells on the tissue specimen is scored to obtain a value N4, which is a normalized value for the number of tumor cells. From the tissue specimen in FIG. 8A, N4=30 was obtained.
[0119] Next, the cell type density is scored (step S105). Here, a value N5 is obtained by scoring the density of tumor cells on the tissue specimen, and if the density of tumor cells on the tissue specimen is higher than a predetermined density, N5=10, and if it is lower than the predetermined density, N5=0. In the tissue specimen of FIG. 8A, N5=10 was obtained because the density of tumor cells was higher than the predetermined density.
[0120] Next, weighting is performed on N4 and N5 based on the intercellular distance (step S106). Here, when the distance between the tumor cells in the tumor region 210 on the tissue specimen and the immune cells in the interstitial region 220 is shorter than a predetermined distance (the cells are close to each other), N6 = (N4 + N5) × 2 is set, and when the distance is longer than the predetermined distance (the cells are far from each other), N6 = (N4 + N5) × 1 is set. In the tissue specimen of FIG. 8A, the intercellular distance was smaller than the predetermined distance, so N6 = (N4 + N5) × 2 = (30 + 10) × 2 = 80 was obtained.
[0121] Next, a biological substance score N7 is calculated (step S108). Here, the biological substance score is a value N7 obtained by scoring the number of bright spots of the phosphor-holding nanoparticles based on the amount of the target substance present, and N7 is a value obtained by normalizing the number of fluorescent bright spots. From the tissue specimen in Figure 8A, N7=50 was obtained.
[0122] The scores calculated as above are added up to obtain an analysis score N (step S108). Here, N=N3+N6+N7. From the tissue specimen in Figure 8A, N = N3 + N6 + N7 = 120 + 80 + 50 = 250 was obtained. The analysis score N=250 is presented to the user. For example, if the numerical value of the analysis score is linked to the malignancy of the cancer in advance, the user can judge the malignancy of the cancer of the subject by referring to the analysis score. In this way, the analysis score can be used as various indexes in pathological diagnosis, etc.
[0123] [Other embodiments] Although the preferred embodiment to which the present invention is applied has been described above, the description of the above embodiment is merely a preferred example of the present invention, and the present invention is not limited to this.
[0124] In the above description, an example has been disclosed in which a HDD or a non-volatile semiconductor memory is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, a portable recording medium such as a CD-ROM can be applied. In addition, a carrier wave can be applied as a medium for providing data of the program according to the present invention via a communication line.
[0125] In addition, the detailed configurations and operations of the devices constituting the pathological diagnosis support system 100 may be modified as appropriate without departing from the spirit of the invention. [Industrial Applicability]
[0126] The present invention can be used in an information providing method, information providing device, and program capable of providing objective, highly accurate support information for various decisions made based on information obtained from tissue sections. [Explanation of symbols]
[0127] 100 Pathological diagnosis support system 1A Microscope image acquisition device 2A Information provision device 21 Control unit (image acquisition unit, score calculation unit, information presentation unit) 22 Control section 23 Display section 24 Communication I / F 25 Memory section N Communication Network
Claims
1. 1. A method for providing support information for supporting a decision based on information obtained from a tissue section, comprising: an image acquisition step of acquiring a digitized bright-field image of the stained tissue section for bright-field observation; a score creation step of acquiring a plurality of types of information from the bright field image and creating an analysis score by combining the plurality of types of information; and an information presenting step of presenting the analysis score as the support information, In the score creation step, a score based on the proportion or ratio of the areas of various regions present in the tissue section, a score based on the presence or absence of various structures, the density of various structures or the number of occurrences of various structures, and a score based on the number of each cell type, the density of each cell type, the distance between different cell types or the distance between a specific cell type and a specific structure are calculated in this order, and the analysis score is created based on the calculated scores. Information provision method.
2. In the image acquisition step, a bright-field image of the entire tissue section is acquired by imaging the entire tissue section using a virtual microscope slide creation device capable of imaging the entire tissue section. The information providing method according to claim 1 .
3. In the score generation step, information regarding the localization relationship between regions, structures, and cell types present in the tissue section is obtained.
3. The information providing method according to claim 1 or 2.
4. the tissue section is stained with a staining reagent in which a biological substance recognition site is bound to a fluorescent substance-holding nanoparticle having a plurality of fluorescent substances accumulated therein, so that a specific biological substance present in the tissue section can be fluorescently observed; The image acquisition step further comprises acquiring a digitized fluorescent image of the tissue section; In the score creation step, information relating to the presence of the specific biological material is further obtained from the fluorescent image, and an analysis score is created by combining the information with the plurality of types of information. The information providing method according to any one of claims 1 to 3.
5. A region visualization step is provided in which biological material present in a specific region on the tissue section is stained with a fluorescent material so as to be fluorescently observable. The information providing method according to any one of claims 1 to 4.
6. An information providing device that provides support information for supporting a decision based on information obtained from a tissue section, comprising: an image acquisition unit that acquires a digitized bright-field image of a tissue section stained to enable bright-field observation; a score creation unit that acquires a plurality of types of information from the bright field image and creates an analysis score by combining the plurality of types of information; an information presenting unit that presents the analysis score as the support information, The score creation unit calculates in this order a score based on the proportion or ratio of areas of various regions present in the tissue section, a score based on the presence or absence of various structures, the density of various structures or the number of occurrences of various structures, and a score based on the number of each cell type, the density of each cell type, the distance between different cell types, or the distance between a specific cell type and a specific structure, and creates the analysis score based on the calculated scores. Information provision device.
7. a computer of an information providing device that provides support information for supporting a decision based on information obtained from a tissue section, an image acquisition unit for acquiring a digitized bright-field image of a tissue section stained to enable bright-field observation; a score creation unit that acquires multiple types of information from the bright field image and creates an analysis score by combining the multiple types of information; an information presenting unit that presents the analysis score as the support information; A program for causing the device to function as a The score creation unit is a program that calculates, in the order of: a score based on the area proportion or ratio of various regions present in the tissue section from the bright field image; a score based on the presence or absence of various structures, the density of various structures or the number of occurrences of various structures; and a score based on the number of each cell type, the density of each cell type, the distance between different cell types, or the distance between a specific cell type and a specific structure, and creates the analysis score based on the calculated scores.
Citation Information
Patent Citations
Pathological tissue diagnosis support device
JP1998197522A
Image analysis for prognosis of solid tumor progression
JP2013537969A
A Machine Learning System Based on Tissue Objects for Automatic Scoring of Digital Whole Slides
JP2016518813A
Method for prognostication of survival of patients with solid tumors
JP2016530505A
Systems and methods for specimen interpretation
US20200160032A1
