Apparatus and method for predicting cell composition of tissue image based on spatial gene expression information

Abstract

An apparatus for predicting the cellular composition of a tissue image based on spatial gene expression information includes a communication module that receives a tissue image of a test subject; a memory that stores a program for predicting cellular composition information from the tissue image; and a processor that executes the program. The program predicts cellular composition information by inputting the tissue image into a cellular composition prediction model that has been trained based on learning data consisting of spatial transcript information and tissue images spatially aligned therewith. The spatial transcript information includes transcript data including spatial information, and tissue image data that shares the spatial information. The spatial information means positional information for a plurality of spots arranged on a two-dimensional plane of the tissue image data, and includes the coordinates of each spot.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for predicting the cell composition of a tissue image based on spatial gene expression information.

Background Art

[0002] A microscopic image of a tissue has information composed of various cells, and each cell has a complex structure depending on its function. In order to find cell types functionally classified from such a complex structure and obtain biomarkers, molecular-specific staining methods (immunohistochemical staining, FISH (Fluorescence in Situ Hybridization), etc.) other than simple tissue images are widely used.

[0003] In the case of such molecular-level pathological images, only one or several types of molecular information can be obtained per experiment, and additional experimental processes and materials for the tissue are required.

[0004] The spatially resolved transcriptome technology, which has been developed and utilized in recent years, can acquire gene expression information of hundreds to tens of thousands of types at once and can acquire gene expression information while preserving tissue position information.

[0005] On the other hand, the labeling work for morphologically classifying the composition of a tissue from a simple tissue image (hematoxylin and eosin staining) is performed by experts in pathological tissue images. Based on this, in recent years, deep learning technologies for predicting the labeling of pathological tissue images from tissue images have been developed.

[0006] In connection with this, Republic of Korea Registered Patent No. 10-2108050 (Title of Invention: Method and Apparatus for Classifying Breast Cancer Histological Images via an Enhanced Convolutional Network) discloses a method for classifying breast cancer histological images via an enhanced convolutional network.

Summary of the Invention

Problems to be Solved by the Invention

[0007] The present invention is for solving the above-described problems, and inputs a general tissue image without spatial transcriptome information into a cell composition prediction model learned based on a spatial transcriptome including a transfer body that shares spatial information and spatial transcriptome information including tissue images, and predicts complex composition information of cells in the tissue. One technical problem is to provide an apparatus and a method for this.

[0008] However, the technical problems to be achieved by this embodiment are not limited to the above-described technical problems, and there may be other technical problems.

Means for Solving the Problems

[0009] As a technical means for solving the above-described technical problems, an apparatus for predicting the cell composition of a tissue image based on spatial gene expression information according to a first aspect of the present invention includes a communication module that receives a tissue image of an object to be inspected; a memory that stores a program for predicting cell composition information from the tissue image; and a processor that executes the program. The program predicts cell composition information by inputting the tissue image into a cell composition prediction model learned based on learning data including spatial transcriptome information and a tissue image spatially aligned therewith. The spatial transcriptome information includes transcriptome data including spatial information and tissue image data that share the spatial information. The spatial information means position information for a plurality of spots arranged on a two-dimensional plane of the tissue image data, and includes coordinates of each spot.

[0010] Also, a method for predicting the cell composition of a tissue image based on spatial gene expression information using the tissue image cell composition prediction device according to the second aspect of the present invention includes: receiving a tissue image of a subject to be examined; and predicting cell composition information by inputting the tissue image into a cell composition prediction model learned based on learning data including spatial transcript information and a tissue image spatially aligned therewith. The spatial transcript information includes transcript data including spatial information and tissue image data sharing the spatial information. The spatial information means position information for a plurality of spots arranged on the two-dimensional plane of the tissue image data and includes the coordinates of each spot.

Advantages of the Invention

[0011] According to an embodiment of the present invention, it is possible to present a learning model that can predict the composition information of various cells only from the morphological information of an easily obtainable tissue image (H&E staining).

[0012] In addition, by using, as learning data, tissue images secured in various diseases and spatial transcript information including transcript data sharing spatial information, it is possible to present a learning model that can predict cell composition information by tissue or disease type.

[0013] On the other hand, previously, a deep learning-based algorithm for estimating the detailed classification of tissues from tissue images (H&E staining) has been reported, but the corresponding technology has the disadvantage that visual interpretation and labeling by tissue image experts are essential. Also, such labeling has the problems of consuming a lot of time and effort and inducing differences among evaluators. In addition, since it does not provide molecular-level information, it has been difficult to develop an algorithm for inferring the distribution under detailed molecular-functional cell classification.

[0014] However, the present invention solves the above-described problems and, firstly, can be utilized as a numerical value for quantifying the diversity of cell populations in various diseases. That is, by inputting only simple tissue images into the learning model, quantitative information regarding the diversity of cell populations can be obtained. This can be variously applied to pathophysiological research on various diseases (such as cancer and inflammatory diseases), research on the development of new treatment techniques, research on the development of diagnostic biomarkers, and the like.

[0015] Secondly, by inputting only simple tissue images into the learning model, molecular functional cells can be quantified. This has utility value as a biomarker that can indicate the characteristics and severity of a specific disease or predict the treatment effect.

[0016] For example, it is well known that when inflammatory cells other than cancer cells are concentrated in a tumor, it is closely related to predicting the reactivity of tumor immunotherapy. That is, the simple tissue image input into the learning model according to the present invention can predict the distribution information of inflammatory cells and can be utilized as a quantitative biomarker.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Best Mode for Carrying Out the Invention

[0018] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. However, the present invention can be embodied in various different forms and is not limited to each embodiment described herein. Also, the accompanying drawings are merely for facilitating understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the accompanying drawings. In the drawings, parts not related to the description are omitted for clarity in explaining the present invention, and the sizes, forms, and shapes of each component shown in the drawings can be variously deformed. Throughout the specification, the same / similar reference numerals are assigned to the same / similar parts.

[0019] Suffixes such as "module" and "unit" for components used in the following description are given or mixed only for ease of preparing the specification and do not have meanings or roles that distinguish them from each other. Also, in explaining the embodiments disclosed in this specification, when it is determined that a specific description of related known technologies may obscure the gist of the embodiments disclosed in this specification, detailed descriptions thereof are omitted.

[0020] Throughout the specification, when one part is "connected (joined, contacted, or coupled)" to another part, this includes not only the case where they are "directly connected (joined, contacted, or coupled)", but also the case where they are "indirectly connected (joined, contacted, or coupled)" with other members interposed therebetween. Also, when one part "includes (comprises or is provided with)" one component, this means that, unless otherwise specified to the contrary, it does not exclude other components and can further "include (comprise or be provided with)" other components.

[0021] Each term indicating an ordinal number such as "first", "second", etc. used in this specification is used only for the purpose of distinguishing one component from another and does not limit the order or relationship of each component. For example, the first component of the present invention can be named the second component, and similarly, the second component can also be named the first component.

[0022] FIG. 1 is a block diagram showing the configuration of a cell composition prediction apparatus for tissue images according to an embodiment of the present invention.

[0023] Referring to FIG. 1, the cell composition prediction apparatus 100 for tissue images includes a communication module 110, a memory 120, and a processor 130, and may further include a database 140. The cell composition prediction apparatus 100 for tissue images receives a tissue image of an object to be inspected and performs an operation of predicting cell composition information using the same.

[0024] For this purpose, the tissue image cell composition prediction device 100 of the tissue image can be embodied in a computer or a portable terminal device that can be connected to a server or other terminals via a network. Here, the computer includes, for example, a notebook computer, a desktop, a laptop, etc. equipped with a web browser (WEB Browser), and the portable terminal device is, for example, a wireless communication device that guarantees portability and mobility, and can include all types of handheld-based wireless communication devices such as various smartphones, tablet PCs, smartwatches, etc.

[0025] The network means a connection structure that enables information exchange between each node such as each terminal and each device, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW: World Wide Web), a wired / wireless data communication network, a telephone network, a wired / wireless TV communication network, etc. An example of the wireless data communication network includes, but is not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC: Visible Light Communication), LiFi, etc.

[0026] The communication module 110 receives a tissue image of the object to be inspected. At this time, the tissue image is a tissue image that can generally be easily obtained through a microscope and does not include spatial transcript information. The communication module 110 can include a device that includes the hardware and software necessary to transmit and receive signals such as control signals or data signals through wired / wireless connection with other network devices.

[0027] The memory 120 stores a program for predicting cell composition information from the tissue image received via the communication module 110. At this time, the program for predicting cell composition information predicts cell composition information by inputting the tissue image into a cell composition prediction model learned based on learning data consisting of spatial transcriptome information and a tissue image spatially aligned therewith. The specific content of the cell composition information will be described later.

[0028] At this time, the memory 120 must be interpreted as a general term for a non-volatile storage device that continues to maintain the stored information even when power is not supplied, and a volatile storage device that requires power to maintain the stored information. The memory 120 can perform a function of temporarily or permanently storing data processed by the processor 130. The memory 120 may include a magnetic storage media or a flash storage media in addition to the volatile storage device that requires power to maintain the stored information, but the scope of the present invention is not limited thereto.

[0029] The processor 130 executes a program for predicting cell composition information stored in the memory 120, and outputs cell composition information for the object as a result of the execution.

[0030] In one example, the processor 130 may be embodied in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an ASIC (application-specific integrated circuit), an FPGA (field programmable gate array), etc., but the scope of the present invention is not limited thereto.

[0031] The database 140 can store tissue images received via the communication module 110 and various data for training the cell composition prediction model. In addition, the database 140 accumulatively stores the cell composition information extracted by the cell composition information extraction program, and enables various applications for quantifying molecular functional cells from tissue images based on such cell composition information.

[0032] Hereinafter, the cell composition prediction model for extracting cell composition information will be described.

[0033] FIG. 2 is a conceptual diagram showing the configuration of a cell composition prediction model according to an embodiment of the present invention. FIG. 3 is a diagram for explaining the image segmentation unit of the cell composition prediction model according to an embodiment of the present invention.

[0034] The spatial transcriptome information 20 includes transcriptome data containing spatial information and tissue image data sharing the spatial information. The spatial information means position information for a plurality of spots 212 arranged on the two-dimensional plane of the tissue image data, and includes the coordinates of each spot 212. Here, the tissue image data is a tissue image taken after performing H&E staining using a special slide containing the coordinates of a plurality of spots, which corresponds to the prior art, and thus a detailed description thereof will be omitted.

[0035] That is, the spatial transcriptome information 20 is data in which hundreds to tens of thousands of transcriptome data are obtained for each spot 212, and the transcriptome data and the tissue image data can be spatially aligned using the coordinates of the spot 212.

[0036] The cell composition prediction model 200 is constructed based on learning data obtained by matching the already collected spatial transcriptome information 20 for different tissues of humans or animals with the cell composition information 240 for the transcriptome data classified according to the coordinates of each spot 212.

[0037] The cell composition prediction model 200 includes an image segmentation unit 210, a molecular marker model unit 220, and a prediction unit 230.

[0038] The image segmentation unit 210 divides the tissue image data into patch tissue images of a preset size.

[0039] The image segmentation unit 210 is constructed to perform a process of matching transcript data and tissue image data based on the coordinates of the spots 212, a process of arranging a square box of a preset size on the tissue image data including a plurality of spots 212, and a process of extracting the tissue image data into at least one or more patch tissue images 211 such that the coordinates of the spot 212 located in the middle among the plurality of spots 212 become the central coordinate value 213 of the square box.

[0040] For example, the size of the patch tissue image 211 may be an image size of 128×128, and the length of one side of the patch may be 510 μm.

[0041] Exemplarily, the image segmentation unit 210 can match the transcript data with the tissue image data based on the coordinates of the spots (spatial units for obtaining the transcript data). Subsequently, based on the central coordinate value 213 of the spots 212, the tissue image data can be divided into a plurality of patch images (patch tissue images) having a square (square box) size of a certain size. Thereafter, the molecular marker model unit 220 can output cell distribution information (cell density) by cell type for each cell group based on the transcript data included in the divided patch tissue images 211.

[0042] The molecular marker model unit 220 outputs cell distribution information by cell type of the cell group labeled through transcript data. Exemplarily, the molecular marker model unit 220 may be configured by a CellDART model, but is not limited thereto, and may be configured by a deep learning-based algorithm that estimates the detailed subclassification of tissues with existing tissue images (H&E staining).

[0043] FIG. 4 is a diagram for explaining the molecular marker model unit of the cell composition prediction model according to an embodiment of the present invention.

[0044] On the other hand, referring to FIG. 4, the molecular marker model unit 220 can be constructed based on learning data obtained by matching the cell distribution information 241 by cell type of the cell group included in the existing tissue image published on the Internet with the transcript data included in each single cell type 242. The molecular marker model unit 220 can output cell composition information 240 using the CellDART model constructed based on learning data consisting of the cell distribution information 241 of the cell group labeled with the published transcript data and the information 242 for each single cell type according to existing research. Exemplarily, the CellDART model includes a feature extractor including a source classifier and a domain classifier. The CellDART model preprocesses an existing transcript dataset and extracts integrated marker genes for each cell cluster. Subsequently, the shared transcript data between the pooled cluster marker and the spatial transcript information is selected for downstream analysis. Next, 8 cells are randomly selected from the single-cell data, and 20,000 similarity points are generated by assigning random weight values.

[0045] The feature extractor is trained to estimate cell fractions at similarity points and distinguish similarity points from spatial spots. First, the weights of the neural network excluding the domain classifier are updated. Next, the data labels for spots and similar spots are inverted, and only the domain classifier is updated. Finally, the learned CellDART model is applied to spatial transcriptome data to estimate the cell ratio of each spot. In the CellDART model, publicly available data can be utilized for single-cell transcriptome data for estimating cells, and cell-specific names labeled by existing studies can be applied. Since this corresponds to publicly available technology, a detailed description thereof will be omitted.

[0046] Referring again to FIG. 2, the prediction unit 230 extracts cell composition information 240 labeled with transcriptome data based on the central coordinate value 213 of the spot 212 among the plurality of spots 212 included in the patch tissue image 211.

[0047] Exemplarily, the prediction unit 230 can include a preprocessing process for the patch tissue image 211. The preprocessing process can perform stain normalization for H&E staining. For example, the patch tissue image 211 can be rotated, horizontally and vertically symmetric, enlarged and reduced (20% range), and changes for each RGB channel can be performed as an arbitrary function for the data input to the learning process of the convolutional neural network to increase the data.

[0048] As an example, the convolutional neural network is based on the ImageNet-based ResNet-50, and after preferentially applying the parameters trained on ImageNet, they can be updated during the training process. Also, 5% of the entire patch tissue image 211 can be utilized for internal validation. During the entire learning process, 64 patch tissue images and cell groups can be input per mini-batch, and in the optimization process, the Adam optimizer can be applied. The learning rate can be set to 0.0001, and the total number of epochs can be set to 100. Also, as the loss function for model training, Poisson Loss was utilized considering the distribution with respect to the cell density.

[0049] FIGS. 5 to 8 are diagrams showing cell composition information predicted from the tissue image input to the cell composition prediction model according to an embodiment of the present invention.

[0050] The program includes, as cell composition information 240, information regarding the types of cell groups predicted from the tissue image 21 and a heatmap tissue image showing the cell distribution information for each type of cell group.

[0051] FIG. 5 is a diagram showing cell composition information for each type of cell group predicted by inputting the H&E image of the tissue into the cell composition prediction model 200 of the present invention.

[0052] As shown in the illustration, by inputting a patch of independent tissue image data into the cell composition prediction model 200 of the present invention, an estimated image of the cell density can be generated.

[0053] This shows the density of the cell population predicted from the transcripts of an arbitrarily selected 5% internal validation set, and the results of the model predicted from the patch tissue images of the H&E images. The horizontal axis represents the values predicted from the deep learning model and the patch tissue images of the H&E images, and the vertical axis represents the density of the cells obtained from the transcript data.

[0054] Figure 6(a) shows information on the types of cell populations predicted from tissue image 21, and Figure 6(b) is a heatmap tissue image showing the cell distribution information for each type of cell population. When the tissue image obtained by Visium is input into the cell composition prediction model 200 of the present invention and applied based on the patch tissue image, a heatmap tissue image estimating the cell distribution information of the myeloid type can be output.

[0055] Figure 7 is data publicly available for evaluating the operation of an independent model. In a dataset that informs whether tumor infiltrating lymphocytes penetrate into lymphocytes according to the opinion of a pathologist for each patch of tissue images of lung adenocarcinoma, it was confirmed that a statistically significantly higher value of T / NK cells appears in the patches where TIL is present.

[0056] Figure 8(a) is a publicly available H&E image of lung adenocarcinoma as independent data. When the image of lung adenocarcinoma publicly available is input into the cell composition prediction model 200 of the present invention, as shown in Figure 8(b), the cell distribution information can be predicted for each type of cell population in the tissue.

[0057] In the following, among the configurations shown in FIGS. 1 to 8 described above, the description of the same configuration will be omitted.

[0058] Figure 9 is a flowchart showing a method for predicting the cell composition of a tissue image according to an embodiment of the present invention.

[0059] A method for predicting the cell composition of a tissue image based on spatial gene expression information using the tissue image cell composition prediction apparatus 100 according to an embodiment of the present invention includes receiving a tissue image 21 of a subject to be examined (S110), and predicting cell composition information by inputting the tissue image 21 into a cell composition prediction model 200 learned based on learning data consisting of spatial transcript information 20 and molecular markers (S120). At this time, the spatial transcript information 20 includes transcript data including spatial information and tissue image data sharing the spatial information. The spatial information means position information for a plurality of spots 212 arranged on the two-dimensional plane of the tissue image data, and includes the coordinates of each spot 212.

[0060] The cell composition prediction model 200 is constructed based on learning data obtained by matching previously collected spatial transcript information 20 for different tissues of humans or animals with cell composition information 240 for transcript data classified according to the coordinates of each spot 212.

[0061] The cell composition prediction model 200 includes an image division unit 210 that divides tissue image data into patch tissue images of a preset size, a molecular marker model unit 220 that outputs cell distribution information for each type of cell group labeled through transcript data, and a prediction unit 230 that extracts cell composition information 240 labeled with transcript data based on the central coordinate value 213 of a spot 212 among the plurality of spots 212 included in the patch tissue image 211.

[0062] The image division unit 210 of the cell composition prediction model 200 is constructed to perform a process of matching transcript data and tissue image data based on the coordinates of the spots 212, a process of arranging a square box of a preset size on the tissue image data including the plurality of spots 212, and a process of extracting the tissue image data into at least one or more patch tissue images 211 such that the coordinates of the spot 212 located in the middle among the plurality of spots 212 become the central coordinate value 213 of the square box.

[0063] The step of predicting cell composition information (S120) includes, as cell composition information 240, information on the types of cell groups predicted from the tissue image 21, and a heatmap tissue image showing cell distribution information for each cell group by type.

[0064] The cell composition prediction method described above can also be embodied in the form of a recording medium including computer-executable instruction words such as program modules executed by a computer. The computer-readable medium may be any available medium accessible by a computer, including all volatile and non-volatile media, separable and non-separable media. Also, the computer-readable medium can include a computer storage medium. The computer storage medium includes all volatile and non-volatile, separable and non-separable media embodied by any method or technology for storing information such as computer-readable instruction words, data structures, program modules, or other data.

[0065] Those of ordinary skill in the art to which the present invention pertains will be able to understand that, based on the above description, it can be easily deformed into other specific forms without changing the technical idea and essential features of the present invention. Therefore, it must be understood that each of the embodiments described above is illustrative in all aspects and not limiting. The scope of the present invention is indicated by the claims described below, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts must be construed as being included within the scope of the present invention.

Claims

1. In an apparatus for predicting the cell composition of a tissue image based on spatial gene expression information, a communication module that receives a tissue image of an object to be inspected; a memory storing a program for predicting cell composition information from the tissue image; and a processor that executes the program; comprising the program predicts the cell composition information by inputting the tissue image into a cell composition prediction model learned based on learning data consisting of spatial transcriptome information and a tissue image spatially aligned therewith, the spatial transcriptome information includes transcriptome data including spatial information and tissue image data sharing the spatial information, the spatial information means position information for a plurality of spots arranged in the tissue image data, and includes the coordinates of each spot, the cell composition prediction model is constructed based on learning data that matches the spatial transcriptome information for tissues of humans or animals already collected and the cell composition information for the transcriptome data classified by the coordinates of each spot, an apparatus for predicting the cell composition of a tissue image.

2. the cell composition prediction model includes an image division unit that divides the tissue image data into patch tissue images of a preset size, a molecular marker model unit that outputs cell distribution information for each type of cell group labeled through the transcriptome data, and a prediction unit that extracts the cell composition information labeled with the transcriptome data based on the central coordinate value of the spot among the plurality of spots included in the patch tissue image, the apparatus for predicting the cell composition of a tissue image according to Claim 1.

3. The image segmentation part of the cell composition prediction model performs the processes of matching the transcript data and the tissue image data based on the coordinates of the spots, placing a square box of a preset size on the tissue image data containing the plurality of spots, and extracting the tissue image data into at least one or more of the patch tissue images such that the coordinates of the spot located in the middle among the plurality of spots become the central coordinate values of the square box. The cell composition prediction device for a tissue image according to claim 2 is constructed to perform these processes.

4. The molecular marker model part of the cell composition prediction model is constructed based on learning data obtained by matching the type-specific cell distribution information of cell groups included in existing tissue images publicly available on the Internet with the transcript data included in each single cell type. The cell composition prediction device for a tissue image according to claim 2.

5. The program includes, as the cell composition information, information on the types of cell groups predicted from the tissue image and a heatmap tissue image showing the type-specific cell distribution information of each cell group. The cell composition prediction device for a tissue image according to claim 2.

6. In a method for predicting the cell composition of a tissue image based on spatial gene expression information using a cell composition prediction device for a tissue image, receiving a tissue image of an object to be inspected; and predicting the cell composition information by inputting the tissue image into a cell composition prediction model learned based on learning data consisting of spatial transcript information and a tissue image spatially aligned therewith, the spatial transcript information includes transcript data containing spatial information and tissue image data sharing the spatial information, the spatial information means position information for a plurality of spots arranged in the tissue image data and includes the coordinates of each spot, The method for predicting the cell composition of a tissue image is such that the cell composition prediction model is constructed based on learning data obtained by matching the spatial transcriptome information for each tissue of a human or animal that has already been collected with the cell composition information for the transcriptome data classified according to the coordinates of each spot.

7. The cell composition prediction model according to claim 6 includes an image segmentation unit that divides the tissue image data into patch tissue images of a preset size, a molecular marker model unit that outputs cell distribution information for each type of cell group labeled through the transcriptome data, and a prediction unit that extracts the cell composition information labeled with the transcriptome data based on the central coordinate value of the spot among the plurality of spots included in the patch tissue image.

8. The image segmentation unit of the cell composition prediction model is constructed to perform a process of matching the transcriptome data and the tissue image data based on the coordinates of the spots, a process of arranging a rectangular box of a preset size on the tissue image data including the plurality of spots, and a process of extracting the tissue image data into at least one or more of the patch tissue images such that the coordinates of the spot located in the middle among the plurality of spots become the central coordinate value of the rectangular box. The method for predicting the cell composition of a tissue image according to claim 7.

9. The molecular marker model unit of the cell composition prediction model is constructed based on learning data obtained by matching the cell distribution information for each type of cell group included in existing tissue images publicly available on the Internet with the transcriptome data included in each single cell type. The method for predicting the cell composition of a tissue image according to claim 7.

10. The step of predicting the cell composition information includes, as the cell composition information, information on the type of cell group predicted from the tissue image and a heatmap tissue image showing the cell distribution information for each type of cell group. The method for predicting the cell composition of a tissue image according to claim 7. Claim 11 A non-transitory computer-readable recording medium on which is recorded a computer program for performing a method for predicting the cell composition of a tissue image according to any one of Claims 6 to 10.

Images