Tumor microenvironment prediction method and analysis device using deep learning model

Abstract

This tumor microenvironment prediction method using a deep learning model comprises steps in which an analysis device: generates a cytokeratin (CK) staining image from a hematoxylin and eosin (H&E) staining image with respect to a region of interest of a sample; generates a leukocyte common antigen (LCA) staining image from the H&E staining image; generates a desmin staining image from the H&E staining image; and calculates a tumor microenvironment index for the region of interest on the basis of the CK staining image, the LCA staining image, and the desmin staining image.

Description

Tumor microenvironment prediction method and analysis device using a deep learning model

[0001] The technology described below is a technique for predicting the tumor microenvironment by converting tumor tissue images into various staining images using a deep learning model.

[0002] The tumor microenvironment refers to the ecosystem surrounding a tumor and includes cells such as immune cells, stroma, blood vessels, and fibroblasts. The tumor microenvironment can interact with tumor cells and promote tumor growth. The tumor microenvironment is being studied as a target for tumor treatment and prevention.

[0003] The technology described below is intended to provide a technique for generating different types of staining images from tissue slide images using a generative model, and for calculating indicators related to the tumor microenvironment using the generated staining images.

[0004] A method for predicting a tumor microenvironment using a deep learning model includes a step of an analysis device receiving an H&E (Hematoxylin & Eosin) staining image for a region of interest of a specific sample, a step of the analysis device inputting the H&E staining image into a first generative model that has been learned in advance to generate a CK (cytokeratin) staining image for the region of interest, a step of the analysis device inputting the H&E staining image into a second generative model that has been learned in advance to generate an LCA (leukocyte common antigen) staining image for the region of interest, a step of the analysis device inputting the H&E staining image into a third generative model that has been learned in advance to generate a desmin staining image for the region of interest, a step of the analysis device distinguishing a tumor region and a stroma region for the region of interest based on the CK staining image, the LCA staining image, and the desmin staining image, and a step of the analysis device calculating a tumor microenvironment index for the region of interest based on the tumor region and the stroma region.

[0005] An analysis device for predicting a tumor microenvironment includes an input device for receiving an H&E staining image for a region of interest of a specific sample, a storage device for storing a first generation model for converting the H&E staining image into a CK staining image, a second generation model for converting the H&E staining image into an LCA staining image, and a third generation model for converting the H&E staining image into a desmin staining image, and a calculation device for distinguishing a tumor region and a stroma region for the region of interest based on the CK staining image for the region of interest generated by inputting the input H&E staining image into the first generation model, the LCA staining image for the region of interest generated by inputting the input H&E staining image into the second generation model, and the desmin staining image for the region of interest generated by inputting the input H&E staining image into the third generation model, and calculating a tumor microenvironment index for the region of interest based on the tumor region and the stroma region.

[0006] The technology described below uses basic tissue staining images to produce various indicators representing the tumor microenvironment, thereby contributing to disease treatment and patient prognosis prediction.

[0007] Figure 1 is an example of a system for predicting tumor microenvironment indicators.

[0008] Figure 2 is an example of a process for generating different types of dyeing images using a generative model.

[0009] Figure 3 is an example of a process for distinguishing target regions in various types of staining images.

[0010] Figure 4 is an example of a process for predicting the prognosis of a subject by calculating tumor microenvironment indices from tissue staining images.

[0011] Figure 5 is an example of an analysis device that produces tumor microenvironment indicators.

[0012] Figure 6 shows the results of generating various types of dyeing images using the generative model.

[0013] The technology described below is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this does not limit the technology described below to specific embodiments, and it should be understood that all modifications, equivalents, and alternatives fall within the spirit and scope of the technology described below.

[0014] Terms such as first, second, A, and B may be used to describe various components, but these components are not limited by these terms and are used solely to distinguish one component from another. For example, without departing from the scope of the technology described below, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0015] As used herein, the singular expressions should be understood to include the plural expressions unless the context clearly dictates otherwise, and the term "comprises" and the like should be understood to mean the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0016] Before going into a detailed description of the drawings, it should be made clear that the division of components in this specification is merely a division based on the main function of each component. In other words, two or more components described below may be combined into a single component, or a single component may be further subdivided into two or more components with more detailed functions. In addition to its own main function, each component described below may additionally perform some or all of the functions of other components, and of course, some of the main functions of each component may be exclusively performed by other components.

[0017] Additionally, in performing a method or method of operation, each process constituting the method may occur in a different order than the stated order, unless the context clearly indicates a specific order. That is, each process may occur in the same order as the stated order, may be performed substantially simultaneously, or may be performed in the opposite order.

[0018] The technology described below analyzes tissue slide images to produce information about the tumor microenvironment.

[0019] The technology described below can convert a first type of stained slide image into other types of stained slide images using a generative model, and can derive tumor microenvironment-related indicators based on the generated other types of stained slide images.

[0020] Generative models are deep learning models that generate input images into different types of images. Generative models can be any of a variety of structural models. For example, variational autoencoders (VAEs), generative adversarial networks (GANs), and diffusion models are representative generative models. For convenience, the following explanation focuses on GANs.

[0021] The technique described below uses a generative model to transform a basic first-type dye image into another type of dye image.

[0022] The first staining image may be an H&E (Haematoxylin and eosin) staining image.

[0023] Other types of staining images may include cytokeratin (CK) staining images, leukocyte common antigen (LCA) staining images, and desmin staining images.

[0024] Tumor microenvironment indices refer to indicators related to the tumor microenvironment that can be derived from tissue staining images. Tumor microenvironment indices may include the tumor-stroma ratio (TSR) and tumor-infiltrating lymphocytes (TILs). Tumor microenvironment indices may include the following:

[0025]

[0026] TSR stands for tumor-to-stromal ratio.

[0027]

[0028] Single-TIL refers to primary or single TIL. TIL refers to the ratio of lymphocyte areas in the tumor area and stromal area.

[0029]

[0030] tTIL (intratumoral lymphocyte region) refers to the ratio of lymphocyte regions within or adjacent to the tumor in the tumor and stromal regions. The intratumoral lymphocyte region can be defined as the lymphocyte region located at a certain distance (~50 μm) from the tumor region.

[0031]

[0032] sTIL (stromal lymphocyte region) refers to the ratio of lymphocyte regions located in the stroma between the tumor region and the stromal region. The stromal lymphocyte region corresponds to the lymphocyte region excluding the intratumoral lymphocyte region among the total lymphocyte region.

[0033] Furthermore, a new (improved, polished) TIL can also be used by combining the aforementioned tTIL and sTIL as in Equation 5 below.

[0034]

[0035] The device that calculates tumor microenvironment indicators using stained images of tissues is called an analysis device. An analysis device is a device capable of image and data processing and can take the form of a PC, smart device, or server.

[0036] Figure 1 is an example of a system (100) for predicting tumor microenvironment indicators. In Figure 1, the analysis device is illustrated as an example of a computer terminal (150) and a server (250).

[0037] Figure 1(A) illustrates a system (100) for predicting tumor microenvironment indicators using a computer terminal (150) by a user (R). The staining image generation device (110) is a device that stains a tissue slide and scans the stained result to generate a staining image. The staining image generation device (110) generates a first staining image (H&E staining image).

[0038] A staining image generation device (110) transmits an H&E staining image to a computer terminal (150) via a wired or wireless network. The computer terminal (150) generates different types of staining images (CK staining images, LCA staining images, and desmin staining images) based on the H&E staining image. The computer terminal (150) calculates a tumor microenvironment index based on the H&E staining image and different types of staining images for the same tissue. The tumor microenvironment index may be at least one of the indices defined in the aforementioned mathematical expressions 1 to 5.

[0039] The computer terminal (150) provides the analysis results to the user (R).

[0040] Figure 1 depicts the dyeing image generation device (110) and the computer terminal (150) as separate objects. However, the dyeing image generation device (110) and the computer terminal (150) may be physically implemented as one device or as connected devices.

[0041] Figure 1(B) illustrates a system (200) for predicting tumor microenvironment indicators by allowing a user to access an analysis server (250) via a user terminal (270). The staining image generation device (110) is a device that stains a tissue slide and scans the stained result to generate a staining image. The staining image generation device (110) generates a first staining image (H&E staining image).

[0042] The dyeing image generation device (110) transmits the H&E dyeing image to the analysis server (250) via a wired or wireless network.

[0043] The analysis server (250) generates different types of staining images (CK staining images, LCA staining images, and desmin staining images) based on the H&E staining images. The analysis server (250) calculates tumor microenvironment indices based on the H&E staining images and different types of staining images for the same tissue. The tumor microenvironment indices may be at least one of the indices defined in the aforementioned mathematical expressions 1 to 5. The analysis server (250) may transmit the analysis results to the user terminal (270). In addition, the analysis server (250) may also transmit the analysis results to the hospital's EMR system (260).

[0044] Fig. 2 is an example of a process (200) for generating different types of dyeing images using a generation model. Fig. 2 shows the process performed in a dyeing image generation device (110 / 210) and an analysis device (150 / 250).

[0045] A staining image generation device (110 / 210) generates an H&E staining image of a tissue of a specific sample (310). The specific sample may be tissue collected from a specific subject.

[0046] The analysis device (150 / 250) can input or receive H&E staining images.

[0047] The analysis device (150 / 250) can generate other types of staining images based on the H&E staining image.

[0048] The analysis device (150 / 250) can generate different types of staining images using a pre-trained generative model. At this time, the different types of staining images may include a CK staining image, an LCA staining image, and a desmin staining image. The pre-trained generative models may be prepared in advance for each different type of staining image. That is, the generative model (G1) can generate a CK staining image based on an input H&E staining image. The generative model (G2) can generate an LCA staining image based on an input H&E staining image. The generative model (G1) can generate a desmin staining image based on an input H&E staining image.

[0049] The analysis device (150 / 250) can generate a CK image using a pre-learned generation model G1 based on the received H&E image (320).

[0050] The analysis device (150 / 250) can generate an LCA staining image using a pre-learned generation model G2 based on the received H&E image (330).

[0051] The analysis device (150 / 250) can generate a desmin staining image using a pre-learned generation model G3 based on the received H&E image (340).

[0052] The analysis device (150 / 250) can then calculate tumor microenvironment indices based on H&E staining images and other types of staining images.

[0053] The aforementioned generative models (G1, G2, and G3) must each be built in advance using training data. Generating model G1 can be generated using training data including H&E staining images and CK staining images of the same tissue. Generating model G2 can be generated using training data including H&E staining images and LCA staining images of the same tissue. Generating model G3 can be generated using training data including H&E staining images and Desmin staining images of the same tissue.

[0054] The generative models (G1, G2, and G3) correspond to the generator of the GAN. The bottom of Figure 2 illustrates the learning process for the GAN model.

[0055] A GAN consists of a generator G and a discriminator D. The generator G is a model that generates data based on input information. The discriminator D is a model that performs classification. In a GAN, the generator G, which generates data, and the discriminator D, which evaluates the generated data, learn opposingly to each other, gradually improving their performance. The generator G and the discriminator D can each be created using any of various machine learning models. For example, the generator G can be implemented using models such as U-net and autoencoders.

[0056] The discriminator D classifies whether the generated data is real or fake. The generator G is trained to generate data by receiving a latent code z as input, but to generate information to fool the discriminator D. The generator G generates data G(z), and the discriminator D generates the discrimination result D(G(z)) for G(z). The generator G has an objective function to minimize 1-D(G(z). Ultimately, it is minimized when D(G(z)) is 1. The generator G is trained so that the discriminator D can mistake G(z) for the original data.

[0057] Meanwhile, GAN can also be in the form of adding certain conditional information (c) to the generator G and the discriminator D respectively (conditional GAN).

[0058] Figure 3 is an example of a process (400) for distinguishing target areas in various types of staining images.

[0059] The analysis device acquires an H&E stained image of the tissue of a specific sample (410). The entire area represented by the H&E stained image is named a region of interest.

[0060] The analysis device inputs the H&E staining image of the area of ​​interest into the generation model G1 to generate a CK staining image (420).

[0061] The analysis device inputs the H&E staining image of the area of ​​interest into the generation model G2 to generate an LCA staining image (430).

[0062] The analysis device inputs the H&E staining image of the area of ​​interest into the generation model G1 to generate a desmin staining image (440).

[0063] The analysis device can generate a uniform immunohistochemical staining image of the H&E stained image of the region of interest on a patch basis.

[0064] The analysis device distinguishes normal epithelium areas from H&E stained images (415). The analysis device can distinguish normal epithelium from the input H&E stained images using a pre-trained classification model. The classification model can classify whether the images are normal epithelium on a patch-by-patch basis. The analysis device divides the input H&E stained images into patches of a certain size and inputs the patch images into the classification model to classify whether the images are normal epithelium on a patch-by-patch basis.

[0065] The CK stained image (CK stained image) includes a normal epithelial area and a tumor area. The analysis device can distinguish the tumor area by removing the normal epithelial area from the CK stained image (450).

[0066] The analysis device can distinguish a CK non-stained region of the region of interest based on the CK staining image (425). The CK non-stained region may include a muscle region that should not be included in the substrate.

[0067] The analysis device can distinguish the stained muscle region in the desmin stained image of the region of interest (435). The analysis device can distinguish the substrate region by removing the muscle region from the CK non-stained region of the region of interest (460).

[0068] The analysis device can distinguish the stained lymphocyte area in the LCA staining image of the region of interest (445).

[0069] The analysis device can distinguish between an intratumoral lymphocyte region and an intratumoral lymphocyte region based on the tumor region and lymphocyte region identified in the region of interest (470). The intratumoral lymphocyte region refers to a lymphocyte region close to the tumor region based on the tumor region. The intratumoral lymphocyte region refers to a lymphocyte region located within a critical distance from the tumor region. The critical distance may be 50 μm.

[0070] The analysis device can distinguish the remaining lymphocyte area, excluding the intratumoral lymphocyte area, from the lymphocyte area of ​​the area of ​​interest as the stromal lymphocyte area (480).

[0071] Figure 4 is an example of a process (500) for predicting the prognosis of a subject by calculating a tumor microenvironment index from a tissue staining image.

[0072] The analysis device receives an H&E-stained slide image (Whole Slide Image, WSI) of a region of interest of a specific sample (510). Fig. 4 illustrates a hotspot indicated by a red circle in the whole H&E-stained image.

[0073] The analysis device generates a CK staining image, an LCA staining image, and a desmin staining image based on the input H&E staining image using each of the aforementioned generation models (520). In addition, the analysis device can distinguish a normal epithelial area in the H&E staining image using the aforementioned classification model (520).

[0074] The analysis device can binarize the normal epithelial region image classified from each stained image and H&E stained image generated using the generative model (530).

[0075] The analysis device can express each binary image in a separate color according to its type and combine them to generate an aggregated map (540). At this time, the aggregated map represents the tumor area, the stromal area, the intratumoral lymphocyte area, the stromal lymphocyte area, the muscle area, and the normal epithelium area in different colors. Meanwhile, the analysis device can generate an aggregated map by combining masks in the order of the CK staining image, the LCA staining image, the desmin staining image, and the normal epithelium image. That is, when different colors (areas) overlap for the same pixel, only the later image can be displayed in color.

[0076] The analysis device can calculate the aforementioned tumor microenvironment indicators (TSR, TIL) based on the cluster map (540). The analysis device can calculate information on specific hot spots using the cluster map (540).

[0077] Additionally, the analysis device can perform disease prediction or prognosis prediction for a sample (subject) based on the produced tumor microenvironment indicator(s) (550).

[0078] Figure 5 illustrates an example of an analysis device (600) that calculates tumor microenvironment indicators. The analysis device (600) corresponds to the aforementioned analysis devices (150 and 250 of Figure 1). The analysis device (600) may be physically implemented in various forms. For example, the analysis device (600) may take the form of a computer device such as a PC, a network server, a data processing chipset, or the like.

[0079] The analysis device (600) may include a storage device (610), a memory (620), a computing device (630), an interface device (640), a communication device (650), and an output device (660).

[0080] The storage device (610) can store an H&E staining image (first staining image) for a target tissue of a specific sample.

[0081] The storage device (610) can store different generation models that generate different types of staining images from H&E staining images. The generation models can include a first generation model that generates a CK staining image, a second generation model that generates an LCA staining image, and a third generation model that generates a desmin staining image. In this case, the generation models can be patch-based generation models.

[0082] The storage device (610) can store a classification model that classifies normal epithelial areas in H&E stained images. The classification model may be a CNN (Convolutional Neural Network)-based model.

[0083] The storage device (610) can store a program or code for preprocessing or postprocessing an image.

[0084] Additionally, the storage device (610) can store commands or program codes for the process of calculating tumor microenvironment indicators through the process described above.

[0085] The storage device (610) can store the tumor microenvironment indicators as analysis results.

[0086] The memory (620) can store data and information generated during the process of the analysis device (600) calculating a tumor microenvironment index.

[0087] The interface device (640) is a device that receives certain commands and data from the outside.

[0088] The interface device (640) can receive an H&E staining image (first staining image) of a specific sample from a physically connected input device or an external storage device.

[0089] The interface device (640) can transmit the produced tumor microenvironment indicator or analysis result (disease diagnosis or prognosis prediction) to an external object.

[0090] The interface device (640) may refer to a configuration that transmits data received through a communication device (650) into the analysis device (600).

[0091] A communication device (650) refers to a configuration that receives and transmits certain information through a wired or wireless network.

[0092] The communication device (650) can receive an H&E staining image (first staining image) of a specific sample from an external object.

[0093] Alternatively, the communication device (650) may transmit the produced tumor microenvironment indicator or analysis result (disease diagnosis or prognosis prediction) to an external object such as a user terminal.

[0094] An output device (660) is a device that outputs certain information. The output device (660) can output interfaces, analysis results, etc. required for the data processing process.

[0095] The computing device (630) can predict tumor microenvironment indicators using commands or program codes stored in the storage device (610).

[0096] The computational device (630) can input an H&E staining image (first staining image) into the first generation model to generate a CK staining image (second staining image).

[0097] The calculation device (630) can input the H&E staining image (first staining image) into the second generation model to generate an LCA staining image (third staining image).

[0098] The computational device (630) can input an H&E staining image (first staining image) into a third generation model to generate a desmin staining image (third staining image).

[0099] The computational device (630) can distinguish a normal epithelial region from an H&E stained image (first stained image) using a classification model. The computational device (630) can classify a normal epithelial region from an H&E stained image (first stained image) on a patch basis.

[0100] The computational device (630) can distinguish a tumor region of the region of interest by removing a normal epithelial region from the CK stained image (second stained image) of the region of interest. This process can also be performed on a patch-by-patch basis.

[0101] The computational device (630) can distinguish the muscle area of ​​interest in the desmin stained image (third stained image). This process can also be performed on a patch basis.

[0102] The computational device (630) can separate the stromal region by removing the muscle region from the CK-unstained region (CK-unstained region) of the region of interest. This process can also be performed on a patch-by-patch basis.

[0103] The computational device (630) can distinguish a lymphocyte region in the LCA staining image (second staining image) of the region of interest. This process can also be performed on a patch basis.

[0104] The computational device (630) can distinguish intratumoral lymphocyte regions based on the tumor region and lymphocyte regions of the region of interest. The computational device (630) can distinguish intratumoral lymphocyte regions within a threshold distance from the tumor region of the region of interest. This process can also be performed on a patch-by-patch basis.

[0105] The computational device (630) can distinguish the stromal lymphocyte region from the lymphocyte region in the region of interest, excluding the intratumoral lymphocyte region. This process can also be performed on a patch basis.

[0106] The computational device (630) can generate a mask by binarizing the CK staining image, the LCA staining image, the desmin staining image, and the epithelial region image. The computational device (630) can sequentially combine different types of images to generate a set map (see FIG. 4). At this time, the computational device (630) can set different colors for different types of images (see FIG. 4).

[0107] The computing device (630) can derive tumor microenvironment indicator(s) using various staining images or cluster maps of the region of interest. Furthermore, the computing device (630) can also perform a diagnosis or prognosis prediction for a sample (subject) based on the derived tumor microenvironment indicator(s).

[0108] The computing device (630) may be a device such as a processor, AP, or chip embedded with a program that processes data and performs certain operations.

[0109] The researcher describes the generative model he built. He used PatchGAN, which generates images on a patch-by-patch basis. PatchGAN takes a patch-by-patch image as input and generates a target patch image.

[0110] The researchers collected data from a cohort of 320 patients who visited Samsung Medical Center (SMC) and a cohort of 186 patients who visited Seoul National University Hospital (SNUH). Table 1 below presents the population information used to build and validate the generative model. The population information includes TNM stage, Lauren classification for gastric cancer, and recurrence status.

[0111]

[0112] The researcher built a generative model using the data set shown in Table 2 below. The data set includes pairs of CK staining images, pairs of LCA staining images, and pairs of desmin staining images. Each image pair consists of a corresponding staining image and an H&E staining image of the same tissue. The researcher identified patches of a certain size (256 × 256 pixels) from each data set. That is, the aforementioned generative model G1 was generated using 90,521 pairs of learning patches, the generative model G2 was generated using 174,968 pairs of learning patches, and the generative model G3 was generated using 94,808 pairs of learning patches. The researcher built the generative model using the loss function below. Equation 6 below is the loss function for the generator, and Equation 7 is the loss function for the discriminator. In the equations below, x is the input H&E staining image patch, and y is the actual corresponding immunohistochemical staining image (CK, LCA, or desmin). G(·) is the virtual image output by the generator, and D(·) is the output value of the discriminator.

[0113]

[0114] L G It uses smooth L1 loss for RGB channels and smooth L1 loss for DAB (diaminobenzidine) color channel. Also, L GIt further uses SSIM loss and BCE (binary cross entropy) loss.

[0115]

[0116] L D uses BCE loss.

[0117]

[0118]

[0119]

[0120]

[0121] The performance of each model was evaluated by the peak signal-to-noise (PSNR) and structure similarity index measure (SSIM) between the original image and the virtual dye image generated by the generative model.

[0122] Meanwhile, a CNN-based classification model classifies whether a patch in an H&E-stained image is normal epithelial or not. This classification model was built using training data consisting of 23,862 normal epithelial patches extracted from 58 slide images and 82,073 patches extracted from other tissues. The classification model exhibited an area under the ROC curve (AUC) of 0.97.

[0123] CKLCADESMINNormal EpitheliumNumber ofWSIs15 pairs17 pairs14 pairs58Number ofPatch Images90521pairs174968pairs94808pairsNormal Epithelium: 23862Other tissue: 82073Model PerformancePSNR: 20.55SSIM:0.49PSNR:17.07SSIM: 0.41PSNR: 18.43SSIM: 0.39AUC: 0.97

[0124] Figure 6 shows the results of generating various types of staining images using the generative model. Figure 6(A) shows a CK staining image generated based on an H&E staining image and an actual CK staining image. Figure 6(B) shows an LCA staining image generated based on an H&E staining image and an actual LCA staining image. Figure 6(C) shows a desmin staining image generated based on an H&E staining image and an actual desmin staining image. In Figures 6(A) to 6(C), the left side is an entire slide image, and the right side is an enlarged image for a specific region (a, b, c, and d) of the entire slide. As can be seen from Figure 6, the virtual staining images generated by the generative model are quite similar to the actual staining images.

[0125] Table 3 below shows the correlation between TSR and tumor phenotype derived from staining images generated using the generative model.

[0126]

[0127] Table 4 below shows the association between single-TIL and tumor phenotype derived from staining images generated using the generative model.

[0128]

[0129] Table 5 below shows the correlation between sTILs and tumor phenotypes derived from staining images generated using the generative model.

[0130]

[0131] Table 6 below shows the correlation between tTILs and tumor phenotypes derived from staining images generated using the generative model.

[0132]

[0133] Examining Tables 3 to 6, TSR showed a high correlation (P<0.0001) with all variables, except chemotherapy response. TNM stage showed a relatively high correlation with tTILs. The Lauren classification showed a relatively high correlation with single-TILs and tTILs. Chemotherapy response showed a relatively high correlation with single-TILs, sTILs, and tTILs.

[0134] Table 7 below analyzes the correlation between TILs and survival (5 years or more). As shown in Table 7, single-TILs and polished-TILs showed a consistent correlation with survival.

[0135]

[0136] Table 8 below analyzes the correlation between TSR and the TIL&TSR composite index and survival (5 years or more). As shown in Table 8, TSR, single-TIL&TSR, and polished-TIL&TSR all showed consistent correlations with survival.

[0137]

[0138] Additionally, the tissue staining image conversion method, medical image analysis method using staining images, and cancer staging method described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided on a temporary or non-transitory computer-readable medium.

[0139] A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on a non-transitory readable medium, such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM), EEPROM (Electrically EPROM), or flash memory.

[0140] Temporarily readable media refers to various types of RAM, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

[0141] The present embodiment and the drawings attached to the present specification only clearly illustrate a part of the technical idea included in the above-described technology, and it is obvious that all modified examples and specific embodiments that can be easily inferred by a person skilled in the art within the scope of the technical idea included in the specification and drawings of the above-described technology are included in the scope of the rights of the above-described technology.

Claims

1. A step in which the analysis device receives an H&E (Hematoxylin & Eosin) stained image of an area of ​​interest of a specific sample; A step in which the above analysis device inputs the H&E staining image into a first generation model that has been learned in advance to generate a CK (cytokeratin) staining image for the region of interest; A step in which the above analysis device inputs the H&E staining image into a pre-learned second generation model to generate an LCA (leukocyte common antigen) staining image for the region of interest; A step in which the above analysis device inputs the H&E staining image into a pre-learned third generation model to generate a desmin staining image for the region of interest; A step in which the analysis device distinguishes a tumor area and a stroma area for the region of interest based on the CK staining image, the LCA staining image, and the desmin staining image; and A method for predicting a tumor microenvironment using a deep learning model, the method comprising a step of the analysis device calculating a tumor microenvironment index for the region of interest based on the tumor region and the stroma region.

2. In paragraph 1, A method for predicting a tumor microenvironment using a deep learning model in which the above analysis device inputs the H&E staining image into a learned classification model to distinguish a normal epithelial region for the region of interest, and removes the normal epithelial region from the CK staining image to distinguish the tumor region.

3. In paragraph 1, A method for predicting a tumor microenvironment using a deep learning model in which the above analysis device distinguishes a CK non-stained area of ​​the area of ​​interest based on the CK stained image, and removes a muscle area of ​​the area of ​​interest distinguished based on the desmin stained image from the CK non-stained area to distinguish the stromal area.

4. In paragraph 1, A method for predicting a tumor microenvironment using a deep learning model in which the above analysis device distinguishes the entire lymphocyte area of ​​the region of interest in the LCA staining image, distinguishes an intratumoral lymphocyte area located at a critical distance from the tumor area among the entire lymphocyte area, and distinguishes a stromal lymphocyte area excluding the intratumoral lymphocyte area among the entire lymphocyte area.

5. In paragraph 1, A method for predicting a tumor microenvironment using a deep learning model, wherein the above tumor microenvironment indicator includes at least one of TSR (Tumor-stroma ratio), TIL (Tumor-Infiltrating Lymphocytes), tTIL (intratumoral lymphocytes region), and sTIL (stromal lymphocytes region).

6. In paragraph 1, A method for predicting a tumor microenvironment using a deep learning model, wherein the above analysis device further includes a step of predicting a diagnosis or prognosis for the specific sample based on the tumor microenvironment indicator.

7. Input device for receiving H&E (Hematoxylin & Eosin) staining images of areas of interest of a specific sample; A storage device storing a first generation model that converts an H&E staining image into a CK (cytokeratin) staining image, a second generation model that converts an H&E staining image into a LCA (leukocyte common antigen) staining image, and a third generation model that converts an H&E staining image into a desmin staining image; and An analysis device for predicting a tumor microenvironment, including a computational device that distinguishes a tumor region and a stroma region for the region of interest based on a CK staining image for the region of interest generated by inputting the input H&E staining image into the first generation model, an LCA staining image for the region of interest generated by inputting the input H&E staining image into the second generation model, and a desmin staining image for the region of interest generated by inputting the input H&E staining image into the third generation model, and calculates a tumor microenvironment index for the region of interest based on the tumor region and the stroma region.

8. In paragraph 7, The above storage device further stores a classification model that classifies normal epithelial areas in H&E stained images, The above-mentioned computing device is an analysis device that predicts a tumor microenvironment by inputting the input H&E staining image into the classification model to distinguish a normal epithelial region for the region of interest, and removing the normal epithelial region from the CK staining image to distinguish the tumor region.

9. In paragraph 7, The above calculation device is an analysis device that predicts a tumor microenvironment by distinguishing a CK non-stained area of ​​the area of ​​interest based on the CK stained image, and removing a muscle area of ​​the area of ​​interest distinguished based on the desmin stained image from the CK non-stained area to distinguish the stromal area.

10. In paragraph 7, The above-mentioned computing device is an analysis device that predicts a tumor microenvironment by distinguishing the entire lymphocyte area of ​​the region of interest in the LCA staining image, distinguishing an intratumoral lymphocyte area located at a critical distance from the tumor area among the entire lymphocyte area, and distinguishing a stromal lymphocyte area excluding the intratumoral lymphocyte area among the entire lymphocyte area.

11. In paragraph 7, The above-mentioned computing device is an analysis device for predicting a tumor microenvironment including at least one of the tumor microenvironment indicators TSR (Tumor-stroma ratio), TIL (Tumor-Infiltrating Lymphocytes), tTIL (intratumoral lymphocytes region), and sTIL (stromal lymphocytes region).

Images