System for detecting tumor infiltrating lymphocytes using deep learning algorithm and method thereof

Abstract

The present invention relates to a system for detecting tumor-infiltrating lymphocytes using a deep learning algorithm and a method thereof. According to the present invention, a tumor-infiltrating lymphocyte detection system using a deep learning algorithm comprises an input unit that receives a patch image to be measured, and a detection unit that inputs the input patch image to be measured into a trained patch classification model to classify the patch image, inputs the regiond image from the patch image into a trained cell classification model to classify whether it corresponds to a tumor-infiltrating lymphocyte, and combines the results of the patch classification model and the cell classification model to detect tumor-infiltrating lymphocytes present in the patch image to be measured.

Description

Technology Field

[0001] The present invention relates to a tumor-infiltrating lymphocyte detection system and method using a deep learning algorithm, and more specifically, to a tumor-infiltrating lymphocyte detection system and method that receives a plurality of patches extracted from a tissue pathology image and detects tumor-infiltrating lymphocytes present in the patches through a patch classification model and a cell classification model. Background Technology

[0002] Tumor-infiltrating lymphocytes (TILs) refer to immune cells found around or inside tumors that play a role in killing cancer cells and preventing the disease from spreading to other parts of the body.

[0003] Research is being conducted to detect tumor-infiltrating lymphocytes by analyzing pathology slide images to determine tumor treatment methods and predict treatment responses, and in particular, technology for detecting tumor-infiltrating lymphocytes through artificial intelligence models is being developed.

[0004] However, conventional AI models had a problem where the accuracy of detecting tumor-infiltrating lymphocytes was reduced because they occasionally classified tissues other than cell nuclei as nuclei.

[0005] The technology forming the background of the present invention is disclosed in Korean Published Patent No. 10-2023-0026950 (published on February 27, 2023). The problem to be solved

[0006] As such, according to the present invention, a tumor-infiltrating lymphocyte detection system and a method are provided, which receive a plurality of patches extracted from tissue pathology images and detect tumor-infiltrating lymphocytes present in the patches through a patch classification model and a cell classification model. means of solving the problem

[0007] According to an embodiment of the present invention for achieving such technical challenges, a tumor-infiltrating lymphocyte detection system using a deep learning algorithm comprises an input unit that receives a patch image to be measured, and a detection unit that inputs the input patch image to be measured into a trained patch classification model to classify the patch image, inputs the regiond image from the patch image into a trained cell classification model to classify whether it corresponds to a tumor-infiltrating lymphocyte, and combines the results of the patch classification model and the cell classification model to detect tumor-infiltrating lymphocytes present in the patch image to be measured.

[0008] It may further include an image collection unit that collects patch images containing tumor-infiltrating lymphocytes and patch images not containing tumor-infiltrating lymphocytes, a preprocessing unit that generates point labels for tumor-infiltrating lymphocytes present in the patch images and obtains regiond images of the tumor-infiltrating lymphocytes from the patch images, and a learning unit that inputs the patch images with generated point labels and regiond images into a constructed patch classification model and a cell classification model to train them.

[0009] The above preprocessing unit can generate a dot label by creating a dot at the center of the tumor-infiltrating lymphocytes included in the patch image.

[0010] The above preprocessing unit can input patch images into a cell nucleus regionalization model to obtain a plurality of first regionalized images of tumor-infiltrating lymphocytes, and apply a circular filter algorithm to the plurality of first regionalized images to obtain a plurality of second regionalized images of tumor-infiltrating lymphocytes.

[0011] The above preprocessing unit can extract a plurality of second regiond images for training a cell classification model from a plurality of first regiond images according to cell size, circularity, convexity, and inertia ratio using a circular filter algorithm.

[0012] The above learning unit can train a patch classification model using patch images and point labels to classify patch images into a value of 0 or 1 depending on the amount of tumor-infiltrating lymphocytes present in the patch images.

[0013] The above learning unit can input a plurality of second regiond images into a cell classification model and train the cell classification model to output a probability value indicating whether the cells included in each second regiond image are tumor-infiltrating lymphocytes.

[0014] The detection unit can determine that if the result value obtained from the patch classification model is 1 and the probability value obtained from the cell classification model is greater than the first threshold value, the cell included in the second regiond image corresponds to a tumor-infiltrating lymphocyte, and if the result value obtained from the patch classification model is 1 and the probability value obtained from the cell classification model is less than the first threshold value, the cell included in the second regiond image does not correspond to a tumor-infiltrating lymphocyte.

[0015] The detection unit can determine that if the result value obtained from the patch classification model is 0 and the probability value obtained from the cell classification model is greater than the second threshold value, the cell included in the second regiond image corresponds to a tumor-infiltrating lymphocyte, and if the result value obtained from the patch classification model is 0 and the probability value obtained from the cell classification model is less than the second threshold value, the cell included in the second regiond image does not correspond to a tumor-infiltrating lymphocyte.

[0016] The first threshold and the second threshold are values ​​set between 0 and 1, and the second threshold can be set to a value greater than the first threshold.

[0017] According to an embodiment of the present invention, a method for detecting tumor-infiltrating lymphocytes using a deep learning algorithm comprises the steps of receiving a patch image to be measured, inputting the input patch image to be measured into a trained patch classification model to classify the patch image, inputting the regiond image from the patch image into a trained cell classification model to classify whether it corresponds to a tumor-infiltrating lymphocyte, and then combining the results of the patch classification model and the cell classification model to detect tumor-infiltrating lymphocytes present in the patch image to be measured.

[0018] The method may further include the steps of collecting patch images in which tumor-infiltrating lymphocytes are present and patch images in which tumor-infiltrating lymphocytes are not present, generating point labels for tumor-infiltrating lymphocytes present in the patch images and obtaining regiond images of tumor-infiltrating lymphocytes from the patch images, and inputting the patch images with generated point labels and the regiond images into the constructed patch classification model and cell classification model to train them. Effects of the invention

[0019] As such, according to the present invention, by determining the amount of tumor-infiltrating lymphocytes in a patch image and classifying whether cells within the same patch image are tumor-infiltrating lymphocytes by regionalizing them, prediction at the object level is possible, and since regionalization of overlapping areas is possible, accurate prediction of the classification of tumor-infiltrating lymphocytes is possible. Brief explanation of the drawing

[0020] FIG. 1 is a configuration diagram illustrating a tumor-infiltrating lymphocyte detection system according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating a method for detecting tumor-infiltrating lymphocytes using a tumor-infiltrating lymphocyte detection system according to an embodiment of the present invention. Figure 3 is a diagram illustrating the patch classification model and cell classification model constructed in step S230 shown in Figure 2. FIG. 4 is an example diagram for explaining step S260 shown in FIG. 2. FIG. 5 is an exemplary diagram showing the prediction results according to an embodiment of the present invention. Specific details for implementing the invention

[0021] Preferred embodiments according to the present invention will be described in detail below with reference to the attached drawings. In this process, the thickness of lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of explanation.

[0022] Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intent or practice of the user or operator. Therefore, the definitions of these terms should be based on the content throughout this specification.

[0023] Hereinafter, a tumor-infiltrating lymphocyte detection system according to an embodiment of the present invention will be described in more detail using FIG. 1.

[0024] FIG. 1 is a configuration diagram illustrating a tumor-infiltrating lymphocyte detection system according to an embodiment of the present invention.

[0025] As illustrated in FIG. 1, a tumor-infiltrating lymphocyte detection system (100) according to an embodiment of the present invention includes an image collection unit (110), a preprocessing unit (120), a learning unit (130), an input unit (140), and a detection unit (150).

[0026] First, the image collection unit (110) collects a patch image in which tumor-infiltrating lymphocytes are present and a patch image in which tumor-infiltrating lymphocytes are not present.

[0027] The preprocessing unit (120) generates dot labels for tumor-infiltrating lymphocytes present in the patch image and obtains a regiond image of the tumor-infiltrating lymphocytes from the patch image.

[0028] The learning unit (130) constructs a patch classification model and a cell classification model, respectively, and then trains the constructed patch classification model and cell classification model by inputting a patch image with a dot label and a regiond image.

[0029] The input unit (140) receives a patch image to be measured.

[0030] Finally, the detection unit (150) inputs the input patch image to be measured into a trained patch classification model to classify the patch image, inputs the regiond image from the patch image into a trained cell classification model to classify whether it corresponds to a tumor-infiltrating lymphocyte, and combines the results of the patch classification model and the cell classification model to detect tumor-infiltrating lymphocytes present in the patch image to be measured.

[0031] Hereinafter, a method for detecting tumor-infiltrating lymphocytes using a tumor-infiltrating lymphocyte detection system (100) according to an embodiment of the present invention will be described in more detail using FIGS. 2 to 5.

[0032] FIG. 2 is a flowchart illustrating a method for detecting tumor-infiltrating lymphocytes using a tumor-infiltrating lymphocyte detection system according to an embodiment of the present invention.

[0033] As illustrated in FIG. 2, a method for detecting tumor-infiltrating lymphocytes according to an embodiment of the present invention includes the step of training a patch classification model and a cell classification model, and the step of detecting tumor-infiltrating lymphocytes using the patch classification model and the cell classification model that have been trained.

[0034] First, looking at the step of training the patch classification model and the cell classification model, the tumor infiltrating lymphocyte detection system (100) collects patch images (S210).

[0035] To elaborate, the image collection unit (110) collects a whole slide image from a hospital server and obtains a patch image in which tumor-infiltrating lymphocytes are present and a patch image in which tumor-infiltrating lymphocytes are not present from the collected whole slide image.

[0036] Next, the preprocessing unit (120) generates dot labels in the tumor-infiltrating lymphocyte regions present in the patch image and obtains a first regiond image from the same patch image (S220).

[0037] To elaborate, the preprocessing unit (120) creates a dot label by creating a dot at the center of the tumor-infiltrating lymphocytes included in the patch image.

[0038] Additionally, the preprocessing unit (120) inputs patch images into a cell nucleus regionalization model to obtain a plurality of first regionalized images of tumor-infiltrating lymphocytes. At this time, the cell nucleus regionalization model is trained using H&E stained tissue pathology images, and various cell nucleus regionalization models previously proposed may be used. Since it is not a core component of the present invention, a detailed description of the cell nucleus regionalization model is omitted.

[0039] Next, the preprocessing unit (120) applies a circular filter algorithm to a plurality of first regiond images to obtain a plurality of second regiond images of tumor-infiltrating lymphocytes corresponding to ground-truth (S230).

[0040] The unintended territorialization of tumor cells and the background is large or irregular in shape, whereas tumor-infiltrating lymphocytes are of a constant size and maintain a circular shape that is not distorted or crumpled.

[0041] Accordingly, the preprocessing unit (120) uses a circular filter algorithm to extract a plurality of second regiond images for training a cell classification model from a plurality of first regiond images according to cell size, circularity, convexity, and inertia ratio.

[0042] That is, the preprocessing unit (120) excludes regiond images in which the cell size is larger than a reference value from a plurality of first regiond images. On the other hand, the preprocessing unit (120) extracts regiond images containing cells that have a circularity greater than 0.2, a convexity greater than 0.2, or an inertia ratio greater than 0.1 as second regiond images.

[0043] When step S230 is completed, the learning unit (130) trains a patch classification model and a cell classification model using the patch image with generated point labels and the final regiond image, i.e., the second regiond image (S240).

[0044] First, the learning unit (130) builds a patch classification model and a cell classification model.

[0045] Figure 3 is a diagram illustrating the patch classification model and cell classification model constructed in step S230 shown in Figure 2.

[0046] As shown in Figure 3, since the patch classification model requires the use of local information at the cell level for accurate prediction, a convolutional network algorithm is applied to the patch classification model and the cell classification model.

[0047] To extract global information from the input image, the initial network applies a large 7x7 kernel. Subsequently, it is structured as a repeating sequence of 3x3 kernels and convolution blocks containing batch normalization and ReLU functions. Each layer with a fixed number of channels consists of 3, 4, 6, and 3 convolution blocks, with 64, 128, 256, and 512 channels, respectively. At the beginning of each layer, the convolution stride is increased proportionally to the increase in the number of channels to reduce spatial resolution. This is intended to enable the extraction of detailed local information. Furthermore, at the end of the network, Global Average Pooling is applied instead of a Fully Connected Layer to reduce the number of parameters.

[0048] Next, the learning unit (130) uses the patch image and dot labels to train the patch classification model to classify the patch image into a value of 0 or 1 according to the amount of tumor-infiltrating lymphocytes present in the patch image.

[0049] To explain this further, if the number of points included in the patch image exceeds 5, the patch classification model determines that tumor-infiltrating lymphocytes exist in the corresponding patch image and outputs a value of 1. On the other hand, if the number of points included in the patch image is 5 or less, the patch classification model determines that tumor-infiltrating lymphocytes do not exist in the corresponding patch image and outputs a value of 0.

[0050] In the present invention, the number of points for determining the presence of tumor-infiltrating lymphocytes is set to 5, but is not limited thereto, and if necessary, the presence of tumor-infiltrating lymphocytes may be determined using values ​​greater or fewer than 5.

[0051] Additionally, the learning unit (130) inputs a plurality of second regiond images into a cell classification model to train the cell classification model to output a probability value indicating whether the cells included in each second regiond image are tumor-infiltrating lymphocytes.

[0052] When the training of the patch classification model and the cell classification model is completed using steps S210 to S240, the tumor infiltrating lymphocyte detection system (100) detects tumor infiltrating lymphocytes present in the corresponding patch image using the trained patch classification model and the cell classification model.

[0053] The input unit (140) receives a patch image to be measured (S250).

[0054] The input patch images have one or more numbers.

[0055] Next, the detection unit (150) inputs the patch image to be measured into a trained patch classification model to classify the patch image, inputs the regiond image from the patch image into a trained cell classification model to classify whether the cells included in the regiond image correspond to tumor-infiltrating lymphocytes, and detects tumor-infiltrating lymphocytes present in the patch image to be measured by combining the results of the patch classification model and the cell classification model (S260).

[0056] FIG. 4 is an example diagram for explaining step S260 shown in FIG. 2.

[0057] As illustrated in FIG. 4, the detection unit (150) inputs a patch image to be measured into a patch classification model. Then, the patch classification model outputs a value of 0 or 1 depending on the presence or absence of tumor-infiltrating lymphocytes.

[0058] Additionally, the detection unit (150) inputs the regiond image from the patch image to be measured into the cell classification model. Then, the cell classification model outputs a probability value indicating whether the tumor-infiltrating lymphocytes are present.

[0059] The detection unit (150) detects tumor-infiltrating lymphocytes present in the patch image to be measured using a preset thresholding value, a result value obtained from a patch classification model, and a probability value obtained from a cell classification model.

[0060] To explain this again, a threshold value for the probability value obtained from the cell classification model is selected based on the result value obtained from the patch classification model.

[0061] If the result value obtained from the patch classification model is 1, the detection unit (150) sets the threshold value for the probability value as the first threshold value.

[0062] On the other hand, if the result obtained from the patch classification model is 0, the detection unit (150) sets the threshold value for the probability value to the second threshold value.

[0063] Here, the first threshold and the second threshold are values ​​set between 0 and 1, and the second threshold is set to a value greater than the first threshold. In one embodiment, the first threshold is 0.5 and the second threshold is 0.8.

[0064] When the threshold value is set as described above, if the result value obtained from the patch classification model is 1 and the probability value obtained from the cell classification model is greater than 0.5, the detection unit (150) determines that the cells included in the regiond image correspond to tumor-infiltrating lymphocytes.

[0065] On the other hand, if the result value obtained from the patch classification model is 1 and the probability value obtained from the cell classification model is less than 0.5, the detection unit (150) determines that the cells included in the regiond image do not correspond to tumor-infiltrating lymphocytes.

[0066] In addition, if the result value obtained from the patch classification model is 0 and the probability value obtained from the cell classification model is greater than 0.8, the detection unit (150) determines that the cells included in the regiond image correspond to tumor-infiltrating lymphocytes.

[0067] On the other hand, if the result value obtained from the patch classification model is 0 and the probability value obtained from the cell classification model is less than 0.8, the detection unit (150) determines that the cells included in the regiond image do not correspond to tumor-infiltrating lymphocytes.

[0068] FIG. 5 is an exemplary diagram showing the prediction results according to an embodiment of the present invention.

[0069] As shown in FIG. 5, the horizontal axis represents the prediction result according to an embodiment of the present invention, and the vertical axis represents a value corresponding to ground truth.

[0070] Referring to FIG. 5, it can be seen that, according to an embodiment of the present invention, tumor-infiltrating lymphocytes present in a patch image can be detected with an accuracy of 0.92 or higher.

[0071] As such, the quantitative measurement system according to the present invention determines the amount of tumor-infiltrating lymphocytes in a patch image and classifies whether cells within the same patch image are tumor-infiltrating lymphocytes by regionalizing them, thereby enabling prediction at the object level and, since regionalization of overlapping areas is possible, accurate prediction of the classification of tumor-infiltrating lymphocytes is possible.

[0072] The present invention has been described with reference to the embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the following claims. Explanation of the symbols

[0073] 100: Tumor-Infiltrating Lymphocyte Detection System 110 : Image collection unit 120 : Preprocessing section 130 : Learning Department 140 : Input section 150 : Detector

Claims

Claim 1 A system for detecting tumor-infiltrating lymphocytes using a deep learning algorithm comprises: an input unit for receiving a patch image to be measured; a detection unit for inputting the received patch image to be measured into a trained patch classification model to classify the patch image, inputting the regiond image from the patch image into a trained cell classification model to classify whether it corresponds to a tumor-infiltrating lymphocyte, and synthesizing the results of the patch classification model and the cell classification model to detect tumor-infiltrating lymphocytes present in the patch image to be measured; wherein the patch classification model is a model trained to receive a patch image, compare the amount of tumor-infiltrating lymphocytes present in the patch image with a preset reference value, and output the patch image as a value of 0 or 1; wherein the cell classification model is a model trained to receive the regiond image and represent whether each cell is a tumor-infiltrating lymphocyte as a probability value; and wherein the detection unit selects one of a preset first and second threshold value according to the result of the patch classification model, compares the selected threshold value with the probability value resulting from the cell classification model, and determines and detects whether the corresponding cell is a tumor-infiltrating lymphocyte, wherein the first and second threshold values ​​are values ​​from 0 to 1, and the second threshold value is the first Tumor-infiltrating lymphocyte detection system with a value greater than the threshold. Claim 2 A tumor-infiltrating lymphocyte detection system according to claim 1, further comprising: an image collection unit that collects a patch image in which tumor-infiltrating lymphocytes are present and a patch image in which tumor-infiltrating lymphocytes are not present; a preprocessing unit that generates point labels for tumor-infiltrating lymphocytes present in the patch image and obtains a regiond image of the tumor-infiltrating lymphocytes from the patch image; and a learning unit that inputs the patch image with the generated point labels and the regiond image to a constructed patch classification model and a cell classification model for training. Claim 3 In claim 2, the preprocessing unit is a tumor-infiltrating lymphocyte detection system that generates a dot label by generating a dot at the center of a tumor-infiltrating lymphocyte included in a patch image. Claim 4 In claim 2, the preprocessing unit inputs a patch image into a cell nucleus regionalization model to obtain a plurality of first regionalized images of tumor-infiltrating lymphocytes, and applies a circular filter algorithm to the plurality of first regionalized images to obtain a plurality of second regionalized images of tumor-infiltrating lymphocytes, thereby forming a tumor-infiltrating lymphocyte detection system. Claim 5 In claim 4, the preprocessing unit extracts a plurality of second regiond images for training a cell classification model among a plurality of first regiond images according to cell size, circularity, convexity, and inertia ratio using a circular filter algorithm, a tumor-infiltrating lymphocyte detection system. Claim 6 In claim 2, the learning unit is a tumor-infiltrating lymphocyte detection system that uses a patch image and a dot label to train a patch classification model to classify a patch image into a value of 0 or 1 according to the amount of tumor-infiltrating lymphocytes present in the patch image. Claim 7 In claim 6, the learning unit inputs a plurality of second-registered images into a cell classification model and trains the cell classification model to output a probability value indicating whether a cell included in each second-registered image is a tumor-infiltrating lymphocyte. Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 A method for detecting tumor-infiltrating lymphocytes using a quantitative measurement system comprises the steps of: receiving a patch image to be measured; inputting the received patch image to be measured into a trained patch classification model to classify the patch image; inputting the regiond image from the patch image into a trained cell classification model to classify whether it corresponds to a tumor-infiltrating lymphocyte; and synthesizing the results of the patch classification model and the cell classification model to detect tumor-infiltrating lymphocytes present in the patch image to be measured. The patch classification model is a model trained to receive a patch image, compare the amount of tumor-infiltrating lymphocytes present in the patch image with a preset reference value, and output the patch image as a value of 0 or 1. The cell classification model is a model trained to receive the regiond image and represent whether each cell is a tumor-infiltrating lymphocyte as a probability value. The detection step comprises selecting one of a preset first and second threshold values ​​according to the result of the patch classification model, comparing the selected threshold value with the probability value resulting from the cell classification model to determine whether the cell is a tumor-infiltrating lymphocyte and detecting it. The first and second threshold values ​​are values ​​from 0 to 1, wherein the second threshold value is greater than the first threshold value Method for detecting large tumor-infiltrating lymphocytes. Claim 12 A method for detecting tumor-infiltrating lymphocytes according to claim 11, further comprising the steps of: collecting a patch image in which tumor-infiltrating lymphocytes are present and a patch image in which tumor-infiltrating lymphocytes are not present; generating dot labels on the tumor-infiltrating lymphocytes present in the patch image and obtaining a regiond image of the tumor-infiltrating lymphocytes from the patch image; and inputting the patch image with the generated dot labels and the regiond image into a constructed patch classification model and a cell classification model to train them.

Images