Tumor segmentation device and method through semi-supervised learning based on mean-teacher model using local guide map
The semi-supervised learning method with a mean-teacher model and local guide map effectively segments small kidney tumors by leveraging unlabeled data and labeled data synergistically, enhancing localization and segmentation accuracy.
Patent Information
- Application Number
- KR1020240016263
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-02-01
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-02-01
AI Technical Summary
Existing kidney tumor segmentation methods face challenges in accurately identifying small tumors in abdominal CT images due to insufficient labeled data, overfitting, and the similarity in shape and brightness values between tumors and surrounding organs, leading to over-segmentation and under-segmentation issues.
A semi-supervised learning approach using a mean-teacher model with a local guide map that generates a kidney local guide map from unlabeled data, incorporating it as an additional input channel to focus on tumor locations, and utilizes both labeled and unlabeled data through supervised and consistency loss functions for improved segmentation.
Enhances the detection of small kidney tumors by improving localization and segmentation performance, particularly for small-sized tumors, while reducing the reliance on extensive labeled data and addressing overfitting.
Smart Images

Figure 112024013200185-PAT00009_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a tumor segmentation device and method using semi-supervised learning based on an mean-teacher model utilizing a local guide map. Specifically, it relates to a tumor segmentation device and method that improves generalization performance and compensates for overfitting by using both labeled and unlabeled data together. Background Technology
[0002] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.
[0003] Kidney cancer accounts for 2 to 3% of all adult tumors, with over 430,000 cases of kidney disease detected worldwide. Since most cases present with no specific symptoms in the early stages, the disease often has already progressed by the time symptoms appear; therefore, early detection and surgical treatment are crucial. In the past, radical nephrectomy, which involves the removal of the kidney and surrounding metastatic tissues or lymph nodes, was common; however, recently, partial nephrectomy, which removes only the tumor while preserving the kidney as much as possible, is being performed more frequently. To plan a partial nephrectomy, it is necessary to divide the kidney tumor, as it is important to identify the location and morphology of the tumor as well as the safety margin for protecting key tissues during surgery.
[0004] However, in Figure 1, which shows the characteristics of the kidney in an abdominal CT image, if we refer to the diagram showing the various locations of the kidney tumor (Figure 1 (a)) and the signal intensity similarity between the kidney tumor and other organs (Figure 1 (b)), it can be seen that there are limitations in segmenting the kidney tumor because tumors of various sizes, large and small, exist in various locations such as inside, outside, and at the boundary of the patient's kidney, and their shape and brightness values are similar to those of surrounding organs such as the small intestine and spleen, which can lead to over-segmentation and outliers in those organs.
[0005] Related research for segmenting kidney tumors in abdominal CT images is generally conducted through supervised learning, and performance is evaluated using public data provided in the 2019 Kidney and Kidney Tumor Segmentation (KiTS19) Challenge. To identify the approximate location of kidney tumors in abdominal 3D CT images, a stepped V-Net consisting of localization and segmentation steps and a U-Net utilizing attention such as a Squeeze and Excitation (SE) module that extracts important feature information and focuses on key parts were used to achieve kidney tumor segmentation performance of 54.16 to 74.32%, but there were limitations in detecting small tumors.
[0006] Furthermore, improved performance of 82.54% to 85.09% was demonstrated for renal tumors by applying residual connections to nnU-Net, which automatically optimizes network parameters based on a self-adaptive framework, or by utilizing a step network that extracts kidney regions from low-resolution 3D images, upsamples them to high-resolution 3D images, and then segments them into kidneys and renal tumors, or by using it for kidney localization and segmentation of kidneys and renal tumors. In the case of renal tumor segmentation using such deep convolution-based supervised learning, performance degradation may occur if labeled data is insufficient, so securing sufficient labeled data is crucial. However, there are limitations in collecting labeled data due to the high time and cost required for labeling medical images. Additionally, while various studies have been conducted to address the problem of overfitting to labeled data during the early stages of training, there are currently no research cases specifically regarding renal tumors. Prior art literature
[0007] 1. Korean Published Patent No. 10-2022-0023323 (March 2, 2022) 2. Korean Published Patent No. 10-2023-0147492 (October 23, 2023) The problem to be solved
[0008] A tumor segmentation device and method based on semi-supervised learning of an mean-teacher model utilizing a local guide map according to an embodiment effectively detects small kidney tumors in situations where label data is insufficient.
[0009] To this end, the kidney tumor segmentation device and method based on semi-supervised learning of an mean-teacher model utilizing a local guide map according to an embodiment generates a kidney local guide map containing location information of the kidney by utilizing unlabeled data together, and provides a mean-teacher network that segments by focusing on tumors around the kidney by utilizing the generated local guide map as an additional input channel.
[0010] In addition, the tumor segmentation device and method based on semi-supervised learning of an mean-teacher model utilizing a local guide map according to the embodiment perform a performance comparison between the basic U-Net and nnU-Net, which shows superior performance in kidney and kidney tumor segmentation, and analyze the results by classifying them into four groups according to tumor size. means of solving the problem
[0011] A tumor segmentation device based on semi-supervised learning of a mean-teacher model utilizing a local guide map according to an embodiment includes: a memory storing at least one instruction for kidney tumor segmentation; and a processor that performs an operation according to the instruction. The processor performs semi-supervised learning of a mean-teacher model utilizing a kidney local guide map in an abdominal CT image, and the mean-teacher model generates a kidney local guide map containing location information of the kidney, and segments the kidney tumor focusing on small-sized tumors around the kidney through the kidney local guide map.
[0012] In addition, the processor can crop the entire abdominal CT image to include only the kidney region to detect the kidney region, adjust the size of the detected kidney region, and adjust the brightness value of the image if the brightness value of the abdominal CT image and the kidney tumor and surrounding tissue in the kidney region are similar to a certain level or higher.
[0013] In addition, the processor adjusts the brightness values of the image by performing brightness value normalization to distinguish between the kidney tumor and surrounding tissues, and the brightness value normalization can be performed using min-max normalization.
[0014] In addition, the mean-teacher model includes a student model and a teacher model, and can perform semi-supervised learning using both labeled and unlabeled images.
[0015] In addition, the student model and the teacher model may include a contracting path and an expanding path consisting of five layers based on a backbone network.
[0016] In addition, at each layer of the shrinkage path, convolution operations, batch normalization and the ReLU activation function can be applied, and feature maps can be generated through max pooling and dropout.
[0017] In addition, the mean-teacher model uses the supervised loss function (L) as the total loss function for learning. S ) and consistency loss function (L C By using ) together, labeled and unlabeled data can be utilized for training.
[0018] In addition, the supervised loss function is a loss function that performs learning on labeled data, and by calculating the difference between the predicted value output by the student model and the correct label, it can train the model so that the predicted result approaches the actual correct answer.
[0019] In addition, the supervised loss function can be calculated as the sum of binomial cross entropy and Dice loss.
[0020] In addition, the consistency loss function is a loss function that performs learning on both labeled and unlabeled data, allowing the model to be trained to maintain consistency by minimizing the difference between the predicted values output by the student model and the teacher model. Effects of the invention
[0021] A kidney tumor segmentation device and method based on a mean-teacher model using a local guide map according to an embodiment improves tumor location identification performance by focusing on the area around the tumor while considering the location information of the kidney.
[0022] In addition, the kidney tumor segmentation device and method using semi-supervised learning based on an average-teacher model utilizing a local guide map according to the embodiment improve segmentation performance, particularly for small-sized tumors.
[0023] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description of the invention or the claims. Brief explanation of the drawing
[0024] Figure 1 is a diagram showing the characteristics of the kidney in an abdominal CT image. FIG. 2 is a diagram illustrating the data processing process of a tumor segmentation device through semi-supervised learning based on an mean-teacher model utilizing a local guide map according to an embodiment. FIG. 3 is a block diagram of a tumor splitting device (100) according to an embodiment. FIG. 4 is a drawing showing the normalization effect according to an embodiment. FIG. 5 is a diagram illustrating the configuration of the mean-teacher model and the data processing process according to an embodiment. Figure 6 is a table showing the results of the kidney tumor segmentation performance indicators for the supervised learning method and the semi-supervised learning method according to the label data ratio. Figure 7 is a table to verify the segmentation performance results by kidney tumor size. Figure 8 shows the qualitative results of renal tumor segmentation by tumor size using supervised and semi-supervised learning methods based on the ratio of labeled data in abdominal CT images. Figure 9 shows the Grad-CAM visualization results of renal tumor segmentation by tumor size using supervised and semi-supervised learning methods based on the ratio of labeled data in abdominal CT images. FIG. 10 is a diagram illustrating the process of kidney tumor segmentation through semi-supervised learning based on an mean-teacher model utilizing a local guide map according to an embodiment. Specific details for implementing the invention
[0025] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.
[0026] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0027] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0028] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0029] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.
[0030] Some of the operations or functions described herein as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.
[0031] In describing the tumor segmentation device and method based on semi-supervised learning using an average-teacher model utilizing a local guide map according to the embodiment, an example applied to the kidney is described, but the tumor segmentation process in the kidney is merely one example. It goes without saying that the tumor segmentation device and method based on semi-supervised learning using an average-teacher model utilizing a local guide map according to the embodiment can be applied to organs other than the kidney. Furthermore, the tumor segmentation device and method based on semi-supervised learning using an average-teacher model utilizing a local guide map according to the embodiment is not limited to CT images but can be applied to other image modalities such as MR.
[0032] Hereinafter, the present invention will be described in detail with reference to the attached drawings.
[0033] Figure 2 is a diagram illustrating the data processing process of a tumor segmentation device through semi-supervised learning based on an average-teacher model utilizing a local guide map according to an embodiment.
[0034] Referring to FIG. 2, a tumor segmentation device (100) according to an embodiment may be configured to include a two-stage mean-teacher model. The mean-teacher model first performs a kidney segmentation step (Stage 1) that generates a kidney local guide map containing location information of the kidney. Then, using the generated kidney local guide map, it performs a kidney tumor segmentation step (Stage 2) that focuses on small-sized tumors around the kidney.
[0035] A tumor segmentation device (100) based on semi-supervised learning using a mean-teacher model utilizing a local guide map according to an embodiment enables the effective detection of small kidney tumors in situations where label data is scarce. To this end, the tumor segmentation device (100) generates a kidney local guide map containing location information of the kidney by utilizing unlabeled data together. Subsequently, a mean-teacher network is proposed that segments by focusing on tumors around the kidney using the generated kidney local guide map as an additional input channel. In addition, a performance comparison is performed between the basic U-Net and nnU-Net, which shows superior performance in kidney and kidney tumor segmentation, and the results are analyzed by classifying them into four groups according to tumor size.
[0036] FIG. 3 is a block diagram of a tumor splitting device (100) according to an embodiment.
[0037] In the embodiment, the tumor splitting device (100) may include a server as a computing system that provides services to other computers or devices in a computer network or stores and manages data. The tumor splitting device (100) accepts requests from other computers or devices called clients and provides responses or data to those requests. The configuration of the tumor splitting device (100) shown in FIG. 3 is merely a simplified example.
[0038] The communication unit (110) can be configured regardless of the mode of communication, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Additionally, the communication unit (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication unit (110) may be responsible for transmitting and receiving data necessary to perform a technique according to one embodiment of the present disclosure.
[0039] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.
[0040] Memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). For example, memory (120) stores RM data and RM protocols according to the user, as will be described later. Additionally, memory (120) stores various types of modules, instruction sets, or models.
[0041] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).
[0042] In an embodiment, the processor (130) performs a two-step mean-teacher model. The processor (130) performs a kidney segmentation step that generates a kidney local guide map containing location information of the kidney, and a kidney tumor segmentation step that focuses on small-sized tumors around the kidney through the kidney local guide map. Each step is described below.
[0043] The processor (130) performs a data preprocessing process to generate data corresponding to the kidney region in order to create a kidney local guide map containing location information of the kidney. Since the kidney occupies a relatively small area in the abdominal CT image, the processor (130) detects the kidney region by cropping the image to include only the kidney region, instead of using the entire abdominal CT image of size 512x512, in order to segment the kidney by focusing on the kidney region information. Afterward, the size of the detected kidney region is adjusted. In an embodiment, the processor (130) can adjust the size of the kidney region to 400x256.
[0044] In an abdominal CT image, if the brightness values of the kidney tumor and surrounding tissue are similar, missegmentation may occur by identifying other organs or surrounding tissue as the kidney tumor. Therefore, in the embodiment, the processor (130) adjusts the brightness values of the image when the brightness values of the kidney tumor and surrounding tissue in the kidney region are similar to the abdominal CT image by a certain level or more.
[0045] As shown in Fig. 4, brightness values are adjusted to normalize brightness values so that the kidney tumor and surrounding tissues can be clearly distinguished.
[0046] FIG. 4 is a diagram showing the normalization effect according to an embodiment, where (a) of FIG. 4 is the original image and (b) is the image after performing min-max normalization. In the embodiment, the processor (130) performs min-max normalization of the brightness values of the image using Equation 1 to normalize the brightness values of the image, and normalizes the brightness values to between 0 and 255.
[0047] [Mathematical Formula 1]
[0048]
[0049] In this case, I is the brightness value of the original image, and I min Wow I max These are the minimum and maximum values of the brightness range to be used, respectively, and are set to -79HU and 304HU, representing the fat and soft tissue ranges, to reduce the impact on tissues unrelated to the height.
[0050] In abdominal CT images, since kidney tumors occur in locations adjacent to the kidney, the location of the kidney tumor can be identified by utilizing the location information of the kidney. Therefore, the processor (130) first segments the kidney using an average-teacher model, and then uses the segmentation result as a kidney local guide map to provide location information of the kidney during the tumor segmentation step.
[0051] Figure 5 is a diagram illustrating the configuration of the mean-teacher model and the data processing process according to an embodiment.
[0052] Referring to FIG. 5, the mean-teacher model according to the embodiment performs semi-supervised learning using both labeled and unlabeled images. The mean-teacher model consists of a Student Model and a Teacher Model, and provides a method to improve performance by utilizing consistency between the Student Model and the Teacher Model. The Student Model is trained through comparisons between the correct answers in the labeled images and the output of the Student Model, and comparisons between the output of the Student Model and the output of the Teacher Model. At each training step, the Teacher Model updates the parameters by applying an exponential moving average to the parameters of the Student Model.
[0053] In the embodiment, the student model and the teacher model use 2D U-Net as the backbone network and have a contracting path and an expanding path consisting of five layers. In each layer of the contracting path, 3x3 convolution operations, batch normalization, and the ReLU activation function are applied, and feature maps are generated through 2x2 max pooling and dropout. In each layer of the expanding path, the feature maps obtained through 2x2 upsampling are combined with the feature maps of the contracting path located in the same layer through skip connections, and 3x3 convolution operations, batch normalization, and ReLU are repeated. The last layer consists of a 1x1 convolution layer using the sigmoid activation function to generate probability maps for prediction. Dropout is applied to provide additional perturbations to improve the generalization performance of the network, and in the case of the teacher model, random noise is applied to the input image.
[0054] In the embodiment, the student model is trained by updating parameters using gradient descent, and the teacher model parameters are updated at every training step using the exponential moving average between the student model parameters and the teacher model parameters of the previous step through Equation 2.
[0055] [Mathematical Formula 2]
[0056]
[0057] θ in mathematical equation 2 t is a parameter of the teacher model and θ t-1 is the teacher model parameter of the previous step, α is the student model parameter, and is the hyperparameter value that controls the moving average and was calculated as 0.99.
[0058] Subsequently, the processor (130) uses a two-channel image generated by combining the CT image and the kidney local guide map as the input image for the mean-teacher model to finally segment the kidney tumor. By using the kidney local guide map as an additional input channel, the location information of the kidney is taken into account, thereby allowing the location of small tumors to be identified by focusing on the area surrounding the kidney.
[0059] In an embodiment, the processor (130) uses Equation 3 as the total loss function for learning the mean-teacher model.
[0060] [Mathematical Formula 3]
[0061]
[0062] The processor (130) obtains the supervised loss function (L) through mathematical formula 3. S ) and consistency loss function (L C ) is used together to utilize both labeled and unlabeled data for training.
[0063] The supervised loss function is a loss function that performs learning on labeled data. It calculates the difference between the predicted value output by the student model and the correct label, thereby training the model so that the results predicted by the model become closer to the actual correct answer. The supervised loss function is calculated as the sum of the binomial cross entropy and the Dice loss through Equation 4.
[0064] [Mathematical Formula 4]
[0065]
[0066] The consistency loss function is a loss function that performs learning on both labeled and unlabeled data, and trains the model to maintain consistency by minimizing the difference between the predicted values output by the student model and the teacher model. The consistency loss function is calculated using the mean-squared error through Equation 5.
[0067] [Mathematical Formula 5]
[0068]
[0069] In the embodiment, loss functions enable the model to learn from unlabeled data by making it unnecessary for the model to have correct labels.
[0070] In mathematical formula 5, yi is the correct label of the i-th data, and is the predicted value output through the student model for the i-th data, and is the predicted value output through the teacher model for the i-th data, and N is the total number of data.
[0071] A tumor segmentation device (100) based on semi-supervised learning using an average-teacher model utilizing a local guide map according to an embodiment simulates kidney tumor segmentation and evaluates the results. To this end, the data used in the experiment is the KiTS19 Challenge data provided by the Minnesota Medical Center, which consists of preoperative abdominal CT arterial phase images of 210 patients who underwent partial nephrectomy or radical nephrectomy. The resolution of these images is 512x512, the number of slices is 19 to 1059, the slice thickness is 0.4375 to 1.041 mm, and the slice interval is 0.5 to 5 mm. The experiment was performed on a server equipped with a GeForce RTX 3090 graphics card using version 1.10.1 of PyTorch, a Python-based deep learning library. When training the network, the epoch was set to 200, the learning rate to 0.003, and the batch size to 8, and the Adam Optimizer was used for optimization. In addition, dropout values of [0.005, 0.1, 0.2, 0.3, 0.5] were applied to each layer of the student and teacher models. The experiment used a hold-out validation method, divided the data into 169 training datasets and 41 test datasets, and was conducted while adjusting the ratio of labeled to unlabeled data in the training dataset.
[0072] In addition, in the embodiment, to evaluate the kidney tumor segmentation device (100) through semi-supervised learning based on a mean-teacher model utilizing a local guide map, a supervised learning method using 2D U-Net and 2D nnU-Net was compared with a proposed semi-supervised learning method that considers kidney location information by adding a mean-teacher model (MT) and a kidney local guide map, and the performance according to the number of labeled data used for network training was analyzed by adjusting the ratio of labeled data. To evaluate the segmentation performance by kidney tumor size, the tumors were divided into four groups based on their size. If the tumor size was less than 2.3 cm, it was set as a small tumor; if it was 2.3 cm or more and less than 4.3 cm, it was set as a medium tumor; if it was 4.3 cm or more and less than 7.4 cm, it was set as a large tumor; and if it was 7.4 cm or more, it was set as a very large tumor. The mean and standard deviation of tumor size for each group were 1.8±0.2cm for small tumors, 3.1±0.5cm for medium tumors, 5.1±0.7cm for large tumors, and 10.5±2.1cm for very large tumors.
[0073] In addition, quantitative and qualitative evaluations were performed to evaluate the performance of the kidney tumor segmentation device (100) through semi-supervised learning based on an average-teacher model utilizing a local guide map according to the embodiment. In the quantitative evaluation, F1-score, recall, and precision were used as performance indicators to evaluate the segmentation performance between the proposed method and the comparison method. Additionally, since the kidney tumor region occupies a relatively small area compared to the background region in abdominal CT images, the ratio of the kidney tumor to the background is unbalanced. Therefore, to equally consider the performance of the kidney tumor region, a Balanced accuracy was additionally evaluated by calculating the accuracy of the kidney tumor and the background separately and taking the average.
[0074] Figure 6 is a table showing the results of kidney tumor segmentation performance indicators for supervised and semi-supervised learning methods according to the ratio of labeled data. Referring to Figure 6, for the supervised learning methods, the F1-score, recall, and precision values of the methods using only 25% labeled data decreased as the ratio of labeled data decreased, compared to the methods using 100% labeled data. The F1-score, recall, and precision of the semi-supervised learning method using 25% labeled data and 75% unlabeled data were 26.23%, 27.50%, and 34.66%, respectively. This indicates that the low performance was due to overfitting caused by the small amount of labeled data during the early stages of training, resulting in low generalization performance and under-segmentation caused by insufficient utilization of unlabeled data. For the proposed method using 25% of labeled data and applying a renal local guide map, the F1-score, recall, and precision values were 75.24%, 72.96%, and 86.57%, respectively. By considering the positional information of the kidney using the renal local guide map, it aided in identifying tumor locations, thereby improving the problem of under-segmentation of small renal tumors that occurred in other comparison methods and demonstrating the best results. Furthermore, the Balanced accuracy of the proposed method was 86.41%, showing a performance 7.46%p higher than nnU-Net using 100% of labeled data, indicating the most significant improvement in renal tumor segmentation accuracy.
[0075] Figure 7 is a table to verify the segmentation performance results according to kidney tumor size. Referring to Figure 7, supervised learning methods such as U-Net and nnU-Net, as well as MT, a semi-supervised learning method that does not use a kidney local guide map, failed to identify the location of the tumor, resulting in under-segmentation of small and medium-sized tumors in abdominal CT images and overall low segmentation performance. On the other hand, the proposed method using a kidney local guide map helped identify the location of the tumor by considering the positional information of the kidney, thereby improving the segmentation accuracy for small and medium-sized tumors, and showed performance similar to nnU-Net using 100% labeled data for large tumors. However, since the kidney local guide map only provides positional information of the kidney, positional information is limited in cases where the tumor is very large compared to the kidney or where only the tumor exists without the kidney; consequently, in very large tumors, it showed a tendency for under-segmentation of the tumor compared to nnU-Net using 100% labeled data, resulting in lower segmentation performance.
[0076] Figure 8 shows the qualitative results of kidney tumor segmentation by tumor size using supervised learning and semi-supervised learning methods based on the ratio of labeled data in abdominal CT images. Figure 8(a) shows the result of U-Net performed based on supervised learning on 100% of labeled data, Figure 8(b) shows the result of U-Net performed based on supervised learning on 25% of labeled data, Figure 8(c) shows the result of nnU-Net performed based on supervised learning on 100% of labeled data, Figure 8(d) shows the result of supervised learning performed on 25% of labeled data, Figure 8(e) shows the result of performing a mean-supervised model of semi-supervised learning using 25% of labeled data and 75% of unlabeled data together, and Figure 8(f) shows the result of performing a mean-supervised model with a kidney local guide map applied using 25% of labeled data and 75% of unlabeled data together. In the case of data 1 to 3, which represent the segmentation results of small tumors, methods (a) to (e) failed to perform proper segmentation because it was difficult to locate the tumor due to the small size of the kidney tumor. In contrast, method (f), which used a kidney local guide map, helped locate the tumor and showed improved segmentation results. In the case of data 4, 5, and 6, which represent the segmentation results of medium tumors, the tumors were larger than small kidney tumors, so methods (a) to (e) showed slight improvement in segmentation, but the problem of under-segmentation still occurred.
[0077] In contrast, method (f), which demonstrates an advantage in tumor localization, showed the best segmentation results. For data 7 and 8, which represent the segmentation of large tumors, methods (b) and (d), which used 25% of the labeled data, exhibited undersegmentation due to the limited amount of labeled data, whereas methods (a) and (c), which used 100% of the labeled data, improved the undersegmentation by utilizing more labeled data. On the other hand, method (f), which uses 25% of the labeled data and employs a renal local guide map, showed similar performance while using less labeled data, although it suffered from oversegmentation issues where peripheral renal tissue was incorrectly segmented as a renal tumor. For data 9 and 10, which represent the segmentation of very large tumors, method (f) tended to undersegment very large tumors because it considers only the positional information of the renal.
[0078] Figure 9 shows the Grad-CAM visualization results of renal tumor segmentation by tumor size using supervised and semi-supervised learning methods based on the ratio of labeled data in abdominal CT images. In the case of supervised learning methods using 25% and 100% of labeled data, it can be observed that they failed to properly identify the location of small tumors surrounding the kidney, resulting in the segmentation focusing only on a part of the renal tumor or even including the renal region as the renal tumor area. However, by considering the positional information of the kidney, the proposed method can be seen to focus on segmenting the renal tumor area compared to other methods, even in environments with low labeled data.
[0079] The method is described in order below. Since the operation (function) of the tumor segmentation method using semi-supervised learning based on an average-teacher model utilizing a local guide map according to the embodiment is essentially the same as the function of the tumor segmentation device using semi-supervised learning based on an average-teacher model utilizing a local guide map, descriptions that overlap with FIGS. 2 to 9 will be omitted.
[0080] FIG. 10 is a diagram illustrating a tumor segmentation process through semi-supervised learning based on an average-teacher model utilizing a local guide map according to an embodiment.
[0081] Referring to FIG. 10, in the embodiment, semi-supervised learning of a Mean Teacher model is performed using a Kidney Local Guided Map in an abdominal CT image. To this end, in step S100, semi-supervised learning of a Mean Teacher model is performed using a Kidney Local Guided Map in an abdominal CT image, and a kidney segmentation step is performed to generate a Kidney Local Guided Map containing location information of the kidney in the Mean Teacher model. Subsequently, in step S200, a tumor segmentation step is performed to segment the kidney tumor focusing on small tumors around the kidney using the Kidney Local Guided Map.
[0082] As used herein, a model may refer to any form of computer program that operates based on a network function, an artificial neural network, and / or a neural network. Throughout this specification, the terms model, neural network, network function, and neural network may be used interchangeably. A neural network is formed in which one or more nodes are interconnected through one or more links to form input and output node relationships within the neural network. The characteristics of a neural network may be determined by the number of nodes and links within the neural network, the relationships between the nodes and links, and the values of the weights assigned to each link. A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may form a layer.
[0083] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, Generative Adversarial Networks (GAN), restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), transformers, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.
[0084] Neural networks can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. The training of a neural network may be a process of applying knowledge to the neural network to perform a specific action.
[0085] Neural networks can be trained to minimize output errors. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, labeled data with correct answers is used for each training point, whereas in unsupervised learning, unlabeled data can be used. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a training cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's training cycle. In addition, to prevent overfitting, methods such as increasing training data, regularization, dropout (which disables some nodes), and batch normalization layers can be applied.
[0086] In one embodiment, the model may borrow at least a part of a transformer. The transformer may be composed of an encoder that encodes embedded data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of data and outputs a series of data of different types after undergoing encoding and decoding steps. In one embodiment, the series of data may be processed into a form that the transformer can compute. The process of processing the series of data into a form that the transformer can compute may include an embedding process. Expressions such as data token, embedding vector, embedding token, etc., may refer to data embedded in a form that the transformer can process.
[0087] To encode and decode a series of data, the encoders and decoders within the transformer can be processed using an attention algorithm. An attention algorithm can refer to an algorithm that calculates the similarity between one or more keys for a given query, applies this similarity to the values corresponding to each key, and then calculates an attention value by performing a weighted sum of the similarity-applied values.
[0088] Various types of attention algorithms can be classified depending on how the query, key, and value are configured. For example, if attention is calculated by setting the query, key, and value identically, this can be referred to as a self-attention algorithm. If attention is calculated by reducing the dimensionality of embedding vectors to process a series of input data in parallel and determining an individual attention head for each partitioned embedding vector, this can be referred to as a multi-head attention algorithm.
[0089] In one embodiment, the transformer may be composed of modules that perform a plurality of multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional components other than attention algorithms, such as embeddings, normalization, and softmax. A method for constructing the transformer using an attention algorithm may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.
[0090] A transformer can be applied to various data domains, such as embedded natural language, segmented image data, and audio waveforms, to convert a series of input data into a series of output data. To convert data with various data domains into a series of data that can be input to the transformer, the transformer can embed the data. The transformer can process additional data that represents the relative positional or phase relationships between the series of input data. Alternatively, the series of input data may be embedded by additionally reflecting vectors that represent the relative positional or phase relationships between the input data. In one example, the relative positional relationships between the series of input data may include, but are not limited to, word order within a natural language sentence, the relative positional relationships of each segmented image, and the temporal order of segmented audio waveforms. The process of adding information that represents the relative positional or phase relationships between the series of input data may be referred to as positional encoding.
[0091] In one embodiment, the model may include, but is not limited to, at least one of a Recurrent Neural Network (RNN), a Long Short Term Memory (LSTM) network, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).
[0092] In one embodiment, the model may be a model trained using a transfer learning method. Here, transfer learning refers to a learning method in which a pre-trained model having a first task is obtained by pre-training a large amount of unlabeled training data using a semi-supervised or self-learning method, and the pre-trained model is fine-tuned to be suitable for a second task, and a target model is implemented by training the labeled training data using a supervised learning method.
[0093] The kidney tumor segmentation device and method based on semi-supervised learning using an average-teacher model utilizing a kidney local guide map in an abdominal CT image according to an embodiment improves tumor localization performance by focusing on the area around the tumor while considering the positional information of the kidney.
[0094] In addition, the kidney tumor segmentation device and method using semi-supervised learning based on an mean-teacher model utilizing a kidney local guide map in abdominal CT images according to the embodiment improve segmentation performance, particularly for small-sized tumors.
[0095] The disclosed content is merely illustrative and can be modified and implemented in various ways by a person skilled in the art without departing from the gist of the claim in the patent claims; therefore, the scope of protection of the disclosed content is not limited to the specific embodiments described above. Explanation of the symbols
[0096] 100: Tumor splitting device
Claims
Claim 1 A tumor segmentation device comprising: a memory storing at least one instruction for tumor segmentation; and a processor that performs an operation according to said instruction, wherein the processor performs semi-supervised learning of a mean-teacher model by utilizing a kidney local guide map in an abdominal CT image, said mean-teacher model generates the kidney local guide map containing kidney location information and segments the kidney tumor focusing on small tumors around the kidney through the kidney local guide map, said processor detects the kidney region by cropping the abdominal CT image to include only the kidney region and adjusts the size of the detected kidney region, and performs brightness value normalization of the abdominal CT image to distinguish the kidney tumor from surrounding tissue, wherein the brightness value normalization is performed using min-max normalization, and finally segments the kidney tumor by using an image consisting of two channels generated by combining the abdominal CT image and the kidney local guide map in the channel direction as an input image of said mean-teacher model. Claim 2 delete Claim 3 delete Claim 4 A tumor segmentation device according to claim 1, wherein the mean-teacher model includes a student model and a teacher model, and performs semi-supervised learning using both labeled images and unlabeled images. Claim 5 In claim 4, the tumor segmentation device comprising a student model and a teacher model, wherein the student model and the teacher model include a contracting path and an expanding path consisting of five layers based on a backbone network. Claim 6 A tumor segmentation device according to claim 5, wherein each layer of the above-mentioned shrinkage path applies convolution operations, batch normalization, and a ReLU activation function, and generates a feature map through max pooling and dropout. Claim 7 In paragraph 4, the mean-teacher model is a supervised loss function (L) as the total loss function for learning. S ) and consistency loss function (L C A tumor segmentation device that utilizes both labeled and unlabeled data for training by using ) together. Claim 8 A tumor segmentation device according to claim 7, wherein the supervised loss function is a loss function that performs learning on labeled data, and trains the model to get closer to the actual correct answer by calculating the difference between the predicted value output by the student model and the correct label. Claim 9 A tumor segmentation device according to claim 7, wherein the map loss function is calculated as the sum of binomial cross entropy and Dice loss. Claim 10 In claim 7, the consistency loss function is a loss function that performs learning on both labeled and unlabeled data, and a tumor segmentation device that trains a model to maintain consistency by minimizing the difference between the predicted values output by the student model and the teacher model. Claim 11 A method for segmenting a kidney tumor through semi-supervised learning based on a mean-teacher model utilizing a kidney local guide map in an abdominal CT image, performed by a device, comprising: a step of performing semi-supervised learning of a mean-teacher model utilizing a kidney local guide map in an abdominal CT image; wherein the step of performing semi-supervised learning comprises a kidney segmentation step of generating a kidney local guide map including kidney location information in the mean-teacher model. A tumor segmentation method comprising: a tumor segmentation step for segmenting a kidney tumor with a focus on small tumors around the kidney through the kidney local guide map; wherein the kidney segmentation step comprises a step of cropping the abdominal CT image to include only the kidney region to detect the kidney region, adjusting the size of the detected kidney region, and performing brightness value normalization of the abdominal CT image to distinguish the kidney tumor from surrounding tissue, wherein the brightness value normalization is performed using min-max normalization; and wherein the tumor segmentation step comprises a step of finally segmenting the kidney tumor by using an image consisting of two channels generated by combining the abdominal CT image and the kidney local guide map in the channel direction as an input image of the mean-teacher model. Claim 12 delete Claim 13 delete Claim 14 A tumor segmentation method according to claim 11, wherein the mean-teacher model includes a student model and a teacher model, and performs semi-supervised learning using both labeled images and unlabeled images. Claim 15 A tumor segmentation method according to claim 14, wherein the student model and teacher model comprise a contracting path and an expanding path consisting of five layers based on a backbone network. Claim 16 A tumor segmentation method according to claim 15, wherein each layer of the above-mentioned shrinkage path applies convolution operations, batch normalization, and a ReLU activation function, and generates a feature map through max pooling and dropout. Claim 17 In paragraph 14, the mean-teacher model uses a supervised loss function (L) as the total loss function for learning. S ) and consistency loss function (L C A tumor segmentation method that utilizes both labeled and unlabeled data for training by using ) together. Claim 18 A tumor segmentation method according to claim 17, wherein the supervised loss function is a loss function that performs learning on labeled data, and trains the model to get closer to the actual correct answer by calculating the difference between the predicted value output by the student model and the correct label. Claim 19 A tumor segmentation method according to claim 17, wherein the above-mentioned map loss function is calculated as the sum of binomial cross entropy and Dice loss. Claim 20 In claim 17, the above consistency loss function is a loss function that performs learning on both labeled and unlabeled data, and a tumor segmentation method that trains a model to maintain consistency by minimizing the difference between the predicted values output by the student model and the teacher model.
Citation Information
Patent Citations
Method for segmentation based on medical image
KR1020220056892A
Device, method and program for segmenting medical images based on artificial intelligence
KR1020230148139A
Method for polyp segmentation with consistency training and continuous update of pseudo-label
KR102538212B1
Source of abdominal pain identification in medical imaging
US20180263585A1
