Appratus and method for segmentation through self-supervised learning based on patch zero-out jigsaw puzzle
The self-supervised learning network using a patch zero-out jigsaw puzzle approach enhances kidney segmentation in abdominal CT images by leveraging unlabeled data for pre-training and refining learning data, addressing the challenges of small kidney area and high labeling costs, achieving improved segmentation accuracy.
Patent Information
- Application Number
- KR1020240016264
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-02-01
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-02-01
AI Technical Summary
Existing kidney segmentation methods in abdominal CT images face challenges due to the small area occupied by the kidney, significant shape variation, unclear boundaries, and the high cost of obtaining labeled data, leading to performance degradation in supervised learning.
A self-supervised learning network using a patch zero-out jigsaw puzzle approach learns texture and location information from unlabeled data, performs supervised learning with labeled data, and selects tile positions by analyzing results, enhancing image understanding and segmentation efficiency.
The method improves learning efficiency and segmentation performance by leveraging unlabeled data for pre-training, refining learning data through patch zero-out, and learning texture information, resulting in improved segmentation accuracy even with reduced labeled data.
Smart Images

Figure R1020240016264_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a segmentation device and method based on patch zero-out jigsaw puzzle-based self-supervised learning. Specifically, it relates to an extended segmentation device and method that improves efficiency by using a self-supervised learning method with patch zero-out and jigsaw puzzles for unlabeled data, even when performing supervised learning with a large amount of labeled data. Background Technology
[0002] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.
[0003] Kidney cancer is one of the most common cancers worldwide, with over 330,000 cases diagnosed annually. Since kidney cancer causes almost no symptoms in its early stages and only manifests when it has progressed significantly and grown large enough to displace organs, it is crucial to apply surgical treatment during the asymptomatic early stages. Previously, radical nephrectomy, which involves removing the kidney and lymph nodes along with the tumor, was the standard treatment; however, recently, partial nephrectomy—which removes a portion of the kidney including the tumor to preserve remaining kidney function as much as possible—is being performed. In this context, it is necessary to automatically segment the kidney within abdominal CT images to establish a surgical plan by identifying the location and shape of the kidney before performing the partial nephrectomy.
[0004] However, as shown in Fig. 1, the kidney occupies a small area in the abdomen, and there are limitations in segmenting the kidney because there is significant variation in shape and position due to the kidney tumor and the brightness values of the kidney and the kidney tumor are similar, making the boundary unclear. Fig. 1 is a diagram showing the kidney in an abdominal CT image, where (a) is an abdominal CT image and (b) is a diagram showing a case where the shape of the kidney is deformed by a kidney tumor.
[0005] Related research on kidney segmentation in abdominal CT images is generally conducted using supervised learning, and performance is evaluated using public data provided in the 2019 Kidney and Kidney Tumor Segmentation (KiTS19) Challenge. These supervised learning-based methods require a sufficient amount of labeled data, and performance degradation may occur if there is insufficient data. However, there is a limitation in that it is difficult to obtain a large amount of labels due to the high cost of obtaining labels. Prior art literature
[0006] 1. Korean Patent Publication No. 10-2023-0125169 (Aug. 29, 2023) 2. Korean Patent Registration No. 10-2481191 (Dec. 21, 2022) The problem to be solved
[0007] The segmentation device and method based on self-supervised learning using a patch zero-out jigsaw puzzle according to an embodiment segments the kidney by learning texture and location information of the kidney through a jigsaw puzzle-based self-supervised learning network that adds a patch zero-out to segment the kidney occupying a small area in the abdominal region using unlabeled data.
[0008] In addition, the partitioning device and method based on self-supervised learning of a patch zero-out jigsaw puzzle according to the embodiment analyzes the results to select an appropriate tile position by performing patch zero-out while changing the position of the tile.
[0009] However, the problem to be solved according to one embodiment is not limited to that mentioned above. means of solving the problem
[0010] A segmentation device using a self-supervised learning network based on a patch zero-out jigsaw puzzle according to the present embodiment includes a processor that performs operations according to instructions, and the processor performs self-supervised learning based on a patch zero-out jigsaw puzzle to learn location information and texture information of the height using unlabeled data, performs supervised learning to segment the height using labeled data, and can select the position of the tile while changing the position of the tile performing patch zero-out by analyzing the result of self-supervised learning.
[0011] Additionally, the segmentation method using a self-supervised learning network based on a patch zero-out jigsaw puzzle according to the present embodiment may include: a step of performing self-supervised learning based on a patch zero-out jigsaw puzzle to learn location information and texture information of the height using unlabeled data; a step of performing supervised learning to segment the height using labeled data; and a step of selecting the position of a tile while changing the position of the tile performing patch zero-out by analyzing the result of the self-supervised learning. Effects of the invention
[0012] The segmentation device and method based on self-supervised learning using a patch zero-out jigsaw puzzle as described above learns various representations from unlabeled data through pre-training via self-supervised learning, thereby enhancing the network's understanding of images through pre-training when performing the downstream task of segmentation.
[0013] In addition, the segmentation device and method based on self-supervised learning of a patch zero-out jigsaw puzzle according to the embodiment improve learning efficiency by refining the learning data by making the patch area within a selected tile among the images (tiles) segmented through the jigsaw puzzle black and blacking out at the location of the organ.
[0014] In addition, the segmentation device and method based on patch zero-out jigsaw puzzle self-supervised learning according to the embodiment improve the efficiency of image reconstruction through supervised learning by blacking out organ regions using a patch zero-out method when performing self-supervised learning on any organ without limitations on the number of patches and organs.
[0015] In addition, in the embodiment, the blackout image tile improves supervised learning performance by enhancing self-supervised learning by setting the position to a tile containing an image of a long period.
[0016] In addition, in the embodiment, shape and texture information regarding the extended region and context information between the damaged region and the existing region can be learned by performing a cutout in the extended region and then performing reconstruction on the damaged region, and spatial relationships regarding the image can be understood by performing permutation prediction on the tiles.
[0017] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description of the invention or the claims. Brief explanation of the drawing
[0018] Figure 1 is a drawing showing the kidneys in an abdominal CT image. FIG. 2 is a diagram illustrating the data processing process of a jigsaw puzzle-based self-supervised learning network used in a partitioning device according to an embodiment. FIG. 3 is a block diagram of a dividing device according to an embodiment. FIG. 4 is a figure showing the structure of a patch zero-out jigsaw puzzle network according to an embodiment. FIG. 5 is a drawing showing a patch zero out generated according to an embodiment. Figure 6 is a diagram showing the results of the extension segmentation performance indicators of the supervised learning method and the self-supervised learning method according to the label data ratio. FIG. 7 is a diagram showing the results of the spanning segmentation performance indicators obtained by performing self-supervised learning by changing the tiles performing patch zero-out in the embodiment. Figure 8 is a diagram showing the qualitative results of kidney segmentation by supervised learning and self-supervised learning methods according to the ratio of labeled data in abdominal CT images. FIG. 9 is a diagram showing the qualitative results of extension segmentation performed by self-supervised learning by changing the area where patch zero-out is performed in the extension device in an embodiment. Figure 10 is a diagram illustrating the kidney segmentation process through a self-supervised learning network based on a patch zero-out jigsaw puzzle of abdominal CT images. Specific details for implementing the invention
[0019] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.
[0020] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0021] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0022] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0023] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.
[0024] Some of the operations or functions described herein as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.
[0025] In describing the segmentation device and method for abdominal CT images based on self-supervised learning using a patch zero-out jigsaw puzzle according to the embodiment, an embodiment applied to the kidney is described, but the segmentation process in the kidney is merely one example. It goes without saying that the segmentation device and method for abdominal CT images based on self-supervised learning using a patch zero-out jigsaw puzzle according to the embodiment can be applied to organs other than the kidney. Furthermore, the segmentation device and method for abdominal CT images based on self-supervised learning using a patch zero-out jigsaw puzzle according to the embodiment can be applied not only to abdominal CT images but also to other image modalities such as MR images.
[0026] Hereinafter, the present invention will be described in detail with reference to the attached drawings.
[0027] FIG. 2 is a diagram showing the data processing process of a jigsaw puzzle-based self-supervised learning network used in a lengthening segmentation device according to an embodiment.
[0028] Referring to FIG. 2, the kidney segmentation device (100) according to the embodiment segments the kidney by learning texture information and location information of the kidney through a jigsaw puzzle-based self-supervised learning network that adds patch zero-out to kidney segmentation occupying a small area in the abdominal region using unlabeled data. In addition, the segmentation device (100) according to the embodiment analyzes the results to select an appropriate tile location by performing patch zero-out while changing the position of the tile.
[0029] As illustrated in FIG. 2, the segmentation device (100) according to the embodiment performs a patch zero-out jigsaw puzzle-based self-supervised learning step of learning positional information and texture information of the height using unlabeled data through a jigsaw puzzle-based self-supervised learning network, and performs a supervised learning step of segmenting the height using labeled data.
[0030] FIG. 3 is a block diagram of a splitting device according to an embodiment.
[0031] In the embodiment, the partitioning device (100) is a computing system that provides services to other computers or devices in a computer network or stores and manages data, and may include a server. The partitioning device (100) accepts requests from other computers or devices called clients and provides responses or data to those requests. The configuration of the partitioning device (100) shown in FIG. 3 is merely a simplified example.
[0032] The communication unit (110) can be configured regardless of the mode of communication, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Additionally, the communication unit (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication unit (110) may be responsible for transmitting and receiving data necessary to perform a technique according to one embodiment of the present disclosure.
[0033] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.
[0034] Memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). For example, memory (120) stores RM data and RM protocols according to the user, as will be described later. Additionally, memory (120) stores various types of modules, instruction sets, or models.
[0035] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).
[0036] In an embodiment, the processor (130) performs self-supervised learning through a self-supervised learning network based on a patch zero-out jigsaw puzzle that learns positional information and texture information of the height using unlabeled data, performs supervised learning to segment the height using labeled data, and selects the position of the tile while changing the position of the tile that performs patch zero-out by analyzing the result of self-supervised learning.
[0037] To this end, the processor (130) performs image preprocessing. In an abdominal CT image, if the brightness values of the kidney and surrounding tissue are similar, the surrounding tissue may be incorrectly segmented as the kidney when kidney segmentation is performed. Therefore, the processor (130) performs brightness value normalization by adjusting the brightness values of the image so that the kidney and surrounding tissue can be clearly distinguished. Additionally, if the brightness values of the kidney and surrounding tissue in the abdominal CT image are similar to a certain level or higher, the processor (130) may adjust the brightness values of the image and perform brightness value normalization.
[0038] In the embodiment, the processor (130) performs min-max normalization on the brightness value of the image through Equation 1 for brightness value normalization, thereby normalizing the brightness value of the image to between 0 and 255.
[0039] [Mathematical Formula 1]
[0040]
[0041] In mathematical formula 1, I is the brightness value of the original image, and I min and I max These are the minimum and maximum values of the brightness range to be used, respectively; to reduce the influence of tissues unrelated to elongation, they are set to -79HU and 304HU, representing the fat and soft tissue ranges. Subsequently, the pixel size of the image with a range of 0.4375 to 1.041mm is normalized to the median value of 0.78125mm, and the slice thickness with a range of 0.5mm to 5mm is normalized to the median value of 3mm. Then, the image with a resolution of 512x512 is downsampled to 256x256 and used as the input image.
[0042] Hereinafter, a jigsaw puzzle-based self-supervised learning network according to an embodiment is described.
[0043] In the embodiment, the processor (130) utilizes a jigsaw puzzle technique, which is one of the pretext tasks of self-supervised learning utilizing unlabeled data. Through this, the processor (130) generates puzzle patches from an image, randomly shuffles them, and then recombines the shuffled images to restore the shuffled images. Since recombination helps identify the location of organs present within the patches, the jigsaw puzzle-based self-supervised learning network according to the embodiment performs jigsaw puzzle-based self-supervised learning to learn the visual representation of unlabeled data.
[0044] In the embodiment, before performing the jigsaw puzzle, the processor (130) performs a data preprocessing step by randomly cropping the image to be input to the model into 225x225 pixels and then dividing it into a 3x3 grid with a size of 75x75 pixels. Within each generated grid, the processor (130) performs random cropping again into 64x64 pixel tiles; by performing cropping and removing the connected parts between tiles, the relative position of each tile can be learned more accurately. Subsequently, the nine tiles are rearranged through a permutation randomly selected from a predefined set of permutations.
[0045] The processor (130) inputs the jigsaw puzzle generated through the process into a context-free network (CFN) model. The CFN consists of nine AlexNet-structured CNNs that share the same weights and a fully-connected layer that combines features generated from the nine CNNs to predict permutations for each patch. At this time, each CNN receives one tile as input.
[0046] Subsequently, the processor (130) performs pre-training of a jigsaw puzzle-based self-supervised learning network. The jigsaw puzzle-based self-supervised learning network is pre-trained using multiple proxy tasks and defined as a multi-objective loss function. FIG. 4 is a diagram showing the structure of a patch zero-out jigsaw puzzle network. The goal of the self-supervised learning proposed in the structure embodiment of the patch zero-out jigsaw puzzle network is to simultaneously learn location information and texture information for the spanning region, as shown in FIG. 4.
[0047] In the embodiment, during the pre-training stage, two projection heads are added after the nine CNN models that are executed on the jigsaw puzzle. The processor (130) numbers the nine tiles that enter as input to the jigsaw puzzle from 1 to 9, selects tiles 4 and 6 which generally have left and right extensions, performs a cutout to the patch size as shown in FIG. 5, and then shuffles the tiles to prepare them as inputs for the nine CNNs. FIG. 5 is a diagram showing the patch zero-out generated according to the embodiment.
[0048] In the embodiment, the processor (130) first takes the correct permutation for the 9 tiles processed from the network to the CNN to restore the tiles to their original positions and then merges them into a 2D image to restore the cutout area in the image. Afterward, the image is restored using the expanding path structure of the 2D U-Net as the reconstruction head. The reconstruction loss calculated in this step uses the MSE loss and is defined as Equation 2.
[0049] [Mathematical Formula 2]
[0050]
[0051] In this case, x represents the original image, and represents a reconstructed image. In the embodiment, to predict the shuffled tiles in the correct permutation order, the processor (130) first adds a fully-connected layer after 9 CNNs and then merges the 9 results through a concatenate. Subsequently, the processor (130) predicts the permutation using a permutation head composed of 2 fc layers and a softmax layer. The permutational loss calculated in this step uses the cross-entropy between the predicted permutation and the correct permutation and is defined as in Equation 3.
[0052] [Mathematical Formula 3]
[0053]
[0054] In mathematical formula 3, y represents the correct permutation index, and represents the predicted permutation index. In the embodiment, the processor (130) calculates the total loss through Equation 4 using the reconstruction loss that performs the reconstruction of the cutout image and the permutational loss that predicts the permutation, and proceeds with learning in a direction that minimizes the total loss.
[0055] [Mathematical Formula 4]
[0056] L total =L rec +L Der
[0057] In the embodiment, shape and texture information regarding the extended region and context information between the damaged region and the existing region can be learned by performing a cutout in the extended region and then performing reconstruction on the damaged region, and spatial relationships regarding the image can be understood by performing permutation prediction on the tiles.
[0058] Subsequently, the processor (130) performs extension splitting through a downstream task. In the embodiment, extension splitting is performed through supervised learning using labeled data, and the splitting network uses a 2D U-Net with AlexNet as the backbone. A CNN network of self-supervised learning based on a patch zero-out jigsaw puzzle is used as the pre-trained network of the U-Net encoder. As the loss function, the Dice similarity coefficient between the predicted label and the correct label is measured and used, and the model is trained to minimize it.
[0059] In addition, in the embodiment, the results are analyzed to select an appropriate tile position by performing patch zero-out while changing the position of the tile.
[0060] For the analysis, the data used in the experiment was the KiTS19 Challenge dataset provided by Minnesota Medical Center, consisting of preoperative abdominal CT arterial phase images of 210 patients who underwent partial or radical nephrectomy. The resolution of these images is 512x512, the number of slices ranges from 19 to 1059, the slice thickness ranges from 0.4375 to 1.041 mm, and the slice interval ranges from 0.5 to 5 mm. The experiment was conducted on a server equipped with a GeForce RTX 3090 graphics card using PyTorch version 1.10.1, a deep learning library based on Python 3.9.12. For network training, the pre-training parameters were set to 1000 epochs, a learning rate of 0.003, and a batch size of 16, and an SGD optimizer was used for optimization. For the downstream task, the parameters were set to 200 epochs, a learning rate of 0.0003, and a batch size of 8, and an Adam optimizer was used for optimization. The experiment used a hold-out validation method, splitting the data into 169 training datasets and 41 test datasets, and conducting experiments while adjusting the proportion of labeled data within the training dataset.
[0061] To evaluate the segmentation device based on self-supervised learning for patch zero-out jigsaw puzzles of abdominal CT images according to the embodiment, a supervised learning method using 2D U-Net and self-supervised learning methods such as Jigsaw Puzzle, SimCLR, and the proposed method were compared, and performance was analyzed by changing the tiles performing patch zero-out. To evaluate the performance of the proposed method, quantitative and qualitative evaluations were performed, and for the quantitative evaluation, F1-score, recall, and precision were used as performance indicators to evaluate the segmentation performance between the proposed method and the comparison method.
[0062] Figure 6 shows the results of the height segmentation performance indicators for supervised and self-supervised learning methods according to the ratio of labeled data. When experiments were conducted using 100% labeled data, the F1-score, recall, and precision were 89.33%, 87.02%, and 92.74%, respectively, when learning using supervised learning. When learning height position information through Jigsaw puzzle-based self-supervised learning, the F1-score, recall, and precision increased slightly to 89.63%, 87.7%, and 92.56%, respectively. SimCLR-based self-supervised learning is a framework for contrast learning of visual representations that learns image-level representations from unlabeled data by maximizing the agreement between augmented views of the same data; when this was performed, the highest performance was observed with F1-score, recall, and precision of 91.93%, 90.67%, and 93.82%, respectively. The proposed method achieved F1-score, recall, and precision of 91.47%, 89.72%, and 93.82%, respectively, by learning texture information of the elongation through patch zero-out and reconstruction. It demonstrated nearly identical performance compared to SimCLR, with an F1-score difference of 0.46%p. Furthermore, the performance improvement of the proposed method compared to supervised learning alone was greater as the proportion of labeled data decreased; in particular, the proposed method performed better when using 50% labeled data than when using 100% labeled data for supervised learning. Additionally, the F1-score of the proposed method increased by 0.68%p when using 50% labeled data and by 1.69%p when using 25% labeled data compared to SimCLR, indicating superior performance as the proportion of labeled data decreased.
[0063] Figure 7 is the result of the spanning segmentation performance indicator in which self-supervised learning was performed by changing the tiles that perform patch zero-out in the embodiment.
[0064] Referring to Figure 7, when patch zero-out was performed by selecting two random tiles from the seven tiles excluding the extension region out of the nine tiles, the lowest performance was observed because information regarding the extension region was not learned during reconstruction. When patch zero-out was performed by selecting two random tiles out of the nine tiles, performance was shown to improve slightly because the texture of some extension regions was learned, as the randomly selected tiles might include an extension region. When patch zero-out was performed by fixing the two tiles containing the extension region, the best performance was observed because texture information was learned by targeting the extension to be segmented.
[0065] Figure 8 shows the qualitative results of kidney segmentation by supervised and self-supervised learning methods based on the ratio of labeled data in abdominal CT images. Referring to Figure 8, for the segmentation results using 100% labeled data (Data 1–3), the 2D U-Net showed under-segmentation and over-segmentation occurring in kidney regions with brightness values similar to tumors and tumor regions located within the kidneys, respectively. In the Jigsaw Puzzle, it was confirmed that under-segmentation was partially improved as the location of the kidneys could be identified, but under- and over-segmentation still occurred. SimCLR, which learned the representation of the entire image, and the proposed method, which learned texture information regarding the kidneys, showed the best performance as under-segmentation and over-segmentation were improved. In the case of the segmentation results using 50% labeled data (Data 4–6) and 25% labeled data (Data 7–9), 2D U-Net showed under-segmentation or over-segmentation into the tumor region when the kidney shape was significantly deformed due to proximity to organs surrounding the kidney or large tumor size. Jigsaw puzzle showed some improvement in under-segmentation, but under- and over-segmentation still occurred.When SimCLR was applied, under-segmentation of the kidney deformed by a large tumor was significantly improved, but over-segmentation occurred in the tumor area, and in the example, the best results were shown with significant improvement in both under-segmentation and over-segmentation.
[0066] Figure 9 shows the qualitative results of kidney segmentation performed through self-supervised learning by changing the area for performing patch zero-out in a kidney device using a self-supervised learning network based on a patch zero-out jigsaw puzzle of an abdominal CT image according to an embodiment. Referring to Figure 9(b), which shows the results of performing patch zero-out by randomly selecting 2 tiles from the remaining 7 tiles excluding the kidney region among the 9 tiles, it was shown that under-segmentation and over-segmentation into the tumor and surrounding areas occurred because the texture information of the kidney was not properly learned. As a result of performing patch zero-out by randomly selecting 2 tiles out of the 9 tiles, it was possible to learn some of the texture information of the kidney as shown in Figure 9(a), showing slightly improved performance compared to Figure 9(b). However, since a lot of texture information regarding non-kidney areas was still learned, a tendency for under-segmentation was observed. As a result of performing patch zero-out in the extension area, as shown in (c) of Fig. 9, the best performance was achieved by learning only texture information for the extension.
[0067] Below, a segmentation method using a self-supervised learning network based on a patch zero-out jigsaw puzzle of an abdominal CT image will be described in turn. Since the operation (function) of the kidney segmentation method using a self-supervised learning network based on a patch zero-out jigsaw puzzle of an abdominal CT image according to the embodiment is essentially the same as the function of the kidney segmentation device using a self-supervised learning network based on a patch zero-out jigsaw puzzle of an abdominal CT image, descriptions that overlap with FIGS. 2 to 9 will be omitted.
[0068] Figure 10 is a diagram illustrating the kidney segmentation process through a self-supervised learning network based on a patch zero-out jigsaw puzzle of abdominal CT images.
[0069] Referring to FIG. 10, in step S100, self-supervised learning based on a patch zero-out jigsaw puzzle is performed to learn the position information and texture information of the kidney using unlabeled data. In step S200, supervised learning to segment the kidney using labeled data is performed, and in step S300, the results of self-supervised learning are analyzed to select the position of the tile while changing the position of the tile performing patch zero-out.
[0070] In this specification, networks and models may refer to any form of computer program that operates based on network functions, artificial neural networks, and / or neural networks. Throughout this specification, the terms model, neural network, network function, and neural network may be used interchangeably. A neural network is formed in which one or more nodes are interconnected through one or more links to form input-node and output-node relationships within the neural network. The characteristics of a neural network may be determined by the number of nodes and links within the neural network, the relationships between the nodes and links, and the values of the weights assigned to each link. A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may form a layer.
[0071] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, Generative Adversarial Networks (GAN), restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), transformers, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.
[0072] Neural networks can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. The training of a neural network may be a process of applying knowledge to the neural network to perform a specific action.
[0073] Neural networks can be trained to minimize output errors. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, labeled data with correct answers is used for each training point, whereas in unsupervised learning, unlabeled data can be used. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a training cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's training cycle. In addition, to prevent overfitting, methods such as increasing training data, regularization, dropout (which disables some nodes), and batch normalization layers can be applied.
[0074] In one embodiment, the model may borrow at least a part of a transformer. The transformer may be composed of an encoder that encodes embedded data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of data and outputs a series of data of different types after undergoing encoding and decoding steps. In one embodiment, the series of data may be processed into a form that the transformer can compute. The process of processing the series of data into a form that the transformer can compute may include an embedding process. Expressions such as data token, embedding vector, embedding token, etc., may refer to data embedded in a form that the transformer can process.
[0075] To encode and decode a series of data, the encoders and decoders within the transformer can be processed using an attention algorithm. An attention algorithm can refer to an algorithm that calculates the similarity between one or more keys for a given query, applies this similarity to the values corresponding to each key, and then calculates an attention value by performing a weighted sum of the similarity-applied values.
[0076] Various types of attention algorithms can be classified depending on how the query, key, and value are configured. For example, if attention is calculated by setting the query, key, and value identically, this can be referred to as a self-attention algorithm. If attention is calculated by reducing the dimensionality of embedding vectors to process a series of input data in parallel and determining an individual attention head for each partitioned embedding vector, this can be referred to as a multi-head attention algorithm.
[0077] In one embodiment, the transformer may be composed of modules that perform a plurality of multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional components other than attention algorithms, such as embeddings, normalization, and softmax. A method for constructing the transformer using an attention algorithm may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.
[0078] A transformer can be applied to various data domains, such as embedded natural language, segmented image data, and audio waveforms, to convert a series of input data into a series of output data. To convert data with various data domains into a series of data that can be input to the transformer, the transformer can embed the data. The transformer can process additional data that represents the relative positional or phase relationships between the series of input data. Alternatively, the series of input data may be embedded by additionally reflecting vectors that represent the relative positional or phase relationships between the input data. In one example, the relative positional relationships between the series of input data may include, but are not limited to, word order within a natural language sentence, the relative positional relationships of each segmented image, and the temporal order of segmented audio waveforms. The process of adding information that represents the relative positional or phase relationships between the series of input data may be referred to as positional encoding.
[0079] In one embodiment, the model may include, but is not limited to, at least one of a Recurrent Neural Network (RNN), a Long Short Term Memory (LSTM) network, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).
[0080] In one embodiment, the model may be a model trained using a transfer learning method. Here, transfer learning refers to a learning method in which a pre-trained model having a first task is obtained by pre-training a large amount of unlabeled training data using a semi-supervised or self-learning method, and the pre-trained model is fine-tuned to be suitable for a second task, and a target model is implemented by training the labeled training data using a supervised learning method.
[0081] The segmentation device and method based on self-supervised learning for patch zero-out jigsaw puzzles of abdominal CT images as described above learn various representations from unlabeled data through pre-training via self-supervised learning, thereby enhancing the network's understanding of the image through pre-training when performing the downstream task of segmentation.
[0082] In addition, the segmentation device and method based on self-supervised learning of a patch zero-out jigsaw puzzle of an abdominal CT image according to the embodiment improves learning efficiency by refining the learning data by making the patch area within a selected tile among the images (tiles) segmented through the jigsaw puzzle black, thereby blacking out the location of the organ.
[0083] In addition, the segmentation device and method for abdominal CT images based on patch zero-out jigsaw puzzle self-supervised learning according to the embodiment improve the efficiency of image reconstruction through supervised learning by blacking out organ regions using a patch zero-out method when performing self-supervised learning on any organ without limitations on the number of tiles and organs.
[0084] In addition, in the embodiment, the blackout image tile improves supervised learning performance by enhancing self-supervised learning through setting the position of the tile containing the long-term image.
[0085] In addition, in the embodiment, shape and texture information regarding the extended region and context information between the damaged region and the existing region can be learned by performing a cutout in the extended region and then performing reconstruction on the damaged region, and spatial relationships regarding the image can be understood by performing permutation prediction on the tiles.
[0086] The disclosed content is merely illustrative and can be modified and implemented in various ways by a person skilled in the art without departing from the gist of the claim in the patent claims; therefore, the scope of protection of the disclosed content is not limited to the specific embodiments described above. Explanation of the symbols
[0087] 100: Splitter
Claims
Claim 1 A memory storing at least one instruction for kidney segmentation through a self-supervised learning network based on a patch zero-out jigsaw puzzle of an abdominal CT image; and includes a processor that performs operations according to the above command, wherein the processor divides the abdominal CT image, for which image preprocessing has been performed to normalize brightness values to a specified Hounsfield Unit (HU) range representing the fat and soft tissue range, into a plurality of tiles, performs patch zero-out to black out the organ's location by making the patch area within the tile containing the kidney black, and then performs shuffle to randomly disrupt the order of the plurality of tiles to generate training data, inputs the shuffled plurality of tiles into a weight-sharing Convolutional Neural Network (CNN) and performs permutational loss to predict the correct permutation order through a fully connected layer and a permutation head, and reconstruction loss to restore the plurality of tiles that have passed through the Convolutional Neural Network (CNN) to their original positions, merges them into a 2D image, and restores the blacked-out and damaged area through a reconstruction head composed of an extended path structure of U-Net, so as to minimize the total loss, unlabeled A segmentation device that performs self-supervised learning based on a patch zero-out jigsaw puzzle to learn position information and texture information of the height using unlabeled data, reuses the pre-trained network of a U-Net encoder with the convolutional neural network (CNN) as a backbone, performs supervised learning to segment the height using labeled data, and selects the position of a tile while changing the position of the tile performing the patch zero-out by analyzing the result of the self-supervised learning. Claim 2 In claim 1, the processor is a segmentation device that adjusts the brightness value of an abdominal CT image and performs brightness value normalization to distinguish between the kidney and surrounding tissues. Claim 3 A segmentation device according to claim 1, wherein the self-supervised learning network performs a jigsaw puzzle technique, which is one of the pretext tasks of self-supervised learning utilizing unlabeled data, and the jigsaw puzzle technique generates puzzle patches from an image, randomly shuffles them, and then recombines the puzzle patches to restore the shuffled image. Claim 4 In paragraph 3, the jigsaw puzzle technique is a segmentation device that performs jigsaw puzzle-based self-supervised learning to learn the visual representation of the unlabeled data and identifies the location of organs existing within the patch according to the learning result. Claim 5 In paragraph 3, the self-supervised learning network is a segmentation device that pre-trains a jigsaw puzzle-based self-supervised learning network using a plurality of proxy tasks, defines it as a multi-objective loss function, and simultaneously learns location information and texture information for an extended region. Claim 6 In claim 5, the self-supervised learning network is a partitioning device that adds two projection heads after nine CNN models performed in a jigsaw puzzle, numbers nine tiles entering as input to the jigsaw puzzle from 1 to 9, selects tiles 4 and 6 that have left and right extensions, performs zero-out with a patch size, and then shuffles the tiles to prepare them as inputs to nine CNNs. Claim 7 In claim 6, the self-supervised learning network is a segmentation device that predicts permutations using a permutation head composed of two fc layers and a softmax layer, after first adding a fully-connected layer after nine CNNs to predict the shuffled tiles in the correct permutation order. Claim 8 In claim 6, the self-supervised learning network is a segmentation device that loads a ground truth permutation for 9 tiles, which have been processed up to a CNN to restore a cropped region in an image, restores the tiles to their original positions, and then merges them into a 2D image. Claim 9 delete Claim 10 In paragraph 5, the above multi-objective loss function is a splitting device that uses a Dice similarity coefficient between a predicted label and a correct label. Claim 11 A kidney segmentation method using a self-supervised learning network based on a patch zero-out jigsaw puzzle of an abdominal CT image, performed by a device, wherein the processor of the device divides the abdominal CT image, which has undergone image preprocessing to normalize brightness values to a specified Hounsfield Unit (HU) range representing fat and soft tissue areas, into a plurality of tiles; wherein the processor performs patch zero-out to black out the location of the organ by making the patch area within the tile containing the kidney black among the plurality of tiles, and then performs shuffle to randomly disrupt the order of the plurality of tiles to generate training data; wherein the processor inputs the shuffled plurality of tiles into a weight-sharing convolutional neural network (CNN) and performs permutational loss to predict the correct permutation order through a fully connected layer and a permutation head, and restores the plurality of tiles that have passed through the convolutional neural network (CNN) to their original positions, merges them into a 2D image, and then restores the blacked-out and damaged area through a reconstruction head composed of an extended path structure of a U-Net. A method for segmentation comprising: a step of performing self-supervised learning based on a patch zero-out jigsaw puzzle to learn position information and texture information of the height using unlabeled data to minimize the total loss, which is the sum of the reconstruction loss; a step in which the processor reuses the pre-trained network of a U-Net encoder with the convolutional neural network (CNN) as a backbone and performs supervised learning to segment the height using labeled data; and a step in which the processor analyzes the results of the self-supervised learning and selects the position of the tile while changing the position of the tile performing the patch zero-out. Claim 12 A segmentation method according to claim 11, wherein the step of performing self-supervised learning comprises the step of the processor adjusting the brightness value of the abdominal CT image and performing brightness value normalization to distinguish the kidney from surrounding tissues. Claim 13 In claim 11, the self-supervised learning network performs a jigsaw puzzle technique, which is one of the pretext tasks of self-supervised learning utilizing the unlabeled data, and the jigsaw puzzle technique is a partitioning method that generates puzzle tiles from an image, randomly shuffles them, and then recombines the puzzle tiles to restore the shuffled image. Claim 14 In claim 13, the above jigsaw puzzle technique is a segmentation method that performs jigsaw puzzle-based self-supervised learning to learn the visual representation of the above unlabeled data and identifies the location of organs existing within the patch according to the learning result. Claim 15 In claim 13, the self-supervised learning network is a segmentation method that pre-trains a jigsaw puzzle-based self-supervised learning network using multiple proxy tasks, defines it as a multi-objective loss function, and simultaneously learns location information and texture information for the spanning region. Claim 16 In claim 15, the self-supervised learning network is a partitioning method that adds two projection heads after nine CNN models performed in a jigsaw puzzle, numbers nine tiles entering as input to the jigsaw puzzle from 1 to 9, selects tiles 4 and 6 that have left and right extensions, performs zero-out with a patch size, and then shuffles the tiles to prepare them as inputs to nine CNNs. Claim 17 In claim 16, the self-supervised learning network is a partitioning method that predicts permutations using a permutation head composed of two fc layers and a softmax layer, after first adding a fully-connected layer after nine CNNs to predict the shuffled tiles in the correct permutation order. Claim 18 In claim 16, the self-supervised learning network is a segmentation method that loads a ground truth permutation for 9 tiles, which have been processed up to a CNN to restore a cropped region in an image, restores the tiles to their original positions, and then merges them into a 2D image. Claim 19 delete Claim 20 In paragraph 15, the above multi-objective loss function is a partitioning method that uses a Dice similarity coefficient between a predicted label and a correct label.
Citation Information
Patent Citations
Liver tumor recognition method based on self-supervised dense convolutional neural network
CN113362295A
Method, apparatus and program to read pneumonia
KR1020230047240A
Systems, methods, and apparatuses for learning semantics-enriched representations via self-discovery, self-classification, and self-restoration in the context of medical imaging
US20210265043A1
Multimodal self-supervised learning for image analysis
US20220067941A1
