Cross-Anatomical Region Organ Incremental Segmentation Method Based on Self-Supervision and Expert Gating

Abstract

The present invention discloses a method for incremental segmentation of organs across anatomical regions based on self-supervision and expert gating, which mainly solves the problem that the existing technology can only segment organs in specific anatomical regions and lacks scalability. The implementation solution is as follows: a large number of unlabeled CT images across anatomical regions are used; a self-supervised encoder with general representation ability is obtained through self-supervised pre-training; taking the trained self-supervised encoder as a fixed shared encoder, an under-complete autoencoder is constructed and added after the shared encoder to achieve incremental segmentation of CT images in multiple anatomical regions; an expert gating system composed of under-complete autoencoders is trained to automatically select the decoder corresponding to the test image in the test stage, and forward the test image to the correct decoder to obtain the segmentation result. The present invention simplifies the process of incremental segmentation of organs across anatomical regions, improves the overall segmentation performance of each anatomical region, and can be used to assist doctors in automatically delineating organs in each anatomical region.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for incremental segmentation of organs across anatomical regions, which can be used to assist doctors in automatically delineating multiple organs before the treatment of abdominal, thoracic and pelvic diseases. Background Art

[0002] In actual clinical applications, a model that can only segment a certain anatomical region is limited. Multi-organ segmentation is widely used in surgical path planning. The classic process of clinical surgical path planning and navigation is as follows: before surgery, the surgical target and other important tissue organs are segmented from CT images for surgical path planning; during the operation, rigid or non-rigid registration is performed on the deformed tissues to accurately locate the surgical target and other important tissues, and the surgical plan and operation are adjusted in real time. Precise surgical path planning and navigation are the keys to realizing robotic interventional surgery. When performing surgical path planning, in order to find the best intervention path, in addition to segmenting the organs in the anatomical region where the surgical target is located, the organs in adjacent anatomical regions should also be segmented. Therefore, there is an urgent clinical need for a segmentation model that can cross anatomical regions.

[0003] With the development of deep learning, since it avoids manual extraction of image features, more and more multi-organ segmentation methods based on deep learning have been proposed. The initial methods often convert the segmentation problem into a classification problem based on patch partitioning, and obtain the prediction of the central pixel of the patch by classifying the entire patch. The article "Segmentation of Organs-at-risk in Head and Neck CT Images Using Convolutional Neural Networks" published by Ibragimov et al. in Medical Physics achieved the segmentation of thirteen risk organs in the head and neck based on a convolutional classification network. A binary classification network was trained for each risk organ, and during training, the patches around the region of interest were used as positive samples, and the patches in other positions were used as negative samples, and CNN was used to extract features and classify. During testing, the class of the central pixel of the patch was determined according to the classification result of the patch.

[0004] With the proposal of the fully convolutional network FCN, the training and testing processes of segmentation networks have become more convenient. The dense connection V-type network proposed by Gibson et al. in the article Automatic Multi-organ Segmentation on Abdominal CT with Dense V-networks published in the journal IEEE Transactions on Medical Imaging improves the performance of abdominal multi-organ segmentation by adopting a dense connection mode and a multi-scale V-type network structure. Recently, Vision Transformers have gradually become the focus of attention. Using Transformers can capture long-range dependencies, making up for the deficiency of FCN in local feature extraction. In addition, applying Transformers, which were previously commonly used in natural language processing tasks, to computer vision is beneficial for multi-modal information fusion.

[0005] Chen et al. proposed Transformers Make Strong Encoders for Medical Image Segmentation in the journal Computer Vision and Pattern Recognition. It combines Transformers with the UNet structure, that is, at the end of the encoder, the feature maps are divided into blocks and encoded and sent into multiple cascaded Transformer modules to obtain long-range self-attention dependencies, giving full play to the advantages of both feature extraction methods. In the abdominal multi-organ dataset, TransUNet achieved better performance than the dense connection V-type network.

[0006] However, all of the above methods can only segment organs in specific anatomical regions and do not have scalability. When it is necessary to segment new anatomical regions, an independent segmentation network must be retrained, which makes the training process more complex and consumes more time and effort. Summary of the Invention

[0007] The purpose of the present invention is to address the above deficiencies of the prior art and propose a cross-anatomical region organ incremental segmentation network based on self-supervision and expert gating, so as to simultaneously segment multiple organs in different anatomical regions through a single network, simplify the cross-anatomical region segmentation process, and improve the segmentation efficiency of the model.

[0008] The technical idea of the present invention is as follows: By using a large number of unlabeled CT images across anatomical regions, a self-supervised network Net0 is trained using a self-supervised learning strategy; By adding an undercomplete autoencoder, the self-supervised network Net0 is extended to a cross-anatomical region organ increment first network Net1 for segmenting multiple abdominal organs; By adding a decoder and an undercomplete autoencoder, the cross-anatomical region organ increment first network Net1 is extended to a cross-anatomical region organ increment second network Net2 to increase the segmentation of multiple thoracic organs; Similarly, the cross-anatomical region organ increment second network Net2 is extended to a cross-anatomical region organ increment third network Net3 to increase the segmentation of multiple pelvic organs; By combining all undercomplete autoencoders into an expert gating system, decoder selection in the prediction stage is achieved.

[0009] According to the above idea, the implementation of the present invention includes the following:

[0010] (1) Obtain a CT dataset and divide it into a training set and a test set:

[0011] 1a) Obtain N publicly available CT datasets from websites and one non-public CT dataset from a hospital, where N ≥ 5, and among them:

[0012] One of the publicly available datasets is labeled with eight abdominal organs: the liver, the left kidney, the spleen, the esophagus, the gallbladder, the stomach, the pancreas, and the duodenum; Another publicly available dataset is labeled with six thoracic organs: the left lung, the right lung, the heart, the esophagus, the trachea, and the spinal cord;

[0013] The non-public dataset is labeled with five pelvic organs: the bladder, the rectum, the sigmoid colon, the small intestine, and the cervix;

[0014] 1b) Perform preprocessing of window adjustment, cropping, and normalization on the N publicly available datasets and the non-public dataset in sequence. Then, merge the preprocessed publicly available datasets without labels into a dataset D and perform tiling on it. Next, perform random perturbation processing on the pixel points in the tiled images, and divide them into an unlabeled image training set Dtr and an unlabeled image test set Dte;

[0015] 1c) Unify the sizes of the images in the two preprocessed labeled publicly available datasets and the non-public dataset, then perform data augmentation on the datasets, and randomly divide the three datasets after data augmentation into three labeled image training sets A1, A2, A3 and three labeled image test sets B1, B2, B3;

[0016] (2) Based on the UNet network structure, construct a self-supervised network Net0 composed of a self-supervised encoder and a decoder;

[0017] (3) Set the loss function during the training process to the mean absolute loss, and use the unlabeled image training set Dtr and the Adam optimizer to iteratively train the self-supervised network Net0 until the value of the loss function no longer decreases, obtaining a trained self-supervised encoder;

[0018] (4) Load all the parameters of the trained self-supervised encoder into the self-supervised network Net0, and add an undercomplete autoencoder in the self-supervised network Net0 to form the cross-anatomical region organ increment first network Net1;

[0019] (5) Train the cross-anatomical region organ increment first network Net1:

[0020] 5a) Freeze all the parameters in the trained self-supervised encoder;

[0021] 5b) Set the decoder loss function to the Dice loss and the undercomplete autoencoder loss to the MSE loss;

[0022] 5c) Use the labeled image first training set A1 and the Adam optimizer to iteratively train the cross-anatomical region organ increment first network Net1 until the values of the Dice loss and the MSE loss no longer decrease, obtaining a trained cross-anatomical region organ increment first network Net1;

[0023] (6) Load all the trained parameters of the cross-anatomical region organ increment first network into the cross-anatomical region organ increment first network Net1, and add a new decoder and a new undercomplete autoencoder in Net1 to form the cross-anatomical region organ increment second network Net2;

[0024] (7) Train the cross-anatomical region organ increment second network Net2:

[0025] 7a) Freeze all the parameters in the trained cross-anatomical region organ increment first network;

[0026] 7b) Set the loss function of the newly added decoder to the Dice loss and the loss of the newly added undercomplete autoencoder to the MSE loss;

[0027] 7c) Use the labeled image second training set A2 and the Adam optimizer to iteratively train the cross-anatomical region organ increment second network Net2 until the value of the loss function no longer decreases, obtaining a trained cross-anatomical region organ increment second network Net2;

[0028] (8) Load the trained parameters W2 of the cross-anatomical region organ increment second network into the cross-anatomical region organ increment second network Net2, and add a new decoder and a new undercomplete autoencoder in Net2 to form the cross-anatomical region organ increment third network Net3;

[0029] (9) Train the third network Net3 for cross - anatomical region organ increment:

[0030] 9a) Freeze all the parameters in the trained second network for cross - anatomical region organ increment;

[0031] 9b) Set the loss function of the newly added decoder to Dice loss and the loss of the newly added under - complete auto - encoder to MSE loss;

[0032] 9c) Use the third training set A3 of labeled images and the Adam optimizer to iteratively train the third network Net3 for cross - anatomical region organ increment until the value of the loss function no longer decreases, and obtain the trained third network Net3 for cross - anatomical region organ increment;

[0033] (10) Make predictions on the test set:

[0034] 10a) Load all the parameters of the trained third network Net3 for cross - anatomical region organ increment into the third network Net3 for cross - anatomical region organ increment;

[0035] 10b) Sequentially input the images in the three labeled test sets B1, B2, and B3 into the third network Net3 for cross - anatomical region organ increment after loading the parameters, and select the correct decoder for the image through the expert gating composed of the under - complete auto - encoder to obtain the segmentation result of the image.

[0036] Compared with the existing technology, the present invention has the following advantages:

[0037] 1. Simplify the cross - anatomical region organ segmentation process.

[0038] The present invention realizes the expansion from the first network Net1 for cross - anatomical region organ increment to the third network Net3 for cross - anatomical region organ increment by using a shared encoder, newly added decoders, and under - complete auto - encoders. Compared with the previous segmentation method of establishing independent models for each anatomical region, the expansion method of the present invention is simpler and has a shorter training time.

[0039] 2. Improve the segmentation performance of cross - anatomical regions.

[0040] The present invention uses a self - supervised learning strategy to train auxiliary tasks to obtain a feature encoder that can generally represent cross - anatomical region organs, improves the network's representation ability for cross - anatomical region images, and thus improves the segmentation performance of cross - anatomical regions;

[0041] Since the incremental expansion method proposed by the present invention does not change the structure of the old network, the segmentation of the new anatomical region will not affect the segmentation performance of the old anatomical region at all, thus improving the overall segmentation performance. Brief Description of the Drawings

[0042] Figure 1 is the implementation flowchart of the present invention;

[0043] Figure 2 is the structural schematic diagram of the self-supervised network Net0 in the present invention;

[0044] Figure 3 is the structural schematic diagram of the under-complete autoencoder in the present invention;

[0045] Figure 4 is the structural schematic diagram of the cross-anatomical region organ incremental third network Net3 in the present invention;

[0046] Figure 5 is the abdominal, chest and pelvic 3D segmentation visualization comparison diagram between the present invention and the existing comparison method;

[0047] Figure 6 is the abdominal, chest and pelvic 2D segmentation visualization comparison diagram between the present invention and the existing comparison method. Detailed Description of the Invention

[0048] The following presents the embodiments and effects of the present invention in conjunction with the drawings. It should be noted that this embodiment is only used for further detailed description of the technical solution of the present invention and does not constitute any limitation to the present invention.

[0049] Refer to Figure 1 , the implementation steps of this example are as follows:

[0050] Step 1. Obtain the CT dataset and divide it into a training set and a test set.

[0051] 1.1) Obtain N publicly available CT datasets from the Internet and one non-public CT dataset from a hospital, where N≥5, and:

[0052] One of the N publicly available CT datasets is labeled with eight abdominal organs: liver, left kidney, spleen, esophagus, gallbladder, stomach, pancreas, and duodenum; another publicly available dataset is labeled with six chest organs: left lung, right lung, heart, esophagus, trachea, and spinal cord;

[0053] The non-public dataset is labeled with five pelvic organs: bladder, rectum, sigmoid colon, small intestine, and cervix;

[0054] 1.2) Limit the CT images in all datasets within the interval [-160HU, 240HU] for cropping to cut off the useless black edges, and then normalize the cropped CT images to obtain the normalized image Y:

[0055]

[0056] Among them, X is the input image, and X min is the maximum gray value of the pixel points of the input image, and X max is the minimum gray value of the pixel points of the input image;

[0057] 1.3) Combine the public datasets of unlabeled organs after normalization into a dataset D, and randomly cut all the images in the dataset D into blocks of size 96×96×96;

[0058] 1.4) Perform random perturbation processing on the pixel points in the cropped images, that is, first take an image from the cropped dataset D, and perform non-linear mapping on the pixel points in the image with the help of a fifth-order Bezier curve; then randomly select multiple windows in the non-linearly mapped image, and shuffle the pixel points within the windows as the new window values; then set the maximum occlusion rate to 30%, and randomly cover the image with shuffled pixel values in a rectangular form;

[0059] 1.5) Divide the unlabeled image dataset D after random perturbation into an unlabeled image training set Dtr and an unlabeled image test set Dte according to the ratio of 8:2;

[0060] 1.6) Unify the sizes of the images in the two labeled public datasets and the unpublic dataset after normalization, that is, use the method of linear interpolation to uniformly adjust the CT images to the size of 96×96×96, and use the method of nearest neighbor interpolation to adjust the labeled images to the size of 96×96×96;

[0061] 1.7) Augment the two labeled public datasets and the unpublic dataset after size adjustment:

[0062] 1.7.1) Randomly select a CT image from the dataset;

[0063] 1.7.2) Randomly rotate and flip the selected CT image, and then scale the image and add the scaled image to the dataset;

[0064] 1.7.3) Repeat step 1.7.2) three times to complete five-fold augmentation of the dataset.

[0065] 1.8) Divide the two labeled public datasets and the unpublic dataset after data augmentation into three labeled image training sets A1, A2, A3 and three labeled image test sets B1, B2, B3 according to the ratio of 8:2.

[0066] Step 2. Based on the UNet network structure, construct a self-supervised network Net0 including a self-supervised encoder and a decoder.

[0067] Refer toFigure 2 , the specific implementation of this step is as follows:

[0068] 2.1) Build a self-supervised encoder composed of a three-dimensional convolutional block, four convolutional encoding blocks, and 12 Transformer layers cascaded in sequence. Each convolutional encoding block is composed of a multi-scale feature extraction module, an SE module, and a pooling layer cascaded. In this multi-scale feature extraction module, convolutional kernels of 1×1×1, 3×3×3, and 5×5×5 are used to extract features respectively;

[0069] 2.2) Build a decoder composed of four convolutional decoding blocks and four upsampling layers cascaded in sequence. This convolutional decoding block has the same structure as the convolutional encoding block. At the same time, skip connections are used to splice the low-level features output by the convolutional encoding block and the high-level features output by the same-layer convolutional decoding block to make the features at all levels complementary. The output of the three-dimensional convolution at the end of this decoder is set to 1 channel, and the Sigmoid activation function is used to normalize the output to between 0 and 1;

[0070] 2.3) Serially connect the self-supervised encoder and the decoder to form a self-supervised network Net0.

[0071] Step 3. Set the loss function during the training process as the mean absolute loss, and use the unlabeled image training set Dtr to train the self-supervised network Net0 to obtain a trained self-supervised encoder.

[0072] 3.1) Take a CT image from the unlabeled training set Dtr as the input of the self-supervised network Net0, obtain the output result of its decoder, and calculate the mean absolute loss L ssl (y, y'):

[0073]

[0074] where N represents the total number of image pixels;

[0075] 3.2) Backpropagate the calculated mean absolute loss L ssl (y, y') to obtain the gradient value, and use the Adam optimizer to update the parameters of the self-supervised network Net0;

[0076] 3.3) Repeat steps 3.1)-3.2) until all data in the unlabeled training set Dtr have been learned, and complete one iteration of learning;

[0077] 3.4) Repeat step 3.3) for multiple iterations of learning until the mean absolute loss value no longer decreases, and obtain a trained self-supervised encoder.

[0078] Step 4. Construct the cross-anatomical region organ incremental first network Net1.

[0079] 4.1) Establish an undercomplete autoencoder composed of a cascade of a fully connected layer, a ReLU activation function, a new fully connected layer, and a Sigmoid activation function, as Figure 3 shown;

[0080] 4.2) Load all the parameters of the trained self-supervised encoder into the self-supervised network Net0, and connect the newly added undercomplete autoencoder in series with the self-supervised encoder in the self-supervised network Net0 to construct the cross-anatomical region organ incremental first network Net1.

[0081] Step 5. Train the cross-anatomical region organ incremental first network Net1.

[0082] 5.1) Freeze all the parameters in the self-supervised encoder;

[0083] 5.2) Take a CT image from the first training set A1 of labeled images and input it into the cross-anatomical region organ incremental first network Net1. Calculate the first Dice loss L between the output result P1 of the decoder and the corresponding true labeled image G1 DC1 , and at the same time calculate the first MSE loss L AE (y1, y'1) of the output y1 of the undercomplete autoencoder and the input y'1 of the undercomplete autoencoder:

[0084]

[0085]

[0086] where N1 represents the length of the output feature vector;

[0087] 5.3) Backpropagate the calculated first Dice loss L DC1 to obtain the gradient value, and use the Adam optimizer to update the parameters of the decoder in the cross-anatomical region organ incremental first network Net1:

[0088] 5.4) Backpropagate the calculated first MSE loss L AE (y1, y'1) to obtain the gradient value, and use the Adam optimizer to update the parameters of the undercomplete autoencoder in the cross-anatomical region organ incremental first network Net1;

[0089] 5.5) Repeat steps 5.2)-5.4) until all the data in the first training set A1 of labeled images have been learned, completing one iteration of learning;

[0090] 5.6) Repeat step 5.5) for multiple iterations of learning until the first Dice loss LDC1 and the first MSE loss L AE (y1, y'1) no longer decreases, and the trained cross-anatomical region organ incremental first network Net1 is obtained.

[0091] Step 6. Construct the cross-anatomical region organ incremental second network Net2.

[0092] 6.1) Establish a new decoder with the same structure as the decoder in the self-supervised network Net0;

[0093] 6.2) Establish a new under-complete autoencoder with the same structure as the under-complete autoencoder in the cross-anatomical region organ incremental first network Net1;

[0094] 6.3) Load all the trained parameters of the cross-anatomical region organ incremental first network into the cross-anatomical region organ incremental first network Net1, and connect the new decoder and the under-complete autoencoder in series with the self-supervised encoder in the cross-anatomical region organ incremental first network Net1 respectively to form the cross-anatomical region organ incremental second network Net2.

[0095] Step 7. Train the cross-anatomical region organ incremental second network Net2.

[0096] 7.1) Freeze all the parameters in the cross-anatomical region organ incremental first network Net1;

[0097] 7.2) Take a CT image from the second training set A2 of labeled images and input it into the cross-anatomical region organ incremental second network Net2. Calculate the second Dice loss L between the output result P2 of the decoder and the corresponding true labeled image G2 DC2 , and at the same time calculate the second MSE loss L between the output y2 of the under-complete autoencoder and the input y’2 of the under-complete autoencoder AE (y2, y'2):

[0098]

[0099]

[0100] where N2 represents the length of the output feature vector;

[0101] 7.3) Backpropagate the calculated second Dice loss L DC2 to obtain the gradient value, and use the Adam optimizer to update the parameters of the decoder in the cross-anatomical region organ incremental second network Net2:

[0102] 7.4) For the calculated second MSE loss L AE(y2, y'2) Backpropagate to obtain the gradient value, and use the Adam optimizer to update the parameters of the undercomplete autoencoder in the cross-anatomical region organ increment second network Net2;

[0103] 7.5) Repeat steps 7.2)-7.4) until all data in the second training set A2 of the labeled images have been learned, completing one iteration of learning;

[0104] 7.6) Repeat step 7.5) for multiple iterations of learning until the second Dice loss L DC2 and the second MSE loss L AE (y2, y'2) no longer decreases, obtaining the trained cross-anatomical region organ increment second network Net2.

[0105] Step 8. Construct the cross-anatomical region organ increment third network Net3;

[0106] Refer to Figure 4 , the implementation of this step includes the following:

[0107] 8.1) Establish a new decoder with the same structure as the decoder in the self-supervised network Net0;

[0108] 8.2) Establish a new undercomplete autoencoder with the same structure as the undercomplete autoencoder in the cross-anatomical region organ increment first network Net1;

[0109] 8.3) Load all the trained parameters of the cross-anatomical region organ increment second network into the cross-anatomical region organ increment second network Net2, and connect the new decoder and undercomplete autoencoder in series with the encoder in the cross-anatomical region organ increment second network Net2 respectively to form the cross-anatomical region organ increment third network Net3.

[0110] Step 9. Train the cross-anatomical region organ increment third network Net3.

[0111] 9.1) Freeze all the parameters in the cross-anatomical region organ increment second network Net2;

[0112] 9.2) Take a CT image from the third training set A3 of the labeled images and input it into the cross-anatomical region organ increment third network Net3, calculate the third Dice loss L DC3 between the output result P3 of the decoder and the corresponding true labeled image G3, and at the same time calculate the third MSE loss L AE (y3, y'3) of the output y3 of the undercomplete autoencoder and the input y’3 of the undercomplete autoencoder:

[0113]

[0114]

[0115] Among them, N3 represents the length of the output feature vector;

[0116] 9.3) For the calculated third Dice loss L DC3 Backpropagate to obtain the gradient value, and use the Adam optimizer to update the parameters of the decoder in the cross-anatomical region organ incremental third network Net3:

[0117] 9.4) For the calculated third MSE loss L AE (y3, y'3), backpropagate to obtain the gradient value, and use the Adam optimizer to update the parameters of the undercomplete autoencoder in the cross-anatomical region organ incremental second network Net2;

[0118] 9.5) Repeat steps 9.2)-9.4) until all data in the third training set A3 of the labeled images have been learned, and complete one iteration of learning;

[0119] 9.6) Repeat step 9.5), perform multiple iterations of learning until the third Dice loss L DC3 and the third MSE loss L AE (y3, y'3) no longer decreases, and obtain the trained cross-anatomical region organ incremental third network Net3.

[0120] Step 10. Make predictions on the test set.

[0121] 10.1) Load all the parameters of the trained cross-anatomical region organ incremental third network Net3 into the cross-anatomical region organ incremental third network Net3;

[0122] 10.2) Input the images in the three labeled test sets B1, B2, and B3 into the cross-anatomical region organ incremental third network Net3 with loaded parameters in sequence;

[0123] 10.3) Calculate the MSE loss between the outputs and inputs of the three undercomplete autoencoders;

[0124] 10.4) Select the decoder corresponding to the undercomplete autoencoder with the smallest loss value as the decoder for the current image, and the output of the decoder is the segmentation result of the image.

[0125] The labels of the above steps are only for clearly explaining the implementation solution of the present invention, and the order of the labels is not limited.

[0126] The effects of the present invention can be further illustrated by the following simulations.

[0127] 1. Simulation conditions

[0128] The experimental simulation platform is a desktop computer with 32GB of memory. Its CPU model is Intel(R) Core(TM) CPU i9-10920X @ 3.50GHZ. Python 3.7 and torch 1.9.1 are used to build and train the neural network model. The NVIDIA GeForce RTX 3090 graphics card with 24GB of video memory is used to accelerate the operation with the Cuda 11.1 computing architecture.

[0129] The segmentation performance evaluation index used in the simulation is the Dice coefficient, and its calculation formula is as follows:

[0130]

[0131] Among them, A represents the true annotation, and B represents the segmentation result.

[0132] 2. Simulation content

[0133] Simulation 1: Using the unlabeled image training set Dtr, the labeled image training sets A1, A2, and A3, the incremental segmentation networks for the abdomen, chest, and pelvis are trained respectively using the incremental segmentation method proposed in the present invention and the existing four incremental segmentation methods of joint training, fine-tuning, feature extraction, and LWF. Then, the labeled image test sets B1, B2, and B3 are used to test the incremental segmentation networks obtained by training with their respective incremental segmentation strategies, and the segmentation results of the test sets for each segmentation method are obtained. The Dice coefficient between the segmentation result of each method and the true label is calculated, and the average Dice coefficient of the CT image segmentation results in each anatomical region is obtained, as shown in Table 1:

[0134] Table 1 Average Dice coefficients (%) of the segmentation results of each anatomical region of the present invention and the existing comparison methods

[0135]

[0136] The differences and segmentation results of the four existing incremental segmentation methods in Table 1 are as follows:

[0137] The so-called joint training refers to training a multi-branch network using the labeled image training sets A1, A2, and A3 at the same time. As can be seen from the first row of Table 1, the average performance of the three anatomical region organs in joint training is 1.13% lower than that of the method proposed in the present invention.

[0138] The so-called fine-tuning refers to fine-tuning both the encoder and decoder of the old network to incrementally segment new organs. As can be seen from the second row of Table 1, using fine-tuning for incremental segmentation results in very low Dice coefficients for the segmentation of the abdomen and chest, indicating that this method causes serious forgetting of the old anatomical regions.

[0139] The so-called feature extraction means that the encoder learned in the first task is fixed, and this encoder is shared by subsequent tasks. As can be seen from the third row of Table 1, the last two tasks, which are quite different from the first task of abdominal multi-organ segmentation, also achieved relatively good results. Perhaps it is because the decoder can adjust according to the encoder features for relevant tasks to obtain relatively good segmentation performance. However, the average performance of the organs in the three anatomical regions is still 1.03% lower than the method proposed in the present invention.

[0140] The so-called LWF refers to a method of protecting old knowledge from being overwritten by new knowledge by imposing knowledge distillation constraints on the loss function of the new network. As can be seen from the fourth row of Table 1, the average performance of using LWF for organ segmentation in the three anatomical regions is 3.31% lower than the method proposed in the present invention.

[0141] Based on the analysis of Table 1 above, the average Dice coefficient of the present invention is increased by 1.03% - 9.94% compared with the existing four segmentation methods, indicating that the present invention significantly improves the incremental segmentation performance.

[0142] Simulation 2: Randomly selected two test cases from the annotated image test sets B1, B2 and test set B3 respectively, and performed 3D visualization on the segmentation results of the three anatomical regions of the present invention and the existing four segmentation methods, and compared their accuracies. The results are as Figure 5 .

[0143] As can be seen from Figure 5 , when the present invention extends the segmentation of the pancreas, it has higher accuracy than the existing four segmentation methods, and the number of missegmentation and under-segmentation is less.

[0144] Simulation 3: Randomly selected two test cases from the test sets B1, B2 and B3 respectively, and performed 2D visualization on the segmentation results of the three anatomical regions of the present invention and the existing four segmentation methods, and compared their accuracies. The results are as Figure 6 .

[0145] As can be seen from Figure 6 , the same conclusion as in Simulation 2 can still be obtained from the 2D slice images of the segmentation results of the present invention and the existing four segmentation methods.

[0146] The above simulation results show that the present invention has higher cross-anatomical region organ incremental segmentation performance than the prior art.

Claims

1. An incremental segmentation method for organs across anatomical regions based on self-supervision and expert gating, characterized in that It includes the following steps: (1) Obtain the CT dataset and divide it into a training set and a test set: 1a) Obtain N publicly available CT datasets from the website and one non-public CT dataset from the hospital, where N ≥ 5, and: Among the publicly available datasets, one publicly available dataset is labeled with eight abdominal organs: liver, left kidney, spleen, esophagus, gallbladder, stomach, pancreas, and duodenum; another publicly available dataset is labeled with six thoracic organs: left lung, right lung, heart, esophagus, trachea, and spinal cord; The non-public dataset is labeled with five pelvic organs: bladder, rectum, sigmoid colon, small intestine, and cervix; 1b) Perform preprocessing of windowing, cropping, and normalization on the N publicly available datasets and the non-public dataset in sequence. Then, merge the preprocessed publicly available datasets without labels into a dataset D and perform block cutting. Then, perform random perturbation processing on the pixel points in the cut images and divide them into an unlabeled image training set Dtr and an unlabeled image test set Dte; 1c) Unify the sizes of the images in the two preprocessed labeled publicly available datasets and the non-public dataset, then perform data augmentation, and randomly divide these three datasets after data augmentation into three labeled image training sets A1, A2, A3 and three labeled image test sets B1, B2, B3; (2) Based on the UNet network structure, construct a self-supervised network Net0 composed of a self-supervised encoder and a decoder; (3) Set the loss function during the training process to the mean absolute loss, and use the unlabeled image training set Dtr and the Adam optimizer to iteratively train the self-supervised network Net0 until the loss function value no longer decreases, and obtain a trained self-supervised encoder; (4) Load all the parameters of the trained self-supervised encoder into the self-supervised network Net0, and add an under-complete autoencoder in the self-supervised network Net0 to form a cross-anatomical region organ increment first network Net1; (5) Train the cross-anatomical region organ increment first network Net1: 5a) Freeze all the parameters in the trained self-supervised encoder; 5b) Set the decoder loss function to the Dice loss and the under-complete autoencoder loss to the MSE loss; 5c) Use the labeled image first training set A1 and the Adam optimizer to iteratively train the cross-anatomical region organ increment first network Net1 until the Dice loss value and the MSE loss value no longer decrease, and obtain a trained cross-anatomical region organ increment first network Net1; (6) Load all the trained parameters of the cross-anatomical region organ increment first network into the cross-anatomical region organ increment first network Net1, and add a new decoder and a new under-complete autoencoder in Net1 to form a cross-anatomical region organ increment second network Net2; (7) Train the cross-anatomical region organ increment second network Net2: 7a) Freeze all the parameters in the trained cross-anatomical region organ increment first network; 7b) Set the loss function of the newly added decoder to the Dice loss and the loss of the newly added under-complete autoencoder to the MSE loss; 7c) Iteratively train the cross-anatomical region organ increment second network Net2 using the second training set A2 of the labeled images and the Adam optimizer until the value of the loss function no longer decreases, and obtain the trained cross-anatomical region organ increment second network Net2; (8) Load the parameters W2 of the trained cross-anatomical region organ increment second network into the cross-anatomical region organ increment second network Net2, and add a new decoder and a new undercomplete autoencoder in Net2 to form the cross-anatomical region organ increment third network Net3; (9) Train the cross-anatomical region organ increment third network Net3: 9a) Freeze all the parameters in the trained cross-anatomical region organ increment second network; 9b) Set the loss function of the newly added decoder to the Dice loss and the loss of the newly added undercomplete autoencoder to the MSE loss; 9c) Iteratively train the cross-anatomical region organ increment third network Net3 using the third training set A3 of the labeled images and the Adam optimizer until the value of the loss function no longer decreases, and obtain the trained cross-anatomical region organ increment third network Net3; (10) Make predictions on the test set: 10a) Load all the parameters of the trained cross-anatomical region organ increment third network Net3 into the cross-anatomical region organ increment third network Net3; 10b) Input the images in the three labeled test sets B1, B2, and B3 into the cross-anatomical region organ increment third network Net3 after loading the parameters in sequence, calculate the MSE loss between the outputs and the inputs of the three undercomplete autoencoders, and select the decoder corresponding to the undercomplete autoencoder with the minimum loss value as the segmentation result of the decoder output for the current image.

2. The method according to claim 1, wherein In the step (1b), the preprocessing of window adjustment, cropping, and normalization is performed on the N public datasets and the unlabeled datasets in sequence, and the following is achieved: (1b1) Crop the CT images in all datasets within the interval [-160HU, 240HU] to crop off the useless black edges; (1b2) Normalize the cropped CT images to obtain the normalized image Y: where X is the input image, X min is the maximum gray value of the pixels in the input image, and X max is the minimum gray value of the pixels in the input image.

3. The method according to claim 1, wherein: In the step (1b), the unlabeled public datasets after preprocessing are merged into a dataset D, and all the images in the dataset D are randomly cropped into blocks of size 96×96×96; In the step (1b), the random perturbation process of the pixel points in the cropped images is to first take out an image from the cropped dataset D and perform a non-linear mapping on the pixel points in the image with the help of a fifth-order Bezier curve; Then randomly select multiple windows in the non-linearly mapped image, shuffle the pixel points within the windows as the new window values; and then set the maximum covering rate to 30%, and randomly cover the image after shuffling the pixel values in a rectangular form.

4. The method according to claim 1, wherein: In step (1c), the images in the two pre - processed annotated public datasets and the un - public dataset are unified in size. The CT images are uniformly adjusted to a size of 96×96×96 using linear interpolation, and the annotated images are adjusted to a size of 96×96×96 using nearest - neighbor interpolation. In step (1c), the two pre - processed annotated public datasets and the un - public dataset after size adjustment are augmented. First, a CT image is randomly selected from the dataset for random rotation and flipping, then the image is scaled, and the scaled image is added to the dataset. Then, the taken - out CT image is repeatedly rotated, flipped, scaled, and added to the dataset three times to complete five - fold augmentation of the dataset.

5. The method according to claim 1, wherein In step (2), based on the UNet network structure, a self - supervised network Net0 including a self - supervised encoder and a decoder is constructed. The implementation steps are as follows: (2a) A self - supervised encoder is established, which is composed of a three - dimensional convolutional block, four convolutional encoding blocks, and 12 Transformer layers cascaded in sequence. Each convolutional encoding block is composed of a multi - scale feature extraction module, an SE module, and a pooling layer cascaded. In the multi - scale feature extraction module, convolutional kernels of 1×1×1, 3×3×3, and 5×5×5 are used to extract features respectively. (2b) A decoder is established, which is composed of four convolutional decoding blocks and four up - sampling layers cascaded in sequence. The convolutional decoding block has the same structure as the convolutional encoding block. At the same time, skip connections are used to splice low - level features and high - level features of the same layer to make features at all levels complementary. The output of the three - dimensional convolution at the end of the decoder is set to 1 channel, and the Sigmoid activation function is used to normalize the output to between 0 and 1. (2c) The self - supervised encoder and the decoder are serially connected to form the self - supervised network Net0.

6. The method according to claim 1, wherein In step (3), the self - supervised network Net0 is iteratively trained using the unannotated training set Dtr and the Adam optimizer. The implementation steps are as follows: (3a) Take a CT image from the unlabeled training set Dtr as the input of the self-supervised network Net0, obtain the output result of its decoder, and calculate the current mean absolute loss L ssl : Where y’ represents the original image, y represents the output image of the self - supervised network Net0, and N represents the total number of image pixels. (3b) Backpropagate the calculated L ssl to obtain the gradient value of the loss value, and use the Adam optimizer to update the parameters of the self-supervised encoder and decoder in the self-supervised network Net0; (3c) Repeat the process of (3a) - (3b) until all data in the unannotated training set Dtr have been learned, completing one iteration of learning. (3d) Repeat the process of (3c) for multiple iterations of learning until the value of the loss function no longer decreases, ending the training to obtain a trained self - supervised encoder.

7. The method according to claim 1, wherein In step (4), a cross - anatomical - region organ incremental first network Net1 is constructed by adding an under - complete auto - encoder. The implementation steps are as follows: (4a) An under - complete auto - encoder is established, which is composed of a fully - connected layer, a ReLU activation function, a new fully - connected layer, and a Sigmoid activation function cascaded. (4b) The newly added under - complete auto - encoder is serially connected with the self - supervised encoder in the self - supervised network Net0 to form the cross - anatomical - region organ incremental first network Net1.

8. The method according to claim 1, wherein , in step (5b), the decoder loss function is set to the Dice loss L DC , the undercomplete autoencoder loss is set to the MSE loss L AE , which are respectively expressed as follows: where G A is the true annotation corresponding to the input image, P A is the output result of the decoder, y is the output result of the undercomplete autoencoder, y’ is the input of the undercomplete autoencoder, and N represents the length of the output feature vector.

9. The method according to claim 1, wherein , In step (5c), the cross - anatomical - region organ incremental first network Net1 is iteratively trained using the first training set A1 of annotated images and the Adam optimizer. The implementation steps are as follows: (5c1) Take a CT image from the first training set A1 of labeled images and input it into the cross-anatomical region organ increment first network Net1 to obtain the output result P of its decoder. A And the output result y of the under-complete autoencoder, calculate the current Dice loss function value and MSE loss function value; (5c2) Backpropagate the calculated Dice loss function value to obtain the gradient value, and use the Adam optimizer to update the parameters of the decoder in the cross-anatomical region organ increment first network Net1; (5c3) Backpropagate the calculated MSE loss function value to obtain the gradient value, and use the Adam optimizer to update the parameters of the undercomplete autoencoder in the cross-anatomical region organ increment first network Net1; (5c4) Repeat the process of (5c1)-(5c3) until all data in the first training set A1 of the labeled image have been learned, completing one iteration of learning; (5c5) Repeat the process of (5c4) for multiple iterations of learning until the loss function value no longer decreases, obtaining the trained cross-anatomical region organ increment first network Net1.

10. The method according to claim 1, characterized in that In step (6), add a new decoder and a new undercomplete autoencoder to the cross-anatomical region organ increment first network Net1 to form the cross-anatomical region organ increment second network Net2. The implementation steps are as follows: (6a) Establish a new decoder with the same structure as the decoder in the self-supervised network Net0; (6b) Establish a new undercomplete autoencoder with the same structure as the undercomplete autoencoder in the cross-anatomical region organ increment first network Net1; (6c) Connect the new decoder and the undercomplete autoencoder in series with the encoder in the cross-anatomical region organ increment first network Net1 respectively to form the cross-anatomical region organ increment second network Net2.

Images