A whole heart segmentation method based on cross-modal domain transfer learning

Abstract

The present invention discloses a whole-heart segmentation method based on cross-modal domain transfer learning, comprising the following steps: S1, performing data preprocessing on initial whole-heart MRI images and CT images; S2, inputting the preprocessed images into a modal transfer network (MTN) for data training to generate new CT images; S3, inputting the new CT images generated by the MTN and the CT heart images with whole-heart labels into a U-shaped multi-attention network (MAUNet), outputting intermediate heart image segmentation results; S4, inputting the intermediate heart image segmentation results into a spatial configuration network (SCN) for data training to output the final heart image segmentation results. The present invention uses multimodal data to segment the heart. Multimodal data has multi-level information due to different imaging mechanisms. Using multimodal data to segment the same tissue can, on the one hand, expand the training data set, making it easier to train a robust network, and on the other hand, better utilize the multi-level information to improve segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a whole heart segmentation method based on cross-modal domain transfer learning. Background Art

[0002] Whole-heart segmentation aims to extract the volume and shape of all cardiac substructures from cardiac CT or MRI images. These substructures include the left ventricle (LV), left ventricular myocardium (Myo), right ventricle (RV), left atrium (LA), right atrium (RA), ascending aorta (AO), and pulmonary artery (PA). Existing whole-heart segmentation algorithms fall into two categories. One is a two-step approach, which first locates the heart, crops the region of interest (ROI) on the cardiac image, and then segments the ROI. This approach has the following drawbacks: 1) it requires training an additional model and a large number of parameters; 2) it segments a single-modality heart image and fails to utilize the multi-faceted information provided by multimodal images, which affects the segmentation effect. The other is the multimodal segmentation method. Multimodal images can provide multi-level information due to different imaging mechanisms. Using multimodal images to segment cardiac tissue can often achieve better segmentation results. However, existing multimodal segmentation methods often perform modal migration on two-dimensional cardiac slices and then splice them into three-dimensional images for segmentation. This method will cause information loss during the splicing process, resulting in modal migration errors and affecting the final segmentation effect.

[0003] Currently, the main difficulties in the field of whole-heart segmentation are: 1) The scarcity of medical samples makes it difficult to train a robust segmentation model; 2) Background pixels in heart images occupy most of the area, which will lead to class imbalance and affect model convergence; 3) Pixel values ​​at the boundaries of cardiac substructures are very close, resulting in blurred boundaries, affecting the model's classification accuracy for such difficult-to-separate pixels. Summary of the Invention

[0004] The purpose of the present invention is to provide a whole heart segmentation method based on cross-modal domain transfer learning, which mainly solves the problem that the blurred boundary pixels affect the segmentation effect in the existing cardiac image segmentation methods.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A whole heart segmentation method based on cross-modal domain transfer learning includes the following steps:

[0007] S1, data preprocessing of the initial whole-heart MRI and CT images;

[0008] S2, input the preprocessed whole heart MRI image and CT image into the modality transfer network MTN for data training to generate a new CT image;

[0009] S3, the new CT image generated by the modality transfer network MTN and the original CT heart image are input into the U-shaped multi-attention network MAUNet with attention gating mechanism and position attention mechanism, and the intermediate heart image segmentation result is output;

[0010] S4: Input the intermediate cardiac image segmentation result into the spatial configuration network (SCN) for data training, and output the final cardiac image segmentation result.

[0011] Furthermore, in step S1, the whole heart MRI image and CT image after data preprocessing are cropped to a size of 128×128×128.

[0012] Furthermore, in step S2, generating a new CT image includes the following steps:

[0013] S21, input the whole heart MRI image and CT image of size 128×128×128 into the generator network of the modality transfer network MTN;

[0014] S22, the generator network increases the number of channels of whole-heart MRI and CT images from 1 to 64 through a convolutional layer;

[0015] S23, the whole heart MRI image and CT image after the convolution layer are input into two downsampling layers connected by the convolution layer. Each layer doubles the number of channels of the whole heart MRI image and CT image and reduces the image size by half;

[0016] S24, the whole heart MRI image and CT image after two downsampling layers are input into nine consecutive residual layers, during which the image size is kept at 32×32×32 and the number of channels is 256;

[0017] S25, the whole heart MRI image and CT image with a size of 32×32×32 and 256 channels are restored to the input size through two consecutive upsampling layers;

[0018] S26, finally, the whole heart MRI image and CT image restored to the input size are input into the convolution layer to restore the number of channels to 1, and a new CT image is generated as output.

[0019] Furthermore, in the step S1, it also includes using the domain discriminator network of the modality transfer network MTN to judge the authenticity of the image; the specific steps are as follows:

[0020] S27, the generated new CT image is input into the domain discriminator network of the modality transfer network MTN. The new CT image goes through three consecutive downsampling layers, each layer reduces the image size by half. The first layer increases the number of channels from 1 to 64, and each subsequent layer doubles the number of channels.

[0021] S28, the image that has passed through three consecutive downsampling layers is passed through a convolutional layer to restore the number of channels to 1, and a feature image of size 16×16×16 is obtained for the discriminator to make true or false judgments.

[0022] Furthermore, in step S3, the step of outputting the intermediate heart image segmentation result includes:

[0023] S31, the preprocessed CT image and the generated new CT image are cropped to a size of 64×64×64 and input into the 4-layer UNet network of the U-shaped multi-attention network MAUNet. The first layer of the UNet network converts the number of channels of the 64×64×64 image from 1 to 64, and the number of channels in each subsequent layer is doubled. A convolution operation with a stride of 2 is performed between each layer, and the image size is reduced by half;

[0024] S32, then perform four upsampling operations on the image that has passed through the 4-layer UNet network, doubling the size of the upsampled image each time until it returns to the input image size;

[0025] S33, the image output by the four upsampling operations is passed through a convolutional layer to output an 8-channel image to obtain the intermediate heart image segmentation result of each substructure;

[0026] Furthermore, in step S4, the step of outputting the final cardiac image segmentation result includes:

[0027] S41, the intermediate heart image segmentation result is input into the spatial configuration network (SCN) consisting of 1 pooling layer, 4 convolutional layers and one upsampling layer;

[0028] S42, the intermediate heart image segmentation result is reduced by 4 times through the pooling layer;

[0029] S43, three convolutional layers are used to extract features from the segmentation result of the intermediate heart image that has been reduced by 4 times. The number of channels is kept at 64, and then an 8-channel image is output through the last convolutional layer.

[0030] S44, the image size of the convolution layer output is restored through the upsampling layer, and its output is multiplied by the intermediate heart image segmentation result to obtain the final segmentation result.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The present invention uses multimodal data to segment the heart. Multimodal data has multi-level information due to different imaging mechanisms. Using multimodal data to segment the same tissue can, on the one hand, expand the training data set, making it easier to train a robust network, and on the other hand, better utilize multi-level information to improve segmentation accuracy.

[0033] (2) The present invention integrates the attention gating mechanism and the position attention mechanism with the U-shaped network Unet to form a multi-attention network MAUnet, which solves the problems of class imbalance and fuzzy boundaries in the heart segmentation task.

[0034] (3) The present invention applies the spatial configuration network (SCN) to the multi-target segmentation task to learn the relative positions between different substructures and optimize the segmentation results. The effectiveness of the algorithm was evaluated on 40 test datasets provided by the MM-WHS challenge. The average Dice index was 0.911, the average Jaccard index was 0.837, and the average Hausdorff distance was 14.386. The proposed method achieved competitive performance considering all metrics, which is sufficient to prove the effectiveness of the proposed method. The segmentation accuracy of the right atrium (RA), left ventricular myocardium (Myo), and pulmonary artery (PA) reached the highest level so far, and the pulmonary artery segmentation accuracy was improved by 2.6% compared with the current best method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the network structure used in the whole heart segmentation method in an embodiment of the present invention.

[0036] Figure 2 The figure shows the segmentation effect of the method of the present invention in the embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

[0038] The present invention discloses a whole heart segmentation method based on cross-modal domain transfer learning, the network structure adopted by the method is as follows Figure 1As shown in the figure, it consists of three main components: 1) Modality Transfer Network (MTN); 2) U-shaped Multi-Attention Network (MAUNet); and 3) Spatial Configuration Network (SCN). The MTN is based on 3DCycleGAN and performs data augmentation through modality transfer. By learning the mapping relationship between the MRI and CT domains, it preserves the anatomical shape of the heart in MRI images while converting its intensity distribution to the style of CT images, thereby increasing training data for subsequent segmentation. The MAUNet receives as input a CT cardiac image with full-heart labels and a CT image generated by the MTN. It extracts a ROI (Region of Interest) from the original image and uses it to accurately segment the seven substructures of the heart. AGs (Attention Gating Mechanism) and PAB (Position Attention Mechanism) are designed in the skip connections and image decoder of the MAUNet to address class imbalance and blurred boundaries. Furthermore, a new hybrid loss combining the Dice loss and the Focal loss is applied to the segmentation stage to further address these issues. The output of the U-shaped multi-attention network MAUNet is input into the spatial configuration network SCN. The spatial configuration network SCN learns the relative positions between different substructures of the whole heart, fine-tunes the segmentation results of the previous step, and further improves the segmentation accuracy.

[0039] In the specific image segmentation processing, the preprocessed three-dimensional cardiac CT and MRI images are cropped to a size of 128×128×128 and input into the generator network in the modality transfer network MTN. The generator network is a codec structure. First, a convolutional layer is used to increase the number of image channels from 1 to 64, and then two downsampling layers are connected. Each layer doubles the number of channels and reduces the image size by half. The image is then input into 9 consecutive residual layers, during which the size is maintained at 32×32×32 and the number of channels is 256. Then, two upsampling layers are used to restore the image to the input size (128×128×128), followed by a convolutional layer to restore the number of channels to 1, and finally the output generates a CT image. In this process, the generated CT image will be immediately input into the domain discriminator network in the modality transfer network MTN. The domain discriminator network is an encoding structure. The image will pass through three consecutive downsampling layers. Each layer will reduce the image size by half. The first layer will increase the number of channels from 1 to 64. The number of channels will be doubled in each layer thereafter. Finally, a convolutional layer will restore the number of channels to 1. Finally, a feature image of size 16×16×16 is obtained for the discriminator to make true or false judgments.

[0040] The U-shaped Multi-Attention Network (MAUNet) is an improved four-layer UNet that integrates AGs (attention gating mechanism) and PAB (positional attention mechanism). Each layer of the encoding layer adopts a residual structure. The preprocessed and generated CT images are cropped to 64×64×64 size before being input to the network. The first layer converts the number of channels of the image from 1 to 64, and the number of channels is doubled in each subsequent layer. Convolution operations with a stride of 2 are performed between each layer, reducing the image size by half. The decoding layer is symmetrical to the encoding layer, performing four upsampling operations, each doubling the image size until the input image size is restored to 64×64×64. Finally, a convolutional layer outputs an 8-channel image to obtain the prediction result for each part. Each convolutional layer uses a kernel size of 3, followed by batch normalization and ReLU activation.

[0041] The output of the U-shaped multi-attention network MAUNet will be input into the spatial configuration network SCN, which consists of 1 pooling layer, 4 convolutional layers and 1 upsampling layer. The pooling layer reduces the input image by 4 times, and the three convolutional layers extract features. The number of channels is kept at 64 during this period. The last convolutional layer outputs an 8-channel image. The upsampling layer restores the image size, and its output is multiplied with the output of the U-shaped multi-attention network MAUNet to obtain the final segmentation result. Figure 2 The segmentation effect is displayed. The first column is the input image, the second column is the segmentation result on the two-dimensional slice, and the third column is the three-dimensional visualization of the segmentation result of the entire heart.

[0042] Through the above design, the present invention uses multimodal data to segment the heart. Multimodal data contains multi-layered information due to different imaging mechanisms. Using multimodal data to segment the same tissue not only expands the training dataset, making it easier to train a robust network, but also better utilizes multi-layered information to improve segmentation accuracy. Therefore, the present invention has outstanding substantive features and significant advancements.

[0043] The above embodiment is only one of the preferred implementation methods of the present invention and should not be used to limit the scope of protection of the present invention. Any changes or modifications that have no substantive meaning made to the main design concept and spirit of the present invention, as long as the technical problems solved are still consistent with the present invention, should be included in the scope of protection of the present invention.

Claims

1. A whole heart segmentation method based on cross-modal domain transfer learning, characterized in that: The following steps are involved: S1, data preprocessing of the initial whole-heart MRI and CT images; S2, input the preprocessed whole heart MRI image and CT image into the modality transfer network MTN for data training to generate a new CT image; The following steps are involved: S21, input the whole heart MRI image and CT image of size 128×128×128 into the generator network of the modality transfer network MTN; S22, the generator network increases the number of channels of whole-heart MRI and CT images from 1 to 64 through a convolutional layer; S23, the whole heart MRI image and CT image after the convolution layer are input into two downsampling layers connected by the convolution layer. Each layer doubles the number of channels of the whole heart MRI image and CT image and reduces the image size by half; S24, the whole heart MRI image and CT image after two downsampling layers are input into nine consecutive residual layers, during which the image size is kept at 32×32×32 and the number of channels is 256; S25, the whole heart MRI image and CT image with a size of 32×32×32 and 256 channels are restored to the input size through two consecutive upsampling layers; S26, finally, the whole heart MRI image and CT image restored to the input size are input into the convolution layer to restore the number of channels to 1, and the output is a new CT image; S3, the new CT image generated by the modality transfer network MTN and the original CT heart image are input into the U-shaped multi-attention network MAUNet with attention gating mechanism and position attention mechanism, and the intermediate heart image segmentation result is output; S4: Input the intermediate cardiac image segmentation result into the spatial configuration network (SCN) for data training, and output the final cardiac image segmentation result.

2. A whole heart segmentation method based on cross-modal domain transfer learning according to claim 1, characterized in that: In step S1 , the whole heart MRI image and CT image after data preprocessing are cropped to a size of 128×128×128.

3. The whole heart segmentation method based on cross-modal domain transfer learning according to claim 2, characterized in that: In the step S1, the domain discriminator network of the modality transfer network MTN is also used to judge the authenticity of the image; the specific steps are as follows: S27, the generated new CT image is input into the domain discriminator network of the modality transfer network MTN. The new CT image goes through three consecutive downsampling layers, each layer reduces the image size by half. The first layer increases the number of channels from 1 to 64, and each subsequent layer doubles the number of channels. S28, the image that has passed through three consecutive downsampling layers is passed through a convolutional layer to restore the number of channels to 1, and a feature image of size 16×16×16 is obtained for the discriminator to make true or false judgments.

4. The whole heart segmentation method based on cross-modal domain transfer learning according to claim 3, characterized in that: In step S3, the step of outputting the intermediate heart image segmentation result includes: S31, the preprocessed CT image and the generated new CT image are cropped to a size of 64×64×64 and input into the 4-layer UNet network of the U-shaped multi-attention network MAUNet. The first layer of the UNet network converts the number of channels of the 64×64×64 image from 1 to 64, and the number of channels in each subsequent layer is doubled. A convolution operation with a stride of 2 is performed between each layer, and the image size is reduced by half; S32, then perform four upsampling operations on the image that has passed through the 4-layer UNet network, doubling the size of the upsampled image each time until it returns to the input image size; S33, the image output by the 4 upsampling operations is passed through a convolutional layer to output an 8-channel image to obtain the intermediate heart image segmentation result of each substructure.

5. The whole heart segmentation method based on cross-modal domain transfer learning according to claim 4, characterized in that: In step S4, the step of outputting the final cardiac image segmentation result includes: S41, the intermediate heart image segmentation result is input into the spatial configuration network (SCN) consisting of 1 pooling layer, 4 convolutional layers and one upsampling layer; S42, the intermediate heart image segmentation result is reduced by 4 times through the pooling layer; S43, three convolutional layers are used to extract features from the segmentation result of the intermediate heart image that has been reduced by 4 times. The number of channels is kept at 64, and then an 8-channel image is output through the last convolutional layer. S44, the image size of the convolution layer output is restored through the upsampling layer, and its output is multiplied by the intermediate heart image segmentation result to obtain the final segmentation result.

Images