A cardiac CT image segmentation method combining multi-view geometry and transfer learning

Abstract

The application provides a heart CT image segmentation method combining multi-view geometry and transfer learning, which is composed of data preprocessing, transfer learning and post-processing parts. The data preprocessing includes data set splitting, irrelevant area removal, position embedding, boundary enhancement and dimension transformation, which is used to obtain a data set for training and testing the network; the transfer learning includes pre-training, logical reorganization and weight fine-tuning, which is used to obtain the final weight for heart segmentation; the post-processing includes inference prediction and point-by-point voting, which is used to complete the heart segmentation of the three-dimensional CT image using the final weight and further optimize the final heart model. The application realizes accurate segmentation of the heart model by using the method of multi-view geometry and transfer learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical images, in particular to a cardiac CT image segmentation method combining multi-view geometry and transfer learning. BACKGROUND

[0002] The heart is an important component of the circulatory system. According to the China Cardiovascular Health and Disease Report 2020, the prevalence of cardiovascular disease in China is on the rise. According to the proportion of resident deaths, 2 out of 5 deaths are due to cardiovascular disease. Whole heart segmentation can be an important means of diagnosis and treatment, and there is an urgent need to seek an efficient and accurate segmentation method to meet the needs of doctors in reading films.

[0003] CT images are a common medical means in the diagnosis and treatment process and can be used for the examination of various diseases, and are of great significance for the examination of cardiovascular diseases. By segmenting the whole heart using cardiac CT, computers can be used to achieve different methods to reduce errors caused by manual work. Traditional medical image segmentation methods include image-based or variable model-based methods, but they all use semi-automatic segmentation methods that are time-consuming and labor-intensive, and have low segmentation accuracy, which cannot meet the clinical needs. With the continuous development of deep learning, many people have researched the feasibility of deep learning methods and have made some progress, but the existing deep learning methods have problems such as difficulty in obtaining databases, blurred boundaries of heart CT image backgrounds and target regions, and differences between different individuals, resulting in long computation time and unsatisfactory output results, which need to be solved. SUMMARY

[0004] To solve the problems of large errors caused by traditional manual segmentation methods, high labor costs, low accuracy of semi-automatic segmentation methods that are difficult to meet clinical needs, and long computation time and poor reliability of existing automatic segmentation algorithms, the present application provides a cardiac CT image segmentation method combining multi-view geometry and transfer learning, which improves segmentation accuracy and precision while reducing the time spent on computation.

[0005] To achieve the above purpose, the technical solutions adopted by the present application are as follows:

[0006] A heart CT image segmentation method combining multi-view geometry and transfer learning, consisting of data preprocessing, transfer learning, and prediction and post-processing; the data preprocessing includes dataset splitting, irrelevant region removal, position embedding, boundary enhancement, and dimension transformation, for obtaining the dataset for training and testing the network; the transfer learning includes pre-training, logical reorganization, and weight fine-tuning, for obtaining the final weight for heart segmentation; the prediction and post-processing include inference prediction and point-by-point voting, for using the final weight to complete the heart segmentation of three-dimensional CT images and further optimizing to obtain the final heart model, specifically comprising the following steps:

[0007] Step 1, split all three-dimensional CT image original data into training set and validation set, the data number of training set is defined as n, and the data number of validation set is defined as m;

[0008] Step 2, use the ReLU function to set the irrelevant region with pixel value less than 0 in the training set and the validation set to 0 to obtain image CT1;

[0009] Step 3, position embedding is performed on image CT1 to obtain image CT2;

[0010] Step 4, boundary enhancement is performed on image CT1 to obtain image CT3;

[0011] Step 5, add a channel dimension to image CT1, image CT2 and image CT3, and splice in the channel dimension to obtain four-dimensional image CT4;

[0012] Step 6, slice image CT4 along X, Y and Z axes to obtain (x+y+z) three-dimensional image CT5, wherein x is the dimension value of the x-axis in the original CT image data, y is the dimension value of the y-axis in the original CT image data, and z is the dimension value of the z-axis in the original CT image data;

[0013] Step 7, perform steps 2 to 6 on each three-dimensional CT image original data in the training set and the validation set to obtain n×(x+y+z) image CT5 and m×(x+y+z) image CT5;

[0014] Step 8, pre-train the backbone network ResNet using the ImageNet dataset to obtain the pre-training weight;

[0015] Step 9, adjust the network architecture of ResNet to realize logical reorganization of the feature maps output by each level of residual block in ResNet to obtain the backbone network ResNet-DD + ;

[0016] Step 10, construct DeepLabV3 +Network framework, using the ResNet-DD backbone network. + As DeepLabV3 + Feature extraction backbone to obtain DeepLabV3 + -DD + ;

[0017] Step 11: Use n×(x+y+z) CT5 images to apply pre-trained weights to DeepLabV3. + -DD + Fine-tuning was performed, and m×(x+y+z) CT5 images were used for testing. The final weights were obtained after convergence.

[0018] Step 12: Perform steps 2 to 6 on the original 3D CT image data for each test to obtain (x+y+z) images CT5, and use DeepLabV3 with the final weights loaded. + -DD + The x, y, and z CT5 images are segmented and stacked along the corresponding X, Y, and Z axes to obtain the CT images. X CT imaging Y CT imaging Z ;

[0019] Step 13, CT imaging X CT imaging Y CT imaging Z The final heart model was obtained by a point-by-point voting method.

[0020] Furthermore, in step 3, the position embedding adopts the cosine position embedding algorithm.

[0021] Furthermore, in step 4, the boundary enhancement employs a three-dimensional erosion and three-dimensional dilation algorithm.

[0022] Furthermore, in step 9, the logical restructuring employs Net-DD. + Architecture.

[0023] The advantages of this invention compared to the prior art are:

[0024] Compared with the traditional manual segmentation method, the application has a fully automatic process, which minimizes the errors caused by human participation and saves labor costs; compared with the semi-automatic segmentation direction, the application has the following advantages: the self-designed preprocessing process eliminates a large number of irrelevant areas and enhances the fuzzy boundary, which reduces the difficulty of subsequent segmentation and achieves higher segmentation accuracy; compared with the existing automatic segmentation-based algorithm, the application includes a logical reorganization of transfer learning, which captures the potential logical relationship between features, improves the robustness of the network and the convergence speed; the point-by-point voting post-processing improves the utilization rate of information and further improves the accuracy of the segmentation result. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a total flow chart of a cardiac CT image segmentation method combining multi-view geometry and transfer learning of the application.

[0026] Figure 2 is a preprocessing process of the application.

[0027] Figure 3 is a transfer learning process of the application, including pre-training, logical reorganization and weight fine-tuning, the logical reorganization adopts Net-DD + architecture.

[0028] Figure 4 is a prediction and post-processing process of the application, including inference prediction and point-by-point voting.

[0029] Figure 5 is an effect diagram of the final cardiac segmentation of the application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the application clearer, the exemplary embodiments of the application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0031] As Figure 1 shown, the cardiac CT image segmentation method combining multi-view geometry and transfer learning of the application includes data preprocessing, transfer learning, prediction and post-processing.

[0032] The data preprocessing includes data set splitting, irrelevant area elimination, position embedding, boundary enhancement and dimension change, and the specific process is shown in Figure 2 , which is specifically divided into the following steps:

[0033] Step 1, first divide all the original data of CT images into a training set and a validation set, define the number of data in the training set as n, and the number of data in the validation set as m;

[0034] Step 2, using the ReLU function can set the irrelevant part of the pixel less than 0 in the training set and the validation set to 0 to obtain the data of image CT1 with a dimension of 256x256x224;

[0035] Step 3, the data of image CT1 with a dimension of 256x256x224 obtained in step 2 is subjected to cosine position embedding to obtain the data of image CT2 with a dimension of 256x256x224, the purpose is to better limit the pixel position, so that the training is better, and it is also helpful to capture the different positions of the pixels on the same latitude, and the different dimensions provide different information of relative position;

[0036] Step 4, using three-dimensional erosion and three-dimensional expansion algorithm to the data of image CT1 with a dimension of 256x256x224 obtained in step 2, to obtain the data of image CT3, the principle is: the irrelevant pixel points at the boundary position are corroded, that is, the irrelevant pixel points are set to background value 0, and the pixel points related to the heart in the boundary position are expanded, that is, the pixel points related to the heart in the boundary position are set to foreground value 1;

[0037] Step 5, the images CT1, CT2 and CT3 obtained in steps 2, 3 and 4 are increased in a channel dimension, spliced in the channel dimension, and linearly interpolated in the Z-axis dimension to obtain a four-dimensional 3x256x256x256 image CT4, that is, a four-dimensional tensor data with a channel number of 3 and a length, width and height of 256;

[0038] Step 6, slicing the image CT4 obtained in step 5 in the X-axis direction can obtain 256 three-dimensional tensors with a dimension of 3x256x256, and the same slicing operation is performed on the Y and Z axes, and finally 3x256x224 three-dimensional tensors with x+y+z=256+256+224 are obtained;

[0039] Step 7, the above steps 2 to 6 are performed on each three-dimensional CT original data in the training set and the validation set to obtain n x (x+y+z) images CT5 and m x (x+y+z) images CT5, wherein x is the dimension value of the x-axis in the CT image original data, y is the dimension value of the y-axis in the CT image original data, and z is the dimension value of the z-axis in the CT image original data; in the present application, x=256, y=256, and z=224;

[0040] Since the CT data is three-dimensional, directly using three-dimensional CT data calculation, 3D convolution kernel parameter quantity is large, cost is high, and there is no large CT data set, so the three-dimensional CT image is subjected to steps 1-7 operations to obtain a data set that can be used for training and testing a 2D network, and then entering the process of transfer learning. The process of transfer learning is as shown in Figure 3 The specific steps are as follows:

[0041] Step 8, the pre-trained weight is obtained by pre-training the backbone network ResNet using the ImageNet data set;

[0042] The pre-training process uses a Nesterov momentum optimizer, the momentum is 0.9, the initial learning rate is 0.05, the rate decay is 0.94 every 2 cycles, and the weight decay is 4x10 -5 , using 50 GPUs for asynchronous training, the batch size of each GPU is 32, and the image size is 224x224.

[0043] Step 9, using Net-DD + The network architecture of ResNet is adjusted to realize logical reorganization of the feature maps output by each level of the residual block in ResNet to obtain a backbone network ResNet-DD + , so as to improve the accuracy of the model;

[0044] Step 10, build DeepLabV3 + network framework, and embed the backbone network ResNet-DD + reorganized in step 9, and use the backbone network ResNet-DD + as the feature extraction backbone of DeepLabV3 + to obtain DeepLabV3 + -DD + ;

[0045] Step 11: using n x (x+y+z) CT5 images in the training set obtained in step 7 to fine-tune the DeepLabV3 + -DD + loaded with pre-trained weights, and using m x (x+y+z) images in the verification set obtained in step 7 for testing, and obtaining the final weight after convergence;

[0046] The fine-tuning process uses a stochastic gradient descent optimizer, the initial learning rate is 0.05, the learning rate is adjusted using a cosine annealing algorithm, the weight decay is 1x10 -4 , using a single GPU for training, the batch size is 16, and the image size is 256x256.

[0047] In the process of the migration learning, the best weight that can be used for heart segmentation is obtained, and the process of prediction and post-processing is entered. The process of prediction and post-processing is as shown in Figure 4 The specific steps are as follows:

[0048] Step 12: Steps 2 to 6 are performed on each tested three-dimensional CT original data to obtain (x+y+z) image CT5, and the DeepLabV3 obtained in step 11 is used as the best weight + -DD + The x image CT5, y image CT5 and z image CT5 are segmented respectively and stacked in the corresponding X, Y and Z axes to obtain image CT X , image CT Y and image CT Z .

[0049] Step 13: The images of image CT X , image CT Y and image CT Z obtained in step 12 are subjected to point-by-point voting, and the pixel points representing the heart images in the three directions are voted. If more than half of the model votes are obtained, that is, more than 2 model votes are obtained in the present application, it is considered that the pixel point can represent the heart, and the final model is obtained.

[0050] Compared with the prior art, the present application has higher segmentation accuracy and speed, and the comparison of various indicators is shown in Table 1, and the segmentation effect is shown in Figure 5 .

[0051] Table 1 Comparison of segmentation accuracy of the present application and other models

[0052] Models LV Myo RV LA RA AA PA Average U_Net 0.796 0.748 0.803 0.857 0.811 0.843 0.752 0.801 AttU_Net_conv1_5Mffp [1] ]]> 0.906 0.905 0.841 0.895 0.887 0.910 0.864 0.887 Seg-CNN [2] ]] 0.924 0.872 0.879 0.910 0.865 0.940 0.837 0.889 3D FCN [3] ]]> 0.877 0.818 0.778 0.845 0.815 0.941 0.826 0.843 CFUN [4] ]]> 0.879 0.822 0.902 0.832 0.844 0.913 0.821 0.859 CliqueNet [5] ]] 0.948 0.892 0.895 0.918 0.875 0.966 0.877 0.910 Ours’ 0.964 0.921 0.949 0.968 0.959 0.978 0.941 0.954

[0053] The contents not described in detail in the specification of the present application belong to the prior art known to those skilled in the art.

[0054] The above embodiments are provided only for the purpose of describing the present application, and are not intended to limit the scope of the present application. The scope of the present application is defined by the appended claims. Various equivalent replacements and modifications made without departing from the spirit and principles of the present application should be included in the scope of the present application.

Claims

1. A method for segmenting cardiac CT images combining multi-view geometry and transfer learning, characterized in that: The system consists of data preprocessing, transfer learning, prediction, and post-processing. Data preprocessing includes dataset splitting, irrelevant region removal, location embedding, boundary enhancement, and dimensionality transformation to obtain datasets for training and testing the network. Transfer learning includes pre-training, logistic reorganization, and weight fine-tuning to obtain the final weights for heart segmentation. Prediction and post-processing include inference prediction and point-by-point voting to use the final weights to segment the heart from 3D CT images and further optimize it to obtain the final heart model. Specifically, it includes the following steps: Step 1: Split all the original 3D CT image data into a training set and a validation set. The number of data in the training set is defined as n, and the number of data in the validation set is defined as m. Step 2: Use the ReLU function to set irrelevant regions with pixel values ​​less than 0 in the training and validation sets to 0 to obtain image CT1; Step 3: Perform position embedding on image CT1 to obtain image CT2; Step 4: Perform boundary enhancement on image CT1 to obtain image CT3; Step 5: Add a channel dimension to each of images CT1, CT2, and CT3, and stitch them together to obtain a four-dimensional image CT4. Step 6: Slice the image CT4 along the X, Y, and Z axes to obtain (x+y+z) three-dimensional images CT5, where x is the dimension value of the x-axis in the original CT image data, y is the dimension value of the y-axis in the original CT image data, and z is the dimension value of the z-axis in the original CT image data. Step 7: Perform steps 2 to 6 above on each original 3D CT image data in the training set and validation set to obtain n×(x+y+z) CT5 images and m×(x+y+z) CT5 images; Step 8: Use the ImageNet dataset to pre-train the backbone network ResNet to obtain pre-trained weights; Step 9: Adjust the ResNet network architecture to perform logical reorganization on the feature maps output by each residual block in ResNet, resulting in the backbone network ResNet-DD. + ; Step 10: Build DeepLabV3 + Network framework, using the ResNet-DD backbone network. + As DeepLabV3 + The feature extraction backbone network is used to obtain DeepLabV3. + -DD + ; Step 11: Use n×(x+y+z) CT5 images to apply pre-trained weights to DeepLabV3. + -DD + Fine-tuning was performed, and m×(x+y+z) CT5 images were used for testing. The final weights were obtained after convergence. Step 12: Perform steps 2 to 6 on the original 3D CT image data for each test to obtain (x+y+z) images CT5, and use DeepLabV3 with the final weights loaded. + -DD + The x, y, and z CT5 images are segmented and stacked along the corresponding X, Y, and Z axes to obtain the CT images. X CT imaging Y CT imaging Z ; Step 13, CT imaging X CT imaging Y CT imaging Z The final heart model was obtained by a point-by-point voting method.

2. The cardiac CT image segmentation method combining multi-view geometry and transfer learning according to claim 1, characterized in that: In step 3, the position embedding adopts the cosine position embedding algorithm.

3. The cardiac CT image segmentation method combining multi-view geometry and transfer learning according to claim 1, characterized in that: In step 4, the boundary enhancement employs a three-dimensional erosion and three-dimensional dilation algorithm.

4. The cardiac CT image segmentation method combining multi-view geometry and transfer learning according to claim 1, characterized in that: In step 9, the logical restructuring employs Net-DD. + Architecture.

Images