A two-stage multi-label segmentation method for AIS spine CT images based on deep learning

Abstract

The present invention discloses a two-stage AIS spine CT image multi-label segmentation method based on deep learning. The method first obtains an AIS spine image dataset, preprocesses the images in the dataset, and creates new labels for vertebrae positioning. Secondly, the preprocessed images and new labels are processed to train a convolutional neural network for vertebrae positioning. The preprocessed images are then tested using the trained convolutional neural network, and the test results are post-processed to obtain the predicted results of the vertebral center of mass in the spine image. The preprocessed original images and labels are then reprocessed to train the constructed segmentation network. Finally, a region of interest is created based on the vertebral center of mass position predicted during the vertebrae positioning process, and the trained segmentation network is used to segment the target vertebrae in the region of interest, and then the multi-label segmentation results are obtained by integration. The present invention can effectively segment vertebrae.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation, and specifically relates to a two-stage AIS spine CT image multi-label segmentation method based on deep learning. Background Art

[0002] The spine is generally composed of 24 vertebrae: 7 cervical, 12 thoracic, and 5 lumbar. It is an important component of the human musculoskeletal system. Spinal diseases can affect patients' normal lives and, in severe cases, can even be life-threatening. Excessive fatigue and prolonged sitting can easily lead to various spinal diseases. Changes in modern working practices have led to a continuous increase in the number of patients with spinal diseases in my country, with the incidence trending towards younger age groups. Scoliosis is a common spinal disease that often occurs in adolescents. In surgeries for spinal diseases, a good multi-label vertebral segmentation tool can provide effective technical support for preoperative planning, intraoperative navigation, and postoperative evaluation. Traditional spinal image analysis is usually performed by experienced physicians, but this method is not only time-consuming but also requires extremely high professional skills from the segmentation personnel. Therefore, the research on automated multi-label vertebral segmentation methods is of great practical significance.

[0003] The current mainstream spinal imaging methods include X-rays, magnetic resonance imaging (MRI), and computed tomography (CT). Among them, CT can most clearly display the vertebral structure of the spine and has therefore become the most commonly used imaging method in the field of spinal medical imaging processing.

[0004] Although some deep learning models have been developed in other medical image segmentation fields that can achieve good segmentation, there is still a lack of research on segmentation methods for scoliotic spine CT images. The large size of spinal CT images, the variety of vertebrae, and the highly similar structures of adjacent vertebrae make automatic multi-label segmentation of spinal CT images very challenging. In addition, compared with the normal spine, the spine of adolescent idiopathic scoliosis (AIS) is spatially rotated and deformed, which increases the difficulty of segmentation. Therefore, an effective multi-label segmentation scheme for scoliotic spine is needed. Summary of the Invention

[0005] This paper proposes a two-stage deep learning-based multi-label segmentation method for AIS spinal CT images to address the following problems in current spinal image segmentation methods: the large size of spinal CT images cannot be directly used as input for training deep learning networks; there are too many types of vertebrae, and directly segmenting images into more than 20 categories is prone to missegmentation and missed segmentation; the curvature of the AIS spine and the severe deformity of the vertebrae increase the difficulty of segmentation.

[0006] To address the above problems, the present invention provides a two-stage multi-label segmentation method for AIS spinal CT images based on deep learning. The overall segmentation scheme includes two stages: vertebra localization and vertebra segmentation. The steps are as follows:

[0007] Step 1: Obtain the AIS spine image dataset and preprocess the images in the dataset.

[0008] The specific preprocessing is as follows: resample all CT images and labels to make their voxel spacing uniform to 1mm, and rotate all images and labels to the same direction.

[0009] Step 2: Create new labels for vertebrae localization based on the preprocessed labels. These new labels are based on the centroid of the vertebrae. In these new labels, non-vertebrae regions have a label value of 0, while vertebrae regions have a label value between 0 and 1. Points closer to the centroid of the vertebrae have a label value closer to 1.

[0010] Step 3: Process the preprocessed images and new labels. The processing process includes cropping, random rotation, and random translation. The processed images and labels are vertebrae cropped images with a size of 128×128×128.

[0011] Step 4: Use the processed images and new labels to train a convolutional neural network for vertebrae localization, and use Attention-U-Net as the network structure.

[0012] Step 5: Use the trained model to process the original image and perform some post-processing on the result to obtain the predicted result of the vertebral center of mass in the spine image, thus realizing the positioning of the vertebrae.

[0013] The post-processing process involves binarizing the convolutional neural network output image, retaining only the region with a median value closest to 1, and converting this region into a point set. K-Means clustering is then used to cluster the point set into clusters equal to the number of vertebrae, with the cluster center of each cluster serving as the predicted vertebral centroid.

[0014] Step 6: Reprocess the original images and labels after preprocessing. The images used to train the network in the vertebrae segmentation stage are obtained by cropping the original images and labels according to the centroid position of the vertebrae.

[0015] Vertebral segmentation involves calculating the centroid of each vertebra based on the preprocessed labels. The preprocessed image and labels are then cropped to create a fixed-size region centered around each centroid. Furthermore, the cropped region is subjected to random rotations and translations as data augmentation.

[0016] Step 7 improves upon the Attention-U-Net architecture by constructing a segmentation network and training it using reprocessed images and labels. The segmentation network replaces the original convolutional layers in the Attention-U-Net encoding layer with 7×7×7 depthwise convolutions to improve the network's ability to capture shape information. Furthermore, residual connections are added to the encoding layer to address potential transition smoothing issues. In the decoding layer, deep convolutional layers are used to process information from each channel separately, followed by a convolutional layer that integrates information from all channels. Residual connections are also used to improve network performance.

[0017] In step eight, a region of interest is created based on the vertebral centroid position predicted during the vertebral positioning process, and the trained segmentation network is used to segment the target vertebra in the region of interest.

[0018] Step nine: Integrate the segmentation results of all regions of interest to obtain the final multi-label segmentation result.

[0019] The integration process is as follows: the segmentation results of each vertebra are added to the same blank image, and the segmentation results of different vertebrae are distinguished by different label values. The vertebrae segmentation results are sorted according to the coordinates of the centroid on the Z axis to obtain the final multi-label segmentation result.

[0020] Beneficial effects of the present invention:

[0021] The proposed method can effectively achieve multi-label segmentation of AIS spinal CT images. Specifically, the proposed vertebral localization method can effectively predict the centroid position of each vertebra in spinal CT images. During the vertebral segmentation phase, the proposed segmentation network can effectively segment vertebrae and achieve higher Dice scores and lower Hausdorff distances compared to conventional segmentation networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall flow chart of the segmentation scheme of the present invention;

[0023] Figure 2 is a flow chart of the vertebra positioning stage of the present invention;

[0024] Figure 3 The segmentation network structure used in the vertebra segmentation stage of the present invention;

[0025] Figure 4 This is a diagram showing the coding layer module structure of the vertebrae segmentation network of the present invention;

[0026] Figure 5 This is a structural diagram of the decoding layer module of the vertebrae segmentation network of the present invention;

[0027] Figure 6This is a comparison chart of the segmentation results of the present invention and the true labels. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with a specific implementation process, and the specific implementation steps are as follows:

[0029] A two-stage deep learning-based multi-label segmentation method for AIS spinal CT images. The overall segmentation scheme includes two stages: vertebra localization and vertebra segmentation. The steps are as follows:

[0030] Step 1: Obtain the AIS spine image dataset and preprocess the images in the dataset: resample all CT images and labels to unify their voxel spacing to 1 mm, and rotate all images and labels to the same orientation.

[0031] Step 2: Create new labels for vertebrae localization based on the preprocessed labels. These new labels are based on the centroid of the vertebrae. In these new labels, non-vertebrae regions have a label value of 0, while vertebrae regions have a label value between 0 and 1. Points closer to the centroid of the vertebrae have a label value closer to 1.

[0032] Step 3: Process the preprocessed images and new labels. The processing process includes cropping, random rotation, and random translation. The processed images and labels are vertebrae cropped images with a size of 128×128×128.

[0033] Step 4: Use the processed images and new labels to train a convolutional neural network for vertebrae localization, and use Attent ion-U-Net as the network structure.

[0034] Step 5: Use the trained model to process the original image, and then perform some post-processing on the result: Binarize the convolutional neural network output image, retaining only the region with a median value closest to 1 and converting this region into a point set. Use K-Means clustering to cluster the point set into clusters equal to the number of vertebrae, and use the cluster center of each cluster as the predicted vertebral centroid.

[0035] Step 6: Reprocess the preprocessed original images and labels: Calculate the centroid of each vertebra based on the preprocessed labels. Crop the preprocessed images and labels to create a fixed-size region centered on each vertebra's centroid. Additionally, perform random rotations and translations on the cropped regions as data augmentation.

[0036] Step 7 improves upon the Attention-U-Net architecture by constructing a segmentation network and training it using reprocessed images and labels. The segmentation network replaces the original convolutional layers in the Attention-U-Net encoding layer with 7×7×7 depthwise convolutions to improve the network's ability to capture shape information. Furthermore, residual connections are added to the encoding layer to address potential transition smoothing issues. In the decoding layer, deep convolutional layers are used to process information from each channel separately, followed by a convolutional layer that integrates information from all channels. Residual connections are also used to improve network performance.

[0037] In step eight, a region of interest is created based on the vertebral centroid position predicted during the vertebral positioning process, and the trained segmentation network is used to segment the target vertebra in the region of interest.

[0038] Step 9: Combine all ROI segmentation results: Add the segmentation results of each vertebra to the same blank image, using different label values ​​to distinguish the segmentation results of different vertebrae. Sort the vertebrae segmentation results by their centroid coordinates on the Z axis to obtain the final multi-label segmentation result.

[0039] Example:

[0040] The overall segmentation process of the present invention is as follows Figure 1 The dataset used is the AIS spine CT image dataset, which contains 34 complete AIS spine CT images and their corresponding labels. All images and labels were divided into training, validation, and test sets. At the beginning of the experiment, all CT images were resampled to 1mm and uniformly rotated to the PIR orientation to ensure consistency.

[0041] Vertebral positioning

[0042] In the labels provided by the AIS spine CT dataset, all voxels on each vertebra are labeled with the same value. To make the labels more intuitive in reflecting the centroid information of the vertebrae, we created new labels for vertebrae localization based on the original labels. The process of creating new labels is as follows: We calculate the average value d of the maximum distance from the centroid of the vertebra to the surface in each image. max The value x of each voxel on the vertebra in the new label is determined by the distance d between the voxel and the center of mass of the vertebra where it is located. x The specific calculation formula is as follows:

[0043]

[0044] In this way, we obtain new labels for vertebrae positioning. We use the property that the label value of the voxel in the new label is closely related to its distance to the center of mass to locate the vertebrae.

[0045] like Figure 2 As shown in Figure 1, the vertebra localization process can be divided into three stages. In the first stage, the cropped images and new labels are fed into the Attention-U-Net model for training. The images from the test set are then fed into the trained neural network to produce a transition image for vertebra localization (b). In the second stage, the transition image is binarized, retaining only points with high grayscale values ​​(indicating proximity to the centroid of the vertebra) (c). To determine the most appropriate threshold for binarization, we tested it on the validation set and ultimately selected the best-performing threshold (0.8) as the final threshold. In the third stage, K-Means clustering is used to cluster all points into clusters equal to the number of vertebrae. The cluster center of each cluster serves as the predicted centroid of the vertebra (d). In particular, since the spatial distribution differences between vertebrae are primarily reflected in the Z direction, the distance calculation formula in the clustering process adds weight to the Z distance in addition to the traditional Euclidean distance to better distinguish individual vertebrae.

[0046] Vertebrae segmentation:

[0047] During vertebra segmentation, a 128×128×128 region of interest (ROI) was created based on the centroids predicted during vertebra localization. The target vertebra was then segmented within this region. To train the network to meet the requirements, the original CT images and labels were cropped based on the actual centroids of the vertebrae. These cropped images and labels were used for neural network training. Furthermore, to simulate the distortion of the AIS spine and the inevitable errors in actual localization, the images and labels were subjected to random rotations and translations.

[0048] This paper improves on the Attention-U-Net. In the encoding layer, deep convolutions with large 7×7×7 kernels replace the original convolutional layers. Large convolutions help capture shape information, which is crucial for vertebrae segmentation. To mitigate transition smoothing issues associated with large convolutions, residual connections are added to the encoding layer. In the decoding layer, deep convolutional layers process information from each channel separately, followed by a convolutional layer that integrates information from all channels. Residual connections are also used to improve network performance. Figure 3 and Figure 4 The encoding and decoding layer structures of the network used are shown.

[0049] After the segmentation network training is completed, multi-label segmentation of the vertebrae is performed. Specifically: first, the proposed vertebrae positioning method is used to obtain the predicted center of mass of each vertebra in the image. Then, multiple regions of interest are created with the predicted center of mass as the center, and each region of interest is segmented using the trained vertebrae segmentation network. The segmentation results are binarized with a threshold of 0.5 to obtain vertebrae segmentation results with the same number of predicted vertebrae. All vertebrae segmentation results are added to a blank image with the same size as the original image, and different vertebrae are distinguished using different label values. If the repeated area of ​​two segmentation results exceeds half of one of the segmentation results, the vertebrae are considered to be repeated, and one of the results is removed. Finally, after detecting that there are no unsegmented vertebrae, the vertebrae number is determined according to the Z-direction coordinate of each segmentation result to complete the multi-label segmentation of the vertebrae.

[0050] The method for detecting whether there is an unsegmented vertebra is as follows: first, Figure 2 The image in .b is binarized using a low threshold (0.4 in our experiment) to obtain the spine segmentation result. All vertebrae segmentation results are superimposed and subtracted from the spine segmentation result. If a connected region with a volume exceeding one-third of the average vertebral volume exists, it is considered that there are unsegmented vertebrae. The centroid of all connected regions that meet the conditions is calculated, and the vertebrae segmentation and unsegmented vertebrae detection process is repeated based on the centroid until no unsegmented vertebrae are found.

[0051] The Loss function is as follows:

[0052] During training, the loss function used is a combination of Dice loss and cross entropy loss. It can be calculated in a voxel-wise manner:

[0053]

[0054] Where I is the number of voxels, J is the number of classes, and Y i,j and G i,j They represent the probability output of class j at voxel i and the corresponding true label in one-hot encoding form.

[0055] Evaluation indicators:

[0056] In order to evaluate the effectiveness of the method of the present invention, the following indicators were used for evaluation: For vertebral positioning, the average positioning distance (d mean ) for evaluation. mean It is defined as the average of the Euclidean distances between the true vertebral centroid and the predicted vertebral centroid. For vertebral segmentation, the Dice coefficient (Dice score) and Hausdorff distance (HD) are used for evaluation, and are calculated as follows:

[0057]

[0058]

[0059] Among them, M gt and M p Denote the actual cone and the predicted cone respectively, and the final Dice coefficient is the average value of the Dice coefficients of all vertebrae. i and T i represents the surface extracted from the voxel mask of the i-th vertebra, and d(p,t) represents the Euclidean distance between points p and t on the two surfaces.

[0060] Experimental results:

[0061] On the AIS spine CT image dataset, the test set images were segmented. First, the vertebrae were located. Figure 5 The visualization results of vertebrae localization are shown. The images labeled "GT" show the true centroid locations of the vertebrae, while the images labeled "Our" show the predicted centroid locations. The calculated average distance between vertebrae localization is 3.56 mm, which meets the requirements for vertebrae localization. After predicting the centroids of the vertebrae, a region of interest (ROI) is created based on the predicted results, and the target vertebrae within the region is segmented individually. All the individual segmentation results are integrated to obtain the final multi-label segmentation result. Figure 6 Some segmentation results are shown, where the images marked as GT are real labels, and the images marked as Our are the segmentation results of this method. By comparison, it can be seen that the present invention can better segment the vertebrae even in areas with severe scoliosis. To fully demonstrate the effectiveness of the segmentation network used in the present invention, the segmentation network is replaced with a commonly used network in the field of spinal segmentation, and the above two evaluation indicators are used for separate tests. The comparison results are shown in Table 1. It can be seen from Table 1 that the proposed network structure has certain advantages in the task of segmenting individual vertebrae.

[0062] Table 1 Comparison of segmentation performance of different networks (Metrics: Dice (%) / HD (mm))

[0063]

Claims

1. A two-stage AIS spine CT image multi-label segmentation method based on deep learning, characterized by: The following steps are involved: Step 1: Obtain the AIS spine image dataset and preprocess the images in the dataset; Step 2: Based on the preprocessed labels, create new labels for vertebral positioning; Step 3: Process the preprocessed images and new labels; Step 4: Use the processed images and new labels to train a convolutional neural network for vertebrae localization. Step 5: Use the trained convolutional neural network to test the preprocessed image, and then post-process the test results to obtain the predicted results of the vertebral center of mass in the spine image to achieve vertebral positioning; Step 6: Reprocess the pre-processed original images and labels; The reprocessing specifically includes: calculating the centroid of the vertebra based on the preprocessed labels, cropping the preprocessed image and labels with each centroid as the center to obtain a fixed-size region centered on the centroid of each vertebra, and performing random rotation and random translation on the cropped region; Step 7: Improve the Attention-U-Net model to build a segmentation network and train it using the reprocessed images and labels. The segmentation network is specifically based on Attention-U-Net. In the coding layer of the network, the original convolution layer is replaced with a deep convolution of size 7×7×7; residual connections are added to the coding layer; in the decoding layer, deep convolution layers are used to process the information of each channel separately, and then a convolution layer is used to integrate the information of all channels. At the same time, residual connections are used to improve the performance of the network. Step 8: Create a region of interest based on the vertebral centroid position predicted during the vertebral positioning process, and use the trained segmentation network to segment the target vertebra in the region of interest; Step nine: Integrate the segmentation results of all regions of interest to obtain the multi-label segmentation results.

2. The two-stage AIS spine CT image multi-label segmentation method based on deep learning according to claim 1 is characterized in that The preprocessing described in step 1 is specifically as follows: resampling all CT images and labels so that the voxel spacing is unified to 1 mm, and rotating all images and labels to the same direction.

3. The two-stage AIS spine CT image multi-label segmentation method based on deep learning according to claim 1 is characterized in that The new label described in step 2 is created based on the centroid of the vertebra. In the new label, the label value of the non-vertebral area is 0, the label value of the vertebral area is between 0 and 1, and the label value of the point closer to the centroid of the vertebra is closer to 1.

4. The two-stage AIS spine CT image multi-label segmentation method based on deep learning according to claim 3 is characterized in that The processing described in step 3 includes cropping, random rotation, and random translation. The processed images and labels are vertebrae cropped images with a size of 128×128×128.

5. The two-stage AIS spine CT image multi-label segmentation method based on deep learning according to claim 1 is characterized in that The convolutional neural network described in step 4 uses Attention-U-Net as the network structure.

6. The two-stage AIS spine CT image multi-label segmentation method based on deep learning according to claim 5 is characterized in that The post-processing process described in step 5 is as follows: binarize the output image of the convolutional neural network, retain the area with the median value closest to 1 in the output image, and convert this area into a point set; use the K-Means clustering method to aggregate the point set into clusters equal to the number of vertebrae, and use the cluster center of each cluster as the predicted vertebral centroid.

7. The two-stage AIS spine CT image multi-label segmentation method based on deep learning according to any one of claims 1 to 6, characterized in that: Step nine, the integration process is: adding the segmentation results of each vertebra to the same blank image, and distinguishing the segmentation results of different vertebrae using different label values; The vertebral segmentation results are sorted according to the coordinates of the centroid on the Z axis to obtain the final multi-label segmentation result.

Images