Rib centerline extraction method and system based on point cloud

Abstract

The application discloses a rib center line extraction method and system based on point cloud, and the method comprises the following steps: step 1: a rib original CT data, an independent rib label and an independent rib center line label are combined to form a data set; step 2: the rib CT data is converted into point cloud data; step 3: the rib point cloud data is down-sampled; step 4: the rib point cloud data is pre-processed; step 5: an independent rib segmentation model based on point cloud is constructed; step 6: the model of step 5 is trained and verified; and step 7: rib CT prediction and rib center line extraction. The application utilizes point cloud to efficiently use spatial convolution, which is beneficial to extracting multi-scale and multi-level local feature information. Meanwhile, the point cloud data operation has the advantage of high efficiency, and can effectively avoid the technical problems of long time consumption and insufficient precision in the previous rib center line extraction method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a rib center line extraction method and system based on point clouds. BACKGROUND

[0002] A human body generally has 12 pairs of ribs, the rear end of which is connected with a spine, and the front end of the upper 10 pairs of ribs is connected with a sternum. The rib center line has a wide range of uses in medicine: based on the anatomical center line of the ribs, the internal coordinate system can be used to locate organs, which is convenient for surgical planning and postoperative evaluation, and can also register pathological information such as lung nodules. Visualization of the rib center line can help detect and treat bone fracture and bone fracture diseases, greatly reducing the burden of clinicians and promoting various clinical applications. Therefore, it is of great significance to accurately extract the rib center line.

[0003] In previous studies, there are mainly two methods to extract the rib center line. The first rib center line extraction method directly uses the voxel-labeled rib center line as input to a 3D segmentation model, and the rib center line template obtained is matched with the original CT to extract the rib center line, and then a separate refinement is performed to obtain a relatively complete rib center line. This method is extremely dependent on the 3D segmentation model. Due to the limitations of convolution operation, there is a defect that global and long-range semantic information interaction cannot be learned, resulting in a certain segmentation bottleneck, so the quality of the rib center line obtained by direct segmentation is low. The second rib center line extraction method first uses a convolutional neural network algorithm to segment the rib region, then uses a connected domain to obtain the rib region corresponding to each rib, calculates and connects the region center of each rib to obtain the center line of the corresponding rib, and then performs smoothing processing through a curve smoothing method. This method uses a two-stage method of first segmenting the rib region and then obtaining the rib center line, which has certain advantages over the first method and has certain improvement in the quality of the obtained rib center line. The limitation of the convolutional network, the rib segmentation accuracy is still low, and the rib center line extraction is extremely dependent on the accuracy of the rib segmentation, and the segmentation accuracy will directly affect the accuracy of the rib center line extraction; due to the density of the rib CT voxel, the entire algorithm takes a long time, and the average prediction time is 68.31 seconds; and the rib center line extraction and post-processing of this method have limited effect on the improvement of the rib center line, so it is not conducive to clinical application.

[0004] In order to meet the needs of current medical clinical applications, a rib center line extraction method with high precision and high quality is needed. SUMMARY

[0005] The density of the voxel leads to a large amount of data and a large amount of calculation, so that the performance and the calculation speed of the voxel-based method are greatly limited, in view of the problem, the efficiency of the point cloud data can save the memory and the calculation amount, therefore, the application provides a rib center line extraction method and system based on point cloud, the application utilizes the point cloud to efficiently use the spatial convolution, and is favorable for extracting the multi-scale and multi-level local feature information. Meanwhile, the point cloud data operation has the advantages of high efficiency, and can effectively avoid the technical problems of long time consumption and insufficient precision in the existing rib center line extraction method.

[0006] The application adopts the following technical scheme:

[0007] A rib center line extraction method based on point cloud, according to the following steps:

[0008] Step 1: the rib original CT data, the independent rib annotation and the independent rib center line annotation are combined to form a data set;

[0009] Step 2: the rib CT data is converted into point cloud data;

[0010] Step 3: the rib point cloud data is down-sampled;

[0011] Step 4: the rib point cloud data is preprocessed;

[0012] Step 5: an independent rib segmentation model based on point cloud is constructed;

[0013] Step 6: the model of step 5 is trained and verified;

[0014] Step 7: rib CT prediction and rib center line extraction.

[0015] Preferably, step 1: rib CT data preparation

[0016] The application is aimed at CT three-dimensional data with independent rib annotation and independent rib center line annotation, which contains complete rib part and segmentation annotation. The rib original CT data and the independent rib annotation and the independent rib center line annotation data set are divided into a training set, a verification set and a test set according to the ratio of 8:1:1.

[0017] The independent rib annotation is to mark the 24 ribs of the human body according to different voxels.

[0018] Preferably, step 2: rib CT is converted into point cloud data

[0019] The HU value size characteristics of the bone part in the original CT of the rib and the sparseness of the rib part are analyzed, the non-bone voxel of the original rib CT is roughly filtered by setting the HU threshold, the non-bone voxel is converted into point cloud data in the form of three-dimensional space absolute coordinates and relative coordinates, and the voxel value label corresponding to the point is recorded. In this way, the local geometric representation is enriched, and the information loss in the point cloud conversion process is reduced.

[0020] Preferably, step 3: downsampling of rib point cloud data

[0021] The amount of point cloud data is generally large, in order to improve the calculation efficiency, the point cloud data is down-sampled to obtain a specified amount of point cloud data, which is input into the PCT point cloud network model which has superior segmentation capability at present.

[0022] Firstly, the point cloud is divided into 30k point batches of equal size; secondly, for batches with insufficient division, a certain number of points are randomly extracted from other batches, and majority voting is applied to the prediction of repeated scores; finally, the point predictions of all batches are spliced to obtain voxel prediction.

[0023] Preferably, step 4: preprocessing of rib point cloud data

[0024] In order to enhance the learning ability of the model, it is necessary to standardize the point cloud data; in order to enhance the robustness of the model, point cloud data enhancement needs to be taken during model training.

[0025] Point cloud data standardization, i.e. point cloud center normalization; point cloud data enhancement, including point cloud random rotation, point cloud random rotation, point cloud random disturbance, point cloud scaling, point cloud tensorization.

[0026] Preferably, step 5: constructing an independent rib segmentation model based on point cloud, specifically as follows:

[0027] Step 5.1: instance segmentation based on point cloud

[0028] The rib point cloud data that has been preprocessed in step 4 is taken as input, the PCT point cloud model is selected, and the input point cloud data features are extracted for independent rib segmentation.

[0029] The PCT point cloud model includes a coordinate-based input embedding module, an optimized offset attention module, and a classification and segmentation module.

[0030] The coordinate-based input embedding module combines the original position encoding and input embedding into a coordinate-based input embedding module in the PCT framework because the point cloud does not have a fixed order. It can generate distinguishable features because each point has a unique coordinate representing its spatial position.

[0031] The optimized offset attention module is an effective upgrade of the original self-attention module. The working principle is to replace the attention feature with the offset between the input of the self-attention module and the attention feature.

[0032] The classification segmentation module directly pools (samples) the features after the optimized offset attention module to obtain global features, and then performs classification and segmentation operations respectively. The classification module feeds the global features into the classification decoder to predict the final score; the segmentation module connects the global features with the point features (features after the attention module) point by point, passes through a segmentation decoder (the same as the classification network decoder), and predicts the final point segmentation score. The label of the point is determined as the maximum one.

[0033] Step 5.2: Constructing the target loss function

[0034] In order to calculate the similarity between the independent labeled rib segmentation point cloud prediction and the point cloud label annotation, NLLLoss is taken as the loss function, and the network parameters are updated by minimizing NLLLoss until the model converges. The NLLLoss loss function is shown in equation (1).

[0035]

[0036] Where y k is the data label after one_hot encoding, N is the number of labels, and log_softmax is the log operation after the softmax function operation on the predicted value.

[0037] Preferably, step 6: model training and verification, as follows:

[0038] Step 6.1: Model training and parameter updating

[0039] In the model training mode, the training set samples of step 4 after rib point cloud data preprocessing are sent to the model constructed in step 5, and the model is trained by setting reasonable hyperparameters and optimizers. The difference between the independent rib segmentation point cloud prediction and the point cloud label annotation is calculated using the loss function, and then the network parameters are updated by minimizing the difference.

[0040] Step 6.2: Model verification and saving optimal weights

[0041] In the model verification mode, the validation set samples of step 4 only with point cloud standardization are sent to the model with weights, and the average class accuracy (mAcc) and the average class intersection over union (mIoU) are calculated to evaluate the prediction ability of the current model, and the model weight with the optimal indicators is saved.

[0042] Step 7: Rib CT prediction and extraction of rib centerline

[0043] After the test set samples are converted into point cloud data, they are sent to the model loaded with the optimal weight in step 6 to obtain independent rib segmentation point cloud prediction. The independent rib segmentation point cloud prediction is converted into voxel prediction, and the independent rib voxel segmentation result is post-processed: first, the spinal column component is removed according to the intensity characteristics of the spinal column component, then the small target components in the segmentation result are removed, and then the refined independent rib segmentation final result is obtained; finally, the post-processed independent rib prediction result with a label is obtained according to the volume, a single rib connected component is obtained, any rib foreground voxel is located, the centerline is guided to pass through the rib volume center, and the rib centerline is extracted through linear smoothing and upsampling.

[0044] In actual application, the rib CT is converted into point cloud data (step 2), the point cloud is down-sampled (step 3), the point cloud data is pre-processed (step 4), the independent rib segmentation is obtained through the model verified in step 6, and finally the rib centerline is obtained through post-processing operation.

[0045] The HU threshold is set to 300, the part with an intensity greater than the threshold in the rib CT is subjected to median filtering, erosion and expansion operation, the spinal column region coordinates are locked, and the spinal column component in the corresponding rib segmentation region is set to 0, so as to remove the possible spinal column component in the rib segmentation region. Because the number of human ribs is large, there are many irrelevant small components in segmentation, therefore, the number of voxel points of each connected domain is counted to evaluate the size of each component in segmentation, and according to the size of a single rib region, the region size threshold is set to 100, the components with a size less than 100 in segmentation are deleted, so as to refine the rib segmentation region. In order to extract the rib centerline, first, the single rib connected component is obtained according to the refined rib region, for each rib, any rib foreground voxel is located, the end of the connected component is taken as a root point, a penalty domain is defined by implementing Euclidean distance transformation to guide the centerline to pass through the center of the rib volume. Then, the shortest path is found to guide the path from the root point to the farthest point; finally, the path is smoothed, and the centerline is up-sampled to 600 points through linear interpolation, so as to obtain a high-quality rib centerline.

[0046] The application also discloses a system based on the rib centerline extraction method.

[0047] Constructing a data set: the rib original CT data, independent rib labeling and independent rib centerline labeling are combined to form a data set;

[0048] Point cloud data conversion module: rib CT data is converted into point cloud data;

[0049] Down-sampling module: down-sampling rib point cloud data;

[0050] Preprocessing module: preprocessing rib point cloud data;

[0051] Independent rib segmentation model construction module: constructing an independent rib segmentation model based on point cloud;

[0052] Training and verification module: training and verifying the independent rib segmentation model;

[0053] Rib centerline extraction module: rib CT prediction and rib centerline extraction.

[0054] The beneficial effects of the present application are as follows:

[0055] The present application designs a rib centerline extraction method and system based on point cloud. First, the candidate voxels of the skeletal structure are converted into point cloud, then the point cloud is down-sampled and sent into the network model, and finally the network model learns local features in an increasing context range, extracts the geometric characteristics of the ribs, reduces the texture bias of the voxel-based CNN, improves the segmentation performance of the rib segmentation model, and obtains high-quality rib centerlines through rib centerline extraction and improvement. The present application can quickly calculate high-precision and high-quality rib centerlines, which is convenient for clinical practical application. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced below, which constitutes a part of the specification, together with the embodiments of the present application, to explain the present application, and does not constitute a limitation to the present application.

[0057] Figure 1 The flow chart of the rib centerline extraction method based on point cloud of the embodiment of the present application.

[0058] Figure 2 The structure diagram of independent rib segmentation using PCT model in the embodiment of the present application.

[0059] Figure 3 The three-dimensional effect diagram of the extracted rib centerline of the embodiment of the present application.

[0060] Figure 4 The block diagram of the rib centerline extraction system based on point cloud of the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the present application will be further described in detail below in combination with embodiments.

[0062] The rib center line extraction method based on the point cloud of the embodiment is as follows:

[0063] Step 1: Rib CT data preparation

[0064] The present application is directed to the public data set RibFrac and the rib unified labeling data set RibSeg, and since the method of the present application needs independent rib labeling annotation, it is necessary to obtain more detailed independent rib labeling based on the labeling.

[0065] For each rib unified labeling CT, the target component in the rib CT is positioned, and the centroid coordinates thereof are calculated, the rib component with a centroid coordinate X component less than 250 is determined as a left rib, the rib component with a centroid coordinate X component greater than 250 is determined as a right rib, then according to the centroid coordinate Z component, the rib is sequentially labeled as a numbered voxel (1-12 and 13-24) in descending order; the labeling annotation is checked by using the visualization software ITK-SNAP, the rib with a labeling error voxel is re-annotated with a correct labeling voxel, and accurate labeling annotation is ensured.

[0066] The rib original CT data and the independent rib labeling are corresponded to the independent rib center line labeling to form a new data set, and the data set has a total of 370 CTs, which are divided into a training set, a validation set and a test set according to a ratio of 8:1:1. That is, the training set has a total of 290 CTs, and the validation set and the test set each have 37 CTs.

[0067] Step 2: Conversion of rib CT to point cloud data

[0068] The HU value of common bone tissue is greater than 200, and the voxel proportion of the rib in the 3D CT is less than 0.5%, so the HU threshold is set to 200, the non-bone tissue in the rib CT is filtered out, the absolute coordinates and relative coordinates of the bone tissue points are saved as point cloud data, and the labeling annotation voxel value corresponding to the point coordinates is saved as point cloud label.

[0069] The point cloud data is in the form of (x, y, z, x, y, z), wherein x, y and z are absolute spatial coordinates of the point, x, y and z are relative coordinates of the point. l l l l l l After the sampling and grouping from the volume are searched by the ball query (query searching) for each point, the absolute coordinates are subtracted to obtain the relative coordinates, so as to reduce the information loss in the conversion of the point cloud from the volume, and enrich the local geometric representation.

[0070] Step 3: Down-sampling of rib point cloud data

[0071] ​​​​​The point cloud data is generally large in quantity. To improve the calculation efficiency, the point cloud data needs to be down-sampled.

[0072] First, the point cloud is divided into 30k point batches of equal size; second, for batches with insufficient division, points are randomly extracted from other batches, and majority voting is applied to the prediction of repeated scores; finally, the point predictions of all batches are spliced to obtain voxel predictions.

[0073] Step 4: Rib point cloud data preprocessing

[0074] To enhance the learning ability of the model, the down-sampled point cloud data needs to be standardized during the model training and prediction phase; to enhance the robustness of the model, only point cloud data augmentation is used during model training.

[0075] Point cloud data standardization, i.e., point cloud center normalization; point cloud data augmentation, including point cloud random rotation, point cloud random rotation, point cloud random disturbance, point cloud scaling, and point cloud tensorization.

[0076] Step 5: Constructing a point cloud-based independent rib segmentation model

[0077] Step 5.1: Point cloud-based instance segmentation

[0078] The preprocessed rib point cloud data from step 4 is input into the PCT model for 25-class segmentation to obtain independent rib segmentation results.

[0079] The 25-class segmentation is divided into 24 different voxel-labeled rib classes and non-rib classes.

[0080] As shown in Figure 2 , given an input point cloud with a size of (30K, 6), first learn 128-dimensional embedding features through the input embedding module, then concatenate the attention outputs of each attention layer through feature dimensions, and then perform linear transformation to form point-by-point features. Perform max pooling and average pooling on the point-by-point features to extract effective global features representing the point cloud. Connect the global features with the point-by-point features, and then input them into the segmentation decoder: first pass through a feedforward neural network LBRD (combined with Linear, BatchNorm, ReLU, and dropout layers, with a dropout layer probability of 0.5), then pass through a feedforward neural network LBR (combined with Linear, BatchNorm, and ReLU layers), and finally pass through a Linear layer. Finally, obtain the final point-by-point segmentation scores, and the part label of the point is determined as the label with the highest score.

[0081] Step 5.2: Constructing the target loss function

[0082] To calculate the similarity between the independent marker rib segmentation point cloud prediction and the point cloud label annotation, NLLLoss is taken as the loss function, and the network parameters are updated by minimizing NLLLoss until the model converges.

[0083] Step 6: Model training and verification

[0084] Step 6.1: Model training and parameter updating

[0085] In the model training mode, the training set samples of rib point cloud data enhancement in step 4 are sent to the constructed model in step 5, and the model is trained by setting reasonable hyperparameters and optimizers. The difference between the independent rib segmentation point cloud prediction and the point cloud label annotation is calculated using the loss function, and then the network parameters are updated by minimizing the difference.

[0086] During training, the Adamw optimizer is used, the total training period is 250, the Batchsize is 4, the initial learning rate is set to 0.001, and the learning rate is attenuated by 0.5 times every 20 cycles, with a lower limit of 0.00001.

[0087] Step 6.2: Model verification and saving optimal weights

[0088] In the model verification mode, the validation set samples only with point cloud standardization in step 4 are sent to the model with weights, and the average class accuracy (mAcc) and average class intersection over union (mIoU) are calculated to evaluate the prediction ability of the current model, and the model weight with the optimal index is saved.

[0089] Step 7: Rib CT prediction and extraction of rib centerline

[0090] In actual application, the rib CT is converted into point cloud data (synchronous with step 2), the point cloud is down-sampled (synchronous with step 3), the point cloud data is preprocessed (synchronous with step 4), and then the independent rib segmentation is obtained through the model verified in step 6, and finally the rib centerline is obtained through post-processing operation.

[0091] The independent rib segmentation point cloud prediction is converted into voxel prediction. In order to obtain refined independent rib segmentation results and extract high-quality rib centerlines, post-processing is needed for the independent rib voxel segmentation results: first, remove the spine component according to its intensity characteristics, then remove the redundant small target components in the segmentation results, and then obtain the refined independent rib segmentation final result. Finally, according to the volume, the independent marker rib prediction result after post-processing is obtained, the single rib connection component is obtained, the rib voxel is positioned, the centerline is guided to pass through the rib volume center, and linear smoothing and up-sampling are realized to achieve high-quality rib centerline.

[0092] As Figure 4As shown, the embodiment discloses a rib center line extraction system based on the above-mentioned embodiment method, which comprises the following modules:

[0093] Building a data set: the rib original CT data, the independent rib label and the independent rib center line label are combined to form a data set;

[0094] Point cloud data conversion module: convert rib CT data into point cloud data;

[0095] Down-sampling module: down-sample rib point cloud data;

[0096] Preprocessing module: preprocessing rib point cloud data;

[0097] Independent rib segmentation model construction module: construct an independent rib segmentation model based on point cloud;

[0098] Training and verification module: training and verification of the independent rib segmentation model;

[0099] Rib center line extraction module: rib CT prediction and rib center line extraction.

[0100] It should be understood that the specific embodiments described above are only for illustration and explanation of the present application, and are not intended to limit the present application.

Claims

1. A method for extracting rib centerlines based on point clouds, characterized in that: Follow these steps: Step 1: Combine the original rib CT data, individual rib annotations, and individual rib centerline annotations into a dataset; Step 2: Convert rib CT data into point cloud data; Step 3: Downsample the rib point cloud data; Step 4: Preprocess the rib point cloud data; Step 5: Construct an independent rib segmentation model based on point cloud; Given an input point cloud, the model first learns 128-dimensional embedding features through its input embedding module. These features are then passed through multiple offset attention modules, and the attention outputs of these modules are concatenated along the feature dimension. A linear transformation is then performed to form point-by-point features. These point-by-point features are then subjected to max pooling and average pooling to extract effective global features representing the point cloud. The global features are then concatenated with the point-by-point features and fed into the model's segmentation decoder. The decoder first passes through a feedforward neural network (LBRD), which includes Linear, BatchNorm, ReLU, and dropout layers. Next, it passes through a feedforward neural network (LBR), which also includes Linear, BatchNorm, and ReLU layers. Finally, it passes through a Linear layer. Step 6: Train and validate the model from Step 5; Step 7: CT prediction of ribs and extraction of rib centerlines; Step 7 is as follows: Convert the point cloud prediction of independent rib segmentation into voxel prediction, and perform post-processing on the voxel segmentation results of independent ribs: First, remove the spinal component based on the strength characteristics of the spinal component, then remove redundant small target components in the segmentation results to obtain the final refined segmentation results of independent ribs; finally, for the post-processed prediction results of independently labeled ribs, obtain the individual rib connection components based on the volume, locate the foreground voxel of any rib, guide the center line to pass through the center of the rib volume, and perform linear smoothing and upsampling to extract the rib center line.

2. The method for extracting rib centerlines based on point clouds as described in claim 1, characterized in that, In step 1: The original rib CT data, independent rib annotations, and independent rib centerline annotations are combined to form a dataset, which is then divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

3. The method for extracting rib centerlines based on point clouds as described in claim 1, characterized in that, Step 2 is as follows: Analyze the HU value characteristics of the skeletal part and the sparsity of the rib part in the original rib CT, filter the non-skeletal voxels of the original rib CT by setting a HU threshold, convert it into point cloud data in the form of absolute and relative coordinates in three-dimensional space, and record the voxel value label corresponding to the point.

4. The method for extracting rib centerlines based on point clouds as described in claim 1, characterized in that, Step 3 is as follows: First, the point cloud is divided into batches of 30k points of equal size; second, for batches with insufficient number of points, points are randomly drawn from other batches, and majority voting is applied to the prediction of duplicate scores; finally, the point predictions of all batches are stitched together to obtain voxel predictions.

5. A method for extracting rib centerlines based on point clouds as described in any one of claims 1-4, characterized in that, Step 4 is as follows: standardize the downsampled point cloud data; perform point cloud data augmentation during model training.

6. The method for extracting rib centerlines based on point clouds as described in claim 5, characterized in that, Step 5 is as follows: Step 5.1: Instance segmentation based on point cloud Using the preprocessed rib point cloud data from step 4 as input, the PCT point cloud model is selected, and the features of the input point cloud data are extracted for independent rib segmentation. Step 5.2: Construct the target loss function We use NLLLoss as the loss function and update the network parameters by minimizing NLLLoss until the model converges.

7. The method for extracting rib centerlines based on point clouds as described in claim 6, characterized in that, Step 6 is as follows: Step 6.1: Model Training and Parameter Update In the model training mode, the training samples that have undergone rib point cloud data preprocessing in step 4 are fed into the model constructed in step 5. The model is trained by setting reasonable hyperparameters and optimizers. The difference between the independent rib segmentation point cloud prediction and the point cloud label annotation is calculated using the loss function, and then the network parameters are updated by minimizing the difference. Step 6.2: Model Validation and Saving of Optimal Weights In model validation mode, the validation samples that have undergone point cloud standardization in step 4 are fed into the weighted model to calculate the average class accuracy and average class intersection-over-union ratio, evaluate the predictive ability of the model, and save the model weights with the best indicators.

8. A system based on the rib centerline extraction method according to any one of claims 1-7, characterized in that: Includes the following modules: Dataset Construction: The dataset is composed of raw rib CT data, individual rib annotations, and individual rib centerline annotations; Point cloud data conversion module: Converts rib CT data into point cloud data; Downsampling module: Downsamples the rib point cloud data; Preprocessing module: preprocesses the rib point cloud data; Independent rib segmentation model construction module: Constructs an independent rib segmentation model based on point cloud; Training and Validation Module: Independent rib segmentation model is trained and validated; Rib centerline extraction module: CT prediction and extraction of rib centerlines.

Images