A deep learning-based hip CT image segmentation method

Abstract

The application provides a hip CT image segmentation method based on deep learning. The method comprises the following steps: acquiring a hip CT image to be processed; performing coarse segmentation on the input image through a neural network segmentation positioning module to obtain coarse segmentation results of the hip bone and the femur and positioning results of the centers of left and right acetabular fossae; performing fine segmentation on the centers of the acetabular fossae through a Transformer-based fine segmentation module to obtain fine segmentation results of the acetabular fossae and the femoral head; replacing the voxel point categories under the corresponding coordinates in the coarse segmentation results with the categories of the fine segmentation results to realize the fusion of the two results and obtain the final segmentation results. Compared with the prior art, the method provided by the application simplifies the model, improves the processing capacity of interlayer information, avoids the influence of noise and artifacts, has better algorithm stability, and effectively improves the segmentation accuracy of the lesion.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and more specifically to a deep learning-based method for hip joint CT image segmentation. Background Technology

[0002] Hip segmentation is a crucial step in the preoperative planning of hip replacement surgery. It provides a three-dimensional model of the hip and femur, which is used for subsequent registration point selection. Simultaneously, the diseased area in the acetabulum can be used for real-time rendering during intraoperative hip bone resurfacing, facilitating the surgeon's control over the resurfacing depth.

[0003] Patent CN111563899A proposes a method for bone segmentation in hip CT images. This method uses a GAN network to segment two-dimensional slices of the hip CT image, and then uses an LSTM network to learn the inter-slice information between the slices. Although the LSTM network has a certain ability to process sequence information, its input is always only local sequence information, and it cannot grasp the global inter-slice information, thus lacking in the ability to process 3D images.

[0004] Patent CN107274389A proposes a method for obtaining anatomical parameters of the femur and acetabulum based on CT three-dimensional sequence images. This method performs coarse segmentation using thresholding, enhances the gap region using a Hessian matrix, and then uses the level set method to obtain fine segmentation results. However, the thresholding method is highly sensitive to image noise and artifacts, and is prone to oversegmentation. The level set method, being an iterative method, is easily trapped in local optima, and requires setting hyperparameters, which cannot adapt to different data, resulting in poor algorithm stability.

[0005] Patent CN114419618A proposes a deep learning-based preoperative planning system for total hip arthroplasty. This system uses a 3D segmentation network in its segmentation module to segment the hip joint. Because CT images are typically large, the lesion area in the acetabulum fossa occupies a small portion of the overall image. Therefore, using a single network to segment the entire image results in better segmentation of larger areas such as the ilium and femoral stem, while the lesion in the acetabulum fossa contributes little to the loss function, leading to poor segmentation of the acetabulum fossa. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art described in the background section and provide a deep learning-based hip joint CT image segmentation method. This method first performs coarse segmentation of the hip joint using a UNet network, which simplifies the model and improves the processing capability of inter-slice information. Then, it automatically locates the position of the acetabulum by detecting the center of the acetabulum and uses a Transformer network to perform fine segmentation of the femoral head and acetabulum at the acetabulum, which effectively improves the segmentation accuracy of the lesion.

[0007] To achieve the above objectives, the present invention employs the following technical solution: a hip joint CT image segmentation method based on deep learning, comprising the following steps:

[0008] Acquire the CT image of the hip joint to be processed;

[0009] The input image is coarsely segmented by a neural network segmentation and localization module to obtain coarse segmentation results of the hip bone and femur, as well as localization results of the centers of the left and right acetabular fossae.

[0010] The center of the acetabulum is finely segmented using a Transformer-based fine segmentation module to obtain the fine segmentation results of the acetabulum and femoral head;

[0011] The voxel categories at corresponding coordinates in the coarse segmentation result are replaced with the categories in the fine segmentation result to achieve the fusion of the two results and obtain the final segmentation result.

[0012] Furthermore, the neural network segmentation and localization module includes: a downsampling part, a neural network structure, a segmentation branch, and a localization branch.

[0013] Furthermore, the downsampling part uses linear interpolation to sample the input image.

[0014] Furthermore, the neural network structure uses UNet as the basic network structure.

[0015] Furthermore, the segmentation branch consists of 3D convolutions with a size of 1*1*1, and the positioning branch consists of 3D convolutions with a size of 3*3*3.

[0016] Furthermore, the loss function for the segmentation branch is the DiceLoss function, and the loss function for the localization branch is the MSELoss function, with both branches sharing the same UNet weights.

[0017] Furthermore, the coarse segmentation of the input image using the neural network segmentation and localization module to obtain coarse segmentation results of the hip bone and femur, as well as localization results of the centers of the left and right acetabular fossae, includes:

[0018] The input image is downsampled by the downsampling part, so that the resolution and interlayer spacing of the sampled image are 2 mm.

[0019] The UNet neural network structure is used to extract semantic features of 3D images in both high and low dimensions, and these features are then fused to obtain a feature image.

[0020] The category of each voxel point is output through the segmentation branch to obtain the coarse segmentation results of the hip and femur;

[0021] The positioning branch outputs a probability heatmap of the acetabular fossa center point. The closer the voxel value is to the center of the acetabular fossa, the closer it is to 1, and vice versa. The voxel coordinate with the highest probability value is selected as the final positioning result of the acetabular fossa center.

[0022] Furthermore, the Transformer-based fine segmentation module includes: an image cropping component, a block embedding component, a Transformer encoder, and a decoder.

[0023] Furthermore, the Transformer encoder includes a normalization function, a fully connected layer, and multi-head self-attention. The normalization function adjusts the data distribution to avoid gradient vanishing; the fully connected layer performs a linear mapping of the block embeddings; and the multi-head self-attention allows the neural network to focus more on the target region, effectively improving the network's image perception ability.

[0024] Furthermore, the decoder consists of a three-dimensional convolution operator with a size of 3*3*3.

[0025] Furthermore, the fine segmentation of the acetabular fossa center using a Transformer-based fine segmentation module to obtain the fine segmentation results of the acetabular fossa and femoral head includes:

[0026] Based on the location result of the center point of the acetabulum, the area with a size of 96*96*96 is cropped from the image and used as the input image of the fine segmentation module;

[0027] The cropped input image is divided into N small blocks, and the N block images are position-encoded and linearly mapped to obtain the block embedding of each block image;

[0028] The block embedding, after being encoded by the Transformer encoder, obtains a strong feature representation capability;

[0029] The decoder decodes the block embedding encoded by the Transformer encoder, converts the block embedding back into an image form, and classifies each voxel in the image.

[0030] The neural network is trained until the loss function converges to obtain the final neural network model, wherein the loss function of the neural network is the BCELoss function;

[0031] The cropped image is segmented using the neural network model to obtain a precise segmentation result of the acetabular fossa and the femoral head.

[0032] Furthermore, the process of replacing the voxel point category at the corresponding coordinates in the coarse segmentation result with the category in the fine segmentation result, thereby fusing the two results to obtain the final segmentation result, includes:

[0033] The result obtained from the coarse segmentation is resampled into an image with a resolution of 1 mm.

[0034] Calculate the coordinates of each foreground voxel in the RAS coordinate system in the fine segmentation result;

[0035] The voxel point category at the corresponding coordinate in the coarse segmentation result is replaced with the category in the fine segmentation result to achieve the fusion of the two results;

[0036] After fusion, the image is Gaussian smoothed to remove burrs at the boundary between the coarse and fine segmentation results, resulting in the final hip joint CT image segmentation result.

[0037] Furthermore, the smoothing radius of the Gaussian smoothing is set to 1 mm.

[0038] This invention provides a deep learning-based method for hip joint CT image segmentation, which has the following advantages compared with existing technologies:

[0039] 1. Using the UNet network to segment targets directly on 3D images simplifies the model and improves the ability to process inter-layer information;

[0040] 2. Deep learning-based methods can effectively avoid the effects of noise and artifacts. Furthermore, deep learning convolution operators and self-attention modules can function similarly to Hessian matrices, eliminating the need for separate processing of gap regions and resulting in better algorithm stability.

[0041] 3. By using a positioning method, the center position of the acetabulum is automatically detected. After cropping the area near the center of the acetabulum, a separate neural network is used for segmentation, which can avoid the influence of other areas of the hip and femur and improve the segmentation accuracy of the acetabulum and femoral head. Attached Figure Description

[0042] The features, advantages and technical effects of exemplary embodiments of the present invention will now be described with reference to the accompanying drawings.

[0043] Figure 1This is a flowchart illustrating the deep learning-based hip joint CT image segmentation method provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the neural network segmentation and localization module performing coarse segmentation on the input image according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the fine segmentation module based on Transformer performing fine segmentation of the acetabular fossa center according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the fusion of the segmentation results provided in an embodiment of the present invention. Detailed Implementation

[0047] The features and exemplary embodiments of various aspects of this disclosure will now be described in detail. To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to explain this disclosure only and not to limit it. For those skilled in the art, this disclosure can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this disclosure by illustrating examples.

[0048] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0049] To better understand the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] Figure 1 This is a flowchart illustrating the deep learning-based hip joint CT image segmentation method provided in this embodiment of the invention.

[0051] like Figure 1 As shown, this invention provides a deep learning-based method for hip joint CT image segmentation, comprising the following steps:

[0052] S101, acquire the hip joint CT image to be processed;

[0053] S102, the input image is coarsely segmented through the neural network segmentation and localization module to obtain the coarse segmentation results of the hip bone and femur and the localization results of the center of the left and right acetabular fossae;

[0054] S103, the center of the acetabulum is finely segmented by the Transformer-based fine segmentation module to obtain the fine segmentation results of the acetabulum and femoral head;

[0055] S104 replaces the voxel point category at the corresponding coordinates in the coarse segmentation result with the category in the fine segmentation result, thereby fusing the two results to obtain the final segmentation result.

[0056] Figure 2 This is a schematic diagram of the neural network segmentation and localization module performing coarse segmentation on the input image according to an embodiment of the present invention.

[0057] like Figure 2 As shown, the neural network segmentation and localization module includes: a downsampling part, a neural network structure, a segmentation branch, and a localization branch.

[0058] As an optional implementation, the downsampling portion uses linear interpolation to sample the input image.

[0059] As an alternative implementation, the downsampling part can also use other algorithms that can achieve the same or similar effects to sample the input image.

[0060] As an optional implementation, the neural network structure uses UNet as the basic network structure.

[0061] As an alternative implementation, the neural network structure can also use other convolutional neural networks as the basic network structure to achieve the same or similar effects.

[0062] As an optional implementation, the segmentation branch consists of 3D convolutions of size 1*1*1, and the positioning branch consists of 3D convolutions of size 3*3*3.

[0063] As an optional implementation, the loss function for the segmentation branch is the DiceLoss function, the loss function for the localization branch is the MSELoss function, and the two branches share the same UNet weights.

[0064] As an optional implementation, the coarse segmentation of the input image by the neural network segmentation and localization module in S102 to obtain coarse segmentation results of the hip bone and femur and localization results of the centers of the left and right acetabular fossae includes:

[0065] The input image is downsampled by the downsampling part, so that the resolution and interlayer spacing of the sampled image are 2 mm.

[0066] The UNet neural network structure is used to extract semantic features of 3D images in both high and low dimensions, and these features are then fused to obtain a feature image.

[0067] The category of each voxel point is output through the segmentation branch to obtain the coarse segmentation results of the hip and femur;

[0068] The positioning branch outputs a probability heatmap of the acetabular fossa center point. The closer the voxel value is to the center of the acetabular fossa, the closer it is to 1, and vice versa. The voxel coordinate with the highest probability value is selected as the final positioning result of the acetabular fossa center.

[0069] Figure 3 This is a schematic diagram of the fine segmentation module based on Transformer used in an embodiment of the present invention to perform fine segmentation of the acetabular fossa center.

[0070] like Figure 3 As shown, the Transformer-based fine segmentation module includes: an image cropping component, a block embedding component, a Transformer encoder, and a decoder.

[0071] As an optional implementation, the Transformer encoder includes: a normalization function, a fully connected layer, and multi-head self-attention. The normalization function adjusts the data distribution to avoid gradient vanishing; the fully connected layer performs a linear mapping of the block embeddings; and multi-head self-attention allows the neural network to focus more on the target region, effectively improving the network's image perception ability.

[0072] As an optional implementation, the decoder consists of a three-dimensional convolution operator with a size of 3*3*3.

[0073] As an optional implementation, the fine segmentation of the acetabular fossa center using a Transformer-based fine segmentation module described in S103 to obtain a fine segmentation result of the acetabular fossa and femoral head includes:

[0074] Based on the location result of the center point of the acetabulum, the area with a size of 96*96*96 is cropped from the image and used as the input image of the fine segmentation module;

[0075] The cropped input image is divided into N small blocks, and the N block images are position-encoded and linearly mapped to obtain the block embedding of each block image;

[0076] The block embedding, after being encoded by the Transformer encoder, obtains a strong feature representation capability;

[0077] The decoder decodes the block embedding encoded by the Transformer encoder, converts the block embedding back into an image form, and classifies each voxel in the image.

[0078] The neural network is trained until the loss function converges to obtain the final neural network model, wherein the loss function of the neural network is the BCELoss function;

[0079] The cropped image is segmented using the neural network model to obtain a precise segmentation result of the acetabular fossa and the femoral head.

[0080] Figure 4 This is a schematic diagram of the fusion of the segmentation results provided in an embodiment of the present invention.

[0081] like Figure 4 As shown in step S104, the voxel point category at the corresponding coordinates in the coarse segmentation result is replaced with the category in the fine segmentation result to achieve the fusion of the two results and obtain the final segmentation result, including:

[0082] The result obtained from the coarse segmentation is resampled into an image with a resolution of 1 mm.

[0083] Calculate the coordinates of each foreground voxel in the RAS coordinate system in the fine segmentation result;

[0084] The voxel point category at the corresponding coordinate in the coarse segmentation result is replaced with the category in the fine segmentation result to achieve the fusion of the two results;

[0085] After fusion, the image is Gaussian smoothed to remove burrs at the boundary between the coarse and fine segmentation results, resulting in the final hip joint CT image segmentation result.

[0086] As an optional implementation, the smoothing radius of the Gaussian smoothing is set to 1 mm.

[0087] This invention provides a deep learning-based method for hip joint CT image segmentation, which has the following advantages compared with existing technologies:

[0088] 1. Using the UNet network to segment targets directly on 3D images simplifies the model and improves the ability to process inter-layer information;

[0089] 2. Deep learning-based methods can effectively avoid the effects of noise and artifacts. Furthermore, deep learning convolution operators and self-attention modules can function similarly to Hessian matrices, eliminating the need for separate processing of gap regions and resulting in better algorithm stability.

[0090] 3. By using a positioning method, the center position of the acetabulum is automatically detected. After cropping the area near the center of the acetabulum, a separate neural network is used for segmentation, which can avoid the influence of other areas of the hip and femur and improve the segmentation accuracy of the acetabulum and femoral head.

[0091] Although the invention has been described with reference to preferred embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no structural conflict. The invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A deep learning-based hip joint CT image segmentation method, characterized in that, Includes the following steps: S101, acquire the hip joint CT image to be processed; S102, the input image is coarsely segmented through the neural network segmentation and localization module to obtain the coarse segmentation results of the hip bone and femur and the localization results of the center of the left and right acetabular fossae; S103, the center of the acetabulum is finely segmented by the Transformer-based fine segmentation module to obtain the fine segmentation results of the acetabulum and femoral head; S104: Replace the voxel point category at the corresponding coordinate in the coarse segmentation result with the category in the fine segmentation result to achieve the fusion of the two results and obtain the final segmentation result; As described in S104, the voxel point category at the corresponding coordinates in the coarse segmentation result is replaced with the category in the fine segmentation result to achieve the fusion of the two results and obtain the final segmentation result, including: The result obtained from the coarse segmentation is resampled into an image with a resolution of 1 mm. Calculate the coordinates of each foreground voxel in the RAS coordinate system in the fine segmentation result; The voxel point category at the corresponding coordinate in the coarse segmentation result is replaced with the category in the fine segmentation result to achieve the fusion of the two results; After fusion, the image is Gaussian smoothed to remove burrs at the boundary between the coarse and fine segmentation results, resulting in the final hip joint CT image segmentation result.

2. The hip joint CT image segmentation method based on deep learning according to claim 1, characterized in that, The neural network segmentation and localization module described in S102 includes: a downsampling part, a neural network structure, a segmentation branch, and a localization branch.

3. The hip joint CT image segmentation method based on deep learning according to claim 2, characterized in that, The downsampling part of the neural network segmentation and localization module uses linear interpolation to sample the input image.

4. The hip joint CT image segmentation method based on deep learning according to claim 2, characterized in that, The neural network structure in the neural network segmentation and localization module uses UNet as the basic network structure.

5. The hip joint CT image segmentation method based on deep learning according to claim 2, characterized in that, The segmentation branch in the neural network segmentation and localization module consists of 3D convolutions of size 1*1*1, and the localization branch consists of 3D convolutions of size 3*3*3.

6. The hip joint CT image segmentation method based on deep learning according to claim 2, characterized in that, The loss function for the segmentation branch in the neural network segmentation and localization module is the DiceLoss function, and the loss function for the localization branch is the MSELoss function. The two branches share the same UNet weights.

7. The hip joint CT image segmentation method based on deep learning according to claim 1, characterized in that, S102 describes using a neural network segmentation and localization module to perform coarse segmentation of the input image, obtaining coarse segmentation results of the hip bone and femur, as well as localization results of the centers of the left and right acetabular fossae, including: The input image is downsampled by the downsampling part, so that the resolution and interlayer spacing of the sampled image are 2mm. The UNet neural network structure is used to extract semantic features of 3D images in high and low dimensions, and these features are then fused to obtain a feature image. By splitting the branches and outputting the category of each voxel point, coarse segmentation results of the hip and femur are obtained; The probability heatmap of the acetabular fossa center point is output by the localization branch. The voxel value closer to the center of the acetabular fossa is closer to 1, and vice versa. The voxel coordinate with the highest probability value is selected as the final localization result of the acetabular fossa center.

8. The hip joint CT image segmentation method based on deep learning according to claim 1, characterized in that, The Transformer-based fine segmentation module described in S103 includes: an image cropping component, a block embedding component, a Transformer encoder, and a decoder.

9. The hip joint CT image segmentation method based on deep learning according to claim 8, characterized in that, The Transformer encoder in the Transformer-based fine segmentation module includes: a normalization function, a fully connected layer, and multi-head self-attention; the normalization function can adjust the data distribution and avoid gradient vanishing; the fully connected layer is used to perform linear mapping on the block embedding; multi-head self-attention can make the neural network pay more attention to the target region, which can effectively improve the network's ability to perceive images.

10. A hip joint CT image segmentation method based on deep learning according to claim 8, characterized in that, The decoder in the Transformer-based fine segmentation module consists of a 3D convolution operator with a size of 3*3*3.

11. The hip joint CT image segmentation method based on deep learning according to claim 1, characterized in that, S103 describes using a Transformer-based fine segmentation module to perform fine segmentation of the acetabular fossa center, obtaining fine segmentation results of the acetabular fossa and femoral head, including: Based on the location result of the center point of the acetabulum, the area with a cropped size of 96*96*96 is used as the input image of the fine segmentation module; The cropped input image is divided into N small blocks, and the N block images are position-encoded and linearly mapped to obtain the block embedding of each block image; The block embedding, after being encoded by the Transformer encoder, obtains a strong feature representation capability; The decoder decodes the block embedding encoded by the Transformer encoder, converts the block embedding back into an image form, and classifies each voxel in the image. The neural network is trained until the loss function converges to obtain the final neural network model, wherein the loss function of the neural network is the BCELoss function; The cropped image is segmented using the neural network model to obtain a precise segmentation result of the acetabular fossa and the femoral head.

12. The hip joint CT image segmentation method based on deep learning according to claim 1, characterized in that, The smoothing radius of the Gaussian smoothing is set to 1 mm.

Images