Full hip joint skeleton model segmentation and reconstruction method based on deep learning neural network

Abstract

The invention provides a full hip joint skeleton model segmentation and reconstruction method based on a deep learning neural network. The method comprises the following steps: firstly, establishing a deep learning neural network model for full hip joint skeleton model segmentation; the model comprises an encoding module, a down-sampling module, a decoding module and a segmented image output module. The coding module comprises a feature extraction module, a deformable hole patch embedding module and a Patch merging module; in each down-sampling stage, a spatial dynamic self-attention module is adopted to process an input feature map, and the spatial dynamic self-attention module comprises a deformable multi-head attention module and a neighborhood multi-head attention module; establishing a training sample, and training the deep learning neural network model; identifying and segmenting the hip joint CT cross section image of the patient to be processed by using the trained deep learning neural network model; and finally, based on the segmented pelvis and femur two-dimensional image sequence, carrying out total hip joint image three-dimensional modeling through a medical image three-dimensional reconstruction technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of total hip joint bone model segmentation and construction, and particularly relates to a total hip joint bone model segmentation and reconstruction method based on a deep learning neural network. BACKGROUND

[0002] Total hip arthroplasty (THA) is a mature surgical method for treating end-stage osteoarthritis, rheumatoid arthritis, avascular necrosis of the femoral head, and hip fractures. By implanting an artificial joint prosthesis into the human skeletal structure through bone cement and screws, the diseased hip joint is replaced, thereby restoring the mechanical function and movement ability of the patient's hip joint. This procedure has been widely used in clinical practice and has proven effective, making it one of the most important methods for treating end-stage hip joint disease.

[0003] Before performing THA, the surgeon needs to plan the procedure to determine the type, position, and orientation of the prosthesis, which is crucial for surgical precision and postoperative functional recovery. Current preoperative planning in clinical practice relies mainly on two-dimensional X-ray images and three-dimensional CT or MRI images for bone structure assessment and prosthesis parameter selection. However, existing preoperative planning techniques still have the following key problems:

[0004] (1) Difficulty in accurately quantifying individual bone anatomical differences:

[0005] Traditional two-dimensional X-ray films can only provide limited planar structural information, making it difficult to fully reflect the complex three-dimensional anatomical features of the hip joint. Even with CT / MRI for three-dimensional reconstruction, segmentation and modeling often rely on manual or semi-automatic operation, which is highly dependent on operator experience, is complex and time-consuming, and has significant subjectivity and poor reproducibility.

[0006] (2) Unable to reflect the dynamic biomechanical properties of the hip joint:

[0007] Although static images can be used to observe bone structure, they lack effective evaluation methods for post-implantation function, impact risk during movement, etc., leading to functional disorders such as limited mobility or pain in some patients after surgery.

[0008] (3) Lack of standardized and intelligent modeling process:

[0009] Current three-dimensional model generation and processing methods vary greatly among different medical institutions, and there is a lack of uniform standards for related procedures and parameters, making it difficult to achieve high consistency in individualized preoperative planning. In addition, traditional modeling methods often have insufficient accuracy when dealing with complex cases such as abnormal bone quality, joint adhesion, and bone deformation.

[0010] With the development of deep learning and artificial intelligence technology, medical image analysis methods based on deep neural networks have made significant progress in automatic segmentation of bones, structure recognition and three-dimensional reconstruction, and have shown better robustness and automation ability than traditional methods. Especially in preoperative planning of THA, individualized hip modeling technology assisted by deep learning can greatly improve the segmentation accuracy and modeling efficiency while reducing human error.

[0011] However, the existing deep learning-based bone segmentation method still has the following problems to be solved:

[0012] (1) When facing structural variations under different pathological conditions (such as acetabular adhesion, femoral head collapse, severe osteoporosis, etc.), the model often has difficulty in maintaining stability, and the segmentation accuracy decreases;

[0013] (2) There are false positives or false negatives in identifying anatomical structure details such as medullary cavity, which affects the modeling and matching of subsequent prosthesis channels;

[0014] (3) There is a lack of complete closed-loop automatic process from image segmentation to 3D model output for clinical actual application, which is difficult to directly interface with preoperative prosthesis design and virtual replacement simulation.

[0015] Therefore, there is an urgent need for a new hip bone segmentation and reconstruction technology with high robustness and automation ability, which can adapt to various pathological forms, improve modeling accuracy, and support standardized preoperative planning process. SUMMARY

[0016] In order to solve the problems of low segmentation accuracy of hip bone model, inability to adapt to different pathological forms, and lack of automatic modeling process in the prior art, the present application proposes a full hip bone model segmentation and reconstruction method based on deep learning neural network, which can realize accurate automatic segmentation and three-dimensional model reconstruction of the patient's hip pelvis, femur and its medullary cavity region, and provide high-quality bone model basis for preoperative planning and prosthesis design.

[0017] The technical scheme of the present application is:

[0018] A full hip bone model segmentation and reconstruction method based on deep learning neural network, comprising the following steps:

[0019] Step 1: Establish a deep learning neural network model for full hip bone model segmentation; the deep learning neural network model comprises an encoding module, a down-sampling module, a decoding module and a segmentation map output module;

[0020] The coding module is used for extracting feature information, including a feature extraction module, a deformable hole patch embedding module and a Patch merging module; the input patient full hip joint two-dimensional cross-sectional image is sequentially processed by the feature extraction module, the deformable hole patch embedding module and the Patch merging module to obtain a fusion feature map;

[0021] The downsampling module gradually downsamples the fusion feature map, and a spatial dynamic self-attention module is used to process the input feature map at each downsampling stage, the spatial dynamic self-attention module including a deformable multi-head attention module and a neighborhood multi-head attention module;

[0022] The decoding module performs image restoration and target structure prediction;

[0023] The segmentation map output module splices the feature map output by the decoding module, and outputs the class label of each pixel after convolution processing to form a labeled medical image segmentation map;

[0024] Step 2: Establish training samples and train the deep learning neural network model established in step 1;

[0025] Step 3: Use the trained deep learning neural network model obtained in step 2 to identify and segment the patient hip joint CT cross-sectional image to be processed to obtain a two-dimensional image sequence of the pelvis and femur after segmentation;

[0026] Step 4: Based on the two-dimensional image sequence of the pelvis and femur after segmentation, perform full hip joint image three-dimensional modeling through medical image three-dimensional reconstruction technology.

[0027] Further, the deformable hole patch embedding module uses two consecutive deformable hole convolution layers to realize, for each deformable hole convolution layer, each pixel point in the input feature map is processed according to the formula:

[0028] ( f 1 * k ) [ p ] = ∑ p k ∈ Ω ϕ ( f 1 ; p + Δ p + p k ) ⋅ k [ p k ] ,

[0029] ; wherein is the input feature map, wherein for the first layer of deformable hole convolution layer, is the feature map output by the feature extraction module, and for the second layer of deformable hole convolution layer, is the feature map output by the first layer of deformable hole convolution layer; is a sampling function, and the gray value of the sample position in the feature map [ p + Δ p + p k ] is obtained by interpolation, [ p + Δ p + p k ] the gray value of the sample position The current pixel position being processed in the input feature map. The offset for sampling the irregular grid. For convolution kernels computed on a grid, For pixels The nearest 1 pixel For pixels The set of the nearest neighboring pixels.

[0030] Furthermore, the processing procedure of the deformable multi-head attention module is as follows:

[0031] Feature maps input to deformable multi-head attention processing According to the formula

[0032]

[0033] The process is performed to obtain the output feature map. ,in

[0034]

[0035] For the parameters that need to be trained, The sampling function is derived from the feature map. Sample location in [ p + Δ p h ] Perform interpolation, such as bilinear or trilinear interpolation, to obtain the sample location. [ p + Δ p h ] grayscale value, It is the first The generation offset of the head. for The hidden dimension.

[0036] Furthermore, the processing procedure of the neighborhood multi-head attention module is as follows:

[0037] Feature maps for the multi-head attention mechanism processing of the input neighborhood According to the formula

[0038]

[0039] Obtain feature maps In, by pixel position Centered The other 8 pixel features within the grid, excluding itself. ;

[0040] According to the formula

[0041]

[0042] compute the center pixel cosine similarity with its neighbor, where represents the pixel feature of pixel position ; compute the weight according to the formula

[0043]

[0044]

[0045] compute the weight according to the formula represents the similarity weight of the pixel feature of pixel position and its neighbor;

[0046] compute the weight according to the formula

[0047]

[0048] compute the reconstructed feature according to the formula represents the pixel feature after reconstruction at pixel position .

[0049] Further, the offset in the deformable multi-head attention module is obtained by passing to a convolutional layer .

[0050] Further, the decoding module adopts a symmetric design with the downsampling module, introduces high-resolution shallow features of the downsampling module through a skip connection manner, and the output layer of the decoding module uses a 1x1 convolution to map the fused features to a category space, generating a segmentation prediction map with the same size as the input image.

[0051] Further, in the segmentation map output module, after obtaining the skeletal structure segmentation map of the pelvis and femur, the segmentation map is subjected to median filter smoothing processing to obtain a smoothed skeletal structure segmentation map.

[0052] Further, in step 2, the process of establishing a training sample is as follows:

[0053] Step 2.1: Obtain a plurality of two-dimensional transverse cross-sectional images of the whole hip joint of a patient, and convert them into an image format that can be input into a deep learning neural network model.

[0054] Step 2.2: Classify and label the two-dimensional transverse cross-sectional images of the whole hip joint obtained in step 2.1; and enhance the training set.

[0055] Further, in step 2.2, the training data is divided according to the degree of acetabular femoral adhesion:​

[0056] (1) Image data of normal acetabulum and femur structure morphology without obvious adhesion, constituting a Normal training set;

[0057] (2) Image data of abnormal conditions such as acetabulum and femur adhesion and structural deformation, together with the Normal training set, constituting a Sick training set.

[0058] Further, in step 3, after completing the segmentation of the pelvic and femoral external structure, threshold segmentation technology is used to automatically identify the femoral marrow cavity region: by analyzing the gray distribution and morphological characteristics of the bone tissue region in the segmentation map, combining morphological operation and region growing algorithm, the bone marrow cavity region in the femur is automatically extracted.

[0059] Advantages

[0060] The full hip joint bone model segmentation and reconstruction method based on the deep learning neural network provided by the application innovatively combines deformable hollow patch embedding and spatial dynamic self-attention mechanism, can adaptively capture the spatial structure characteristics of different bone deformations, balances local precision and global consistency, can enhance the recognition robustness of fuzzy boundaries and multi-morphology targets in medical images; the spatial dynamic self-attention mechanism combines deformable multi-head attention and neighborhood multi-head attention, and balances the modeling ability of local and long-range dependence relationship, adapts to the segmentation requirements of complex hip joint structures, so that the model can more flexibly capture multi-scale and multi-shaped target objects, and effectively enhance the coding ability of the model to irregular deformation structures and multi-scale distribution characteristics.

[0061] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter in the description. BRIEF DESCRIPTION OF DRAWINGS

[0062] The above and / or additional aspects and advantages of the application will become apparent and be readily understood from the following description, taken in connection with the accompanying drawings, in which:

[0063] Figure 1 : Data enhancement effect diagram;

[0064] Figure 2 : Model prediction diagram after data enhancement;

[0065] Figure 3 : Hollow convolution principle diagram;

[0066] Figure 4 : Deep learning neural network architecture;

[0067] Figure 5 : Bone marrow cavity segmentation result;

[0068] Figure 6 : Output the three-dimensional model diagram (Femur.stl model is blue, and Pelvic.stl model is yellow). DETAILED DESCRIPTION

[0069] Embodiments of the present application are described in detail below with reference to examples illustrated in the accompanying drawings, in which the same or similar elements or elements having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0070] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0071] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features referred to. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0072] The full hip joint bone model segmentation and reconstruction method based on deep learning neural network proposed in the embodiment specifically includes the following steps:

[0073] Step 1: Establish a deep learning neural network model for full hip joint bone model segmentation.

[0074] The deep learning neural network model established in the embodiment includes an encoding module, a down-sampling module, a decoding module, and a segmentation map output module.

[0075] Encoding module:

[0076] The encoding module is used to extract feature information, including a feature extraction module, a deformable hole patch embedding module, and a Patch merging module:

[0077] The feature extraction module is used to extract features from the input two-dimensional cross-sectional image of the patient's entire hip joint to obtain a feature map; the feature extraction module is implemented using a convolutional layer, where the parameters are the parameters that need to be trained.

[0078] The deformable hole patch embedding module is implemented using two consecutive deformable hole convolutional layers, which can adaptively adjust the convolution offset to obtain a more flexible convolutional representation while better capturing local features.

[0079] For each deformable dilated convolutional layer, it applies the following formula to each pixel in the input feature map:

[0080] ( f 1 * k ) [ p ] = ∑ p k ∈ Ω ϕ ( f 1 ; p + Δ p + p k ) ⋅ k [ p k ] ,

[0081] Processing is carried out; among which The input feature map is the first deformable dilated convolutional layer. The feature map output by the feature extraction module is used for the second deformable dilated convolutional layer. This is the feature map output by the first deformable dilated convolutional layer. The sampling function is derived from the feature map. Sample location in [ p + Δ p + p k ] Perform interpolation, such as using bilinear or trilinear interpolation, to obtain the sample location. [ p + Δ p + p k ] grayscale value, The current pixel position being processed in the input feature map. The offset is sampled for an irregular grid; the offset can be obtained through a single convolutional layer. get, For dilated convolution; For convolution kernels computed on a grid, For pixels The nearest 1 pixel For pixels The set of nearest neighbor pixels; in this embodiment, The set contains 3 elements.

[0082] The Patch merging module fuses the feature maps output by the deformable hole patch embedding module to output a fused feature map.

[0083] Two consecutive deformable hole patch embedding modules are used in the encoding module to replace the single layer patch embedding layer of the traditional SwinUNet. Through the overlapping patch embedding of two consecutive deformable hole convolution layers, the offset adaptation can be realized and the local features can be extracted more effectively. Compared with the single layer rigid convolution, the deformable hole convolution can adaptively adjust the receptive field, has better modeling ability for local areas with complex geometric shapes and irregular structures, and thus obtains richer and more accurate local representation.

[0084] The present application uses a multi-scale deformable position encoding module to embed the spatial sampling position at different scales, ensures the continuity of the spatial geometric structure and the dissection accuracy in the segmentation process, is particularly suitable for fine modeling of areas with dramatic changes in lesion morphology, so that the model can better adapt to the target changes in the medical image and ensure the accuracy of the spatial relationship.

[0085] Downsampling module:

[0086] After obtaining the fused feature map through the encoding module, the fused feature map is stepwise downsampled, and convolution is used to improve the perception of multi-scale features and better capture the multi-scale spatial information of the medical image. The size of the fused feature map is 512x512, and through four times of downsampling, the output image size of each downsampling stage is 128x128, 64x64, 32x32, 16x16 and 8x8.

[0087] A spatial dynamic self-attention module is used to process the input feature map at each downsampling stage, and the spatial dynamic self-attention module includes a deformable multi-head attention module (DMSA) and a neighborhood multi-head attention module (NMSA). The DMSA is a mechanism improved on the basis of the standard multi-head self-attention mechanism, which uses a deformable sampling strategy to dynamically focus on different regions by learning the offset, thereby enhancing the ability of the model to capture different position information in the input feature map. The NMSA reduces the computational complexity by focusing on the elements in the local neighborhood while maintaining the capture of local features. This spatial dynamic self-attention mechanism takes into account the modeling ability of local and long-range dependencies at the same time, adapts to the segmentation requirements of the complex structure of the hip joint, and enables the model to more flexibly capture multi-scale and multi-shaped target objects.

[0088] The specific processing process of the spatial dynamic self-attention module includes a deformable multi-head attention processing process and a neighborhood multi-head attention mechanism processing process:

[0089] The deformable multi-head attention processing process is:

[0090] The feature map input into the deformable multi-head attention processing process is: According to the formula

[0091]

[0092] performing processing to obtain an output feature map wherein

[0093]

[0094] is a parameter to be trained, is a sampling function, which is obtained by interpolating the sample position in the feature map [ p + Δ p h ] to obtain the gray value of the sample position [ p + Δ p h ] is the generated offset of the th head, is the hidden dimension of .

[0095] The offset in the DMSA can be obtained by passing to the convolutional layer : By using the offset, the DMSA can perform non-uniform sampling on the key positions of the feature map, that is, it can adaptively select more important local regions for attention. This is different from the traditional fixed sampling method, which may not be able to make the optimal sampling selection for different image contents. At the same time, the sampling function is used to process the irregularly sampled features, further enhancing the adaptability of the model to various complex shape and position features, so that the model can better handle irregular local feature patterns.

[0096] The neighborhood multi-head attention mechanism processing procedure is as follows:

[0097] For the feature map input to the neighborhood multi-head attention mechanism processing procedure, the other 8 pixel features in the grid centered at the pixel position

[0098]

[0099] except itself are obtained according to the formula ;

[0100] The center pixel

[0101]

[0102] is calculated according to the formula ​​​​Cosine similarity with its neighborhood where denotes the pixel feature at pixel position ;

[0103] is computed according to the formula

[0104]

[0105] where denotes the similarity weight of the pixel feature at pixel position with its neighborhood

[0106] is computed according to the formula

[0107]

[0108] where denotes the reconstructed pixel feature at pixel position .

[0109] The neighborhood multi-head attention mechanism is used to realize neighborhood feature reconstruction in the bone segmentation task, which not only improves the expression ability of local features, but also provides more stable support for the recognition of structure boundaries. In the bone segmentation task, the original features may be disturbed by blurred boundaries, tissue heterogeneity or imaging artifacts. In order to effectively alleviate these problems, the neighborhood weighted similarity feature reconstruction mechanism is used in the neighborhood multi-head attention mechanism processing process, the feature updating process is guided by explicitly modeling the semantic similarity between pixels, so as to improve the robustness of the model.

[0110] The deformable hole patch embedding and spatial dynamic self-attention mechanism are combined in the encoding module and the downsampling module, so that the global modeling capability is significantly enhanced while ensuring the local detail capturing capability, and the medical image segmentation precision is improved. In the downsampling stage, the overlapping convolution is used to replace the traditional patch-merging mode, the overlapping convolution kernel can more fully fuse the information of adjacent regions, avoid the loss of local information caused by simple patch merging, and be helpful to the accurate learning and utilization of image features by the subsequent network.

[0111] The decoding module:

[0112] After multi-scale feature extraction by the encoding module and the down-sampling module, the image is restored and the target structure is predicted by the decoding module. The decoding module is designed symmetrically with the down-sampling module, and combines a multi-scale feature fusion mechanism to introduce high-resolution shallow features of the down-sampling module through a skip-connection mode, effectively compensating for the possible loss of detail information in the image reconstruction process and enhancing the ability to restore anatomical boundaries.

[0113] The decoding module gradually restores the feature map size through multiple transposed convolutions, and fuses the down-sampled features of the corresponding stage. This structure not only preserves spatial information, but also improves the model's ability to adapt to multi-scale skeletal morphology in images. The final output layer uses a 1x1 convolution to map the fused features to the class space, generating a segmentation prediction map of the same size as the input image, improving the model's convergence speed and enhancing robustness.

[0114] Segmentation map output module:

[0115] The segmentation map output module uses an efficient prediction head composed of a convolution block and a residual block to splice the feature map output by the decoding module, and outputs to the last convolution layer. After processing, the class label of each pixel is output to form a labeled medical image segmentation map.

[0116] In this embodiment, a median filter smoothing process is also added in the segmentation map output module, that is, after obtaining the skeletal structure segmentation map of the pelvis and femur, the median filter is applied to the segmentation map to output a smoothed segmentation map. The median filter algorithm is as follows:

[0117] Assume that the input image is I(x, y), the output image is O(x, y), and the filter window size is n x n (usually an odd number, such as 3 x 3, 5 x 5, etc.). For each pixel (x, y) in the image, define its neighborhood as:

[0118]

[0119] That is, the set of all pixel values in the n x n neighborhood centered at (x, y).

[0120] Then sort the pixel values in the neighborhood to get the sorted list S, and take the median value as the pixel value of the corresponding position of the output image:

[0121]

[0122] By taking the median value of the neighborhood, the median filter can better preserve the edge details of the image, while eliminating irregular or jagged edges caused by imaging noise, improving the continuity of the structure and the visual effect.

[0123] Step 2: Establish training samples and train the deep learning neural network model established in step 1.

[0124] Step 2.1: Obtain several patient full hip joint two-dimensional cross-sectional images and convert them into image formats that can be input into the neural network.

[0125] Since the original data of the patient's full hip joint two-dimensional cross-sectional images is DICOM data, and the neural network input requires PNG format, the Mimics software is used to output the patient's original DICOM data as BMP format pictures, and then batch convert the BMP format files to PNG format pictures.

[0126] Step 2.2: Classify and label the full hip joint two-dimensional cross-sectional images obtained in step 2.1.

[0127] Considering that the acetabulum-femur structure of patients varies significantly due to disease types and adhesion levels, a unified model may lead to recognition bias, so the training data is divided into:

[0128] (1) Image data with normal acetabulum and femur structure and no obvious adhesion, constituting the Normal training set;

[0129] (2) Image data with acetabulum and femur adhesion and structural deformation, together with the Normal training set, constituting the Sick training set.

[0130] Use three-dimensional image processing software (such as Mimics) to slice and label patient CT image data, the specific process is as follows:

[0131] (1) Label the pelvic region and femoral region in each hip joint two-dimensional cross-sectional image separately;

[0132] (2) Label the pelvic region as yellow (RGB value 255, 255, 0) and the femoral region as blue (RGB value 0, 0, 255);

[0133] (3) Export the labeled image as BMP format and batch convert it to PNG format as training label.

[0134] And in order to enhance the robustness and generalization ability of the model to image interference, a variety of enhancement operations are performed on the training images and their corresponding labels simultaneously, including Gaussian blur, image brightness adjustment, horizontal and vertical flip, affine transformation, etc. Among them, Gaussian blur only acts on the image, and other enhancement operations are applied to the image and the corresponding label simultaneously, obtaining the enhanced Normal training set and the enhanced Sick training set, thereby simulating the imaging variability under real clinical conditions and improving the stability of the model.

[0135] The deep learning neural network model established in step 1 is trained using the enhanced Normal training set and the enhanced Sick training set obtained above to enhance the model's recognition robustness for hip joint images under different structural conditions, and finally a trained deep learning neural network model is obtained.

[0136] Step 3: Using the trained deep learning neural network model obtained in step 2, the patient's hip joint CT cross-sectional images to be processed are recognized and segmented to obtain a two-dimensional image sequence of the pelvis and femur after segmentation:

[0137] (1) The patient's hip joint CT cross-sectional images to be processed are input into the trained deep learning neural network model, which automatically recognizes and completes segmentation.

[0138] (2) The segmented two-dimensional cross-sectional images are output, which include the clearly marked pelvis region and femur region.

[0139] In this embodiment, after completing the segmentation of the pelvis and femur shape structure, further threshold segmentation technology is used for automatic recognition of the femoral marrow cavity region: by analyzing the gray scale distribution and morphological characteristics of the bone tissue region in the segmentation image, combining morphological operation and region growing algorithm, the femoral marrow cavity region is automatically extracted.

[0140] Select the CT image sequence containing the marrow cavity, and use the femur mask obtained by segmentation as the bone segmentation region. Through threshold segmentation technology, the region of the marrow cavity is accurately extracted, providing key geometric information for subsequent prosthesis design, and outputting the final bone segmentation label image with marrow cavity. The specific steps are as follows:

[0141] (2.1) Determine the segmentation target: first, determine the femur region as the segmentation region of interest through the femur mask. This step can help exclude other interference regions and ensure that the segmentation process focuses on the region where the marrow cavity is located.

[0142] (2.2) Gray value analysis: by analyzing the gray histogram of the image, the gray mean values of the foreground and background regions under different thresholds are calculated respectively. In CT images, the gray value of the marrow cavity is low, and it is usually significantly different from the high gray value of the bone. Therefore, selecting an appropriate threshold can maximize the gray difference between the foreground and background.

[0143] (2.3) Maximize the difference: when selecting the threshold, the gray mean difference under different thresholds is calculated to optimize the segmentation effect. Specifically, this difference can be quantified by calculating the variance of the foreground region and the background region. The larger the variance, the more obvious the gray difference between the foreground and the background, and the more ideal the segmentation effect.

[0144] (2.4) Select the best threshold: by calculating the variance difference of foreground and background under different thresholds, find the threshold that maximizes the variance of foreground and background regions. This threshold can effectively separate the bone marrow cavity from the bone, and maximize the contrast between foreground and background, so as to obtain a clear bone marrow cavity region.

[0145] (3) Save the segmentation result in a medical image special format (such as.dcm), and arrange it according to the original CT image sequence.

[0146] (4) At the same time, generate corresponding.png format images for subsequent image visualization and reconstruction processing.

[0147] Step 4: Based on the completed segmentation of the pelvic and femoral two-dimensional image sequence, adopt volume rendering (volumerendering) and other medical image three-dimensional reconstruction techniques to perform full hip joint image three-dimensional modeling:

[0148] Extract the gray value of the different color marked regions in the image, and extract the pelvic and femoral regions respectively; import the obtained pelvic and femoral segmentation results into the three-dimensional modeling module, and reconstruct a complete three-dimensional bone model according to the continuous cross-sectional images. The three-dimensional modeling process adopts MarchingCubes reconstruction algorithm to convert two-dimensional segmentation graph into voxel model, and further generates triangular facet structure. Through smoothing processing and topological optimization of the model, ensure that the generated model has continuity and smoothness on the basis of maintaining the accuracy of anatomical structure, and output as STL format, including Pelvic.stl and Femur.stl two files, convenient for subsequent prosthesis matching, mechanical simulation and other operations.

[0149] The present application combines intelligent recognition of bone marrow cavity and three-dimensional reconstruction process, realizes the automatic process from two-dimensional image input to precise three-dimensional bone structure output, effectively improves the modeling efficiency and accuracy, significantly reduces the manual intervention, and has good clinical practical value and popularization potential.

[0150] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the principles and purposes of the present application within the scope of the present application.

Claims

1. A deep learning neural network-based total hip bone model segmentation and reconstruction method, characterized in that: The method comprises the following steps: Step 1: a deep learning neural network model for full hip joint bone model segmentation is established; the deep learning neural network model comprises an encoding module, a down-sampling module, a decoding module and a segmentation map output module; The encoding module is used for extracting feature information, comprising a feature extraction module, a deformable hole patch embedding module and a Patch merging module; the input patient full hip joint two-dimensional cross-sectional image is sequentially processed by the feature extraction module, the deformable hole patch embedding module and the Patch merging module to obtain a fused feature map; The down-sampling module gradually down-samples the fused feature map, and a spatial dynamic self-attention module is used to process the input feature map at each down-sampling stage; the spatial dynamic self-attention module comprises a deformable multi-head attention module and a neighborhood multi-head attention module; The decoding module performs image restoration and target structure prediction; The segmentation map output module splices the feature map output by the decoding module, and outputs the class label of each pixel after convolution processing to form a labeled medical image segmentation map; Step 2: training samples are established to train the deep learning neural network model established in step 1; Step 3: the trained deep learning neural network model obtained in step 2 is used to identify and segment the patient hip joint CT cross-sectional image to be processed to obtain a two-dimensional image sequence of the pelvis and femur after segmentation; Step 4: based on the two-dimensional image sequence of the pelvis and femur after segmentation, a full hip joint image three-dimensional modeling is performed through a medical image three-dimensional reconstruction technology.

2. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 1, characterized in that: The deformable hole patch embedding module uses two consecutive deformable hole convolution layers to realize the processing; for each deformable hole convolution layer, each pixel point in the input feature map is processed according to the formula: The processing process of the deformable multi-head attention module is as follows: wherein is the input feature map, wherein for the first deformable dilated convolution layer, is the feature map output by the feature extraction module, wherein for the second deformable dilated convolution layer, is the feature map output by the first deformable dilated convolution layer; is a sampling function, which obtains the gray value of the sample position by interpolating the sample position in the feature map , is the current processing pixel point position in the input feature map, is the offset of the irregular grid sampling, is the convolution kernel for calculation on the grid, is the pixel point most adjacent to the pixel point , is the pixel point most adjacent to the pixel point , is the set of pixel points most adjacent to the pixel point .

3. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 1, characterized in that: The processing process of the neighborhood multi-head attention module is as follows: Feature maps are input to a deformable multi-head attention processing procedure According to the formula processing to obtain an output feature map wherein is a parameter to be trained, is a sampling function, which obtains the gray value of the sample position by interpolating the sample position in the feature map , is the generation offset of the th head, is the hidden dimension of .

4. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 3, characterized in that: According to the formula Feature maps for input neighborhood multi-head attention mechanism processing procedure , according to the formula acquiring feature maps with a pixel position at the center of the other 8 pixel features within the grid, except for itself ; According to the formula Calculate the center pixel Its first a neighbor The cosine similarity, where Indicates pixel position Pixel features; According to the formula Calculate the weights, where Indicates pixel position Pixel features and the first The similarity weight of each neighbor; The decoding module is designed symmetrically with the down-sampling module, high-resolution shallow features of the down-sampling module are introduced through a jump connection mode, and a 1x1 convolution is used in the output layer of the decoding module to map the fused features to a class space to generate a segmentation prediction map with the same size as the input image. The reconstructed feature is computed, where represents the pixel feature after reconstruction at pixel position represents the pixel feature after reconstruction at pixel position 5. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 3, characterized in that: The offset in the deformable multi-head attention module is obtained by passing to a convolutional layer is obtained.

6. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 1, characterized in that: In the segmentation map output module, after obtaining the bone structure segmentation map of the pelvis and femur, the segmentation map is subjected to median filter smoothing processing to obtain a smoothed bone structure segmentation map.

7. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 1, characterized in that: In step 2, the process of establishing training samples is as follows:

8. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 1, characterized in that: Step 2.1: a plurality of patient full hip joint two-dimensional cross-sectional images are obtained and converted into an image format that can be input into the deep learning neural network model; Step 2.2: the full hip joint two-dimensional cross-sectional images obtained in step 2.1 are classified and labeled; and the training set is enhanced. In step 2.2, the training data is divided into:

9. The deep learning neural network-based total hip bone model segmentation and reconstruction method according to claim 8, characterized in that: (1) image data with normal acetabulum and femur structure and no obvious adhesion, constituting a Normal training set; ​ (2) The image data of abnormal conditions such as the cohesiveness and structural deformation of the acetabulum and femur, together with the normal training set, constitute the sick training set.

10. The full hip bone model segmentation and reconstruction method based on deep learning neural network according to claim 1, characterized in that: In step 3, after the segmentation of the pelvic and femoral shape structure is completed, the threshold segmentation technique is used to automatically identify the femoral marrow cavity region: by analyzing the gray distribution and morphological characteristics of the bone tissue region in the segmentation graph, combining morphological operation and region growing algorithm, the femoral marrow cavity region is automatically extracted.

Images