End-to-end orthopedic medical image segmentation and three-dimensional reconstruction method and system, electronic equipment and computer readable medium

Abstract

The invention discloses an end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method and system, electronic equipment and a computer readable medium, and belongs to the field of medical image processing, and the method comprises the following steps: S1, obtaining orthopedic three-dimensional medical image data, and carrying out standardized preprocessing to obtain a standard medical image set; s2, constructing an improved 3D (three dimensional)-Trans UNet network architecture; s3, performing network training based on a multi-task cooperative training strategy; s4, performing post-processing optimization on the three-dimensional model to obtain a segmentation body mask in a standard format and a regularized three-dimensional model; and S5, system integration and clinical deployment. By adopting the end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method and system, the electronic equipment and the computer readable medium, integrated efficient execution of orthopedic medical image high-precision segmentation and regular three-dimensional reconstruction is realized.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to end-to-end orthopedic medical image segmentation and three-dimensional reconstruction methods, systems, electronic devices and computer-readable media. Background Technology

[0002] In the current precision medicine system, medical image segmentation, as the core hub connecting image data and clinical decision-making, is a key prerequisite technology for achieving accurate disease diagnosis, personalized treatment planning, and intraoperative navigation. Among various segmentation frameworks, U-Net and its derivative architectures based on convolutional neural networks (CNNs), with their symmetrical encoder-decoder structure and cross-level skip connection design, can effectively fuse low-level spatial details and high-level semantic information, and have become the mainstream solution in the field of medical image segmentation, demonstrating excellent performance in organ and soft tissue image segmentation tasks. However, the core computational unit of CNNs (convolution operation) has inherent local receptive field characteristics, and its feature capture range is limited by the size of the convolution kernel, making it difficult to model long-distance spatial dependencies in images—a limitation that is particularly prominent in orthopedic image processing. Skeletal tissue not only has complex anatomical structures (such as the axial extension of long bones, the irregular curvature of articular surfaces, and the segmental arrangement of the spine), but also exhibits significant individual morphological variations. At the same time, lesions such as fractures and tumors further disrupt structural continuity, leading to problems such as boundary fragmentation and loss of anatomical structural integrity in traditional CNN segmentation, which cannot meet the stringent requirements for segmentation accuracy in orthopedic diagnosis and treatment.

[0003] In orthopedic clinical practice, 3D reconstruction technology based on modalities such as CT and MRI is a core technological support for preoperative surgical planning, intraoperative real-time navigation, and personalized implant design and manufacturing. Its reconstruction accuracy directly determines the safety and effectiveness of the treatment plan. Currently, mainstream orthopedic 3D reconstruction methods can be divided into two main categories: The first category is reconstruction schemes based on traditional image processing algorithms, represented by threshold segmentation and region growing methods. These methods rely on manual experience to set grayscale thresholds or seed points, using pixel / voxel grayscale similarity for region segmentation and surface reconstruction. However, noise, artifacts, and soft tissue interference commonly found in skeletal images lead to extremely poor robustness in threshold selection. Furthermore, these methods struggle to adapt to individual differences in skeletal anatomy, resulting in edge distortion in the reconstruction results. They also have low automation, requiring extensive manual post-processing corrections, which is not only inefficient but also introduces subjective errors due to differences in operator experience.

[0004] The second category is reconstruction schemes based on deep learning. Although these schemes break through the reliance on manual intervention in traditional methods and achieve a certain degree of end-to-end automation, the current mainstream schemes still do not break away from the serial process of "segmentation first, reconstruction later". Because errors in the segmentation stage (such as boundary offset and omission of small structures) are directly transmitted to the subsequent reconstruction stage and are further accumulated and amplified during the surface reconstruction process, the three-dimensional reconstruction model will eventually have serious artifacts such as holes, burrs, and structural discontinuities. These defects will directly affect the fitting accuracy of implants and bones, the reliability of preoperative planning, and the accuracy of surgical navigation, making it difficult to meet the clinical needs of precision diagnosis and treatment in orthopedics.

[0005] In summary, existing technologies lack targeted optimization for orthopedic images (complex anatomical structures, individual morphological variations, and lesion interference), and have failed to solve the problem of synergistic optimization of segmentation and reconstruction tasks. They have not yet formed an integrated solution for orthopedic image segmentation and three-dimensional reconstruction that takes into account high precision, high automation, and strong robustness, making it difficult to meet the clinical application requirements of precise orthopedic diagnosis and treatment. Summary of the Invention

[0006] The purpose of this invention is to provide an end-to-end method, system, electronic device, and computer-readable medium for orthopedic medical image segmentation and three-dimensional reconstruction, thereby solving the aforementioned technical problems.

[0007] To achieve the above objectives, this invention provides an end-to-end method for orthopedic medical image segmentation and three-dimensional reconstruction, comprising the following steps: S1. Acquisition and Standardization Preprocessing of Orthopedic 3D Medical Imaging Data: Acquire the original medical images of the patient's orthopedic parts, and perform standardization, isotropic resampling, and region of interest cropping in sequence after layering and annotation to obtain a standard medical image set. S2. Construct an improved 3D-Trans UNet network architecture. The improved 3D-Trans UNet network is an end-to-end architecture that integrates 3D convolutional local feature extraction, Transformer global dependency modeling, and dual-task parallel output. It consists of a 3D CNN encoder, a 3D Patch embedding and position encoding module, an improved 3D Transformer encoder, a segmentation decoder, and a reconstruction decoder cascaded in sequence. S3. Network training based on multi-task collaborative training strategy: The improved 3D-Trans UNet network constructed in S2 is trained using a standard medical image set, and the network parameters are optimized using a multi-task collaborative loss function. S4. Post-processing optimization of 3D model: Post-process the segmentation volume data and 3D mesh output by the forward propagation of the improved 3D-Trans UNet network after training convergence to obtain a standard format segmentation volume mask and a regularized 3D model. S5. System Integration and Clinical Deployment: The standardized 3D model is encapsulated into a standardized inference engine and integrated into the hospital PACS system or independent medical workstation. After receiving DICOM sequence data, the inference engine automatically executes S1-S4. After processing, the original image, segmentation overlay image and interactive 3D model are synchronously displayed on the system interface.

[0008] Preferably, step S1 specifically includes the following steps: S11. Obtain CT images or MRI three-dimensional tomographic images of the patient's orthopedic areas; S12, Layered labeling; S121, First layer annotation: Medical image annotators perform voxel-level annotation on the skeletal structure contour of the original medical image on a 3D medical image annotation platform to form a complete three-dimensional mask as the basic ground truth for the segmentation task. S122, Second layer of annotation: Based on the first layer of annotation, orthopedic surgeons or experts further distinguish important anatomical substructures inside the bone; S123. Correction: The medical images after the second layer of annotation are reviewed and corrected by radiologists; S13. Data Standardization Processing: When selecting CT images, the HU values ​​of the CT images are mapped to a unified target range through linear transformation. ; In the formula, This represents the standardized voxel value; Indicates the original voxel value; and These represent the minimum and maximum values ​​of the HU value, respectively. and These represent the lower and upper bounds of the objective normalization, respectively; S14. Isotropic resampling processing: The cubic spline interpolation method is used to resample the original anisotropic voxel size to the preset isotropic voxel size to ensure consistent resolution in three directions in three-dimensional space. S15. Region of Interest (ROI) cropping: Threshold setting based on the high HU value characteristics of the skeleton. Binarization is performed to obtain the initial mask. : ; Then set the initial mask The three-dimensional bounding box is expanded outward by a preset number of voxels along each dimension to obtain the final clipping region; S16. Dataset partitioning operation: The preprocessed dataset is randomly divided into training set, validation set and test set in a ratio of 7:2:1.

[0009] Preferably, in step S14, the voxel size after the normalization process in S13 is set to... The target isotropic size is Then for any target location Its coordinates in the original medical image ; And targeting When the corresponding original medical image has non-integer coordinates, the target location is obtained by selecting the surrounding voxel points and weighting the intensity values ​​of these adjacent voxels using a cubic spline interpolation algorithm. The corresponding voxel intensity value.

[0010] Preferably, the 3D CNN encoder described in step S2 is divided into multiple coding levels, each containing two 3D residual units, which are used to achieve downsampling through 3D convolution, batch normalization, and ReLU activation to obtain multi-scale local feature maps. , These represent the local feature maps of layers 1, 2, 3, and 4, respectively. Set input voxel data ,Right now The output local feature map The expressions for the changes in size and number of channels are as follows: ; In the formula, Indicates the first Layer local feature map; This indicates the second-level residual downsampling operation; , and These represent the depth, height, and width of a three-dimensional medical image, respectively. This represents the encoding level of the 3D CNN encoder, and ; The 3D Patch embedding and positional encoding module is used to extract the deepest feature maps from the output of the 3D CNN encoder. Extract the size as Non-overlapping 3D blocks are obtained, and after flattening them, a sequence is obtained through linear projection. ,in, Number of blocks The projection dimension is then followed by a learnable 3D position code. This encoding is generated through an independent learnable lookup table, which enhances the continuity of the Z-axis and yields the initial feature sequence. ; The improved 3D Transformer encoder employs an axial attention mechanism for the initial feature sequence of the input. Three unidirectional subsequences were obtained by rearranging the orthogonal anatomical directions of the orthopedic images. , and Then, self-attention operations are performed on each unidirectional subsequence to obtain the attention outputs for the three orthogonal anatomical directions. , and Finally , and The intermediate feature sequence is formed by concatenating the data along the channel dimension. After processing by a feedforward network and layer normalization, the output is a feature sequence that fuses global dependencies in three directions. : ; In the formula, Presentation layer normalization operation; Indicates a feedforward network; This indicates a channel splicing operation; Taking axial-x attention as an example, for a fixed sequence of coordinates Its self-attention output is: ; In the formula, , and These represent the query vector, key vector, and value vector of the self-attention mechanism, respectively, which are derived from... Obtained through linear transformation; A segmentation decoder is used to segment feature sequences. Reconstruct the 3D feature map while ensuring its size matches that of the deepest feature map. The dimensions are consistent, and then the first upsampling is performed through 3D transposed convolution, while the number of channels is adjusted through convolution to make it consistent with the original size. Channel number matching is performed, and then the upsampled feature map is compared with the feature map of the same resolution layer of the 3D CNN encoder. Perform channel splicing and fuse feature sequences. Global semantic features and feature maps The local detailed features are repeatedly processed to gradually restore the feature map to a resolution consistent with the preprocessed CT image; finally, a voxel-level multi-class segmentation mask is output through 1×1×1 3D convolution and the Softmax activation function; and the upsampling operation expression is as follows: ; In the formula, and These represent the segment decoder's... Level and First Feature map after level upsampling; This represents a 3D transpose convolution operation; This represents a 1×1×1 3D convolution operation; Indicates channel splicing; Reconstruction decoder, used to convert feature sequences Inputting a fully connected layer, the vertex coordinate matrix of the initial coarse mesh is predicted. Patch connection matrix The initial geometric shape of the skeleton was constructed. Indicates the initial number of vertices; then... As input, the vertex coordinates are iteratively optimized through multi-level graph convolutional layers, and the final refined 3D mesh model is output. ; Among them, the The optimized calculation formula for vertex coordinates of a level graph convolutional layer is as follows: ; ; In the formula, Indicates the first The vertex coordinate matrix output by the level graph convolutional layer; Indicates the activation function; Degree matrix; Indicates the first Add self-connected adjacency matrices at each level; Indicates the first The vertex coordinate matrix output by the level graph convolutional layer; Indicates the first Learnable weights of hierarchical graph convolutional layers; Represents the original adjacency matrix; Represents the identity matrix.

[0011] Preferably, the expression for the multi-task collaborative loss function in step S3 is as follows: ; in, ; ; ; In the formula, Indicates the total loss; , and This represents the dynamic weight of the loss for each task within the total loss; Indicates the segmentation loss; Indicates the losses incurred during reconstruction; Indicates the regularization loss; Indicates Dice loss, and , This represents the number of categories in the segmentation task. Indicates the first Individual elements belong to the first Predicted probability of class Indicates the first Individual elements belong to the first The true label of a class, i.e., its basic truth value. Represents a constant; This represents the balancing hyperparameter between Dice loss and weighted cross-entropy loss; This represents the weighted cross-entropy loss, and , Indicates the total number of voxels. Indicates the first Class weights; and The hyperparameter representing the balance between chamfer distance and normal consistency loss; This represents the chamfer distance loss, and , and Let represent the vertex set of the predicted mesh and the vertex set of the real mesh, respectively. Represents vertices With vertex Euclidean distance; This represents the loss of normal consistency, and , and Representing vertices respectively and vertex The normal vector, Represents vertices The set of adjacent vertices; and This represents the hyperparameter that balances the edge length regularization loss and the Laplace smoothing loss. Let represent the loss due to edge length regularization, and , Indicates the first The first piece of the face Edge, Indicates the preset target side length; This indicates the Laplace smoothing loss, and , and Representing vertices respectively and vertex The coordinates; During training, a dynamic adjustment strategy based on the normalization of the magnitude of each loss term is adopted to balance the training weights of the segmentation task, reconstruction task and mesh quality constraints, and to iteratively optimize the network parameters until the model converges.

[0012] Preferably, step S4 specifically includes the following steps: S41. Mesh simplification and optimization using an edge collapse algorithm: Calculate the folding cost of each edge based on a quadratic error metric. Iterate through the edges that minimize the cost until the number of mesh patches is reduced to the target value. Furthermore, the Hausdorff distance between the simplified mesh and the original mesh is lower than a preset threshold. ; S42. Format standardization operation: Outputs the simplified 3D mesh as a standard 3D model file in STL, PLY, or OBJ format, and outputs the segmentation volume data as a segmentation mask file in NIFTI format.

[0013] A system of end-to-end orthopedic medical image segmentation and 3D reconstruction methods, including: Data acquisition and standardization preprocessing module: used to acquire the original medical images of the patient's orthopedic parts, and after hierarchical annotation, perform standardization, isotropic resampling and region of interest cropping in sequence to obtain a standard medical image set; An improved 3D-Trans UNet network architecture building module is used to construct an improved 3D-Trans UNet network architecture building module consisting of a 3D CNN encoder, a 3D Patch embedding and position encoding module, an improved 3D Transformer encoder, a segmentation decoder, and a reconstruction decoder cascaded in sequence. The network training module is used to train the improved 3D-Trans UNet network using a standard medical image set and to optimize the network parameters using a multi-task collaborative loss function. The 3D model post-processing optimization module is used to post-process the segmentation volume data and 3D mesh output by the forward propagation of the improved 3D-Trans UNet network after training convergence, to obtain a standard format segmentation volume mask and a regularized 3D model. The system integration and clinical deployment module is used to encapsulate the regularized 3D model into a standardized inference engine, which is then integrated into the hospital PACS system or a standalone medical workstation. After receiving DICOM sequence data, the inference engine automatically executes the process and displays the original image, segmentation overlay image, and interactive 3D model on the system interface.

[0014] An electronic device, characterized in that it comprises: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement an end-to-end orthopedic medical image segmentation and 3D reconstruction method.

[0015] A computer-readable medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements an end-to-end method for orthopedic medical image segmentation and three-dimensional reconstruction.

[0016] Therefore, the present invention, employing the aforementioned end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method, system, electronic device, and computer-readable medium, has the following beneficial effects: 1. Significantly improved segmentation accuracy: The improved 3D-Trans UNet network integrates the local anatomical details of 3D CNN with the global structural dependency features of 3D Transformer. The axial attention mechanism accurately models the long-distance spatial relationship of bones. Combined with Dice-weighted cross-entropy hybrid loss to solve the class imbalance problem, it achieves voxel-level accurate segmentation of fine bone structures such as cortical bone, cancellous bone, and articular surfaces, and greatly improves the segmentation Dice coefficient and cross-union ratio. 2. 3D Reconstruction Quality and Regularity Optimization: An end-to-end reconstruction mode is adopted to avoid the transmission of segmentation errors to the reconstruction stage. The mesh vertices are optimized through multi-level graph convolution iteration, and loss constraints such as edge length regularization and Laplacian smoothing are used to output a 3D mesh with regular topology, smooth surface, and geometric accuracy that meets clinical requirements. This solves the problems of burrs, distortion, and irregular patches in traditional reconstruction meshes and is suitable for clinical applications such as 3D printing and surgical navigation. 3. Improved computational efficiency and lightweight model: Axial attention is used instead of traditional 3D full attention, which greatly reduces the computational complexity and number of parameters of Transformer; combined with CT image ROI cropping preprocessing to reduce redundant data input, the speed of model training and inference is improved, the hardware computing power requirements are reduced, and it is suitable for real-time clinical processing scenarios. 4. Enhanced data robustness and generalization ability: By standardizing HU values ​​of CT images and performing isotropic resampling preprocessing, data heterogeneity caused by different scanning equipment and protocols is eliminated; based on training on multi-center, multi-site orthopedic datasets, the model's generalization ability to heterogeneous data and complex skeletal anatomical variations is significantly improved, adapting to clinical data from different hospitals and different equipment. 5. Improved clinical adaptability and ease of application: Outputs NIFTI format segmentation mask and STL / PLY / OBJ standard 3D model files, which can be directly connected to hospital PACS systems, 3D printing equipment, surgical navigation systems and preoperative planning software without additional format conversion and secondary processing, simplifying clinical workflow and meeting clinical needs such as bone quantitative assessment, personalized implant design and preoperative simulation. 6. Multi-task collaborative performance balance optimization: A multi-task collaborative training strategy of segmentation-reconstruction-regularization is adopted to achieve feature sharing and mutual constraints between tasks, avoid the performance degradation of another task caused by the optimization of a single task, and simultaneously improve segmentation accuracy and reconstruction quality, so as to achieve the best overall performance of the end-to-end process.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart of the end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method described in this invention; Figure 2 The original unlabeled data of the original medical image training set described in the embodiment (only for illustration purposes; the actual training data covers multiple parts such as the spine, hip, and knee), where (a) is a lateral medical image of the upper limb, (b) is an anteroposterior medical image of the hand, (c) is a lateral medical image of the elbow joint, (d) is an oblique medical image of the hand joint, and (e) is an anteroposterior medical image of the fingers. Figure 1 (f) is an anteroposterior medical image of the wrist joint; (g) is an anteroposterior medical image of a long bone; (h) is an oblique medical image of the forearm; (i) is a lateral medical image of the wrist joint; (j) is an anteroposterior medical image of the shoulder joint; (k) is an anteroposterior medical image of the finger joint. Figure 2 (l) is a lateral medical image of a finger joint; Figure 3 This is a voxel-level multi-class segmentation result of the hand CT image described in the embodiment; Figure 4 This is a 3D mesh model of the hand generated based on the segmentation results, as described in the embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0020] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as a process, method, system, product, or server that includes a series of steps or units, not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] like Figure 1 As shown, the end-to-end orthopedic medical image segmentation and 3D reconstruction method includes the following steps: S1. Acquisition and Standardization Preprocessing of Orthopedic 3D Medical Imaging Data: Acquire the original medical images of the patient's orthopedic parts, and perform standardization, isotropic resampling, and region of interest cropping in sequence after layering and annotation to obtain a standard medical image set. S2. Construct an improved 3D-Trans UNet network architecture. The improved 3D-Trans UNet network is an end-to-end architecture that integrates 3D convolutional local feature extraction, Transformer global dependency modeling, and dual-task parallel output. It consists of a 3D CNN encoder, a 3D Patch embedding and position encoding module, an improved 3D Transformer encoder, a segmentation decoder, and a reconstruction decoder cascaded in sequence. S3. Network training based on multi-task collaborative training strategy: The improved 3D-Trans UNet network constructed in S2 is trained using a standard medical image set, and the network parameters are optimized using a multi-task collaborative loss function. S4. Post-processing optimization of 3D model: Post-process the segmentation volume data and 3D mesh output by the forward propagation of the improved 3D-Trans UNet network after training convergence to obtain a standard format segmentation volume mask and a regularized 3D model. S5. System Integration and Clinical Deployment: The standardized 3D model is encapsulated into a standardized inference engine and integrated into the hospital PACS system or independent medical workstation. After receiving DICOM sequence data, the inference engine automatically executes S1-S4. After processing, the original image, segmentation overlay image and interactive 3D model are synchronously displayed on the system interface.

[0023] Step S1 specifically includes the following steps: S11. Obtain CT images or MRI three-dimensional tomographic images of the patient's orthopedic areas; S12, Layered labeling; S121, First layer annotation: Medical image annotators perform voxel-level annotation on the skeletal structure outline of the original medical image on a 3D medical image annotation platform to form a complete three-dimensional mask as the basic ground truth for the segmentation task; this layer of annotation focuses on the overall shape and spatial position of the bone. S122, Second Layer Annotation: Building upon the first layer of annotation, orthopedic surgeons or experts further differentiate important anatomical substructures within the bone. These important substructures include, but are not limited to: the distinction between cortical and cancellous bone, the marking of articular surface regions, and the delineation of internal cavities (such as the medullary cavity). This layer of annotation not only improves the precision of segmentation but also provides richer semantic information for subsequent high-quality 3D reconstruction. S123. Correction: The medical images after the second layer of annotation are reviewed and corrected by radiologists; S13. Data Standardization Processing: When selecting CT images, the HU values ​​of the CT images are mapped to a unified target range through linear transformation. ; In the formula, This represents the standardized voxel value; Indicates the original voxel value; and These represent the minimum and maximum values ​​of the HU value, respectively. and These represent the lower and upper bounds of the objective normalization, respectively; S14. Isotropic resampling processing: The cubic spline interpolation method is used to resample the original anisotropic voxel size to the preset isotropic voxel size to ensure consistent resolution in three directions in three-dimensional space. S15. Region of Interest (ROI) cropping: Threshold setting based on the high HU value characteristics of the skeleton. Binarization is performed to obtain the initial mask. : ; Then set the initial mask The three-dimensional bounding box is expanded outward by a preset number of voxels along each dimension to obtain the final clipping region; S16. Dataset partitioning operation: The preprocessed dataset is randomly divided into training set, validation set and test set in a ratio of 7:2:1.

[0024] In step S14, the voxel size after the normalization process in S13 is set to... The target isotropic size is Then for any target location Its coordinates in the original medical image ; And targeting When the corresponding original medical image has non-integer coordinates, the target location is obtained by selecting the surrounding voxel points and weighting the intensity values ​​of these adjacent voxels using a cubic spline interpolation algorithm. The corresponding voxel intensity value.

[0025] The 3D CNN encoder described in step S2 is divided into multiple coding levels, each containing two 3D residual units, which are used to achieve downsampling through 3D convolution, batch normalization, and ReLU activation to obtain multi-scale local feature maps. , These represent the local feature maps of layers 1, 2, 3, and 4, respectively. Set input voxel data ,Right now The output local feature map The expressions for the changes in size and number of channels are as follows: ; In the formula, Indicates the first Layer local feature map; This indicates the second-level residual downsampling operation; , and These represent the depth, height, and width of a three-dimensional medical image, respectively. This represents the encoding level of the 3D CNN encoder, and ; The 3D Patch embedding and positional encoding module is used to extract the deepest feature maps from the output of the 3D CNN encoder. Extract the size as Non-overlapping 3D blocks are obtained, and after flattening them, a sequence is obtained through linear projection. ,in, Number of blocks The projection dimension is then followed by a learnable 3D position code. The initial feature sequence is obtained. ; The improved 3D Transformer encoder employs an axial attention mechanism for the initial feature sequence of the input. Three unidirectional subsequences were obtained by rearranging the orthogonal anatomical directions of the orthopedic images. , and Then, self-attention operations are performed on each unidirectional subsequence to obtain the attention outputs for the three orthogonal anatomical directions. , and Finally , and The intermediate feature sequence is formed by concatenating the data along the channel dimension. After processing by a feedforward network and layer normalization, the output is a feature sequence that fuses global dependencies in three directions. : ; In the formula, Presentation layer normalization operation; Indicates a feedforward network; This indicates a channel splicing operation; A segmentation decoder is used to segment feature sequences. Reconstruct the 3D feature map while ensuring its size matches that of the deepest feature map. The dimensions are consistent, and then the first upsampling is performed through 3D transposed convolution, while the number of channels is adjusted through convolution to make it consistent with the original size. Channel number matching is performed, and then the upsampled feature map is compared with the feature map of the same resolution layer of the 3D CNN encoder. Perform channel splicing and fuse feature sequences. Global semantic features and feature maps The local detailed features are repeatedly processed to gradually restore the feature map to a resolution consistent with the preprocessed CT image; finally, a voxel-level multi-class segmentation mask is output through 1×1×1 3D convolution and the Softmax activation function; and the upsampling operation expression is as follows: ; In the formula, and These represent the segment decoder's... Level and First Feature map after level upsampling; This represents a 3D transpose convolution operation; This represents a 1×1×1 3D convolution operation; Indicates channel splicing; Reconstruction decoder, used to convert feature sequences Inputting a fully connected layer, the vertex coordinate matrix of the initial coarse mesh is predicted. Patch connection matrix The initial geometric shape of the skeleton was constructed. Indicates the initial number of vertices; then... As input, the vertex coordinates are iteratively optimized through multi-level graph convolutional layers, and the final refined 3D mesh model is output. ; Among them, the The optimized calculation formula for vertex coordinates of a level graph convolutional layer is as follows: ; ; In the formula, Indicates the first The vertex coordinate matrix output by the level graph convolutional layer; Indicates the activation function; Degree matrix; Indicates the first Add self-connected adjacency matrices at each level; Indicates the first The vertex coordinate matrix output by the level graph convolutional layer; Indicates the first Learnable weights of hierarchical graph convolutional layers; Represents the original adjacency matrix; Represents the identity matrix.

[0026] The expression for the multi-task collaborative loss function mentioned in step S3 is as follows: ; in, ; ; ; In the formula, Indicates the total loss; , and This represents the dynamic weight of the loss for each task within the total loss; Indicates the segmentation loss; Indicates the losses incurred during reconstruction; Indicates the regularization loss; Indicates Dice loss, and , This represents the number of categories in the segmentation task. Indicates the first Individual elements belong to the first Predicted probability of class Indicates the first Individual elements belong to the first The true label of a class, i.e., its basic truth value. Represents a constant; This represents the balancing hyperparameter between Dice loss and weighted cross-entropy loss; This represents the weighted cross-entropy loss, and , Indicates the total number of voxels. Indicates the first Class weights; and The hyperparameter representing the balance between chamfer distance and normal consistency loss; This represents the chamfer distance loss, and , and Let represent the vertex set of the predicted mesh and the vertex set of the real mesh, respectively. Represents vertices With vertex Euclidean distance; This represents the loss of normal consistency, and , and Representing vertices respectively and vertex The normal vector, Represents vertices The set of adjacent vertices; and This represents the hyperparameter that balances the edge length regularization loss and the Laplace smoothing loss. Let represent the loss due to edge length regularization, and , Indicates the first The first piece of the face Edge, Indicates the preset target side length; This indicates the Laplace smoothing loss, and , and Representing vertices respectively and vertex The coordinates; During training, a dynamic adjustment strategy based on the normalization of the magnitude of each loss term is adopted to balance the training weights of the segmentation task, reconstruction task and mesh quality constraints, and to iteratively optimize the network parameters until the model converges.

[0027] Step S4 specifically includes the following steps: S41. Mesh simplification and optimization using an edge collapse algorithm: Calculate the folding cost of each edge based on a quadratic error metric. Iterate through the edges that minimize the cost until the number of mesh patches is reduced to the target value. Furthermore, the Hausdorff distance between the simplified mesh and the original mesh is lower than a preset threshold. ; S42. Format standardization operation: Outputs the simplified 3D mesh as a standard 3D model file in STL, PLY, or OBJ format, and outputs the segmentation volume data as a segmentation mask file in NIFTI format.

[0028] Example In this embodiment, 100 complete 3D CT images of bone regions are acquired to form the original medical images, which cover common orthopedic clinical scenarios, such as... Figure 2 The model includes the hand, wrist, shoulder, and finger joints, ensuring broad anatomical adaptability.

[0029] The original hand image acquired in this embodiment was processed using the method described in this invention to obtain the following result: Figure 3 and Figure 4 As shown. Among them, Figure 3 The segmentation clearly shows the precise division of cortical bone, cancellous bone and background tissue. The segmentation mask is highly consistent with the hand anatomy in the original image, with no edge leakage or structural fragmentation issues. This fully verifies the segmentation advantages of the improved 3D-TransUNet network in fusing local details and global semantic features, and can accurately capture the boundaries of fine hand structures.

[0030] Figure 4 The mesh surface is smooth and free of burrs, wrinkles, and artifacts; the patches have uniform side lengths and regular topology; and the deviation from the actual geometry of the hand meets the accuracy requirements for clinical 3D printing. It can clearly reproduce the hand bones and supports observation and quantitative structural measurement at any angle. Meanwhile, Figure 4 The 3D mesh model shown can be directly exported to the STL standard format, allowing for seamless integration with 3D printing equipment and preoperative planning software without secondary topology repair. The segmentation results are output in NIFTI format and can be directly imported into a medical imaging workstation for clinical quantitative assessments such as volume calculation of hand lesion areas and bone density distribution analysis. These results demonstrate that the method of this invention can achieve end-to-end integrated processing of hand CT image segmentation and reconstruction, balancing accuracy, efficiency, and clinical adaptability. It effectively solves problems such as error propagation during segmentation and reconstruction, insufficient model generalization ability, and cumbersome clinical translation in traditional processes, and has significant practical value for scenarios such as preoperative planning in hand surgery and personalized implant design.

[0031] Table 1 Comparison results between the present invention and the prior art

[0032] As shown in Table 1, the present invention has the following advantages: 1. Advantages of Technological Integration and Innovation: This invention creatively integrates the global modeling capability of Transformer, the local feature extraction capability of CNN, and the geometric learning capability of graph convolutional networks into a unified 3D architecture. This deep integration, rather than simple splicing, allows the information flow within the network to be optimized simultaneously in voxel space (segmentation task) and manifold space (reconstruction task), achieving a synergistic effect of "1+1>2".

[0033] 2. Strong clinical applicability: It directly outputs high-quality, 3D-printable mesh models, seamlessly integrating with digital surgical planning and personalized implant manufacturing processes. This shortens the cycle from image to physical object, saving time for emergency orthopedic surgery or complex deformity correction.

[0034] 3. Optimized computational efficiency: By adopting strategies such as axial attention and performing Transformer computation at the feature layer instead of the original voxel layer, the computational and memory overhead of 3D Transformer is significantly reduced while maintaining a large model capacity, enabling the technology to be deployed and run on conventional medical GPU workstations.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An end-to-end method for orthopedic medical image segmentation and three-dimensional reconstruction, characterized by: Includes the following steps: S1. Acquisition and Standardization Preprocessing of Orthopedic 3D Medical Imaging Data: Acquire the original medical images of the patient's orthopedic parts, and perform standardization, isotropic resampling, and region of interest cropping in sequence after layering and annotation to obtain a standard medical image set. S2. Construct an improved 3D-Trans UNet network architecture. The improved 3D-Trans UNet network is an end-to-end architecture that integrates 3D convolutional local feature extraction, Transformer global dependency modeling, and dual-task parallel output. It consists of a 3D CNN encoder, a 3D Patch embedding and position encoding module, an improved 3D Transformer encoder, a segmentation decoder, and a reconstruction decoder cascaded in sequence. S3. Network training based on multi-task collaborative training strategy: The improved 3D-Trans UNet network constructed in S2 is trained using a standard medical image set, and the network parameters are optimized using a multi-task collaborative loss function. S4. Post-processing optimization of 3D model: Post-process the segmentation volume data and 3D mesh output by the forward propagation of the improved 3D-Trans UNet network after training convergence to obtain a standard format segmentation volume mask and a regularized 3D model. S5. System Integration and Clinical Deployment: The standardized 3D model is encapsulated into a standardized inference engine and integrated into the hospital PACS system or independent medical workstation. After receiving DICOM sequence data, the inference engine automatically executes S1-S4. After processing, the original image, segmentation overlay image and interactive 3D model are synchronously displayed on the system interface.

2. The end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Obtain CT images or MRI three-dimensional tomographic images of the patient's orthopedic areas; S12, Layered labeling; S121, First layer annotation: Medical image annotators perform voxel-level annotation on the skeletal structure contour of the original medical image on a 3D medical image annotation platform to form a complete three-dimensional mask as the basic ground truth for the segmentation task. S122, Second layer of annotation: Based on the first layer of annotation, orthopedic surgeons or experts further distinguish important anatomical substructures inside the bone; S123. Correction: The medical images after the second layer of annotation are reviewed and corrected by radiologists; S13. Data Standardization Processing: When selecting CT images, the HU values ​​of the CT images are mapped to a unified target range through linear transformation. ； In the formula, This represents the standardized voxel value; Indicates the original voxel value; and These represent the minimum and maximum values ​​of the HU value, respectively. and These represent the lower and upper bounds of the objective normalization, respectively; S14. Isotropic resampling processing: The cubic spline interpolation method is used to resample the original anisotropic voxel size to the preset isotropic voxel size to ensure consistent resolution in three directions in three-dimensional space. S15. Region of Interest (ROI) cropping: Threshold setting based on the high HU value characteristics of the skeleton. Binarization is performed to obtain the initial mask. : ； Then set the initial mask The three-dimensional bounding box is expanded outward by a preset number of voxels along each dimension to obtain the final clipping region; S16. Dataset partitioning operation: The preprocessed dataset is randomly divided into training set, validation set and test set in a ratio of 7:2:

1.

3. The end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method according to claim 2, characterized in that: In step S14, the voxel size after the normalization process in S13 is set to... The target isotropic size is Then for any target location Its coordinates in the original medical image ; And targeting When the corresponding original medical image has non-integer coordinates, the target location is obtained by selecting the surrounding voxel points and weighting the intensity values ​​of these adjacent voxels using a cubic spline interpolation algorithm. The corresponding voxel intensity value.

4. The end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method according to claim 3, characterized in that: The 3D CNN encoder described in step S2 is divided into multiple coding levels, each containing two 3D residual units, which are used to achieve downsampling through 3D convolution, batch normalization, and ReLU activation to obtain multi-scale local feature maps. , These represent the local feature maps of layers 1, 2, 3, and 4, respectively. Set input voxel data ,Right now The output local feature map The expressions for the changes in size and number of channels are as follows: ； In the formula, Indicates the first Layer local feature map; This indicates the second-level residual downsampling operation; , and These represent the depth, height, and width of a three-dimensional medical image, respectively. This represents the encoding level of the 3D CNN encoder, and ; The 3D Patch embedding and positional encoding module is used to extract the deepest feature maps from the output of the 3D CNN encoder. Extract the size as Non-overlapping 3D blocks are obtained, and after flattening them, a sequence is obtained through linear projection. ,in, Number of blocks The projection dimension is then followed by a learnable 3D position code. The initial feature sequence is obtained. ; The improved 3D Transformer encoder employs an axial attention mechanism for the initial feature sequence of the input. Three unidirectional subsequences were obtained by rearranging the orthogonal anatomical directions of the orthopedic images. , and Then, self-attention operations are performed on each unidirectional subsequence to obtain the attention outputs for the three orthogonal anatomical directions. , and Finally , and The intermediate feature sequence is formed by concatenating the data along the channel dimension. After processing by a feedforward network and layer normalization, the output is a feature sequence that fuses global dependencies in three directions. : ； In the formula, Presentation layer normalization operation; Indicates a feedforward network; This indicates a channel splicing operation; A segmentation decoder is used to segment feature sequences. Reconstruct the 3D feature map while ensuring its size matches that of the deepest feature map. The dimensions are consistent, and then the first upsampling is performed through 3D transposed convolution, while the number of channels is adjusted through convolution to make it consistent with the original size. Channel number matching is performed, and then the upsampled feature map is compared with the feature map of the same resolution layer of the 3D CNN encoder. Perform channel splicing and fuse feature sequences. Global semantic features and feature maps The local detailed features are repeatedly processed to gradually restore the feature map to a resolution consistent with the preprocessed CT image; finally, a voxel-level multi-class segmentation mask is output through 1×1×1 3D convolution and the Softmax activation function; and the upsampling operation expression is as follows: ； In the formula, and These represent the segment decoder's... Level and First Feature map after level upsampling; This represents a 3D transpose convolution operation; This represents a 1×1×1 3D convolution operation; Indicates channel splicing; Reconstruction decoder, used to convert feature sequences Inputting a fully connected layer, the vertex coordinate matrix of the initial coarse mesh is predicted. Patch connection matrix The initial geometric shape of the skeleton was constructed. Indicates the initial number of vertices; then... As input, the vertex coordinates are iteratively optimized through multi-level graph convolutional layers, and the final refined 3D mesh model is output. ; Among them, the The optimized calculation formula for vertex coordinates of a level graph convolutional layer is as follows: ； ； In the formula, Indicates the first The vertex coordinate matrix output by the level graph convolutional layer; Indicates the activation function; Degree matrix; Indicates the first Add self-connected adjacency matrices at each level; Indicates the first The vertex coordinate matrix output by the level graph convolutional layer; Indicates the first Learnable weights of hierarchical graph convolutional layers; Represents the original adjacency matrix; Represents the identity matrix.

5. The end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method according to claim 4, characterized in that: The expression for the multi-task collaborative loss function mentioned in step S3 is as follows: ； in, ； ； ； In the formula, Indicates the total loss; , and This represents the dynamic weight of the loss for each task within the total loss; Indicates the segmentation loss; Indicates the losses incurred during reconstruction; Indicates the regularization loss; Indicates Dice loss, and , This represents the number of categories in the segmentation task. Indicates the first Individual elements belong to the first Predicted probability of class Indicates the first Individual elements belong to the first The true label of a class, i.e., its basic truth value. Represents a constant; This represents the balancing hyperparameter between Dice loss and weighted cross-entropy loss; This represents the weighted cross-entropy loss, and , Indicates the total number of voxels. Indicates the first Class weights; and The hyperparameter representing the balance between chamfer distance and normal consistency loss; This represents the chamfer distance loss, and , and Let represent the vertex set of the predicted mesh and the vertex set of the real mesh, respectively. Represents vertices With vertex Euclidean distance; This represents the loss of normal consistency, and , and Representing vertices respectively and vertex The normal vector, Represents vertices The set of adjacent vertices; and This represents the hyperparameter that balances the edge length regularization loss and the Laplace smoothing loss. Let represent the loss due to edge length regularization, and , Indicates the first The first piece of the face Edge, Indicates the preset target side length; This indicates the Laplace smoothing loss, and , and Representing vertices respectively and vertex The coordinates; During training, a dynamic adjustment strategy based on the normalization of the magnitude of each loss term is adopted to balance the training weights of the segmentation task, reconstruction task and mesh quality constraints, and to iteratively optimize the network parameters until the model converges.

6. The end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method according to claim 5, characterized in that: Step S4 specifically includes the following steps: S41. Mesh simplification and optimization using an edge collapse algorithm: Calculate the folding cost of each edge based on a quadratic error metric. Iterate through the edges that minimize the cost until the number of mesh patches is reduced to the target value. Furthermore, the Hausdorff distance between the simplified mesh and the original mesh is lower than a preset threshold. ; S42. Format standardization operation: Outputs the simplified 3D mesh as a standard 3D model file in STL, PLY, or OBJ format, and outputs the segmentation volume data as a segmentation mask file in NIFTI format.

7. A system for performing the end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method according to any one of claims 1-6, characterized in that: include: Data acquisition and standardization preprocessing module: used to acquire the original medical images of the patient's orthopedic parts, and after hierarchical annotation, perform standardization, isotropic resampling and region of interest cropping in sequence to obtain a standard medical image set; An improved 3D-Trans UNet network architecture building module is used to construct an improved 3D-Trans UNet network architecture building module consisting of a 3D CNN encoder, a 3D Patch embedding and position encoding module, an improved 3D Transformer encoder, a segmentation decoder, and a reconstruction decoder cascaded in sequence. The network training module is used to train the improved 3D-Trans UNet network using a standard medical image set and to optimize the network parameters using a multi-task collaborative loss function. The 3D model post-processing optimization module is used to post-process the segmentation volume data and 3D mesh output by the forward propagation of the improved 3D-Trans UNet network after training convergence, to obtain a standard format segmentation volume mask and a regularized 3D model. The system integration and clinical deployment module is used to encapsulate the regularized 3D model into a standardized inference engine, which is then integrated into the hospital PACS system or a standalone medical workstation. After receiving DICOM sequence data, the inference engine automatically executes the process and displays the original image, segmentation overlay image, and interactive 3D model on the system interface.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the end-to-end orthopedic medical image segmentation and three-dimensional reconstruction method as described in any one of claims 1-6.

Images