Bone CT image processing method and system based on multi-task network

Abstract

The embodiment of the invention provides a skeleton CT image processing method based on a multi-task network. The method comprises the following steps: S1, constructing and training a skeleton segmentation network model; the main segmentation network decoder and the auxiliary segmentation network decoder are connected with the encoder and are connected in parallel; a main segmentation network decoder of the main segmentation network comprises three layers of 3 * 3 * 3 three-dimensional residual convolutional layers, three layers of up-sampling operators and a Transformer module, an auxiliary decoder of the auxiliary segmentation network comprises four layers of 3 * 3 * 3 three-dimensional residual convolutional layers and three layers of up-sampling operators, and the three layers of 3 * 3 * 3 three-dimensional residual convolutional layers and the three layers of up-sampling operators are fused with the encoder through jump connection; and S2, inputting a target image, and obtaining a target segmentation result output by the main segmentation network. According to the embodiment of the invention, the limitation of a traditional image processing method is effectively relieved by combining a 3D-UNet architecture with a Transform attention mechanism, meanwhile, the segmentation quality of the skeleton CT image is effectively improved by adaptively adjusting the sampling frequency of a complex region in training, and the method has significant advantages for complex anatomical structures with three-dimensional skeleton gaps.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a bone CT image processing method and system based on a multi-task network. Background Art

[0002] In traditional medical image segmentation, the segmentation of healthy bones can usually be achieved with high accuracy through conventional networks. However, bone segmentation for patients with hip arthritis faces significant challenges. First, the bone morphology of patients varies greatly due to individual differences and different stages of disease progression, making it difficult for the model to adapt to the diversity of different cases. Secondly, the pathological characteristics of hip arthritis, such as cartilage degeneration, narrowing of the joint space, bone collision, bone deformation and osteophyte formation, will destroy the normal anatomical contours of the bones and produce irregular surface structures. These lesion areas often show blurred boundaries in CT images, especially in areas with narrow joint spaces. The close proximity of the bones makes it extremely difficult to distinguish between the femoral head and the acetabulum.

[0003] Among them, CN118918123A discloses a "brain tumor image segmentation method based on an improved NestedUNet network and Transformer attention mechanism", which solves the limitations of the traditional U-Net model in feature extraction and segmentation accuracy by introducing the Transformer attention mechanism, multi-level residual blocks and channel attention mechanism into the network. This method effectively captures multi-scale features through up and down sampling operations, and uses the Transformer and channel attention mechanisms to enhance the expression of important features. However, the existing technology is mostly based on the processing of 2D images, and has certain limitations in processing complex case images with three-dimensional bone gaps, such as bone diseases.

[0004] Under existing technology, especially in total hip replacement surgery planning, the precise segmentation of the hip and femur is directly related to the matching degree of the intraoperative cutting template. This places dual conflicting demands on the algorithm: on the one hand, in areas of osteophyte hyperplasia and bone deformation, the segmentation range needs to be appropriately expanded to ensure complete coverage of the diseased tissue to avoid residual abnormal bone affecting the positioning of the prosthesis during surgery; on the other hand, areas of narrow joint space caused by cartilage degeneration require sufficiently precise segmentation boundaries. Any excess may mistakenly adhere adjacent bones as a whole, resulting in deviations in the subsequent prosthesis angle design. This contradiction often causes existing algorithms to face the dilemma of "over-segmentation" and "under-segmentation". Currently, this problem still relies heavily on manual correction by radiologists. However, with the increasing complexity of the case, the time and effort required for manual adjustment increase nonlinearly, highlighting the urgent need to develop targeted segmentation methods. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a bone CT image processing method based on a multi-task network to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a method for processing bone CT images based on a multi-task network is provided, the method comprising: 1. A method for processing bone CT images based on a multi-task network, the method comprising: S1. constructing and training a bone segmentation network model, the bone segmentation network model comprising: an encoder comprising a plurality of 3*3*3 three-dimensional residual convolutional layers; a main segmentation network decoder and an auxiliary segmentation network decoder connected to the encoder and connected in parallel with each other; the main segmentation network decoder of the main segmentation network comprises three layers of 3*3*3 three-dimensional residual convolutional layers, three layers of upsampling operators and a Transformer module, and is fused with the encoder via a skip connection; the auxiliary decoder of the auxiliary segmentation network comprises four layers of 3*3*3 three-dimensional residual convolutional layers and three layers of upsampling operators, and is fused with the encoder via a skip connection;

[0007] The training method of the skeleton segmentation network model includes:

[0008] S11. Input a training set containing overall skeleton features and skeleton end features, wherein the images containing overall skeleton features are used to train the main segmentation network, and the images containing skeleton end features are used to train the auxiliary segmentation network;

[0009] S12. Constructing a combined loss function for updating the overall network parameters, wherein the combined loss function is a fusion of the main segmentation network loss function and the auxiliary segmentation network loss function;

[0010] S13. Constructing a sampling strategy to dynamically adjust the sampling probability of each region of the bone image to increase the network's ability to recognize and extract features of the bone ends;

[0011] S2. Input the target image and obtain the target segmentation result output by the main segmentation network.

[0012] Preferably, the combined loss function includes:

[0013] The combined loss function is composed of the loss functions of the main segmentation network and the auxiliary segmentation network, and is expressed by the following formula:

[0014] L total =L1+γ·L2,

[0015] L total It is used to characterize the total loss function, L1 is used to characterize the total loss function of the main segmentation network, L2 is used to characterize the total loss function of the auxiliary segmentation network, and γ is used to characterize the focus factor.

[0016] Preferably, the loss function of the main segmentation network is:

[0017] L1=α·L C +(1-α)·L D ,

[0018] L1 is used to characterize the total loss function of the main segmentation network, L C Used to characterize the weighted cross entropy loss function, L D It is used to represent the quality adjustment loss function, and α is used to represent the balance factor;

[0019] Among them, the weighted cross entropy loss function includes:

[0020]

[0021] C is used to represent the total number of label categories, y i One-hot encoding value used to represent the target label, p i Used to represent the predicted matching probability value;

[0022] The quality adjustment loss function includes:

[0023]

[0024] L D is used to represent the quality adjustment loss function, N is used to represent the total number of voxels, and p i Used to represent the predicted value of the i-th pixel, g i Used to represent the true value of the i-th pixel, where i is a natural number greater than 0.

[0025] Preferably, the loss function of the auxiliary segmentation network is expressed as follows:

[0026] L2=α·L C +(1-α)·L D ,

[0027] L2 is used to characterize the total loss function of the auxiliary segmentation network, L C Used to characterize the weighted cross entropy loss function, L D It is used to represent the quality adjustment loss function, and α is used to represent the balance factor;

[0028] The sample weights are adjusted using a dynamic scaling factor using the following formula:

[0029] L F =-∝(1-p t ) γ log p t ,

[0030] α is used to characterize the balance factor, γ is used to characterize the focus factor, and p t Used to characterize the actual predicted matching probability;

[0031] The boundary position is optimized by the following formula:

[0032] L B =∫ Ω Φ G (q)s θ (q)dq,

[0033] Ω is used to represent the image domain, Φ G (q) is used to characterize the distance map generated by the real boundary, s θ (q) is the output used to represent the predicted matching probability.

[0034] Preferably, the training set of the bone end features includes:

[0035] The bone images are preprocessed using a cropping strategy to obtain a training set of bone end features;

[0036] Preferably, the sampling strategy includes:

[0037] Dynamically adjusting the sampling probability of each area of the bone image using a dynamic sampling mechanism based on the judgment result of the edge sampling trigger condition;

[0038] Wherein, determining whether the edge sampling trigger condition is met is performed by the following steps: comparing a first threshold value with a parameter value, if the parameter value is less than the first threshold value, the edge sampling trigger condition is met, otherwise, the edge sampling trigger condition is not met;

[0039] If the edge sampling trigger condition is met, sampling is performed from the first area with a first preset probability; if the edge sampling trigger condition is not met, conventional random slice selection is performed.

[0040] According to a second aspect of an embodiment of the present application, a multi-task network-based bone CT image processing system is provided, which is used to implement the multi-task network-based bone CT image processing method as described in the first aspect.

[0041] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, and a memory for storing a program, wherein the program comprises instructions that, when executed by the processor, cause the processor to execute the method described in the first aspect.

[0042] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0043] In summary, the embodiments of the present application effectively alleviate the limitations of traditional image processing methods through the 3D-UNet architecture combined with the Transformer attention mechanism. At the same time, during training, the sampling frequency of complex areas is adaptively adjusted to effectively improve the segmentation quality of bone CT images, and it has significant advantages when facing complex anatomical structures such as those with three-dimensional bone gaps. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0045] Figure 1 is a general flow chart of a multi-task network-based bone CT image processing method according to an exemplary embodiment of the present application;

[0046] Figure 2 A schematic structural diagram of an electronic device according to an exemplary embodiment of the present application;

[0047] Figure 3 This is a box plot of the test set segmentation results of the relevant loss function of an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and in detail described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0049] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0050] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0051] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0052] It should be noted that the modifiers "one" and "plurality" mentioned in this disclosure are illustrative and non-restrictive. Those skilled in the art will understand that unless the context clearly indicates otherwise, they should be understood as "one or more." The names of the messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0053] Existing methods for processing skeletal CT images primarily include threshold segmentation methods that extract bone tissue using fixed CT value intervals, regional segmentation methods that rely on seed point diffusion, edge segmentation methods based on grayscale gradients, and statistical shape model segmentation methods that incorporate anatomical prior knowledge. While traditional methods still have application value in small sample sizes or specific anatomical scenarios, deep learning methods, due to their high degree of automation and strong adaptability to complex structures, have become the mainstream research direction.

[0054] Encoder-decoder architectures, such as U-Net, have been widely used for skeleton segmentation tasks, improving segmentation accuracy through a skip-connection mechanism and multi-scale feature fusion. Similarly, networks such as VNet and ResUNet have further improved segmentation performance by introducing residual modules, demonstrating excellent performance in accurately segmenting complex skeletal structures. In addition, “Zeng G, Yang X, Li J, et al. 3D U-net with multi-level deep supervision: fully automatic segmentation of proximal femur in 3D MR images [C] / / Machine Learning in Medical Imaging: 8th International Workshop, MLMI 2017, Held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 10, 2017, Proceedings 8. Springer International Publishing, 2017: 274-282.” The literature mentioned that the introduction of deep supervision mechanism in 3D-UNet effectively enhanced the learning ability of intermediate layer features, and “

[22] Chen F, Liu J, Zhao Z, et al. Three-dimensional feature-enhanced network for automatic femur segmentation [J]. IEEE journal of biomedical and health informatics, 2017, 23(1): 243-252." The literature mentioned that by introducing a boundary-aware branch, the network is made more sensitive in identifying bone edges. At the same time, the improvement of the training strategy also has an important impact on the segmentation effect.

[0055] While deep learning has made significant progress in healthy bone segmentation, patients with hip arthritis experience degeneration of articular cartilage, which leads to narrowing of the joint space and increased friction between bones. This in turn causes osteophytes and skeletal morphological distortion, complicating the hip joint anatomy. CT images of these patients exhibit irregular bone tissue boundaries, and the contrast between the joint space and surrounding soft tissue is significantly reduced, resulting in blurred joint boundaries. These pathological anatomical features make the aforementioned methods challenging to segment bones in patients with arthritis.

[0056] This solution provides a skeleton segmentation network model based on the 3D-UNet architecture by introducing the Transformer attention mechanism and combining it with the boundary optimization network. The specific implementation of the embodiment of this application is further explained below with reference to the accompanying drawings.

[0057] Specifically, see Figure 1 The skeleton CT image processing method based on the multi-task network of this embodiment mainly includes the following steps:

[0058] Step S100: construct a skeleton segmentation network model.

[0059] Specifically, a skeleton segmentation network model is constructed and trained, wherein the skeleton segmentation network model includes an encoder of several 3*3*3 residual convolution layers. More specifically, a main segmentation network decoder and an auxiliary segmentation network decoder are provided, which are connected to the encoder and connected in parallel with each other. Specifically, the main segmentation network decoder of the main segmentation network includes 3 layers of 3*3*3 three-dimensional residual convolution layers, 3 layers of upsampling operators and a Transformer module, and is fused with the encoder through a jump connection; the auxiliary segmentation network auxiliary decoder includes 4 layers of 3*3*3 three-dimensional residual convolution layers and 3 layers of upsampling operators, and is fused with the encoder through a jump connection.

[0060] Step S102: inputting the training set containing the overall skeleton features and the skeleton end features into the skeleton segmentation network model to train the skeleton segmentation network model.

[0061] The bone ends include the joint areas and the areas around the joints.

[0062] Specifically, the skeleton segmentation network model is trained through step S11. More specifically, S11. a training set containing overall skeleton features and end skeleton features is input, wherein the images containing overall skeleton features are used to train the main segmentation network, and the images containing end skeleton features are used to train the auxiliary segmentation network;

[0063] The training set of overall skeleton features is obtained based on the skeleton image. Furthermore, the training set of skeleton end features includes: pre-processing the skeleton image using a cropping strategy to obtain the training set of skeleton end features.

[0064] Among them, the bone image is preprocessed through the cropping strategy. Specifically, the bone ends of the complete bone image are cropped through the cropping strategy in the bone segmentation network, and the cropped image is used as a training set for the bone end features.

[0065] It is understandable that the sampling strategy is mainly used to capture areas in bone images that are difficult to segment. For example, the hip joint is a ball-and-socket joint, and its lesions are often accompanied by complex distortions of the acetabulum and femoral head and three-dimensional osteophyte hyperplasia surrounding the joint cavity. At the same time, the coverage of deep muscles and soft tissues significantly reduces the grayscale contrast between bones and surrounding tissues in CT images, resulting in more prominent problems of blurred boundaries and feature loss. Therefore, this scheme proposes a sampling strategy for capturing areas in bone images that are difficult to segment. It is understandable that the areas in bone images that are difficult to segment are identified as key areas. In practical applications, this scheme uses an oversampling strategy to dynamically adjust the sampling frequency of each image area to dynamically increase the sampling frequency of the key area, that is, the sampling frequency of the area that is difficult to segment is relatively higher than that of other areas. It should be clear that the training set of bone end features includes bone end areas, where the bone ends cover each bone joint area and the area around the joint.

[0066] Specifically, the sampling strategies include:

[0067] Based on the judgment result of edge sampling trigger condition, the dynamic sampling mechanism is used to dynamically adjust the sampling probability of each area of the bone image;

[0068] Wherein, determining whether the edge sampling trigger condition is met is performed by the following steps: comparing a first threshold value with a parameter value, if the parameter value is less than the first threshold value, the edge sampling trigger condition is met, otherwise, the edge sampling trigger condition is not met;

[0069] If the edge sampling trigger condition is met, sampling is performed from the first area with a first preset probability; if the edge sampling trigger condition is not met, conventional random slice selection is performed.

[0070] Optionally, the parameter value is a number between 0 and 1 obtained based on random generation, and is compared with a fixed threshold value (i.e., a first threshold value) set between 0 and 1. It can be understood that based on the comparison between the random parameter value and the first threshold value, it is further determined whether the edge sampling condition is met, that is, the sampling is triggered. In some embodiments, the random parameter is directly generated by the Random.random() function in Python. It should be clear that the generation of the parameter value can be arbitrarily modified according to actual conditions, and this solution is not limited here.

[0071] Example 1: In the CT images of hip bone tissue, there are significant differences in the structural complexity of different anatomical regions: the semantic segmentation difficulty of the femoral shaft region is relatively low due to its regular bone morphology and simple texture features; while the image features of the acetabular edge, femoral head contact surface, and distal femur are complex and changeable due to their complex anatomical structure, shape differences between individuals, and typical pathological characteristics of arthritis, making them difficult areas for model learning.

[0072] This solution solves the problem of complex and changeable image features in key areas to a certain extent through a dynamic sampling mechanism. Specifically, it is a dynamic mechanism based on priority sampling of complex areas. Its core is to set a threshold, further compare the parameter value with the set threshold, and dynamically adjust the slice sampling probability of the head and tail areas according to the comparison result. Specifically, when the parameter value is less than the first threshold, sampling is performed from the first area with a first preset probability, otherwise conventional random slice selection is performed. It can be understood that this strategy decides whether to focus on the complex area based on the parameter value at each sampling. Specifically, in some embodiments, the first preset probability is set to 50%, and the first area is set to the hip joint area. It can be understood that when the parameter value is lower than the first threshold, the system selects the starting slice from the hip joint area with a probability of 50%, and in other cases, conventional random slice selection is performed. More specifically, when the parameter value is not lower than the first threshold, it is selected from the distal femoral area. This mechanism can significantly improve the probability of sampling slices containing key anatomical landmarks (such as the acetabular edge and the femoral head contact surface), while retaining the randomness of normal area sampling to avoid overfitting.

[0073] S12. Construct a combined loss function for updating the encoder network parameters, wherein the combined loss function comprises a fusion of the loss functions of the primary segmentation network and the auxiliary segmentation network. It is understood that the loss functions of the primary segmentation network and the auxiliary segmentation network operate synchronously, and the encoder network parameters are synchronously iteratively updated. It is understood that the target segmentation results output by the skeleton segmentation network model are synchronously iteratively updated until the constraints of the loss function are met, at which point the iterative update ceases.

[0074] Specifically, the loss function of the main segmentation network includes:

[0075] The main segmentation network is optimized using the following formula:

[0076] L1=α·L C +(1-α)·L D ,

[0077] L1 is used to characterize the total loss function of the main segmentation network, L C Used to characterize the weighted cross entropy loss function, L D is used to characterize the quality adjustment loss function, and α is used to characterize the balance factor. In some embodiments, α is set to 0.5.

[0078] Among them, the weighted cross entropy loss function includes:

[0079]

[0080] C is used to represent the total number of label categories, y i One-hot encoding value used to represent the target label, p iIt is used to represent the predicted matching probability value so that the main segmentation network can achieve the training goal, that is, to maximize the matching probability of the correct voxel and the corresponding label.

[0081] L2=α·L C +(1-α)·L D ,

[0082] L2 is used to characterize the total loss function of the auxiliary segmentation network, L C Used to characterize the weighted cross entropy loss function, L D It is used to represent the quality adjustment loss function, and α is used to represent the balance factor.

[0083] In bone CT image processing, due to the serious imbalance in the number of foreground and background voxels, simply using the cross entropy loss function can easily cause the model to be biased towards the background area, thereby reducing the segmentation accuracy of the target area. To this end, this solution uses a combination of WeightedCross-Entropy Loss and Dice Loss as the loss function for the main task. Weighted Cross-EntropyLoss alleviates the class imbalance problem by adjusting the category weights and improves the classification performance; Dice Loss optimizes the quality of the model's segmentation results by maximizing the overlap between the predicted area and the true area. The calculation formula of Dice Loss is L D , that is, the quality adjustment loss function includes:

[0084]

[0085] L D is used to represent the quality adjustment loss function, N is used to represent the total number of voxels, and p i Used to represent the predicted value of the i-th pixel, g i Used to represent the true value of the i-th pixel, where i is a natural number greater than 0.

[0086] Furthermore, to optimize the accuracy of boundary segmentation between bones, for example, the accuracy of boundary segmentation between the acetabulum and the femoral head, this scheme constructs a boundary optimization auxiliary task and achieves collaborative training by sharing encoder parameters with the main task.

[0087] Specifically, in some embodiments, the boundary optimization task network consists of three upsampling operators and four residual convolution modules, and finally outputs the auxiliary segmentation result through a 1×1×1 convolution layer. Its loss function is integrated into the global optimization objective and jointly drives model training with the main task.

[0088] Cross-Entropy loss, Focal loss, and Boundary loss were selected and combined with Dice loss for comparative experimental analysis. Among them, Focal Loss enhances the weight of difficult samples by dynamically scaling factors, which can alleviate the problem of Dice loss's lack of sensitivity to local boundary features. Specifically, the loss function of the auxiliary segmentation network includes:

[0089] The sample weights are adjusted using a dynamic scaling factor using the following formula:

[0090] L F =-∝(1-p t ) γ log p t ,

[0091] α is used to characterize the balance factor, γ is used to characterize the focus factor, and p t Used to characterize the actual predicted matching probability;

[0092] It is understandable that Boundary Loss focuses on optimizing the boundary position. By finely depicting the boundary area, it reduces the blurring or misclassification of the boundary and improves the segmentation accuracy of the model on the bone boundary. Therefore, the boundary position is optimized by the following formula:

[0093] L B =∫ Ω Φ G (q)s θ (q)dq,

[0094] Ω is used to represent the image domain, Φ G (q) is used to characterize the distance map generated by the real boundary, s θ (q) is the output used to represent the predicted matching probability.

[0095] Step S104: construct a combined loss function for updating the encoder network parameters to make the output result of the skeleton segmentation network model more accurate. Specifically, the combined loss function is expressed by the following formula:

[0096] L total =L1+γ·L2,

[0097] L total It is used to characterize the total loss function, L1 is used to characterize the total loss function of the main segmentation network, L2 is used to characterize the total loss function of the auxiliary segmentation network, and γ is used to characterize the focus factor. In some embodiments, γ is set to 0.3.

[0098] Step S106, input the target image and obtain the target segmentation result output by the main segmentation network. It can be understood that the target image is input into the trained bone segmentation network for processing and outputs the target segmentation result, wherein the target segmentation result is a bone CT image with clear bone boundaries.

[0099] The target segmentation results are obtained by the following steps:

[0100] The long-range skeleton dependency of the target image is constructed by embedding a Transformer module at the end of the encoder; the multi-scale features of the target image are extracted by hierarchical downsampling; the decoders of the main segmentation network and the auxiliary segmentation network are used to perform feature fusion based on the long-range skeleton dependency, and further output the target segmentation result.

[0101] It is understandable to people in this field that bone CT images contain related image information when output by different imaging devices, such as "lower limb CT image, upper limb CT image" and other information. In some embodiments, long-range bone dependencies are established through semantic association. It is understandable that long-range dependency refers to the semantic association between areas that are far apart in the image. For example, in lower limb CT, the symmetry between the left hip joint and the right hip joint, the anatomical association between the femoral head of the hip joint and the distal femur, by dividing the three-dimensional feature map output by the encoder into 3D patches, inputting it into the Transformer module, calculating the dependency strength between different patches through the self-attention mechanism, and finally weighting and fusion of the Value vector to generate a globally perceived embedding vector. The embedding vector is derived from the high-dimensional semantic features output by the encoder, and the associated features are obtained by calculating the self-attention weight matrix. The high weight represents a strong anatomical association between the two. It should be clear that the establishment of long-range bone dependencies can be modified arbitrarily according to actual conditions, and this solution is not limited here.

[0102] As you can understand, the self-attention weight matrix refers to the trainable Q / K / V matrix in the transformer module that projects the input into the query, key, and value spaces. Attention scores are calculated by taking the dot product of the query and key, and then normalized using softmax to generate the self-attention weight matrix, which dynamically reflects the dependency strength between different patches. Ultimately, these weights are weighted and fused with the value vector to generate a globally context-aware embedding vector.

[0103] It should be clear that the main segmentation network and the auxiliary segmentation network share the same encoder. Furthermore, the main segmentation network includes an encoder and several main segmentation network decoders, and the auxiliary segmentation network includes an encoder and several auxiliary segmentation network decoders.

[0104] For example, a Transformer module is embedded at the end of the encoder to establish long-range skeletal dependencies. Specifically, the encoder output feature map is divided into fixed-size 3D patches. A self-attention mechanism is used to generate a global context-aware embedding vector, which simultaneously captures the correlation between narrow joint gap areas and distal skeletal morphology. This module significantly enhances the representation of skeletal edges and complex structures through cross-region semantic association.

[0105] In some embodiments, the skeleton segmentation network model adopts an encoder-decoder architecture and integrates a Transformer module to enhance global modeling capabilities. Among them, the encoder contains a 4-level residual convolution module and 3 2×2×2 maximum pooling layers, and extracts skeleton features through hierarchical downsampling. Specifically, each residual module contains dual 3×3×3 convolution layers, ReLU activation and GroupNorm layers. Furthermore, 1×1×1 convolution jump connections are used to fuse cross-layer features to improve training stability. The decoder fuses the multi-scale features of the encoder through multi-level upsampling and gradually restores the spatial resolution: each decoding block contains an upsampling operator and a residual convolution module. In the initial stage, the low-dimensional features output by the Transformer are mapped to a high-dimensional space through 3×3×3 convolutions, and finally the target segmentation result is output through 1×1×1 convolutions.

[0106] In practice, patient skeletal CT images are obtained from various imaging devices. The images are typically 512×512 pixels in size, with a pixel size range of 0.2–0.8 mm. They contain an average of 270 slices with a slice spacing of 0.4–1 mm. Some scans are bilateral, while others are unilateral. To unify the training samples, the data is preprocessed. Specifically, all data is cropped to unilateral lower limb slices, and then z-axis resampling is used to unify the spatial resolution to 1 mm. Window level adjustment and intensity normalization are used to standardize the data. This processing strategy reduces GPU memory requirements and standardizes the data format. Due to GPU memory limitations, a 128×128×128 cube is extracted from the original CT image as the network input.

[0107] The dataset was split into a training set (104 samples) and a test set (10 samples) in a 9:1 ratio. Five-fold cross-validation was used to evaluate the model's segmentation performance. To improve model generalization, dynamic data augmentation strategies were implemented during training, including horizontal and vertical flipping, random 90° rotation, Gaussian noise injection, and gamma correction, effectively increasing sample diversity.

[0108] Model training was performed in Python 3.8 and PyTorch 1.12.1 on an Nvidia GeForce RTX 4070Ti Super GPU. Parameters were updated using the Adam optimizer, with an initial learning rate of 0.001, 200 training epochs, and a batch size of 1. During training, the model parameters with the highest validation accuracy were automatically saved, and the optimal cross-validation result was selected as the final model output.

[0109] Example 3: In some embodiments, a validity evaluation system is further provided, and the validity of the target bone image is verified by using the validity evaluation system. If the verification is passed, a valid evaluation result is output; otherwise, an invalid evaluation result is output.

[0110] Specifically, to quantitatively evaluate the accuracy of segmentation results, this approach uses the Dice coefficient as the primary quantitative evaluation metric and constructs a multi-dimensional evaluation system combining Precision, Recall, ASD, and HD95. This metric covers four key dimensions: volumetric overlap, error type, boundary accuracy, and extreme risk. This system not only meets common standards in medical image segmentation (such as DSC), but also introduces boundary-sensitive metrics (ASD and HD95) tailored to the clinical needs of bone segmentation. By quantifying performance from multiple perspectives, the practicality and robustness of the algorithm can be comprehensively evaluated, providing a reliable basis for subsequent model optimization and clinical deployment. Dice measures the degree of spatial overlap between predictions and true labels, assessing the overall consistency of segmented regions. Precision measures the proportion of predicted bone regions that are correct, reflecting the model's ability to resist false detections. Recall reflects the proportion of true bone regions detected, reflecting the model's ability to resist missed detections. ASD (Average Surface Distance) measures the average surface distance between the predictions and the true surface, quantifying the fineness of the segmentation boundary. HD95 (95% Hausdorff distance) measures the maximum local deviation between the predictions and the true surface, assessing the model's stability in complex regions.

[0111] In the existing technology, the relevant document 1: "Marzorati D, Sarti M, Mainardi L, et al. Deep3D convolutional networks to segment bones affected by severe osteoarthritis in CT scans for PSI-based knee surgical planning [J]. IEEE Access, 2020, 8: 196394-196407." has achieved remarkable results in knee arthritis bone segmentation. Using the 3D-UNet model, it achieved a Dice coefficient of more than 96%. Related Reference 2: "Marsilio L, Faglia A, Rossi M, et al. CEL-Unet: A novel CNN architecture for 3D Segmentation of Knee Bones affected by Severe Osteoarthritis for PSI-Based Surgical Planning [C] / / 2022 44th Annual International Conference of the IEEE Engineering in Medicine & BioIogy Society (EMBC). IEEE, 2022: 5039-5042." A CEL-UNet variant was further developed, achieving Dice coefficients of 97% and 96% for the femur and tibia, respectively. This approach performs comparable to the aforementioned methods, but it should be noted that complex bone images are difficult to process. For example, the unique three-dimensional structure of the hip joint can easily cause multidirectional morphological distortion of the acetabulum and femoral head when diseased, and osteophytes often present as three-dimensional proliferations surrounding the joint cavity. Consequently, knee and hip arthritis differ significantly in their anatomical structures and pathological characteristics. This approach is adapted to the specific needs of hip arthritis through model architecture optimization and improved data preprocessing strategies. The experimental data are shown in Table 1. Figure 3 As shown:

[0112] Table 1 Results of bone segmentation using different loss functions parameter Cross-Entropy Loss Focal Loss Boundary Loss Femoral Dice (%) 96.35±0.97 95.84±3.07 94.64±2.85 Hip Dice (%) 93.68±1.24 92.70±1.35 92.67±1.15 Precision(%) 96.05±0.98 94.83±5.43 93.73±5.32 Recall (%) 96.66±1.52 97.03±1.10 95.73±1.61 ASD(mm) 0.60±0.40 1.09±0.74 0.94±0.56 HD95(mm) 4.09±6.15 8.25±10.08 11.84±13.91 Dice+Weighted Cross-Entropy Loss achieved Dice coefficients of 96.35% and 93.68% in the femur and hip segmentation tasks, respectively, significantly outperforming Focal Loss (95.84% / 92.70%) and Boundary Loss (94.64% / 92.67%). The combination achieved both 96.05% Precision and 96.66% Recall, demonstrating its excellent and balanced performance in controlling over-segmentation and under-segmentation errors. Quantification results of ASD (0.60mm) and HD95 (4.09mm) further confirmed that the segmentation boundaries maintained high consistency with the gold standard in both local details and global morphology. Comprehensive analysis showed that the Dice+Cross Entropy loss function achieved superior prediction accuracy in femoral volume segmentation by accurately integrating edge information, and its multi-dimensional indicator advantages verified the effectiveness of this loss function combination. It can be understood that the method proposed in this scheme meets the accuracy requirements of orthopedic surgical navigation (ASD < 1mm) in terms of indicators such as Dice coefficient (femur Dice coefficient 96.98%, hip Dice coefficient 95.20%) and ASD (0.40mm). In summary, this method provides new possibilities for the clinical diagnosis and treatment of hip arthritis, especially showing significant advantages in dealing with complex anatomical variations and pathological characteristics. These technological innovations provide high-precision three-dimensional model support for computer-assisted total hip arthroplasty (THA).

[0113] In summary, the embodiments of the present application effectively alleviate the limitations of traditional image processing methods through the 3D-UNet architecture combined with the Transformer attention mechanism. At the same time, during training, the sampling frequency of complex areas is adaptively adjusted to effectively improve the segmentation quality of bone CT images, and it has significant advantages when facing complex anatomical structures such as those with three-dimensional bone gaps.

[0114] Another embodiment of the present application further provides a skeletal CT image processing system based on a multi-task network to implement any corresponding operation in the above-mentioned multiple method embodiments.

[0115] See also Figure 2 , shown is a schematic diagram of an electronic device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0116] like Figure 2 As shown, the electronic device may include: a processor (processor) 202 , a communication interface (Communications Interface) 204 , a memory (memory) 206 , and a communication bus 208 .

[0117] in:

[0118] The processor 202 , the communication interface 204 , and the memory 206 communicate with each other via the communication bus 208 .

[0119] The communication interface 204 is used to communicate with other electronic devices or servers.

[0120] The processor 202 is configured to execute the program 210 , and specifically to execute the relevant steps in the embodiment of the above-mentioned multi-task network-based bone CT image processing method.

[0121] Specifically, the program 210 may include program codes, which include computer operation instructions.

[0122] The processor 202 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. A smart device includes one or more processors, which may be of the same type, such as one or more CPUs, or different types, such as one or more CPUs and one or more ASICs.

[0123] The memory 206 is used to store the program 210. The memory 206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0124] The program 210 may include multiple computer instructions. Specifically, the program 210 may enable the processor 202 to execute operations corresponding to the multi-task network-based bone CT image processing method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0125] The specific implementation of each step in program 210 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0126] The electronic device of the embodiment of the present application can adapt to the diversity of different cases and has significant advantages in edge capture and global shape restoration of complex anatomical structures such as three-dimensional bone gaps.

[0127] This application also provides a computer-readable storage medium storing instructions for causing a machine to perform the method described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0128] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.

[0129] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0130] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0131] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0132] The above-described method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the navigation method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the navigation method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the navigation method shown herein.

[0133] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0134] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A bone CT image processing method based on a multi-task network, characterized in that: The method comprises: S1. Build and train a skeleton segmentation network model, the skeleton segmentation network model includes: An encoder containing several 3*3*3 three-dimensional residual convolutional layers; A main segmentation network decoder and an auxiliary segmentation network decoder connected to the encoder and in parallel with each other; The main segmentation network decoder of the main segmentation network includes three layers of 3*3*3 three-dimensional residual convolution layers, three layers of upsampling operators and Transformer modules, and is fused with the encoder through skip connections; The auxiliary decoder of the auxiliary segmentation network includes 4 layers of 3*3*3 three-dimensional residual convolution layers and 3 layers of upsampling operators, and is fused with the encoder through skip connections; The training method of the skeleton segmentation network model includes: S11. Input a training set containing overall skeleton features and skeleton end features, wherein the images containing overall skeleton features are used to train the main segmentation network, and the images containing skeleton end features are used to train the auxiliary segmentation network; S12. Constructing a combined loss function for updating the overall network parameters, wherein the combined loss function is a fusion of the main segmentation network loss function and the auxiliary segmentation network loss function; S13. Constructing a sampling strategy to dynamically adjust the sampling probability of each region of the bone image to increase the network's ability to recognize and extract features of the bone ends; S2. Input the target image and obtain the target segmentation result output by the main segmentation network.

2. The method according to claim 1, characterized in that The combined loss function includes: The combined loss function is composed of the loss functions of the main segmentation network and the auxiliary segmentation network, and is expressed by the following formula: L total =L1+γ·L2, L total It is used to characterize the total loss function, L1 is used to characterize the total loss function of the main segmentation network, L2 is used to characterize the total loss function of the auxiliary segmentation network, and γ is used to characterize the focus factor.

3. The method according to claim 2, characterized in that The loss function of the main segmentation network is: L1=·L C +(1-α)·L D , L1 is used to characterize the total loss function of the main segmentation network, L C Used to characterize the weighted cross entropy loss function, L D It is used to represent the quality adjustment loss function, and α is used to represent the balance factor; Among them, the weighted cross entropy loss function includes: C is used to represent the total number of label categories, y i One-hot encoding value used to represent the target label, p i Used to represent the predicted matching probability value; The quality adjustment loss function includes: L D is used to represent the quality adjustment loss function, N is used to represent the total number of voxels, and p i Used to represent the predicted value of the i-th pixel, g i Used to represent the true value of the i-th pixel, where i is a natural number greater than 0.

4. The method according to claim 2-3, characterized in that The loss function of the auxiliary segmentation network is expressed as follows: L2=α·L C +(1-a)·L D , L2 is used to characterize the total loss function of the auxiliary segmentation network, L C Used to characterize the weighted cross entropy loss function, L D It is used to represent the quality adjustment loss function, and α is used to represent the balance factor; The sample weights are adjusted using a dynamic scaling factor using the following formula: L F =-∝(1-p t ) γ log p t , α is used to characterize the balance factor, γ is used to characterize the focus factor, and p t Used to characterize the actual predicted matching probability; The boundary position is optimized by the following formula: L B =∫ Ω Φ G (q)s θ (q)dq, Ω is used to represent the image domain, Φ G (q) is used to characterize the distance map generated by the real boundary, s θ (q) is the output used to represent the predicted matching probability.

5. The method according to claim 1, wherein The training set of the bone end features includes: The bone images are preprocessed using a cropping strategy to obtain a training set of bone end features.

6. The method according to claim 1, characterized in that The sampling strategy includes: Dynamically adjusting the sampling probability of each area of the bone image using a dynamic sampling mechanism based on the judgment result of the edge sampling trigger condition; Wherein, determining whether the edge sampling trigger condition is met is performed by the following steps: comparing a first threshold value with a parameter value, if the parameter value is less than the first threshold value, the edge sampling trigger condition is met, otherwise, the edge sampling trigger condition is not met; If the edge sampling trigger condition is met, sampling is performed from the first area with a first preset probability; if the edge sampling trigger condition is not met, conventional random slice selection is performed.

7. A multi-task network-based bone CT image processing system, characterized in that: The multi-task network-based bone CT image processing system is used to implement the multi-task network-based bone CT image processing method described in any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method steps according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images