Clinical vertebra image segmentation method and apparatus for assisting pedicle screw placement surgery

Abstract

A clinical vertebra image segmentation method for assisting pedicle screw placement surgery, said method comprising: constructing a VerseDiff-UNet end-to-end framework, the framework being integrated with a denoising diffusion probabilistic model (DDPM); combining a noise-added image with a marked mask by using the VerseDiff-UNet framework, and guiding a diffusion direction toward a target region; and introducing a shape priors module on the basis of the DDPM, and extracting structural semantic information from an input spine image. In order to capture specific anatomical prior information in a medical image, the shape priors module is combined and the module effectively extracts the structural semantic information from the input spine image, thereby enabling more accurate anatomical structure segmentation, and facilitating accurate diagnosis and treatment of spinal disorders.

Description

A clinical spine image segmentation method and device for assisting in a spine screw placement operation TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a clinical spine image segmentation method and device for assisting in a spine screw placement operation. BACKGROUND

[0002] The spine is the second lifeline of the human body, supporting the normal operation of the body's organs, however, spinal diseases are common and have a high rate of disability, seriously affecting the quality of life of patients, and if not effectively treated, it will bring heavy burden to the patient's family and society, therefore, the treatment of spine-related diseases has become a problem to be solved. In the treatment of spinal diseases, doctors usually use CT (computed tomography) images or X-ray images to observe the spine condition of the patient, however, the display of these images is limited and it is difficult to provide enough intuitive and clear information, therefore, if an algorithm can be used to directly segment the tissues and organs that are beneficial to the doctor's diagnosis in the image and extract effective information from it, it will help the doctor to observe more clearly and diagnose more quickly, further improving the disease diagnosis rate and cure rate.

[0003] Segmentation of vertebrae and intervertebral discs (IVD) in spine images is widely used in the evaluation, diagnosis, surgical planning and image-guided intervention of spinal diseases. The widely used spine clinical screw placement operation in the clinic is to display the position of the screw in the two-dimensional image through intraoperative X-ray fluoroscopy to assist the surgeon in determining the screw placement point. However, due to the lack of three-dimensional images and the presence of ghosting and noise in X-ray images, this technology has limitations and it is quite challenging to determine the screw position.

[0004] The current common research on spine images is roughly divided into image preprocessing, intervertebral disc positioning and segmentation, and vertebral block positioning and segmentation. The core of the spine segmentation workflow is image segmentation, however, spine medical images have characteristics such as image blur, uneven gray scale distribution, multiple noise, and low image contrast, in addition, the spine structure is complex, when designing a spine medical image processing method, it is often necessary to design specific processing methods for different types of images or different types of parts to improve and optimize traditional image processing methods, the commonly used image processing algorithms are not ideal when applied to spine medical images, and it is difficult to form a unified and effective processing algorithm. In addition, in the case of corresponding exogenous implantation surgical instruments on the spine, the use of classic artificial intelligence assisted segmentation algorithms often has the problem of low segmentation accuracy or inability to complete the recognition and segmentation of the spine.

[0005] The state-of-the-art spine segmentation methods can be divided into three categories according to the structure of the segmentation object: vertebral body and intervertebral disc distribution segmentation, vertebral body and intervertebral disc simultaneous segmentation, and semantic image representation-based segmentation. Among them, the segmentation methods include three categories based on model-driven, machine learning and deep learning.

[0006] 1. Model-driven segmentation

[0007] This method relies on mathematical models and traditional manual features for vertebral body positioning and detection, such as active contour models and custom filters. For example, using active contour models and morphological operators to list all vertebral body inclination; using label fusion to combine the labels of the deformed atlas to obtain the final segmentation of the target dataset; using custom filters to extract the required anatomical features, etc.

[0008] 2. Machine learning-based segmentation

[0009] In terms of machine learning, regression forests and probabilistic graph models are used to detect vertebral bodies in arbitrary field-of-view CT scans, a supervised classification forest-based vertebral body positioning and recognition method is used; image displacement is estimated by jointly optimizing training and test displacement values in a data-driven manner and utilizing visual features of three-dimensional image patches to classify image voxels around the center of the intervertebral disc.

[0010] 3. Deep learning-based segmentation

[0011] Deep learning-based methods for pixel-level classification of spine images can accept input images of any size, use deconvolution layers to upsample the feature maps of the last convolution layer, and restore them to the same size as the input image, so that a prediction can be made for each pixel while preserving the spatial information in the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature maps to achieve accurate segmentation of vertebral bodies and intervertebral discs. With the advent of UNet networks, medical segmentation algorithms typically use an encoder-decoder structure combined with skip connections to enable the decoder to reuse the features extracted by the encoder. However, due to its structure based on convolutional neural networks, it cannot effectively extract global features. On the contrary, the structure of Transformer has significantly improved the performance of global feature modeling, but the corresponding dynamic extraction of global features not only increases the computational load but also causes compatibility to decline, iteration to be difficult to converge, and other problems. Recently, Diffusion Probabilistic Model (DPM) has gained popularity as a powerful generative model capable of generating high-quality and diverse images. Inspired by its success, some researchers have applied DPM to the field of medical image segmentation.

[0012] In summary, the shortcomings of the prior art are as follows:

[0013] To distinguish the spine from the background in the X-ray image in the spine screw placement surgery, many different spine medical image segmentation methods have been proposed. Previous methods based on graphics and statistics mainly model the pixel information of the original image, which divides the image into several regions by setting specific image intensity values, such as foreground and background of binary classification, involving threshold adjustment, fitting function, but due to the high difficulty, high complexity and asymmetric low precision of the operation, it has gradually been eliminated. In addition, the modeling ability of manually constructed features is limited, and the segmentation method based on traditional machine learning has low robustness, and the spine image segmentation needs to extract features manually, which misses the task specificity and limits the performance of segmentation.

[0014] With the progress of deep learning technology, some studies have successfully applied neural network-based models. The classic convolutional neural network (CNN) usually adopts an encoder-decoder structure in medical segmentation algorithms and combines a skip connection to enable the decoder to reuse the features extracted by the encoder. But due to its structure based on convolutional neural network, it cannot effectively extract global features. On the contrary, the structure of Transformer has significantly improved the performance of global feature modeling, but the corresponding dynamic extraction of global features not only leads to an increase in computational complexity, but also causes compatibility to decline, iteration to be difficult to converge and other problems.

[0015] Recently, the diffusion probabilistic model (DPM) has gained popularity as a powerful generative model that can generate high-quality and diverse images. Inspired by its success, some researchers have applied DPM to the field of medical image segmentation. Most of these methods are based on the classic UNet decoding and encoding structure, and some researchers have introduced the Transformer architecture to extract deeper semantic segmentation features from the original image, but often ignore the difficulty of designing a Transformer segmentation architecture to meet compatibility and the geometric multiplicative growth of computational complexity.

[0016] SUMMARY

[0017] The embodiment of the application provides a clinical spine image segmentation method and device for assisting spine screw placement surgery, so as to at least solve the technical problem of poor image segmentation effect.

[0018] According to an embodiment of the application, a clinical spine image segmentation method for assisting spine screw placement surgery is provided, comprising the following steps:

[0019] S101: An end-to-end framework of VerseDiff-UNet is constructed, and the framework is integrated with a denoising diffusion probabilistic model (DDPM);

[0020] S102: Combine the image with added noise with the labeled mask using the VerseDiff-UNet framework to guide the diffusion direction to the target region.

[0021] S103: Introduce a shape prior module based on the denoising diffusion probability model DDPM to extract structural semantic information from the input spine image.

[0022] According to another embodiment of the present application, a clinical spine image segmentation device for assisting in a spine screw placement operation is provided, comprising:

[0023] A framework construction unit is configured to construct an end-to-end framework of the VerseDiff-UNet, which integrates a denoising diffusion probability model DDPM.

[0024] A mask combination unit is configured to combine the image with added noise with the labeled mask using the VerseDiff-UNet framework to guide the diffusion direction to the target region.

[0025] A semantic extraction unit is configured to introduce a shape prior module based on the denoising diffusion probability model DDPM to extract structural semantic information from the input spine image.

[0026] Further, the semantic extraction unit comprises:

[0027] A feature acquisition unit is configured to obtain multi-scale features through an encoder given the original image data.

[0028] A feature fusion unit is configured to input information to the network through a traditional down-sampling and up-sampling structure to extract information, and then through each layer of the shape prior module to extract information and sum up the corresponding decoder features of the same level to obtain multi-scale fusion features.

[0029] A result prediction unit is configured to enhance the skipped original features through interaction with the global shape prior to generate discriminative shape representations and global context, extract vertebral anatomical prior features of the original medical image using the shape prior model, and finally obtain a prediction result.

[0030] The clinical spine image segmentation method and device for assisting in a spine screw placement operation in the embodiments of the present application combine a shape prior module in order to capture specific anatomical prior information in medical images. The module effectively extracts structural semantic information from the input spine image, realizes more accurate anatomical structure segmentation, and promotes accurate diagnosis and treatment of spinal diseases. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0032] Fig. 1 is a flow chart of a clinical spine image segmentation method for assisting a spine screw placement surgery according to the present application;

[0033] Fig. 2 is a network architecture diagram of the present application;

[0034] Fig. 3 is a schematic diagram of a training and sampling process according to the present application;

[0035] Fig. 4 is a diagram of segmentation results of different medical image segmentation methods according to the present application;

[0036] Fig. 5 is a module diagram of a clinical spine image segmentation device for assisting a spine screw placement surgery according to the present application;

[0037] Fig. 6 is a module diagram of a clinical spine image segmentation device for assisting a spine screw placement surgery according to the present application. DETAILED DESCRIPTION

[0038] In order to make the technical personnel of the present application better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0039] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] Embodiment 1

[0041] According to an embodiment of the present application, a clinical spine image segmentation method for assisting a spine screw placement surgery is provided, as shown in Fig. 1, comprising the following steps:

[0042] S101: An end-to-end framework of VerseDiff-UNet is constructed, and a denoising diffusion probability model DDPM is integrated into the framework;

[0043] S102: The image with added noise is combined with the labeled mask using the VerseDiff-UNet framework, and the diffusion direction is guided to the target region.

[0044] S103: A shape prior module is introduced based on the denoising diffusion probability model DDPM to extract structural semantic information from the input spine image.

[0045] The clinical spine image segmentation method for assisting in the spine screw placement surgery in the embodiment of the application combines a shape prior module to capture specific anatomical prior information in a medical image. The module effectively extracts structural semantic information from an input spine image, realizes more accurate anatomical structure segmentation, and promotes accurate diagnosis and treatment of spinal diseases.

[0046] Specifically, to solve the technical problems existing in the prior art, the application proposes an end-to-end framework named VerseDiff-UNet, which integrates a denoising diffusion probability model (DDPM) into a standard U-shaped architecture. In each step, the application combines an image with added noise with a labeled mask to accurately guide the diffusion direction to the target region. In addition, to capture specific anatomical prior information in a medical image, the application combines a shape prior module. The module effectively extracts structural semantic information from an input spine image, realizes more accurate anatomical structure segmentation, and promotes accurate diagnosis and treatment of spinal diseases. In the general medical image segmentation background, the application first introduces a shape prior module on the DDPM-based model to significantly improve the segmentation performance of the denoising diffusion probability model. In order to improve the performance parameters of the model for spine segmentation under the X-ray imaging condition in the clinical screw placement surgery and further assist the doctor in positioning the spine, the application introduces a shape prior module to enhance the feature capture ability of the spine, which significantly improves the spine segmentation performance of the diffusion model.

[0047] The basic content of the technical scheme of the application is as follows:

[0048] As shown in FIG. 2, the UNet in the VerseDiff-UNet proposed by the application comprises two parts, an encoder and a decoder. First, given the original image data, multi-scale features are obtained through the encoder. The information input into the network is not only extracted through the traditional down-sampling and up-sampling structure, but also extracted through the shape prior science of each layer and summed with the corresponding decoder features of the same level to obtain multi-scale fusion features. Since the encoder features and the decoder features of the same level contain the same number and size of features, the skipped original features can be further enhanced through interaction with the global shape prior, thereby helping to generate discriminative shape representation and global context, thereby helping the model to better extract the vertebral anatomical prior features of the original medical image, and finally obtaining the prediction result. For the task of medical image segmentation, the network of the application directly generates a prediction mask as the output.

[0049] As shown in FIG. 3, the main architecture of the VerseDiff-UNet is based on the encoder-decoder of the UNet. The encoder-decoder is composed of three corresponding down-sampling and up-sampling modules. The diffusion model uses time steps to perform variational inference on Markov processes to learn the training data distribution. In the training phase, for each step, the anatomical structure is induced by adding the input image to the noise segmentation mask. Specifically, the model is input with a pair of original spine images with a manually labeled mask in the training phase, and the mask is gradually added with noise. The framework consists of a forward and backward process, also known as a training and sampling process. In the forward process at each time step, Gaussian noise is added to the image until the image becomes an isotropic Gaussian. In the reverse process, a Markov chain is constructed using the UNet network structure to obtain a clean image by constructing an incremental denoising sequence. Using different Gaussian noise, n different reasonable masks are generated.

[0050] The technical solutions of the application are described in detail as follows:

[0051] 1. Data augmentation

[0052] Before training, the application uses dynamic data augmentation to avoid network overfitting and improve the robustness of the model. Three types of data augmentation are included: the image is randomly rotated from -30° to 30° to simulate the rotation variance; the image is randomly translated by 1-5% to simulate the translation variance; random elastic deformation, random contrast adjustment to improve the generalization ability of the model. In order to ensure that all methods use the same augmented training data, the application uses the same random state for all methods when performing data augmentation.

[0053] 2. Training phase one

[0054] Unlike traditional medical image segmentation methods that directly input raw image data to predict the corresponding segmentation label map, the diffusion model uses the raw image and the segmentation label map with noise as input to learn the denoising process. In the diffusion forward process, the present application adds a continuous t-step noise to the converted multi-channel label.

[0055] According to the formula: The label map x with t-step noise is obtained t After that, the present application will predict the clear label map x0 from x t and the original image data through the denoising module.

[0056] 3. Training phase two

[0057] The UNet in the VerseDiff-UNet proposed by the present application also contains two parts, an encoder and a decoder. First, given the original image data I∈R N×D×W×H , where N is the number of modal images, I and the noise one-hot label x t are concatenated to the encoder of the VerseDiff-UNet to obtain multi-scale features I E . The information input into the network not only passes through the traditional down-sampling and up-sampling structure, but also passes through the S f information extraction of each layer shape priori learning module, and is added to the corresponding decoder feature I D of the same layer to obtain multi-scale fusion features I f . Because the encoder feature I E and the decoder feature I D of the same level contain the same number and size, through the interaction with the global shape priori, the original feature that is skipped can be further enhanced, so as to promote the generation of features with discriminative shape representation and global context, which is helpful for the model to better extract the spinal anatomy priori features of the original medical image and finally obtain the prediction result In the design of the present application, the task of medical image segmentation, the network of the present application directly generates the predicted mask as the output.

[0058] Wherein, the step index t is integrated with the added embedding and decoder function. In each of them, it is embedded using a shared learning lookup table. Specifically, it is:

[0059] 4. Training phase three

[0060] The training and sampling architecture is shown in FIG. 3. The main architecture of VerseDiff-UNet is based on an encoder-decoder of UNet. The encoder-decoder is composed of three corresponding down-sampling and up-sampling modules respectively. The diffusion model performs variational inference on the Markov process to learn the training data distribution p(x0) with T time steps. In the training phase, the input image is added to the noise segmentation mask x t every t steps, thereby inducing anatomical structures. Specifically, the training phase inputs the model with pairs of original spine images and manually labeled masks, and gradually adds noise to the masks. In the forward process, Gaussian noise is added to the image x0~p(x0) every time step in T until the image becomes an isotropic Gaussian image. This forward noise process is represented as:

[0061] where (x0, x1, x2…x T ) represents T steps in the Markov chain, and a is a noise scheduler that controls the variance of the noise.

[0062] In the reverse process, an incremental denoising sequence is established using the Markov chain and the UNet network structure to obtain clean images. The parameters are as follows:

[0063] These are obtained by minimizing the KL-divergence between the forward distribution and the reverse distribution of all time steps.

[0064] The loss function is defined as:

[0065] The number of iterations is sampled from a uniform distribution, and e is sampled from a Gaussian distribution.

[0066] In summary, for the segmentation of clinical medical images of the human spine, the present application mainly solves the following problems:

[0067] 1. In clinical practice, medical images of the spine have the characteristics of image blur, uneven gray scale distribution, high noise, and low image contrast. In addition, since the spine structure is composed of a series of structurally similar but differently classified vertebrae, current segmentation methods for these images still face problems such as inter-class similarity, intra-class variation, and spatial bias, and the accuracy of segmentation using traditional machine learning methods is low;

[0068] 2. Due to the complexity of the automatic segmentation task of the spine, it is often difficult to obtain a sufficient number and quality of labeled data sets, and manual segmentation of these data sets is time-consuming and labor-intensive. In addition, there may be a problem of class imbalance in the data set, for example, due to the shape of the spine, which results in a large aspect ratio of the corresponding medical image, further exacerbating the difficulty in spinal radiological analysis;

[0069] 3. In the clinical vertebral implantation surgery, accurate identification and segmentation of the spinal structure is crucial for the spine surgery physician. In the case of corresponding exogenous implantation surgical instruments on the vertebra, some classic artificial intelligence assisted segmentation algorithms often have low segmentation accuracy or even cannot complete the identification and segmentation of the vertebra.

[0070] The key points and points to be protected of the present application are:

[0071] 1) The DPM model in the vertebra image segmentation first introduces a shape prior module, which effectively extracts structural semantic information from the input spine image for training the diffusion model;

[0072] 2) The diffusion model and the shape prior module are combined for segmentation network training, which is a new type of medical image segmentation architecture;

[0073] 3) In the preprocessing stage of the model, eight types of data enhancement methods are used, and the same random state is used for training data enhancement by introducing rotation, translation, elastic deformation and other changes to enhance the diversity, robustness and generalization ability of the model, reduce overfitting, and make the model more suitable for various situations in actual application, and improve the learnability of the image.

[0074] Compared with the prior art, the present application has the following advantages:

[0075] 1. The present application proposes a general end-to-end 2D medical image segmentation framework named VerseDiff-UNet, which uses a denoising diffusion probability model (DDPM) to meet the demand for accurate and diversified vertebra multi-type medical image segmentation templates, so that the model has better robustness and accuracy;

[0076] 2. In order to capture specific anatomical prior information in medical images, the present application combines a shape prior module in the Unet framework, which effectively extracts structural semantic information from the input spine image to significantly improve the segmentation effect;

[0077] 3. It is proved that the diffusion model trained effectively can obviously surpass other deep learning models in extracting effective information features for vertebra segmentation in the case of bone screw coverage on the vertebra in the vertebra implantation surgery.

[0078] The present application has been proved to be feasible through experiments, simulations and use, and has a high degree of automation and has been effectively verified through a large number of experiments. The quantitative calculation result is stable and accurate. The vertebra segmentation method proposed in the present application is different from the traditional medical image segmentation method of directly inputting original image data to predict the corresponding segmentation label image. The diffusion model uses the original image and the segmentation label image with noise as input to learn the denoising process.

[0079] To demonstrate the superiority of the proposed model, the present application conducts comparative experiments with other state-of-the-art methods on the disclosed CT spine dataset. The dataset consists of 609 anteroposterior X-ray images of the spine, with the spine markers provided by two professional doctors at the London Health Sciences Center. Each vertebra is located by four markers relative to four corners. All image sizes are 250x750. In addition, due to the extremely serious noise artifacts in the dataset and the fact that some data is even mixed with medical equipment on the spine, in order to further test the proposed model in this special case, the present application only targets the vertebral region and performs 0 / 1 binary transformation on the region to perform the original labeling. Although this method of operation may cause the loss of labeled areas in certain parts of the spine, the final experimental results effectively test the model's ability to segment occluded spine images under such medical imaging conditions.

[0080] When evaluating the results, the present application uses the Dice Similarity Coefficient (DSC) and the Intersection over Union (IoU) as quantitative indicators for evaluating segmentation performance. Both indicators are calculated for individual objects in the original image space and then averaged for all objects.

[0081] From the evaluation results, different models show different performances in terms of Dice coefficient and IoU. In terms of Dice coefficient, the Verse-diffu model achieves the highest score of 0.7865, demonstrating its excellent performance in X-ray image segmentation tasks. On the other hand, the UNet model performs relatively weakly in this regard. For the IOU indicator, the Verse-diffu model (0.6765) performs best, while the UNet model (0.4375) performs relatively poorly. The Deeplabv3-resnet50 model achieves satisfactory performance in both indicators, particularly in IoU (0.5791). The SAM model performs relatively consistently, slightly better than the Swin-transformer model.

[0082] Table 1: Comparative experimental indicators of segmentation methods on X-ray dataset

[0083] In addition to using quantitative indicators, the present application randomly selects three samples for visual analysis and subjective evaluation, as shown in Figure 4. Due to the high noise and high artifacts of the dataset, some images in the background pixels show a very high degree of similarity between them. The Swin-transformer model is very sensitive to relative position information and fails to effectively extract key feature information of the spine region during actual training, resulting in slow convergence and difficulty in effectively learning information about the semantic target region.

[0084] In this case, the SAM model takes the target region and the background region as a whole, as shown in the upper circle of FIG. 4, resulting in the loss of semantic segmentation, thereby reducing the segmentation accuracy of the entire data set. The UNet model is difficult to accurately segment the spine blocks, thereby causing them to be connected together. The FCN and Deeplabv3 models are subject to the loss of spine blocks to varying degrees. In contrast, the Verse-diffu model proposed in the present application can effectively extract the spine features in this case, but there is also a certain degree of loss of detailed features.

[0085] In addition, as shown in the lower circle of FIG. 4, the FCN and Deeplabv3 lose the ability to capture the features of the spine region under the imaging condition of the bone nail covering the spine. Compared with the original image, the only inferred spine part is the edge of the region not covered by the bone nail.

[0086] In contrast, the model proposed in the present application can still directly infer the vertebral region in a given image. This is because the model learns only the specific features in the spine region when learning the spine features during the noise addition process, and considers other parts of the region as noise, so in the reverse denoising stage, all the pixels above are removed as noise, and therefore only the feature region containing the vertebral target is smoothly segmented by the feature information obtained in the training stage under this condition. Therefore, the diffusion model effectively trained in the vertebral nail placement surgery can significantly outperform other deep learning models under the imaging condition of the bone nail covering the spine, thereby extracting effective information features and performing vertebral segmentation.

[0087] Embodiment 2

[0088] According to another embodiment of the present application, there is provided a clinical vertebral image segmentation device for assisting vertebral nail placement surgery, as shown in FIG. 5, comprising:

[0089] The framework construction unit 201 is configured to construct an end-to-end framework of the VerseDiff-UNet, and the framework is integrated with a denoising diffusion probability model DDPM;

[0090] The mask combination unit 202 is configured to combine the image with added noise and the labeled mask using the VerseDiff-UNet framework, and guide the diffusion direction to the target region;

[0091] The semantic extraction unit 203 is configured to introduce a shape prior module based on the denoising diffusion probability model DDPM, and extract structural semantic information from the input spine image.

[0092] The clinical spine image segmentation device for assisting in the spine screw placement operation in the embodiment of the application combines a shape prior module in order to capture specific anatomical prior information in medical images. The module effectively extracts structural semantic information from the input spine image, realizes more accurate anatomical structure segmentation, and promotes accurate diagnosis and treatment of spinal diseases.

[0093] The semantic extraction unit 203 comprises:

[0094] The feature acquisition unit 2031 is configured to obtain multi-scale features through an encoder given original image data;

[0095] The feature fusion unit 2032 is configured to input information to the network first through a traditional down-sampling and up-sampling structure to extract information, and then extract information through each layer of a shape prior module and sum up the decoder features corresponding to the same level to obtain multi-scale fusion features;

[0096] The result prediction unit 2033 is configured to enhance the skipped original features through interaction with a global shape prior to generate discriminative shape representations and global contexts, extract vertebral anatomical prior features of the original medical image using a shape prior model, and finally obtain a prediction result.

[0097] Specifically, in view of the technical problems existing in the prior art, the application proposes an end-to-end framework named VerseDiff-UNet, which integrates a denoising diffusion probability model (DDPM) into a standard U-shaped architecture. In each step, the application combines the image with added noise with a labeled mask to accurately guide the diffusion direction to the target area. In addition, in order to capture specific anatomical prior information in medical images, the application combines a shape prior module. The module effectively extracts structural semantic information from the input spine image, realizes more accurate anatomical structure segmentation, and promotes accurate diagnosis and treatment of spinal diseases. In the general medical image segmentation background, the application first introduces a shape prior module on the DDPM-based model, which significantly improves the segmentation performance of the denoising diffusion probability model. In order to improve the performance parameters of the model for spine segmentation under the X-ray imaging condition in the clinical screw placement operation and further assist the doctor in positioning the spine, the application introduces a shape prior module to enhance the feature capture ability of the spine, which significantly improves the spine segmentation performance of the diffusion model.

[0098] The basic content of the segmentation device technical solution of the application is as follows:

[0099] As shown in FIG. 2, the UNet in the VerseDiff-UNet proposed by the application comprises two parts, an encoder and a decoder. First, given the original image data, multi-scale features are obtained through the encoder. The information input into the network is not only extracted through the traditional down-sampling and up-sampling structure, but also extracted through the shape prior science of each layer and summed with the corresponding decoder features of the same level to obtain multi-scale fusion features. Since the encoder features and the decoder features of the same level contain the same number and size of features, the skipped original features can be further enhanced through interaction with the global shape prior, thereby helping to generate discriminative shape representation and global context, thereby helping the model to better extract the vertebral anatomical prior features of the original medical image, and finally obtaining the prediction result. For the task of medical image segmentation, the network of the application directly generates a prediction mask as the output.

[0100] As shown in FIG. 3, the main architecture of the VerseDiff-UNet is based on the encoder-decoder of the UNet. The encoder-decoder is composed of three corresponding down-sampling and up-sampling modules respectively. The diffusion model uses time steps to perform variational inference on Markov processes to learn the training data distribution. In the training stage, for each step, the anatomical structure is induced by adding the input image to the noise segmentation mask. Specifically, the model is input with a pair of original spine images with a manually labeled mask in the training stage, and the mask is gradually added with noise. The framework consists of a forward and backward process, also known as a training and sampling process. In the forward process at each time step, Gaussian noise is added to the image until the image becomes an isotropic Gaussian. In the reverse process, a Markov chain is constructed using the UNet network structure to obtain a clean image by constructing an incremental denoising sequence. Using different Gaussian noise, n different reasonable masks are generated.

[0101] The segmentation device technical solution of the application is described in detail as follows:

[0102] 1. Data augmentation

[0103] Before training, the application uses dynamic data augmentation to avoid overfitting of the network and improve the robustness of the model. There are three types of data augmentation: the image is randomly rotated from -30° to 30° to simulate the rotation variance; the image is randomly translated by 1-5% to simulate the translation variance; random elastic deformation, random contrast adjustment to improve the generalization ability of the model. In order to ensure that all methods use the same augmented training data, the application uses the same random state for all methods when performing data augmentation.

[0104] 2. Training phase one

[0105] Unlike traditional medical image segmentation methods that directly input raw image data to predict the corresponding segmentation label map, the diffusion model uses the raw image and the segmentation label map with noise as input to learn the denoising process. In the diffusion forward process, the present application adds a continuous t-step noise to the converted multi-channel label.

[0106] According to the formula: The label map x with t-step noise is obtained t After that, the present application will predict the clear label map x0 through the denoising module according to x t and the original image data.

[0107] 3. Training phase two

[0108] The UNet in the VerseDiff-UNet proposed by the present application also contains two parts, an encoder and a decoder. First, given the original image data I∈R N×D×W×H , where N is the number of modal images, I and the noise one-hot label x t are concatenated to the encoder of the VerseDiff-UNet to obtain multi-scale features I E . The information input into the network not only passes through the traditional down-sampling and up-sampling structure, but also passes through the S f extracted by each layer of the shape prior learning module, and is added to the corresponding decoder feature I D of the same layer to obtain multi-scale fusion features I f . Because the encoder feature I E and the decoder feature I D of the same level contain the same number and size, through the interaction with the global shape prior, the original feature that is skipped can be further enhanced, so as to promote the generation of features with discriminative shape representation and global context, which is helpful for the model to better extract the spinal anatomy prior features of the original medical image and finally obtain the prediction result In the design of the present application, the task of medical image segmentation, the network of the present application directly generates the predicted mask as the output.

[0109] Wherein, the step index t is integrated with the added embedding and decoder function. In each of them, it is embedded using a shared learning lookup table.

[0110] The calculation process is as follows:

[0111] 4. Training phase three

[0112] The training and sampling architecture is shown in FIG. 3. The main architecture of VerseDiff-UNet is based on an encoder-decoder of UNet. The encoder-decoder is composed of three corresponding down-sampling and up-sampling modules respectively. The diffusion model performs variational inference on the Markov process, and learns the training data distribution p(x0) with T time steps. In the training phase, the input image is added to the noise segmentation mask x t every t steps, thereby inducing anatomical structures. Specifically, the training phase inputs the model with pairs of original spine images and manually labeled masks, and gradually adds noise to the masks. In the forward process, Gaussian noise is added to the image x0~p(x0) every time step in T, until the image becomes an isotropic Gaussian image. This forward noise process is represented as:

[0113] where (x0, x1, x2…x T ) represents T steps in the Markov chain, and a is a noise scheduler that controls the variance of the noise.

[0114] In the reverse process, an incremental denoising sequence is established using the Markov chain and the UNet network structure to obtain clean images. The parameters are as follows:

[0115] These are obtained by minimizing the KL-divergence between the forward distribution and the reverse distribution of all time steps.

[0116] The loss function is defined as:

[0117] The number of iterations is sampled from a uniform distribution, and ∈ is sampled from a Gaussian distribution.

[0118] In summary, for the segmentation of clinical medical images of the human spine, the present application mainly solves the following problems:

[0119] 1. In clinical practice, medical images of the spine have the characteristics of image blur, uneven gray scale distribution, high noise, and low image contrast. In addition, since the spine structure is composed of a series of structurally similar but different types of vertebrae, the current segmentation methods for these images still face problems such as inter-class similarity, intra-class variation, and spatial bias, and the accuracy of segmentation using traditional machine learning methods is low;

[0120] 2. Due to the complexity of the automatic segmentation task of the spine, it is often difficult to obtain a sufficient number and quality of labeled data sets, and manual segmentation of these data sets is time-consuming and labor-intensive. In addition, there may be a problem of class imbalance in the data set, for example, due to the shape of the spine, which leads to a further increase in the difficulty of spinal imaging analysis due to the large aspect ratio of the medical image;

[0121] 3. In the clinical vertebral implantation surgery, accurate identification and segmentation of the spinal structure is crucial for the spine surgery physician. In the case of corresponding exogenous implantation surgical instruments on the vertebra, some classic artificial intelligence assisted segmentation algorithms often have low segmentation accuracy or even cannot complete the identification and segmentation of the vertebra.

[0122] The key points and points to be protected of the present application are:

[0123] 1) The DPM model in the vertebra image segmentation first introduces a shape prior module, which effectively extracts structural semantic information from the input spine image for training the diffusion model;

[0124] 2) The diffusion model and the shape prior module are combined for segmentation network training, which is a new type of medical image segmentation architecture;

[0125] 3) In the preprocessing stage of the model, eight types of data enhancement methods are used, and the same random state is introduced for training data enhancement by introducing rotation, translation, elastic deformation and other changes, which can enhance the diversity, robustness and generalization ability of the model, reduce overfitting, and make the model more suitable for various situations in actual application, and improve the learnability of the image.

[0126] Compared with the prior art, the present application has the following advantages:

[0127] 1. The present application proposes a general end-to-end 2D medical image segmentation framework named VerseDiff-UNet, which uses a denoising diffusion probability model (DDPM) to meet the demand for accurate and diversified vertebra multi-type medical image segmentation templates, so that the model has better robustness and accuracy;

[0128] 2. In order to capture specific anatomical prior information in medical images, the present application combines a shape prior module in the Unet framework, which effectively extracts structural semantic information from the input spine image to significantly improve the segmentation effect;

[0129] 3. It is proved that the diffusion model trained effectively can obviously surpass other deep learning models in extracting effective information features for vertebra segmentation in the case of bone screw coverage on the vertebra in the vertebra implantation surgery.

[0130] The present application has been proved to be feasible through experiments, simulations and use, and has a high degree of automation and has been effectively verified through a large number of experiments. The quantitative calculation results are stable and accurate. The vertebra segmentation method proposed in the present application is different from the traditional medical image segmentation method of directly inputting original image data to predict the corresponding segmentation label image. The diffusion model uses the original image and the segmentation label image with noise as input to learn the denoising process.

[0131] To demonstrate the superiority of the proposed model, the present application conducts comparative experiments with other state-of-the-art methods on the published CT spine dataset. The dataset consists of 609 anteroposterior X-ray images of the spine, with the spine markers provided by two professional doctors at the London Health Sciences Center. Each vertebra is located by four markers relative to four corners. All image sizes are 250x750. In addition, due to the extremely serious noise artifacts in the dataset and the fact that some data is even mixed with medical equipment on the spine, in order to further test the proposed model in this special case, the present application only targets the vertebral region and performs 0 / 1 binary transformation on this region to perform the original labeling. Although this method of operation may cause the loss of labeled areas in certain parts of the spine, the final experimental results effectively test the model's ability to segment occluded spine images under such medical imaging conditions.

[0132] When evaluating the results, the present application uses the Dice Similarity Coefficient (DSC) and the Intersection over Union (IoU) as quantitative indicators for evaluating segmentation performance. Both indicators are calculated for individual objects in the original image space and then averaged for all objects.

[0133] From the evaluation results, different models show different performances in terms of Dice coefficient and IoU. In terms of Dice coefficient, the Verse-diffu model achieved the highest score of 0.7865, demonstrating its excellent performance in X-ray image segmentation tasks. On the other hand, the UNet model performed relatively weakly in this regard. For the IOU indicator, the Verse-diffu model (0.6765) performed best, while the UNet model (0.4375) performed relatively poorly. The Deeplabv3-resnet50 model achieved satisfactory performance on both indicators, particularly in terms of IoU (0.5791). The SAM model performed relatively consistently, slightly better than the Swin-transformer model.

[0134] Table 1: Comparative experimental indicators of segmentation methods on X-ray dataset

[0135] In addition to using quantitative indicators, the present application randomly selects three samples for visual analysis and subjective evaluation, as shown in Figure 4. Due to the high noise and high artifacts of the dataset, some images in the background pixels show a very high degree of similarity between them. The Swin-transformer model is very sensitive to relative position information and fails to effectively extract key feature information of the spine region during actual training, resulting in slow convergence and difficulty in effectively learning information about the semantic target region.

[0136] In this case, the SAM model takes the target region and the background region as a whole, as shown in the upper circle of FIG. 4, resulting in the loss of semantic segmentation, thereby reducing the segmentation accuracy of the entire data set. The UNet model is difficult to accurately segment the spine blocks, thereby causing them to be connected together. The FCN and Deeplabv3 models are subject to the loss of spine blocks to varying degrees. In contrast, the Verse-diffu model proposed in the present application can effectively extract the spine features in this case, but there is also a certain degree of loss of detailed features.

[0137] In addition, as shown in the lower circle of FIG. 4, the FCN and Deeplabv3 lose the ability to capture the features of the spine region under the imaging condition of the bone nail covering the vertebrae. Compared with the original image, the only inferred spine part is the edge of the region not covered by the bone nail.

[0138] In contrast, the model proposed in the present application can still directly infer the vertebrae region in a given image. This is because the model learns only the specific features in the spine region when learning the spine features during the noise addition process, and considers other parts of the region as noise, so in the reverse denoising stage, all the pixels above are removed as noise, and therefore only the feature region containing the vertebrae target is smoothly segmented by the feature information obtained in the training stage under this condition. Therefore, the diffusion model effectively trained in the vertebroplasty procedure can significantly outperform other deep learning models under the imaging condition of the bone nail covering the vertebrae, thereby extracting effective information features and performing vertebrae segmentation.

[0139] Embodiment 3

[0140] A storage medium storing a program file capable of implementing any of the above-mentioned auxiliary vertebroplasty clinical spine image segmentation methods.

[0141] Embodiment 4

[0142] A processor for running a program, wherein the program performs the above-mentioned auxiliary vertebroplasty clinical spine image segmentation method when running.

[0143] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0144] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0145] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the system embodiments described above are only illustrative, for example, the division of units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection between the units or modules through some interfaces, and can be electrical or other forms.

[0146] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0147] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0148] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or all or part of the technical solutions that contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The storage medium mentioned above includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0149] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A clinical spinal image segmentation method for assisting spinal screw placement surgery, characterized in that, The method comprises the following steps: S101: An end-to-end framework of VerseDiff-UNet is constructed, and a denoising diffusion probability model DDPM is integrated into the framework; S102: The image with added noise is combined with the labeled mask using the VerseDiff-UNet framework, and the diffusion direction is guided to the target region; S103: A shape prior module is introduced based on the denoising diffusion probability model DDPM, and structural semantic information is extracted from the input spine image.

2. The clinical image segmentation method for assisting a spine drilling surgery according to claim 1, wherein, Step S103 specifically comprises: Given the original image data, multi-scale features are obtained through an encoder; The information input into the network is first extracted through a traditional down-sampling and up-sampling structure, and then information is extracted through each layer of the shape prior module and summed with the corresponding decoder features of the same level to obtain multi-scale fusion features; The original features are enhanced through interaction with the global shape prior to generate discriminative shape representations and global context, the shape prior model is used to extract the vertebral anatomical prior features of the original medical image, and finally the prediction result is obtained.

3. The method of claim 2, wherein the method further comprises: In step S103, the encoder-decoder is composed of three corresponding down-sampling and up-sampling modules; the denoising diffusion probability model DDPM uses time steps to perform variational inference on Markov processes to learn the training data distribution, and in the training stage, for each step, the anatomical structure is induced by adding the input image to the noise segmentation mask.

4. The clinical image segmentation method for assisting a spine drilling surgery according to claim 3, wherein, In the training stage, the model is input with a pair of original spine images with manually labeled masks, and the masks are gradually added with noise; in the forward process of each time step, Gaussian noise is added to the image until the image becomes an isotropic Gaussian; In the reverse process, an incremental denoising sequence is constructed using the Markov chain of the UNet network structure to obtain a clean image; different Gaussian noise is sampled n times to generate n different reasonable masks.

5. The clinical image segmentation method for assisting a spine drilling surgery according to claim 3, wherein, Before training, dynamic data augmentation is used to avoid network overfitting and improve the robustness of the model, including three types of data augmentation: randomly rotating the image from -30° to 30° to simulate the rotation variance; randomly translating the image by 1-5% to simulate the translation variance; random elastic deformation, random contrast adjustment to improve the generalization ability of the model.

6. The clinical image segmentation method of claim 3, wherein, The denoising diffusion probability model DDPM uses the original image and the segmentation label image with noise as input to learn the denoising process.

7. The method of claim 2, wherein the method further comprises: Step S103 specifically comprises: Given the original image data I∈R N×D×W×H , where N is the number of modal images, I and noise single-hot label x t are concatenated into the encoder of VerseDiff-UNet to obtain multi-scale features I E ; The information of the input network is first processed by a traditional down-sampling and up-sampling structure, and then is extracted by a shape prior learning module in each layer S f and the decoder features I of the corresponding layer D are added to obtain multi-scale fusion features I f ; By interaction with the global shape prior, the original features that are skipped are enhanced to generate discriminative shape representations and global context, the shape prior model is used to extract the vertebral anatomical prior features of the original medical image, and finally the prediction result is obtained wherein the step index t is integrated with the added embedding and decoder functionality; in each of which the embedding uses a shared learned lookup table:

8. The clinical image segmentation method for assisting a spine drilling surgery according to claim 7, wherein, In the forward process, each time step in T adds Gaussian noise to the image x0~p(x0) until the image becomes an isotropic Gaussian image; the forward noise process is denoted as: where (x0, x1, x2...x T ) denotes T steps in Markov chain, and a is a noise scheduler controlling the noise variance; In the inverse process, an incremental denoising sequence is established by using Markov chain and UNet network structure to obtain a clean image; the parameters are as follows: The KL-divergence between the forward distribution and the reverse distribution of all time steps is minimized to obtain; where the loss function is defined as: The number of iterations is sampled from a uniform distribution, and ∈ is sampled from a Gaussian distribution.

9. A clinical spinal image segmentation device for assisting spinal screw placement surgery, characterized in that, It comprises: a framework construction unit, configured to construct an end-to-end framework of VerseDiff-UNet, and integrate a denoising diffusion probability model DDPM into the framework; a mask combination unit, configured to combine the image with added noise with the labeled mask using the VerseDiff-UNet framework, and guide the diffusion direction to the target region; a semantic extraction unit, configured to introduce a shape prior module based on the denoising diffusion probability model DDPM, and extract structural semantic information from the input spine image.

10. The clinical spinal image segmentation apparatus for assisting a spinal drill surgery of claim 9, wherein, The semantic extraction unit comprises: The feature acquisition unit is configured to obtain multi-scale features through an encoder for given original image data; The feature fusion unit is configured to input information to the network through a traditional down-sampling and up-sampling structure to extract information, and then through each layer of a shape prior module to extract information and sum up with corresponding decoder features of the same level, so as to obtain multi-scale fusion features. The result prediction unit is configured to enhance the skipped original features through interaction with a global shape prior, generate discriminative shape representation and global context, extract vertebral anatomical prior features of the original medical image by using a shape prior model, and finally obtain a prediction result.

Images