Deep learning-based spinal interbody fusion detection apparatus and method
By generating spinal healing images using deep learning technology and combining them with clinical data, the accuracy problem of intervertebral fusion detection has been solved, enabling precise judgment of fusion status and prediction of healing trends, thus reducing medical costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU ZEKANG ZHIGU TECH CO LTD
- Filing Date
- 2022-07-19
- Publication Date
- 2026-05-15
AI Technical Summary
Existing methods for detecting intervertebral fusion lack unified judgment criteria, are highly subjective, and lead to inaccurate detection.
A deep learning-based spinal intervertebral fusion detection device is used to generate spinal healing images using deep convolutional neural networks and generative adversarial networks. Combined with self-supervised contrastive learning and temporal neural ensemble networks, accurate fusion status judgment and healing trend prediction are performed.
It improves the accuracy of intervertebral fusion detection, refines the criteria for fusion assessment, predicts healing trends and time, guides scientific postoperative rehabilitation treatment, and reduces medical costs.
Smart Images
Figure CN115187557B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical imaging technology, and more specifically to a deep learning-based device and method for detecting intervertebral fusion of the spine. Background Technology
[0002] Modern society has brought about tremendous changes in people's lifestyles and work habits, which have also significantly impacted spinal bone diseases. According to surveys, the incidence of cervical spondylosis is 25% in people under 40; approximately 50% in those over 40; and over 85% in those over 60. There are over 2 million potential patients requiring cervical spondylosis surgery annually, with a total of 50 million patients. Lumbar disc herniation, another major category of spinal-related diseases, is one of the most common causes of lower back and leg pain. Prolonged sitting, long hours at a desk, and extended driving are contributing to the increasing incidence of lumbar disc herniation, while the age of onset is also gradually decreasing. Therefore, spinal-related orthopedic surgeries are also on the rise, with interbody fusion being the most typical example.
[0003] Bone healing is an important indicator for follow-up observation after orthopedic surgery. Generally speaking, bone healing refers to the formation of a continuous bone bridge connection between bones. For spinal fusion surgery, the main methods for judging the intervertebral fusion status include various imaging methods such as X-ray, CT and MRI. The criteria for judging fusion mainly include: (1) a bridge-like trabecular connection between endplates; (2) no radiolucent area between the implant and the endplate; (3) displacement between vertebral bodies in cervical flexion-extension radiographs; (4) displacement of the spinous process in cervical flexion-extension radiographs. Among them, criteria (1) and (2) are the most commonly used indicators, but they are highly subjective. Although criteria (3) and (4) are more objective, different studies have different definitions of displacement. It can be seen that the existing intervertebral fusion detection lacks a unified judgment standard and is highly subjective, resulting in inaccurate spinal intervertebral fusion detection. Summary of the Invention
[0004] This invention aims to address the problem of poor accuracy in existing intervertebral fusion detection methods by proposing a deep learning-based spinal intervertebral fusion detection device and method.
[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0006] On the one hand, a deep learning-based spinal interbody fusion detection device is provided, comprising:
[0007] The acquisition module is used to acquire images of the patient's spine;
[0008] The image segmentation module is used to perform image recognition and segmentation on the spinal image based on a deep convolutional neural network model, to obtain the first bone bridge image and its adjacent upper and lower vertebral body images in the spinal image;
[0009] An image generation module is used to generate a spinal healing image of the patient based on the first bone bridge image and its adjacent upper and lower vertebral body images and an adversarial generative network model, and to determine the second bone bridge image in the spinal healing image.
[0010] A determination module is used to determine the similarity between the first bone bridge image and the second bone bridge image, and to determine the intervertebral fusion status of the patient's spine based on the similarity.
[0011] Furthermore, the deep convolutional neural network model is the Point-UNet model, which is an improved network model of the UNet network, that is, the upsampling module of the UNet network is modified into the PointRend sub-network.
[0012] Furthermore, the Point-UNet model uses unlabeled data and is pre-trained based on self-supervised contrastive learning.
[0013] Furthermore, the adversarial generative network model includes multiple discriminators, which are used for weight co-training and data co-training.
[0014] Furthermore, the acquisition module is also used to acquire the patient's clinical data and time-series spinal images;
[0015] The determining module is further configured to determine the spinal intervertebral healing trend and healing time of the patient based on the clinical data and time-series spinal images and a time-series neural ensemble network model.
[0016] Furthermore, the clinical data includes at least the patient's age, sex, height, weight, and bone condition.
[0017] Furthermore, the acquisition module is also used to acquire real healing images of the patient; the system further includes:
[0018] An optimization module is used to generate real-world labeled data based on the real healing image and its corresponding time information, and to optimize the adversarial generative network model and the temporal neural ensemble network model based on the real-world labeled data.
[0019] On the other hand, a deep learning-based method for detecting intervertebral fusion of the spine is provided, including the following steps:
[0020] Obtain images of the patient's spine;
[0021] The spinal image is recognized and segmented based on a deep convolutional neural network model to obtain the first bone bridge image and its adjacent upper and lower vertebral body images in the spinal image;
[0022] Based on the first bone bridge image and its adjacent upper and lower vertebral body images, and using an adversarial generative network model, a spinal healing image of the patient is generated, and a second bone bridge image is determined in the spinal healing image.
[0023] The similarity between the first bone bridge image and the second bone bridge image is determined, and the intervertebral fusion status of the patient's spine is determined based on the similarity.
[0024] Furthermore, the deep convolutional neural network model is the Point-UNet model, which is an improved network model of the UNet network, that is, the upsampling module of the UNet network is modified into the PointRend sub-network.
[0025] Furthermore, the Point-UNet model uses unlabeled data and is pre-trained based on self-supervised contrastive learning.
[0026] Furthermore, the adversarial generative network model includes multiple discriminators, which are used for weight co-training and data co-training.
[0027] Furthermore, the method also includes:
[0028] Acquire the patient's clinical data and time-series spinal images;
[0029] Based on the clinical data and time-series spinal images, and using a time-series neural ensemble network model, the patient's spinal intervertebral healing trend and healing time were determined.
[0030] Furthermore, the clinical data includes at least the patient's age, sex, height, weight, and bone condition.
[0031] Furthermore, the method also includes:
[0032] Obtain real images of the patient's healing process;
[0033] Based on the real healing images and their corresponding time information, real labeled data is generated, and the adversarial generative network model and the temporal neural ensemble network model are optimized based on the real labeled data.
[0034] The beneficial effects of this invention are as follows: The deep learning-based spinal interbody fusion detection device and method described in this invention can help clinicians accurately determine the fusion status of the surgical segment after spinal fusion surgery. By incorporating image recognition technology into the criteria for determining interbody fusion, the criteria for determining fusion can be further subdivided, thereby enabling more precise fusion detection and improving the accuracy of fusion detection. At the same time, by combining the imaging images from previous follow-ups with the patient's clinical data, the healing trend and time of the patient can be predicted, allowing for early scientific intervention and more scientific guidance for postoperative rehabilitation treatment, effectively reducing medical costs. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the deep learning-based intervertebral fusion detection device for the spine according to an embodiment of the present invention;
[0036] Figure 2 This is a flowchart illustrating the deep learning-based intervertebral fusion detection method for the spine according to an embodiment of the present invention.
[0037] Figure 3 This is a schematic diagram of the structure for detecting healing status using the Point-Unet model as described in an embodiment of the present invention;
[0038] Figure 4 This is a schematic diagram of the structure of the contrastive learning framework described in an embodiment of the present invention;
[0039] Figure 5 This is a schematic diagram of the structure for generating spinal healing images based on a GAN model according to an embodiment of the present invention;
[0040] Figure 6 This is a schematic diagram of the structure for predicting healing trends and healing time based on a temporal neural ensemble network model, as described in an embodiment of the present invention.
[0041] Figure 7 This is a schematic diagram of the remote intelligent detection system for intervertebral fusion of the spine according to an embodiment of the present invention. Detailed Implementation
[0042] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0043] The deep learning-based spinal interbody fusion detection device described in this embodiment of the invention, such as... Figure 1 As shown, it includes: an acquisition module, an image segmentation module, an image generation module, and a determination module.
[0044] The acquisition module is used to acquire images of the patient's spine; the image segmentation module is used to perform image recognition and segmentation on the spine images based on a deep convolutional neural network model to obtain a first bone bridge image and its adjacent upper and lower vertebral body images; the image generation module is used to generate a spinal healing image of the patient based on the first bone bridge image and its adjacent upper and lower vertebral body images and an adversarial generative network model, and to determine a second bone bridge image in the spinal healing image; the determination module is used to determine the similarity between the first bone bridge image and the second bone bridge image, and to determine the intervertebral fusion status of the patient's spine based on the similarity.
[0045] Based on the aforementioned spinal interbody fusion detection system, the deep learning-based spinal interbody fusion detection method described in this embodiment, as follows: Figure 2 As shown, it includes the following steps:
[0046] Step 1: Obtain images of the patient's spine; these images can be medical diagnostic images of the patient's spine, such as X-ray images, CT images, or MRI images.
[0047] Step 2: Perform image recognition and segmentation on the spinal image based on a deep convolutional neural network model to obtain the first bone bridge image and its adjacent upper and lower vertebral body images in the spinal image;
[0048] The spine is divided into cervical, thoracic, and lumbar vertebrae, and each vertebra in different locations is composed of multiple vertebral bodies, meaning there are many types of vertebrae in the spine with relatively small differences in shape. Accurately segmenting the pixel regions of vertebrae at different locations and the pixel regions of newly formed bone (bone bridges) between vertebrae, and performing multi-class classification of bone bridge images, are key problems to be solved. This embodiment designs and trains a deep convolutional neural network-based model to perform multi-task training on spinal image data containing different bone bridge states, including image segmentation, bone bridge recognition, and bone bridge classification. By improving the classic model structure and parameters, the model's segmentation efficiency and classification performance are enhanced.
[0049] It is understandable that the UNet network, due to its unique network structure, can achieve good learning results on medical images. However, because the morphology of bone bridges varies greatly during the generation process, especially in the early stages of healing, the edges of bone bridges are blurred, making it difficult to extract precise boundaries. This embodiment, based on the UNet network, incorporates PointRend's ability to consider the image rendering process itself and optimize image segmentation for object edges into the UNet upsampling process, designing the Point-UNet model, such as... Figure 3As shown, in this embodiment, the PointRend subnetwork is used as an upsampling module of the Unet network, enabling it to achieve better segmentation performance on the edges of vertebrae and bone bridges that are difficult to segment. The first bone bridge image and its adjacent upper and lower vertebrae images are obtained by segmentation and classification by the Point-Unet model.
[0050] Supervised deep learning requires a large amount of labeled data, but this is often difficult to achieve in medical image analysis due to factors such as high labeling costs. This limits the solution of medical image analysis tasks based on supervised learning to some extent. Therefore, in this embodiment, a variant of self-supervised learning (SSL), "contrastive learning" (CL) framework, is used to pre-train the Point-UNet model using unlabeled data.
[0051] Self-supervised learning does not require manually labeled category information; it directly utilizes the data itself as supervision to learn the feature representations of the sample data. Contrastive learning, on the other hand, first learns the general feature representations of images on the unlabeled dataset, and then fine-tunes them using a small number of labeled images to improve the model's performance on a given task. In this embodiment, the contrastive learning method consists of three parts: data augmentation, encoding, and loss minimization. Figure 4 As shown, the specific steps include:
[0052] First, generate batches of size N from the original image. Figure 4 Example N=2), and two enhancement combinations are performed to obtain a pair of enhanced images that are essentially different representations of the same image.
[0053] Then, a feature vector is created for each image in the pair, and each image is processed by an encoder to obtain h. i and h j The feature vector z is obtained by using a non-linear fully connected layer. i and z j .
[0054] Finally, the two vectors z are maximized by minimizing the contrastive loss function. i and z j The similarity between the images is calculated, where for each pair of enhanced images, the contrast loss is calculated twice, the second time the loss for image position swapping is calculated, and finally, the average loss of all pairs of images in the batch is calculated.
[0055] By using a contrastive learning framework, the neural network's dependence on labeled data is reduced, allowing the model to still achieve high-accuracy fusion detection requirements.
[0056] Step 3: Generate the patient's spinal healing image based on the first bone bridge image and its adjacent upper and lower vertebral body images and an adversarial generative network model, and determine the second bone bridge image in the spinal healing image;
[0057] It is understandable that, due to the relative scarcity of fully healed intervertebral fusion images and the variability in healing morphology among patients, the prediction of healing trends for each patient should first be based on current digital imaging and a correct assessment of their healing morphology. Therefore, to more clearly determine the different states and conditions of intervertebral healing, this embodiment employs a Generative Adversarial Network (GAN) model to generate fully healed spinal healing images for each patient based on their postoperative spinal images, and to generate customized images of new bone bridges between vertebrae for each patient. This provides a reliable criterion for fusion detection during follow-up, further improving the accuracy of fusion detection, while also avoiding the impact of insufficient labeled data on accuracy.
[0058] High-fidelity image synthesis tasks typically require large amounts of training data. Training generative models in scenarios with limited data often leads to overfitting of the discriminator, significantly degrading the quality of the generated images. Therefore, this embodiment proposes a method for learning from small sample data of intervertebral fusion images by collaboratively training multiple discriminators to achieve different but complementary data learning. Specifically, this includes collaborative training from two perspectives: collaborative training of model weights and collaborative training from a data perspective. The model weight approach encourages different discriminators to learn diverse weights, while the data collaboration perspective encourages learning from data from different viewpoints to help alleviate overfitting.
[0059] like Figure 5 As shown, the overall framework for GAN-based image generation consists of four parts: data sampling, data generation, model weight coordination, and data coordination.
[0060] Weighted co-training uses the same input data and encourages the two discriminators, D1 and D2, to learn weights that are significantly different by using a weight distance metric. Then, cosine similarity is used to measure the distance between the two.
[0061] The data co-training uses different input data. D1 takes the original training data as input, and the data co-training D1 and the weight co-training D1 share parameters. D3 takes new data obtained by frequency domain transformation, which uses random frequency component suppression.
[0062] Step 4: Determine the similarity between the first bone bridge image and the second bone bridge image, and determine the intervertebral fusion status of the patient's spine based on the similarity.
[0063] This embodiment calculates the similarity between the first bone bridge image in the spinal image and the second bone bridge image in the spinal healing image, maps the similarity to the healing area, and finally enters the softmax function to obtain a refined fusion detection result.
[0064] Based on clinical testing, the similarity in this embodiment corresponds to five fusion states, such as... Figure 3 As shown, the grades are as follows: Grade 0 - completely unhealed, no new bone formation; Grade 1 - unhealed, with new bone formation, but no continuous bone bridge, and the new bone area is less than 50%; Grade 2 - unhealed, with new bone formation, but no continuous bone bridge, and the new bone area is greater than 50%; Grade 3 - healed, with continuous bone bridge formation, and the bone connection area is less than 50%; Grade 4: excellent healing - with continuous bone bridge formation, and the bone connection area is greater than 50%. The greater the similarity, the higher the grade of fusion, and vice versa. By subdividing the criteria for judging fusion status, more accurate fusion detection can be achieved.
[0065] To effectively predict the trajectory and trend of intervertebral fusion, this embodiment also includes:
[0066] Acquire the patient's clinical data and time-series spinal images; determine the patient's spinal intervertebral healing trend and healing time based on the clinical data and time-series spinal images and a time-series neural ensemble network model.
[0067] Considering that intervertebral fusion is a continuous healing process, this embodiment classifies and detects the bone healing status at different stages, while further predicting its healing trend and time. Specifically, a temporal neural ensemble network model is used to learn the patient's follow-up medical images in a temporal manner, and the patient's clinical feature encoding is integrated into the temporal neural ensemble network model, ultimately achieving classification prediction of the patient's intervertebral healing trend and regression prediction of healing time.
[0068] like Figure 6As shown, this embodiment designs and constructs an LSTM temporal neural ensemble network model. Spinal images with a time dimension are input into the LSTM model. High-dimensional feature representations between images are obtained by using follow-up images characterizing the patient's healing process (1 month post-surgery, 3 months post-surgery, 6 months post-surgery, etc.). Simultaneously, the patient's clinical data (age, gender, height, bone condition, etc.) are represented using a multilayer perceptron (MLP). Finally, a multilayer feedforward neural network fuses all features for calculation, and a healing trend prediction is obtained using a softmax function, while a regression prediction of healing time is obtained using a linear function. The healing trend includes positive, stagnant, and negative trends, i.e., improving, no change, and worsening. The healing time unit can be represented by weeks.
[0069] like Figure 2 As shown, to further improve the accuracy of fusion detection, this embodiment also includes:
[0070] Acquire real healing images of the patient; generate real labeled data based on the real healing images and their corresponding time information, and optimize the adversarial generative network model and the temporal neural ensemble network model based on the real labeled data.
[0071] As time progresses, the amount and changes in patient healing image data are continuously increasing. These changes mainly include two aspects: firstly, the morphology of newly formed bone in patients continuously changes until complete healing; secondly, specialists can verify predicted trends or timelines, continuously obtaining realistically labeled data. Therefore, this constantly evolving realistically labeled data can serve as training and validation datasets for deep neural networks. These datasets can be fed into GAN and temporal neural network models respectively, and the model parameters can be trained and optimized using online time-incremental learning to continuously improve model performance and further enhance the accuracy of fusion detection.
[0072] Based on the aforementioned deep learning-based spinal interbody fusion detection device, this embodiment can also construct a remote intelligent detection system for spinal interbody fusion based on a big data platform. This system can be deployed in the cloud to achieve distributed data storage and remote multi-terminal services. The system structure is as follows: Figure 7 As shown, from bottom to top, the layers are: cloud server cluster, data storage layer, data processing layer, core model layer, and application layer. Each layer is described below:
[0073] ①Underlying architecture: Based on cloud server cluster architecture, using Internet or 5G network for communication;
[0074] ② Data storage layer: Structured and unstructured data are stored in relational databases and HDFS file systems respectively;
[0075] ③ Data Processing Layer: Historical data can be processed in batch (offline) mode for model iteration; real-time data, i.e., follow-up images uploaded by patients via mobile devices, can be processed in stream mode for remote diagnosis and prediction. For raw multi-source data, data cleaning is performed first to remove useless and dirty data, then multi-source data is fused to enrich the data, and finally image data enhancement operations are performed to obtain a richer dataset.
[0076] ④ Core Model Layer: This mainly includes the aforementioned deep learning-based spinal intervertebral fusion detection device, namely, image healing diagnosis based on a deep convolutional network model, healing trend and healing time prediction based on a temporal neural ensemble network model, and continuous model learning based on constantly expanding new data. In healing diagnosis, self-supervised contrastive learning to reduce data labeling workload is employed, and GANs are used to generate fully healed images. Healing classification and diagnosis are achieved through image similarity calculation. The foundation for classification diagnosis and prediction is image instance segmentation; therefore, constructing a high-precision segmentation algorithm is fundamental to the performance of the entire model layer.
[0077] ⑤ Top-level application: The web-based application presents intelligent analysis and statistical data on patients' intervertebral fusion in a visual manner to professional physicians, assisting them in making rapid diagnoses and allowing for manual verification and feedback of the system's results; the mobile application allows patients to upload follow-up data and, through core services, view automatic diagnoses of the current healing status as well as predictive analysis of healing trends and time.
[0078] In summary, the deep learning-based spinal interbody fusion detection device and method described in this embodiment can achieve graded detection of interbody fusion. By evaluating the patient's medical images, it can accurately compare and calculate the bone formation morphology with the target morphology, obtaining a detailed graded classification of the healing status, enabling graded diagnosis and treatment during postoperative follow-up. Simultaneously, by analyzing the patient's follow-up images through a temporal neural network model, it can effectively analyze the patient's bone formation trend and predict the healing time, providing positive guidance for scientific prognosis. This embodiment can also be used to construct a remote intelligent detection system for spinal interbody fusion. This system can be deployed in the cloud and interface with the hospital's radiology department information system and medical record system to achieve information and data sharing and interoperability. It can provide corresponding services to specialists and patients through multiple terminals, thereby better realizing the downward flow of medical resources and establishing a graded diagnosis and treatment system, reducing the burden on large hospitals, and bringing convenience to patients. It can help clinicians accurately determine the fusion status of the surgical segment after spinal fusion surgery. By incorporating image recognition technology into the criteria for determining intervertebral fusion, the criteria for determining fusion can be subdivided, thereby making fusion detection more precise and improving the accuracy of fusion detection. At the same time, by combining the imaging images from previous follow-up visits with the patient's clinical data, the healing trend and time of the patient can be predicted, allowing for early scientific intervention and more scientific guidance for postoperative rehabilitation treatment, effectively reducing medical costs.
Claims
1. A deep learning-based spinal interbody fusion detection device, characterized in that, include: The acquisition module is used to acquire images of the patient's spine; The image segmentation module is used to perform image recognition and segmentation on the spinal image based on a deep convolutional neural network model, to obtain the first bone bridge image and its adjacent upper and lower vertebral body images in the spinal image; An image generation module is used to generate a spinal healing image of the patient based on the first bone bridge image and its adjacent upper and lower vertebral body images and an adversarial generative network model, and to determine the second bone bridge image in the spinal healing image. A determination module is used to determine the similarity between the first bone bridge image and the second bone bridge image, and to determine the intervertebral fusion status of the patient's spine based on the similarity. The deep convolutional neural network model is the Point-UNet model, which is an improved network model of the UNet network, that is, the upsampling module of the UNet network is modified into a PointRend sub-network; The adversarial generative network model includes multiple discriminators, which are used for weight co-training and data co-training. The acquisition module is also used to acquire the patient's clinical data and time-series spinal images; The determining module is further configured to determine the spinal intervertebral healing trend and healing time of the patient based on the clinical data and time-series spinal images and a time-series neural ensemble network model.
2. The deep learning-based spinal interbody fusion detection device as described in claim 1, characterized in that, The Point-UNet model uses unlabeled data and is pre-trained based on self-supervised contrastive learning.
3. The deep learning-based spinal interbody fusion detection device as described in claim 1, characterized in that, The clinical data include at least the patient's age, sex, height, weight, and bone condition.
4. The deep learning-based spinal interbody fusion detection device as described in claim 1, characterized in that, The acquisition module is further configured to acquire real healing images of the patient; the device also includes: An optimization module is used to generate real-world labeled data based on the real healing image and its corresponding time information, and to optimize the adversarial generative network model and the temporal neural ensemble network model based on the real-world labeled data.
5. A deep learning-based method for detecting intervertebral fusion in the spine, characterized in that, The method is performed by a deep learning-based spinal interbody fusion detection device and includes the following steps: Obtain images of the patient's spine; The spinal image is recognized and segmented based on a deep convolutional neural network model to obtain the first bone bridge image and its adjacent upper and lower vertebral body images in the spinal image; Based on the first bone bridge image and its adjacent upper and lower vertebral body images, and using an adversarial generative network model, a spinal healing image of the patient is generated, and a second bone bridge image is determined in the spinal healing image. Determine the similarity between the first bone bridge image and the second bone bridge image, and determine the intervertebral fusion status of the patient's spine based on the similarity. The deep convolutional neural network model is the Point-UNet model, which is an improved network model of the UNet network, that is, the upsampling module of the UNet network is modified into a PointRend sub-network; The adversarial generative network model includes multiple discriminators, which are used for weight co-training and data co-training. The method further includes: Acquire the patient's clinical data and time-series spinal images; determine the patient's spinal intervertebral healing trend and healing time based on the clinical data and time-series spinal images and a time-series neural ensemble network model.
6. The deep learning-based spinal interbody fusion detection method as described in claim 5, characterized in that, The Point-UNet model uses unlabeled data and is pre-trained based on self-supervised contrastive learning.
7. The deep learning-based method for detecting intervertebral fusion of the spine as described in claim 5, characterized in that, The clinical data include at least the patient's age, sex, height, weight, and bone condition.
8. The deep learning-based spinal interbody fusion detection method as described in claim 5, characterized in that, The method further includes: Obtain real images of the patient's healing process; Based on the real healing images and their corresponding time information, real labeled data is generated, and the adversarial generative network model and the temporal neural ensemble network model are optimized based on the real labeled data.