Osteoporotic Vertebral Refracture Prediction System Based on Deep Learning of CT Images

Abstract

The present invention relates to an osteoporosis vertebral refracture prediction system based on deep learning of CT images, comprising the following steps: S1, collecting CT image data; S2, using a convolutional neural network to extract features from the CT image data; S3, inputting the extracted standardized CT image data features into a deep neural network model to obtain a final classification judgment; its advantages are as follows: The present invention combines clinical CT image data and uses a convolutional neural network to optimize model training, thereby establishing a simple, time-saving, low-consumption, convenient, fast, efficient and accurate prediction system.

Description

Technical Field

[0001] The present invention relates to the technical field of osteoporosis fracture prediction, and specifically, to an osteoporosis vertebral refracture prediction system based on deep learning of CT images. Background Art

[0002] According to the report of the International Osteoporosis Foundation (IOF) in [year]: one osteoporosis fracture occurs every 3 seconds globally. Osteoporotic vertebral compression fracture (OVCF) has always been considered the most common manifestation of osteoporosis, accounting for nearly 50% of osteoporosis fractures. Osteoporosis fractures can cause pain and disability, significantly reducing the quality of life of patients. Existing treatment methods include conservative treatment and surgical treatment. Conservative treatment requires long-term bed rest and immobilization, increasing the risk of adverse events such as hypostatic pneumonia, deep vein thrombosis, and pressure ulcers, and increasing the mortality rate of patients; while surgical treatment, mainly referring to vertebroplasty, can quickly relieve pain but also faces certain surgical and anesthesia risks. Research shows that about 50% of women and 20% of men will experience a first osteoporosis fracture after the age of 50, and 50% of patients with a first osteoporosis fracture will have a second osteoporosis fracture. Therefore, it is particularly important to predict osteoporosis vertebral fractures in advance and intervene early, which has important clinical significance and can avoid the pain caused by fractures and surgeries for patients.

[0003] Existing prediction methods mainly summarize and analyze clinical data, such as the patient's body mass index (BMI), age, osteoporosis index (BMD), and underlying diseases combined by the patient, and also include some data related to surgery, such as surgical methods, whether bone cement leaks, and the recovery of vertebral height. Through the analysis of these data, it is possible to generally judge whether a patient will have a refracture, but it is difficult to accurately predict which specific vertebra will fracture.

[0004] For example, the Fracture Risk Assessment Tool (FRAX) recommended by the World Health Organization is an online fracture risk prediction tool that determines the patient's fracture probability in the next ten years by comprehensively evaluating age, gender, weight, height, fracture history, parental hip fracture, smoking, glucocorticoids, rheumatoid arthritis, secondary osteoporosis, alcohol consumption, and / or femoral neck bone mineral density (BMD).

[0005] Methods for predicting spinal fractures using images have also been reported. Most of them use finite element analysis methods to obtain vertebral bone mass, bone strength, etc., and then predict the risk of fracture through logistic regression analysis methods.

[0006] Searching current patents, Chinese Patent Application: CN108538393B discloses a big data-based bone mass assessment expert system and a prediction model establishment method, including a server internally provided with a prediction model module, a data collection module, and a bone expert module. A database and an intelligent terminal are connected to the server; the database stores user identity data, characteristic data of historical osteoporosis fracture patients, and detection data of patients to be predicted; the prediction model module establishes an osteoporosis fracture prediction model; the data collection module is used to obtain a prediction request and detection data of patients to be predicted, and the osteoporosis fracture prediction model obtains an osteoporosis fracture prediction level according to the detection data; the bone expert module outputs prevention suggestions to patients to be predicted. Beneficial effects: Realize fracture prediction for osteoporosis patients, which is convenient and does not require queuing. And it can be operated anytime and anywhere, intelligent and convenient. However, this invention is based on big data, including clinical, imaging, etc., to obtain the osteoporosis fracture prediction level, that is, the probability of a patient having a fracture, similar to the Fracture Risk Assessment Tool (FRAX) recommended by the World Health Organization. Therefore, this model is not aimed at spinal osteoporosis compression fractures and cannot accurately predict the probability of a certain vertebral body fracture.

[0007] Chinese Patent Application: CN101720468A discloses a method for processing data derived from images of at least a part of the spine, for estimating the risk of future fractures in the vertebrae of the spine. Process position data related to at least four adjacent vertebrae of the spine. Calculate the curvature of the spine of at least two of the adjacent vertebrae. Calculate different curvature values to obtain a value representing the degree of irregularity in the curvature of the spine, and use the degree of irregularity to provide an estimate of the risk of future fractures in the vertebrae of the spine. A higher degree of irregularity indicates a higher risk of future fractures. This invention also predicts vertebral fractures based on spinal images, but in terms of method, it only predicts the incidence of fractures by calculating the curvature of the spine, while the present invention is based on deep learning and artificial intelligence algorithms of CT images.

[0008] Therefore, in summary, there is an urgent need for a system that can combine clinical CT image data, optimize model training using convolutional neural networks, and establish a precise, efficient, easy-to-operate, time-saving and low-consumption osteoporosis vertebral re-fracture prediction system based on deep learning of CT images. However, there has been no report on such an osteoporosis vertebral re-fracture prediction system based on deep learning of CT images. Summary of the Invention

[0009] The objective of the present invention is to combine clinical CT image data, optimize model training using a convolutional neural network, and establish a precise, efficient, easy-to-operate, time-saving, and low-consumption prediction system. Specifically, it is to establish a convenient, fast, efficient, and precise prediction system for osteoporotic vertebral fractures based on CT images.

[0010] To achieve the above objective, the technical solution adopted by the present invention is as follows:

[0011] An osteoporotic vertebral refracture prediction system based on deep learning of CT images, comprising the following steps;

[0012] S1, collect CT image data;

[0013] S2, use a convolutional neural network to extract features from the CT image data;

[0014] S3, input the extracted standardized CT image data features into a deep neural network model to obtain a final classification judgment.

[0015] In the above-mentioned osteoporotic vertebral refracture prediction system based on deep learning of CT images, as a preferred solution, the S1 step comprises the following sub-steps; S11, select the format of the CT image; S12, obtain the CT image.

[0016] In the above-mentioned osteoporotic vertebral refracture prediction system based on deep learning of CT images, as a preferred solution, in the S11 step, the standard for selecting the format of the CT image is: select the group with osteoporotic vertebral refracture and the first surgical method being vertebroplasty, and the required format of the selected CT image is CT cross-section, bone window display, without captions.

[0017] In the above-mentioned osteoporotic vertebral refracture prediction system based on deep learning of CT images, as a preferred solution, in the S12 step, the specific method for obtaining the CT image is: download all vertebral CT cross-sections before the second fracture from the imaging system according to the CT image format in the S11 step, name the vertebral body of the second fracture as the positive vertebral body, and collect the CT cross-section images within the corresponding range of the vertebral body, and the range includes one intervertebral disc above and below the vertebral body.

[0018] In the above-mentioned osteoporotic vertebral refracture prediction system based on deep learning of CT images, as a preferred solution, the S2 step is specifically: further comprising an image preprocessing module, and the input data of the image preprocessing module is the gray-scale image of the vertebral CT cross-section, and its numerical representation is [C, H, W], where C is the image color channel, H is the image height, and W is the image width.

[0019] In the osteoporosis vertebral refracture prediction system based on CT image deep learning described above, as a preferred solution, the specific steps of S3 are as follows: It further includes a neural network module for receiving the picture preprocessing module. The input data thereof is a standardized picture tensor, specifically represented as [1, 512, 512], and the final prediction classification is obtained through the (already trained) neural network.

[0020] The advantages of the present invention are as follows:

[0021] 1. The present invention combines clinical CT image data and uses a convolutional neural network for model training optimization, thereby establishing a set of prediction systems for osteoporotic vertebral fractures that are easy to operate, time-saving, low-consumption, convenient, fast, efficient, and accurate.

[0022] 2. Focusing on the patient group with refracture after OVCF, for the first time, based on the deep learning of CT images, a simple and effective refracture prediction model is established using a convolutional neural network to guide the treatment and prevention of clinical OVCF.

[0023] 3. The reported prediction systems or models can only predict the overall fracture probability of patients and cannot be accurate to a certain vertebra. This prediction model uses CT cross-sections for analysis and can accurately predict the fracture probability of each vertebra. Therefore, this model is more refined. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG Figure 1 is a schematic diagram of a partial structure of the osteoporosis vertebral refracture prediction system based on CT image deep learning described in the present invention.

[0025] FIG Figure 2 is a schematic diagram of the model training and testing process in the osteoporosis vertebral refracture prediction system based on CT image deep learning described in the present invention.

[0026] FIG Figure 3 is an image of the CT picture imaging requirements selected in the experimental group described in the present invention.

[0027] FIG Figure 4 is a CT cross-section image selected in the experimental group described in the present invention.

[0028] FIG Figure 5 is a case diagram (I) for judging positive vertebrae described in the present invention.

[0029] FIG Figure 6 is a case diagram (II) for judging positive vertebrae described in the present invention.

[0030] FIG Figure 7 is 33 CT cross-section images obtained after screening pictures described in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content recorded in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0032] Please refer to the attached Figure 1 , the attached Figure 2 As shown, the attached Figure 1 is a partial structural schematic diagram of the osteoporosis vertebral re-fracture prediction system based on CT image deep learning described in the present invention. The attached Figure 2 is a schematic diagram of the model training and testing process in the osteoporosis vertebral re-fracture prediction system based on CT image deep learning described in the present invention.

[0033] In recent years, with the rapid development of computer technology, there have been some applications of machine learning in the medical field, especially in assisting clinical diagnosis. In the present invention, we consider applying the deep learning method to the establishment of an osteoporosis vertebral fracture system, which can integrate the features extracted from the training data, discover the distributed features of the data through the combination of multi-layer linear transformation and non-linear transformation, and has excellent capabilities for collecting and processing large data and universality. The convolutional neural network (CNN) is a type of feedforward neural network with a deep structure containing convolutional calculations, which can utilize the property of establishing local connections through convolutional operations to effectively establish an end-to-end data processing and recognition system and has extensive applications in image processing; the present invention combines clinically CT image data and then uses the convolutional neural network to optimize the model training, and establishes a set of accurate, efficient, easy-to-operate, time-saving and low-consumption prediction systems. Specifically, a convenient, fast, efficient and accurate prediction system for osteoporosis vertebral fractures based on CT images is established, which mainly includes the following steps;

[0034] S1, collect CT image data;

[0035] S2, use the convolutional neural network to extract features from the CT image data;

[0036] S3, input the extracted standardized CT image data features into the deep neural network model to obtain the final classification judgment.

[0037] In this embodiment, it is preferred that the step S1 includes the following sub-steps: S11, select the format of the CT image; S12, obtain the CT image. In the step S11, the standard for selecting the format of the CT image is: select the group of patients with osteoporotic vertebral refracture and the first surgical method being vertebroplasty. The required format of the selected CT image is CT cross-section, bone window display, and no captions. In the step S12, the specific method for obtaining the CT image is: download all the vertebral CT cross-sections before the second fracture from the imaging system according to the CT image format in the step S11, name the vertebra of the second fracture as the positive vertebra, and collect the CT cross-section images within the corresponding range of this vertebra, and the range includes one intervertebral disc above and below the vertebra.

[0038] Specifically:

[0039] (1) Set up the experimental group;

[0040] Patient selection: Select patients with osteoporotic vertebral refracture, and the first surgical method is vertebroplasty;

[0041] Requirements for CT images: CT cross-section, bone window display, and no captions, as shown in the appendix Figure 3 as follows;

[0042] Obtaining of CT images: First, download all the vertebral CT cross-sections before the second fracture from the imaging system according to the required image format, name the vertebra of the second fracture as the positive vertebra, and collect the CT cross-section images within the corresponding range of this vertebra, and the range includes one intervertebral disc above and below the vertebra. The method is as shown in the appendix Figure 4 as follows:

[0043] A total of 91 cases of secondary fractures were collected, and a total of 4,602 images were used as the image set for the model architecture.

[0044] (2) Set up the control group:

[0045] Collect outpatient or inpatient elderly patients who have visited the Department of Orthopedics of Tongji Hospital since 2017 and have lumbar CT examination results. The age is greater than 60 years old, both men and women are included, and patients with spinal fractures are excluded. Randomly select 70 patients, download their lumbar CT images in the UniWeb Server imaging system of Tongji Hospital Affiliated to Tongji University, and output the images according to the required image format. A total of 18,781 photos were obtained.

[0046] In this embodiment, it is preferably that the specific steps of S2 are as follows: There is also an image preprocessing module, and the input data of the image preprocessing module is the grayscale image of the vertebral CT cross-section, and its numerical representation is [C, H, W], where C is the image color channel, H is the image height, and W is the image width. For the grayscale image of the vertebral CT cross-section, C is 1. However, since the input data of different batches may have different heights H and widths W, this module will convert the input data into a standard tensor representation of [1, 512, 512]. The specific method is as follows:

[0047] ① Use the bicubic interpolation algorithm to convert the image tensor of [1, H, W] into [1, 512, 512]

[0048] ② Perform a regularization operation on the image tensor after the interpolation operation:

[0049] x = x / 255.0

[0050] where x is the image tensor.

[0051] It is preferably that the specific steps of S3 are as follows: There is also a neural network module, which is used to receive the image preprocessing module. Its input data is the standardized image tensor, specifically represented as [1, 512, 512], and the final predicted classification is obtained through the (already trained) neural network.

[0052] The operation process of the osteoporosis vertebral fracture prediction model based on deep learning of the present invention is as follows:

[0053] ① Input the grayscale image of the vertebral CT cross-section (downloaded from the UniWebServer of Tongji Hospital Affiliated to Tongji University) into the image preprocessing module to obtain the standardized image data;

[0054] ② The standardized image data directly enters the neural network module, and the neural network automatically extracts effective features according to the data to obtain the final classification judgment.

[0055] Its classification judgment: (judgment criterion)

[0056] Regarding the judgment basis of vertebral fractures: All patients admitted to the hospital due to repeated fractures have undergone magnetic resonance imaging (MRI). Vertebrae showing high signals on T2 images, low signals on T1 images, and high signals on fat-suppressed images meet the diagnosis of fresh fractures.

[0057] Among them, the classification judgment includes:

[0058] Label Description 0 No secondary fracture occurs 1 Secondary fracture occurs

[0059] Based on the training data constructed above, the already trained neural network structure is as follows:

[0060]

[0061]

[0062] Sub-network module 1

[0063] Type Convolution kernel (quantity) size / stride (or annotation) Convolution (4)3x3 / 1 Convolution (8)3x3 / 1

[0064] Sub-network module 2

[0065]

[0066] Sub-network module 3

[0067]

[0068]

[0069] Sub-network module 4

[0070]

[0071] Sub-network module 5

[0072]

[0073] Sub-network module 6

[0074]

[0075] Sub-network module 7

[0076]

[0077] Sub-network module 8

[0078]

[0079]

[0080] Sub-network module 9

[0081]

[0082] Sub-network module 10

[0083] Type Convolution kernel (quantity) size / stride (or annotation) Convolution (1024)3x3 / 1 Convolution (2048)3x3 / 1

[0084] Next, the mathematical representation of the neural network model will be introduced. The adopted symbol rules are as follows:

[0085]

[0086]

[0087] The expression of model sub-network module 1 (m1) is:

[0088] z 1，1 = Conv 1，1 (x);

[0089] a 1，1 = Relu(z 1，1 )

[0090] z 1，2 = Conv 1，2 (a 1，1 )

[0091] a 1，2 = Relu(z 1，2 )

[0092] u = a 1，2 .

[0093] The expression of model sub-network module 2 (m2) is:

[0094] z 2，1；s = Conv 2，1；s x;

[0095] z 2，1 = DSConv 2，1 (x);

[0096] a 2，1 = Relu(z 2，1 )

[0097] z 2，2 = DSConv 2，2 (a 2，1 )

[0098] p 2，1 = Pool(z 2，2 )

[0099] u = p 2，1 + z 2，1；s .

[0100] The expression of model sub-network module 3 (m3) is:

[0101] z 3，1；s = Conv 3，1；s x;

[0102] z 3，1 = DSConv 3，1 (x);

[0103] a 3，1 = Relu(z 3，1 )

[0104] z 3，2 = DSConv 3，2 (a 3，1 );

[0105] p 3，1 = Pool(z 3，2 );

[0106] u = p 3，1 + z 3，1；s .

[0107] The expression of the model sub - network module 4 (m4) is:

[0108] z 4，1；s = Conv 4，1；s x;

[0109] z 4，1 = DSConv 4，1 (x);

[0110] a 4，1 = Relu(z 4，1 );

[0111] z 4，2 = DSConv 4，2 (a 4，1 );

[0112] p 4，1 = Pool(z 4，2 );

[0113] u = p 4，1 + z 4，1；s .

[0114] The expression of the model sub - network module 5 (m5) is:

[0115] z 5，1；s = Conv 5，1；s x;

[0116] z 5，1 = DSConv 5，1 (x);

[0117] a 5，1 = Relu(z 5，1 );

[0118] z 5，2 = DSConv 5，2 (a 5，1 );

[0119] p 5，1 = Pool(z 5，2 );

[0120] u = p5，1 +z 5，1；s 。

[0121] The expression of the model sub-network module 6 (m6) is:

[0122] z 6，1；s = Conv 6，1；s x;

[0123] z 6，1 = DSConv 6，1 (x);

[0124] a 6，1 = Relu(z 6，1 );

[0125] z 6，2 = DSConv 6，2 (a 6，1 );

[0126] p 6，1 = Pool(z 6，2 );

[0127] u = p 6，1 + z 6，1；s 。

[0128] The expression of the model sub-network module 7 (m7) is:

[0129] z 7，1；s = Conv 7，1；s x;

[0130] z 7，1 = DSConv 7，1 (x);

[0131] a 7，1 = Relu(z 7，1 );

[0132] z 7，2 = DSConv 7，2 (a 7，1 );

[0133] p 7，1 = Pool(z 7，2 );

[0134] u = p 7，1 + z 7，1 ; s 。

[0135] The expression of the model sub-network module 8 (m8) is:

[0136] a 8，1 = Relu(x)

[0137] z 8，1 = DSConv 8，1 (a 8，1 );

[0138] a 8，2 = Relu(z 8，1 );

[0139] z 8，2 = DSConv 8，2 (a 8，2 );

[0140] a 8，3 = Relu(z 8，2 );

[0141] z 8，3 = DSConv 8，3 (a 8，3 );

[0142] u = z 8，3 + x。

[0143] The expression of model sub - network module 9 (m9) is:

[0144] z 9，1；s = Conv 9，1；s x;

[0145] a 9，1 = Relu(x);

[0146] z 9，1 = DSConv 9，1 (a 9，1 );

[0147] a 9，2 = Relu(z 9，1 );

[0148] z 9，2 = DSConv 9，2 (a 9，1 );

[0149] p 9，1 = Pool(z 9，2 );

[0150] u = p 9，1 + z 9，1；s 。

[0151] The expression of model sub - network module 10 (m 10 ) is:

[0152] z 10，1 = Conv 10，1 (x);

[0153] a 10，1 = Relu(z 10，1 );

[0154] z 10，2 = Conv 10，2 (a 10，1 );

[0155] a 10，2 = Relu(z 10，2 );

[0156] u = a 10，2 。

[0157] The overall expression of the model is:

[0158] u1 = m1(x);

[0159] u2 = m2(u1);

[0160] u3 = m3(u2);

[0161] u4 = m4(u3);

[0162] u5 = m5(u4);

[0163] u6 = m6(u5);

[0164] u7 = m7(u6);

[0165] u 8，1 = m 8，1 (u7);

[0166] u 8，2 = m 8，2 (u 8，1 );

[0167] u 8，3 = m 8，3 (u 8，2 );

[0168] u8 = m 8，3 (u 8，3 );

[0169] u9 = m9(u8);

[0170] u 10 = m 10 (u9);

[0171] p = GPool(u 10 );

[0172] z w = Fc(p);

[0173]

[0174] The method for training its model is as follows:

[0175] Train the neural network on 2 NVIDIA GTX 1080Ti graphics cards using the Pytorch framework. The training optimizer is the Adam optimizer, and the corresponding training parameters are: the learning rate is 0.001, beta1 is 0.9, beta2 is 0.999, and epsilon is 1e-8.

[0176] Beneficial effects achieved:

[0177] Case 1: The accuracy of the model of the framework for the test data (a total of 4666 images of fractures and control groups) reaches 0.839, the prediction accuracy for secondary fractures reaches 0.719, and the prediction accuracy for non-secondary fractures reaches 0.867.

[0178] Case 2: After the successful model framework, 12 samples of patients with secondary fractures were collected, a total of 905 CT images, and set as an independent test set. Using this prediction model for prediction, the accuracy reaches 0.817.

[0179] Specific illustration of the drawings:

[0180] 1. Example of the data used for training: (shown)

[0181] Example of the process for obtaining positive case data

[0182] The first step: Determine the positive vertebra

[0183] Patient Zhang XX, on March 15, 2019, had an L2 fracture and received vertebroplasty, as shown in Attachment Figure 5 and Attachment Figure 6 shown;

[0184] On December 5, 2019, the patient had an L4 fracture;

[0185] The second step: Download the images and screen the pictures to obtain 33 CT cross-sectional images, as shown in Attachment Figure 7 shown.

[0186] Example of the process for obtaining control case data:

[0187] Collect outpatient or inpatient elderly patients who have visited the Department of Orthopedics of Tongji Hospital since 2017 and have lumbar CT examination results. The age is greater than 60 years old, both men and women are included, excluding patients with spinal fractures. Randomly select 150 patients, download their lumbar CT images in the UniWeb Server imaging system of Tongji Hospital Affiliated to Tongji University, and output the pictures according to the required picture format.

[0188] The third import program is used for training (please refer to the appendix Figure 2 as shown).

[0189] It should be noted that: The present invention combines clinically CT image data, and uses a convolutional neural network to optimize model training, and establishes a convenient, fast, efficient and accurate prediction system for osteoporotic vertebral fractures based on CT images; The main innovation points in the present invention mainly include: (1) Focusing on the patient group of refracture after OVCF surgery, for the first time, a simple and effective refracture prediction model is established based on deep learning of CT images, using a convolutional neural network to guide the treatment and prevention of clinical OVCF; (2) The reported prediction systems or models can only predict the overall fracture probability of patients and cannot be accurate to a certain vertebra, while this prediction model uses CT cross-sections for analysis and can accurately predict the fracture probability of each vertebra. Therefore, this model is more refined.

[0190] The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and supplements can be made, and these improvements and supplements should also be regarded as the protection scope of the present invention.

Claims

1. An osteoporosis vertebral refracture prediction system based on deep learning of CT images, characterized in that, Including the following steps; S1. Collect CT image data; S2. Use a convolutional neural network to extract features from the CT image data; S3. Input the extracted standardized CT image data features into a deep neural network model to obtain a final classification judgment. The S1 step includes the following sub-steps: S11. Select the format of the CT image; S12. Obtain the CT image. In the S11 step, the standard for selecting the format of the CT image is: select the group with osteoporotic vertebral refracture and the first surgical method being vertebroplasty. The required format of the selected CT image is the CT cross-section, bone window display, without captions. In the S12 step, the specific method for obtaining the CT image is: download all vertebral CT cross-sections before the second fracture from the imaging system according to the CT image format in the S11 step. Name the vertebra of the second fracture as the positive vertebra, and collect the CT cross-section images within the corresponding range of this vertebra, and the range includes one intervertebral disc above and below the vertebra. The S2 step is specifically: It also includes an image preprocessing module. The input data of the image preprocessing module is the gray-scale image of the vertebral CT cross-section, and its numerical representation is [C, H, W], where C is the image color channel, H is the image height, and W is the image width. For the gray-scale image of the vertebral CT cross-section, C is 1. However, due to different input data in different batches may have different heights H and widths W, the image preprocessing module will convert the input data into a standard tensor representation of [1, 512, 512]. The S3 step is specifically: It also includes a neural network module for connecting to the image preprocessing module. Its input data is the standardized image tensor, specifically represented as [1, 512, 512], and the final predicted classification is obtained through the trained neural network.

Images