Deep learning method for detecting activity of thyroid-associated ophthalmopathy based on spect / ct

Abstract

The present application relates to a kind of deep learning method based on SPECT / CT detection thyroid-related eye disease activity, including two stages of eye muscle segmentation stage and activity staging classification, in the eye muscle segmentation stage, first, the CT image of eye is three-dimensional reconstruction, and training eye muscle semantic segmentation model, finally using the eye muscle segmentation model of well-trained in test set eye muscle mask;In the classification stage of judging whether thyroid-related eye disease is in active period, three-channel data are formed by using eye muscle mask, SPECT and CT image, and three-channel data are used to train classification model, SPECT / CT image and the eye muscle mask of patient are combined into three-channel image and input into classification model, and finally the activity of thyroid-related eye disease is output by the trained classification model.Through the deep learning method of two stages, the activity of thyroid-related eye disease is automatically classified.

Description

Technical Field

[0001] This invention relates to the technical field of medical image classification, specifically to a deep learning method for detecting the activity of thyroid-related eye diseases based on SPECT / CT. Background Technology

[0002] Thyroid-associated ophthalmopathy, also known as thyroid eye disease or Graves' eye disease, is the most common orbital disorder in adults. It is now widely accepted that immune-mediated inflammation is the primary pathogenesis of thyroid-associated ophthalmopathy, characterized by thickening, congestion, and edema of the extraocular muscles and orbital fat. Most patients with thyroid-associated ophthalmopathy will experience both active and non-active inflammatory phases. For patients with inactive thyroid-associated ophthalmopathy, supportive treatment such as lubricating eye drops is sufficient; while for patients with active thyroid-associated ophthalmopathy, corticosteroids or radiation therapy are believed to reduce post-inflammatory sequelae.

[0003] The Clinical Activity Score (CAS) is widely used to assess the activity of thyroid-associated ophthalmopathy (TAO). However, CAS is subjective and largely dependent on the opinion of an ophthalmologist. Orbital computed tomography (CT) is a rapid and cost-effective imaging method for orbital diseases, providing accurate morphological information, such as changes in extraocular muscles. This information can aid in the diagnosis of TAO, but it is difficult to accurately assess the inflammatory activity of TAO. Radionuclide imaging, such as orbital single-photon emission computed tomography using 99mTc-labeled diethylenetriaminepentaacetic acid (DIPA), has been reported as a useful biomarker for diagnosing and further staging TAO activity due to its cost-effectiveness, simplicity, and accuracy. SPECT / CT is a fusion imaging technique that combines SPECT for functional images with CT for anatomical images, thereby improving the diagnostic accuracy of the test. Recently, hybrid diagnostic imaging using CT and SPECT has been reported to promise the assessment of inflammatory activity through semi-quantitative image analysis of extraocular muscles in patients with TAO. Furthermore, SPECT / CT may be superior to CAS in predicting the efficacy of periorbital glucocorticoid therapy in patients with TAO.

[0004] In recent years, deep learning has been widely applied in the field of medical imaging, including ocular imaging. Numerous studies have demonstrated that deep learning can automatically screen and diagnose various eye diseases, such as cataracts, diabetic retinopathy, glaucoma, age-related macular degeneration, and retinopathy of prematurity. However, the use of deep learning-based techniques for orbital imaging in the diagnosis and activity assessment of thyroid-related eye diseases has been rarely investigated. Some researchers have developed deep learning models for the diagnosis of thyroid-associated ophthalmopathy (TAOD), training the models with 784 sets of orbital CT images (625 normal individuals and 168 TAOD patients), and using 114 and 227 sets of orbital CT images as validation and test sets, respectively. These studies achieved good results in the diagnostic task of TAOD (accuracy, 87%; sensitivity, 88%; specificity, 85%) [see: Song X, Liu Z, Li L, Gao Z, Fan X, Zhai G, et al. Artificial intelligence CT screening model for thyroid-associated ophthalmopathy and tests under clinical conditions. International journal of computer assisted radiology and surgery. 2021; 16:323-30.]. However, the sample size of TAOD patients in this study was small, and the activity of TAOD patients was not staged.

[0005] Another researcher proposed an algorithm based on deep convolutional neural networks to detect the activity of thyroid-associated ophthalmopathy. The model was trained using 160 orbital MRI images (50 active and 110 inactive) from patients with thyroid-associated ophthalmopathy. 80% of the images were used for training and validation, and 20% for testing. The optimal model achieved accuracy, precision, sensitivity, specificity, and F1 score of 85.5±1.8%, 64.0±3.3%, 82.1±7.1%, 86.5±4.0%, and 0.72±0.04, respectively [see: Lin C, Song X, Li L, Li Y, Fan X. Detection of active and inactive phases of thyroid-associated ophthalmopathy using deep convolutional neural network. BMC Ophthalmology. 2021; 21]. This researcher also acknowledged that their model suffers from some overfitting, and that the method requires significant human intervention.

[0006] Experiments have shown that the automation and accuracy of the above methods in detecting the activity of thyroid-associated ophthalmopathy still need to be improved. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention provides a deep learning method for detecting the activity of thyroid-associated ophthalmopathy based on SPECT / CT.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A deep learning method for detecting the activity of thyroid-associated ophthalmopathy (TAO) based on SPECT / CT includes two stages: extraocular muscle segmentation and activity staging classification. In the extraocular muscle segmentation stage, 3D reconstruction of the ocular CT images is first performed, and an extraocular muscle semantic segmentation model is trained. Finally, the trained extraocular muscle segmentation model is used to obtain the extraocular muscle mask in the test set. In the classification stage, which determines whether TAO is in an active phase, the extraocular muscle mask, SPECT, and CT images are combined into a three-channel data set. This three-channel data is used to train the classification model. The SPECT / CT images and the patient's extraocular muscle mask are combined into a three-channel image and input into the classification model. Finally, the trained classification model outputs the TAO activity level. The specific steps are as follows:

[0010] Step (1): Preprocess the SPECT and CT images respectively to restore them to their true size;

[0011] Step (2): Divide the CT images into training set, validation set and test set. Several experienced operators manually outline the superior rectus muscle, inferior rectus muscle, medial rectus muscle and lateral rectus muscle in the CT images of the training set and validation set, and the doctor checks and adjusts them again.

[0012] Step (3): Construct the SV-Net semantic segmentation network for extraocular muscle segmentation. Use the training set and validation set from step (2) to train and evaluate the semantic segmentation network to obtain the optimized extraocular muscle segmentation model.

[0013] Step (4): Input the CT images of the test set into the trained segmentation model to obtain the extraocular muscle mask of the CT images of the test set;

[0014] Step (5): Use the SPECT and CT images from step (1) and the extraocular muscle mask from step (4) to form a three-channel image, and divide the data into a training set, a validation set, and a test set for activity staging;

[0015] Step (6): Construct a TAO-Net classification network for activity staging, train and evaluate the classification network using the training and validation sets of activity staging, obtain the optimized activity staging model, and finally obtain the activity staging of patients with thyroid-associated ophthalmopathy.

[0016] Further, in step (1), firstly, the SPECT image and CT image are registered to unify the pixel spacing of the SPECT image and CT image and unify them under the same three-dimensional coordinate system; secondly, the SPECT and CT three-dimensional images are longitudinally cut to obtain their coronal plane.

[0017] Furthermore, the SV-Net semantic segmentation network mentioned in step (3) is an improved V-Net network, consisting of a decoder and an encoder. The encoder and decoder are composed of convolutional layers with a kernel size of 3×3×3. The encoders at different levels obtain features of different depths from the image. These features are then restored to the size of the original image by the encoder, thereby obtaining the probability map of the extraocular muscles. The probability map is then segmented with a threshold of 0.5 and converted into a binary segmentation result. The binary image is a mask of the extraocular muscles.

[0018] Furthermore, the TAO-Net classification network described in step (6) is built based on a three-dimensional residual module. TAO-Net includes a convolutional module, a residual module, and a classifier module. The convolutional module obtains shallow features from the image. The residual module consists of two three-dimensional residual blocks. In each three-dimensional residual block, the three-dimensional feature map is passed to a three-dimensional convolution with a batch canonical layer and a residual connection containing the three-dimensional convolution. The classifier module contains three three-dimensional convolutional layers and a fully connected layer with a Softmax activation function. The feature map obtained by the residual module is fed into the classifier module for learning, and the weights of the obtained features are reassigned. Finally, the features are output as the classification result through the fully connected layer.

[0019] The beneficial effects of this invention are:

[0020] (1) Compared with the prior art, the present invention uses a two-stage deep learning method. First, the binary image of the patient's extraocular muscles is obtained through a segmentation network, and then combined with the SPECT / CT image to form a three-channel image, which is input into the classification network. The extraocular muscle mask is added as prior knowledge to the training of the model to guide the optimization of the model and make the model pay more attention to the region of the patient's extraocular muscles.

[0021] (2) Compared with the prior art, the present invention uses multimodal images. Compared with single-modal images, multimodal images can provide more information. Specifically, SPECT images provide patient functional characteristics, and CT images provide patient morphological characteristics. Attached Figure Description

[0022] Figure 1 This invention is a deep learning method for detecting the activity of thyroid-associated ophthalmopathy based on SPECT / CT.

[0023] Figure 2 (a) is a schematic diagram of manually sketching the outline of the extraocular muscles in a CT image; Figure 2 (b) is a binary image generated based on the contour of the extraocular muscles;

[0024] Figure 3 It is the construction of the SV-Net network;

[0025] Figure 4 This is the construction of the TAO-Net network. Detailed Implementation

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] Example:

[0028] like Figure 1 As shown, a deep learning method for detecting the activity of thyroid-associated ophthalmopathy (TAO) based on SPECT / CT includes two stages: extraocular muscle segmentation and activity staging classification. In the extraocular muscle segmentation stage, the CT images of the eye are first reconstructed in three dimensions, and an extraocular muscle semantic segmentation model is trained. Finally, the trained extraocular muscle segmentation model is used to obtain the extraocular muscle mask in the test set. In the classification stage, which determines whether TAO is in an active phase, the extraocular muscle mask, SPECT, and CT images are combined into a three-channel data set. This three-channel data is used to train the classification model. The SPECT / CT images and the patient's extraocular muscle mask are combined into a three-channel image and input into the classification model. Finally, the trained classification model outputs the activity of TAO. The specific steps are as follows:

[0029] Step (1): Preprocess the SPECT and CT images respectively to restore them to their true size.

[0030] First, SPECT and CT images are registered to unify the pixel spacing and align them to the same 3D coordinate system. Second, the SPECT and CT 3D images are longitudinally segmented to obtain their coronal planes. To enhance image contrast, the grayscale values ​​of each coronal plane image are transformed to the range of 0-255 using a standard normalization formula.

[0031]

[0032] Where f(x,y) is the grayscale value of the input image, g(x,y) refers to the grayscale value of the image after image processing, and I max This represents the maximum grayscale value of the image.

[0033] Step (2): Divide the CT images into training, validation, and test sets. Since manually outlining the extraocular muscles of all patients is very time-consuming and labor-intensive, 20% of the CT images were randomly selected from all data for manual outlining, and the remaining 80% of the CT images were used as the segmentation test set. Multiple experienced operators manually outlined the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles in the training and validation set CT images, such as... Figure 2 As shown in (a), and further examined and adjusted by a doctor; based on the binary images required for these extraocular muscle contour generation training, such as Figure 2 As shown in (b), where the background is 0 and the extraocular muscle is 1, these binary images are used as the gold standard for segmentation.

[0034] Step (3): Construct the SV-Net semantic segmentation network for extraocular muscle segmentation. Train and evaluate the semantic segmentation network using the training and validation sets from step (2) to obtain the optimized extraocular muscle segmentation model. Details are as follows:

[0035] The SV-Net network is built using the PyTorch framework based on Python. The input is a 256×256×N 3D image, where N is the number of coronal images. The detailed structure of SV-Net is as follows... Figure 3 As shown, the SV-Net semantic segmentation network is an improved version of the V-Net network, consisting of a decoder and an encoder. The encoder and decoder are composed of convolutional layers with a kernel size of 3×3×3. Encoders at different levels extract features of different depths from the image, and these features are then restored to the size of the original image by the encoder, thus obtaining a probability map of the extraocular muscles. The probability map is then thresholded with a threshold of 0.5 and converted into a binary segmentation result. The binary image is a mask for the extraocular muscles.

[0036] The chosen loss function and evaluation metric are both Dice similarity coefficients, used to assess the difference between the predicted results and the labels. Defined as:

[0037]

[0038] Where P ij and Q ij These represent the predicted output value and label value of pixel (i,j) in the image, respectively.

[0039] During training, the learning rate was set to 0.0001, the number of iterations was 600, and the batch size was 8. The SV-Net model was trained using the segmented training set in step (2) to obtain the converged weights. The samples in the validation set were then input into the trained SV-Net model for evaluation. The average Dice coefficient between the predicted extraocular muscle segmentation result and the gold standard was 0.92.

[0040] Step (4): Input the CT images of the test set into the trained segmentation model to obtain the extraocular muscle mask of the CT images of the test set.

[0041] Step (5): Use the SPECT and CT images from step (1) and the extraocular muscle mask from step (4) to form a three-channel image, and divide the data into a training set, a validation set, and a test set for activity staging. Randomly select 20% of the data from all the data as the classification test set, and divide the remaining data into a classification training set and a validation set through five-fold cross-validation.

[0042] Step (6): Construct a TAO-Net classification network for activity staging, train and evaluate the classification network using the training and validation sets of activity staging, obtain the optimized activity staging model, and finally obtain the activity staging of patients with thyroid-associated ophthalmopathy through the optimized activity staging model.

[0043] The TAO-Net classification network is built based on 3D residual modules, such as... Figure 4 As shown, TAO-Net, a 3D neural network that takes SPECT, CT, and extraocular muscle masks as input, includes a convolutional module, a residual module, and a classifier module. The convolutional module consists of a regular 3D convolution (kernel size 5×7×7), a batch canonical layer, and a pooling layer, extracting shallow features from the image. The 5×7×7 kernel size follows the AlexNet and ResNet settings, which helps preserve rich local visual information. The residual module consists of two 3D residual blocks, avoiding gradient vanishing while extracting deep features from the image; in each 3D residual block, the 3D feature map is passed to a 3D convolution with a batch canonical layer and a residual connection containing the 3D convolution. The classifier module contains three 3D convolutional layers and a fully connected layer with a Softmax activation function. The feature maps obtained by the residual module are fed into the classifier module for learning. Utilizing the learning ability of the neural network, the weights of the obtained features are reassigned, and the final features are output as the classification result through the fully connected layer.

[0044] To verify the performance of the method proposed in this invention, samples from the test set were input into the trained TAO-Net model, and the results were evaluated using accuracy, sensitivity, F1 score, and AUC. The specific results are shown in the table below.

[0045] Accuracy (%) Sensitivity (%) F1 ratings AUC Validation set 85.01±2.35 83.63±1.54 0.83±0.02 0.919±0.09 test set 83.35±1.53 84.63±0.84 0.83±0.01 0.923±0.10 Existing methods 64.00±3.30 82.10±7.10 0.72±0.04 0.890±0.31

[0046] Experimental data revealed that the method proposed in this invention outperforms other methods in four indicators, enabling accurate assessment of thyroid-associated ophthalmopathy activity.

[0047] This invention is not limited to the preferred embodiments described above. Anyone can derive other products in various forms under the guidance of this invention. However, regardless of any changes in shape or structure, any technical solution that is the same as or similar to this application falls within the protection scope of this invention.

Claims

1. A deep learning method for detecting activity of thyroid-associated ophthalmopathy based on SPECT / CT, characterized in that: The method comprises an eye muscle segmentation stage and a classification stage of activity classification. In the eye muscle segmentation stage, the CT image of the eye is first reconstructed in three dimensions, and an eye muscle semantic segmentation model is trained, and finally the eye muscle mask in the test set is obtained by using the trained eye muscle segmentation model. In the classification stage of judging whether the thyroid associated ophthalmopathy is in the active stage, the eye muscle mask, SPECT and CT images are combined into three channel data, and a classification model is trained using the three channel data. The SPECT / CT image and the eye muscle mask of the patient are combined into three channel images and input into the classification model, and finally the activity of the thyroid associated ophthalmopathy is output by the trained classification model. The specific steps are as follows: Step (1): The SPECT image and the CT image are preprocessed respectively, and the SPECT image and the CT image are restored to the true size; Step (2): The CT image is divided into a training set, a validation set and a test set, and the outlines of the superior rectus muscle, the inferior rectus muscle, the medial rectus muscle and the lateral rectus muscle in the training set and the validation set CT images are manually outlined by multiple experienced operators, and then adjusted by a doctor; Step (3): An SV-Net semantic segmentation network for eye muscle segmentation is constructed, and the training set and the validation set in step (2) are used to train and evaluate the semantic segmentation network, and an optimized eye muscle segmentation model is obtained; Step (4): The CT image of the test set is input into the trained segmentation model, so as to obtain the eye muscle mask of the test set CT image; Step (5): The SPECT, CT images in step (1) and the eye muscle mask in step (4) are used to form a three channel image, and the data is divided into a training set, a validation set and a test set for activity classification; Step (6): A TAO-Net classification network for activity classification is constructed, and the training set and the validation set for activity classification are used to train and evaluate the classification network, and an optimized activity classification model is obtained, and finally the activity classification of the thyroid associated ophthalmopathy patient is obtained; the TAO-Net classification network is based on a three-dimensional residual module, and the TAO-Net comprises a convolution module, a residual module and a classifier module. The convolution module obtains shallow features from the image; the residual module is composed of two three-dimensional residual blocks. In each three-dimensional residual block, the three-dimensional feature map is transmitted to a three-dimensional convolution with a batch normalization layer and a residual connection comprising a three-dimensional convolution; The classifier module comprises three three-dimensional convolution layers and a fully connected layer with a Softmax activation function; the feature map obtained by the residual module is sent to the classifier module for learning, the obtained feature is re-assigned weights, and finally the feature is output to the classification result through the fully connected layer.

2. The deep learning method for detecting the activity of thyroid-associated ophthalmopathy based on SPECT / CT according to claim 1, characterized in that: In step (1), first, the SPECT image and the CT image are registered, the pixel spacing of the SPECT image and the CT image is unified, and they are unified in the same three-dimensional coordinate system; Secondly, the SPECT and CT three-dimensional images are longitudinally cut to obtain their coronal planes. 3.The deep learning method for detecting the activity of thyroid-associated ophthalmopathy based on SPECT / CT according to claim 1, wherein: The SV-Net semantic segmentation network in step (3) is an improved V-Net network, which is composed of a decoder and an encoder, the encoder and the decoder are composed of convolutional layers with a convolution kernel size of 3*3*3, the encoders at different levels obtain features at different depths from the image, and the features are restored to the size of the original image by the encoder, so as to obtain a probability map of the extraocular muscle; the probability map is threshold segmented with a threshold of 0.5 to be converted into a binary segmentation result, and the binary image is a mask of the extraocular muscle.

Images