Thyroid-related eye disease auxiliary verification method based on SPECT / CT and federated learning

Abstract

The invention discloses a thyroid-related eye disease auxiliary verification method based on SPECT / CT and federal learning. The thyroid-related eye disease auxiliary verification method comprises the steps that A, image data standardization and orbit area extraction are carried out; b, extracting a preliminary segmentation mask of each rectus in the CT image; c, fusing the structure prior mask, and obtaining an extraocular muscle fine segmentation mask through multi-mode collaborative segmentation; d, utilizing extraocular muscle fine segmentation masks, SPECT and CT images, fusing segmentation prior and multi-modal features to perform deep discrimination, performing automatic prediction of thyroid-related eye disease activity staging, and outputting a prediction probability; and E, based on a dynamic model combination strategy of style similarity and uncertainty quantization, obtaining a dynamic combination model of segmentation and classification adapted to new domain feature distribution, and predicting TAO activity. According to the method, accurate segmentation of extraocular muscle structures and accurate prediction of TAO activity in an invisible data center can be realized, auxiliary verification data support is provided for diagnosis of thyroid-related eye diseases, and the diagnosis accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of auxiliary verification of thyroid-related eye diseases, and in particular to an auxiliary verification method for thyroid-related eye diseases based on SPECT / CT and federated learning. Background Art

[0002] Thyroid-associated ophthalmopathy (TAO) is a common orbital disease associated with thyroid dysfunction, typically manifested by thickening, congestion, and edema of the extraocular muscles, with two phases: active and quiescent. Traditional diagnosis and assessment rely primarily on the physician's clinical experience and subjective judgment, with high operational complexity and considerable subjectivity, resulting in poor diagnostic accuracy and consistency. In recent years, the rapid development of multimodal medical imaging technology has provided richer anatomical and functional information for TAO diagnosis. SPECT / CT combines the anatomical information of CT with the metabolic functional information of SPECT, becoming an effective hybrid imaging modality. However, how to automatically analyze these images remains an urgent problem.

[0003] At present, there have been many studies and patent applications on auxiliary verification methods for thyroid-related eye diseases. For example, CN117936101A discloses a scoring calculation method based on multiple data sources to improve diagnostic efficiency, but this method relies on a lot of manual intervention and lacks automated processing of image data. CN118052819A proposed an analysis method based on facial images and orbital CT to predict the risk of thyroid-related eye diseases, but its utilization of multimodal information is still insufficient. CN116597937A combines deep learning with imaging omics to predict the effect of glucocorticoid treatment by fusion features, but it requires a large amount of labeled data and lacks privacy protection in multi-center data applications. CN116364292A uses Transformer to perform time series analysis to predict thyroid function indicators. Although it improves the dynamic evaluation capability, the in-depth utilization of multimodal image features is still insufficient. CN114219754A proposes an automatic segmentation method based on CT slices, which improves the degree of diagnostic automation by segmenting the contours of the eyeball and orbit. However, it is limited to single-modality image analysis and fails to fully utilize multimodal information. Summary of the Invention

[0004] The purpose of the present invention is to provide an auxiliary verification method for thyroid-related eye diseases based on SPECT / CT and federated learning. By constructing a structural multimodal fine segmentation and prior fusion network, integrating a TAO activity staging multimodal 3D neural network, and introducing a dynamic model combination strategy in the federated reasoning stage, accurate segmentation of extraocular muscle structure and accurate prediction of TAO activity can be achieved in an invisible data center, providing auxiliary verification data support for the diagnosis of thyroid-related eye diseases and improving diagnostic accuracy.

[0005] The present invention adopts the following technical solutions:

[0006] A method for assisting in the verification of thyroid-related eye diseases based on SPECT / CT and federated learning, comprising the following steps:

[0007] A: Perform image data standardization and orbital region extraction;

[0008] B: Structural processing is performed on the orbital region in the CT image, and preliminary segmentation masks of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles are automatically extracted as structural prior masks;

[0009] C: Fusion of structural prior masks to achieve multimodal collaborative segmentation of SPECT and CT images and obtain fine segmentation masks of extraocular muscles;

[0010] D: Using the fine segmentation mask of extraocular muscles, SPECT images and CT images, the segmentation prior is integrated with multimodal features for deep discrimination, automatic prediction of the activity stage of thyroid-related eye disease and output of prediction probability;

[0011] E: A dynamic model combination strategy based on style similarity and uncertainty quantification is used to obtain a dynamic combination model of segmentation and classification that adapts to the feature distribution of the new domain, and accurately segment the extraocular muscle structure and predict TAO activity in an invisible data center.

[0012] In step A, the object detection model is used to automatically detect the left and right eye sockets in the standardized image and output the bounding box coordinates; and the left and right eye socket image blocks of a set size are cropped based on the center of the bounding box.

[0013] In step B, the seed points of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles in the extraocular muscles are determined by morphological matching. Then, a dynamic region growing algorithm based on grayscale similarity and position constraints is used to generate preliminary segmentation regions of each rectus muscle in the extraocular muscles. Based on the preliminary segmentation regions, a preliminary segmentation mask of the extraocular muscles is obtained as a structural prior mask.

[0014] Step B includes the following specific steps:

[0015] B1: Anisotropic diffusion filtering is performed on the orbital region CT image to reduce background noise and enhance anatomical edges;

[0016] B2: The orbital region is extracted using an adaptive threshold segmentation method based on the grayscale histogram. Morphological opening and closing operations are then used to remove isolated regions and fill in edge holes to generate an orbital mask.

[0017] B3: Calculate the center of mass of the orbital mask and divide the candidate regions of the superior rectus muscle, inferior rectus muscle, medial rectus muscle and lateral rectus muscle based on the center of mass;

[0018] B4: Using morphological matching method to locate the rectus muscle seed point in each candidate region;

[0019] B5: Based on grayscale similarity and spatial position constraints, a dynamic region growing algorithm is used to generate the preliminary segmentation mask of each rectus muscle.

[0020] In step C, a multimodal fine segmentation and prior fusion network is used to achieve multimodal collaborative segmentation of SPECT and CT images. The multimodal fine segmentation and prior fusion network consists of two independent encoder branches and a shared decoder module. The two encoder branches are used to process CT images and SPECT images, respectively, to extract anatomical structure information and metabolic function features.

[0021] Each encoder branch contains multiple levels of convolutional modules. The shallow structure in the convolutional module is used to extract low-level features, and the deep structure is used to extract high-level semantic features. The fusion module is used to splice the feature maps generated by the convolutional module in the channel dimension and weightedly adjust the fusion ratio of CT and SPECT feature maps. The decoder consists of multiple deconvolution layers to restore the spatial resolution and fine edge details of the image. Each layer of the decoder receives high-resolution fused features from the corresponding layer of the encoder through jump connections, and finally outputs a preliminary segmentation mask of the extraocular muscles.

[0022] Step C includes the following steps:

[0023] C1: Concatenate the CT image and SPECT image with the structural prior mask in the channel dimension to construct a multi-channel input containing the original image and structural prior information;

[0024] C2: Extract anatomical and metabolic features of CT and SPECT images respectively;

[0025] C3: Fusion of anatomical and metabolic features of CT and SPECT to obtain a fusion feature map;

[0026] C4: Upsample and reconstruct the fused feature map layer by layer to restore the spatial resolution and detailed structure of the segmentation mask, and finally obtain a fine segmentation mask of the extraocular muscles.

[0027] In step D, the TAO activity staging multimodal 3D neural network is used to automatically identify and output the activity staging of thyroid-associated eye disease based on the TAO activity staging model.

[0028] TAO activity staging multimodal 3D neural network, including a multimodal input module, a multi-scale feature encoding module, a gated fusion module, a 3D residual learning module, and a prediction output module;

[0029] The multimodal input module is used to stitch the extraocular muscle fine segmentation mask, SPECT image and CT image in the channel dimension to construct a three-channel input containing anatomical structure information, metabolic function information and segmentation prior information; the multi-scale feature encoding module is used to extract multi-level features in three-dimensional space; the gated fusion module is used to fuse multi-scale fine-grained features; the three-dimensional residual learning module and the prediction output module are used to perform deep semantic extraction of the fused multi-scale features and finally output the TAO activity staging probability.

[0030] Step D comprises the following steps:

[0031] D1: Using the multimodal input module, the extraocular muscle fine segmentation mask, SPECT image, and CT image are spliced ​​in the channel dimension to construct a three-channel input containing anatomical structure information, metabolic function information, and segmentation prior information;

[0032] D2: Use the multi-scale feature encoding module to extract multi-level features in three-dimensional space;

[0033] D3: Use the gated fusion module to perform multi-scale fine-grained feature fusion;

[0034] D4: Utilize the three-dimensional residual learning module and the prediction output module to perform deep semantic extraction on the fused multi-scale features and ultimately output the TAO activity stage probability.

[0035] Step E includes the following steps:

[0036] E1: During the federated learning training phase, each center independently trains a local extraocular muscle segmentation model and TAO activity staging model. All centers periodically upload local model parameters, which are aggregated on the server to form a global model. After training, the last round of local segmentation and classification model parameters are cached.

[0037] E2: Perform style feature extraction and uncertainty score calculation; the global style feature is the mean of the style vectors of all samples at each center, and the uncertainty score is the mean of the variance vectors between samples;

[0038] E3: Based on style similarity and uncertainty, a bilateral weighting strategy is used to calculate the portfolio weight and perform weight distribution;

[0039] E4: Based on the combined weights, the local segmentation model parameters and staging model parameters of all source centers are weightedly fused to obtain a dynamic combination model of segmentation and classification that adapts to the feature distribution of the new domain and is directly used for the extraocular muscle structure segmentation and TAO activity staging inference of the new domain data, completing the accurate segmentation of the extraocular muscle structure and the prediction of TAO activity in the invisible data center.

[0040] Step A includes the following steps:

[0041] A1: Acquire and import original SPECT and CT images from each data center;

[0042] A2: Resample the original SPECT image and CT image;

[0043] A3: Map the resampled image to the same 3D coordinate system to obtain a standardized 3D image;

[0044] A4: Cut the standardized 3D image longitudinally to obtain the coronal image;

[0045] A5: Perform grayscale normalization on SPECT and CT images respectively;

[0046] A6: In the normalized image, locate and crop the orbital region.

[0047] This invention combines multimodal images, multi-center data, federated learning and prior knowledge to achieve accurate segmentation of extraocular muscle structure and accurate prediction of TAO activity in an invisible data center, providing auxiliary verification data support for the diagnosis of thyroid-related eye diseases and improving diagnostic accuracy and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that these embodiments are only used to illustrate some embodiments of the present invention and do not constitute a limitation of the present invention. For those skilled in the art, any modification, replacement or equivalent deformation made without departing from the spirit and essence of the present invention should be included in the scope of protection of the present invention.

[0050] The terms "including", "comprising", etc. used in this specification should be interpreted as open expressions, meaning "including but not limited to", the term "based on", etc. should be understood as "at least partially based on", and the terms "one embodiment", "another embodiment", "some embodiments", etc. all indicate that there may be multiple embodiments and should not be understood as uniqueness or exclusivity.

[0051] In addition, the expressions "first" and "second" mentioned in this specification are only used to distinguish different technical features, modules or steps, and do not represent their order or logical relationship, nor should they be understood as limitations on the implementation methods or execution order, unless otherwise specified in the context.

[0052] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0053] like Figure 1 As shown, the thyroid-related eye disease auxiliary verification method based on SPECT / CT and federated learning of the present invention includes the following steps:

[0054] A: Perform image data standardization and orbital region extraction;

[0055] In the present invention, SPECT images and CT images from each data center are first acquired and standardized to achieve spatial and grayscale consistency of multi-center, multi-modal medical images. Then, the orbital area is located and cropped to provide high-quality input data for subsequent models.

[0056] In the present invention, step A comprises the following steps:

[0057] A1: Acquire and import original SPECT and CT images from each data center;

[0058] A2: Resample the original SPECT and CT images and uniformly adjust all SPECT and CT images to the preset voxel spacing to make their resolution consistent in three-dimensional space, facilitating subsequent multimodal fusion and analysis.

[0059] A3: Map the resampled images to the same 3D coordinate system to obtain standardized 3D images to achieve spatial alignment of cross-modal images and ensure spatial consistency of data collected between different centers and different devices.

[0060] A4: The standardized three-dimensional images are cut longitudinally to obtain coronal images, which facilitates subsequent analysis and model construction of specific anatomical structures.

[0061] A5: Perform grayscale normalization on both SPECT and CT images, linearly mapping the original grayscale values ​​to the [0, 1] interval to eliminate the effects of device and parameter differences on grayscale distribution. The normalization formula is:

[0062]

[0063] Among them, g(x,y) is the normalized grayscale value, I max is the upper limit of grayscale, which can be 1.0; f(x,y) is the original grayscale value, min(g(x,y)) and max(g(x,y)) are the minimum and maximum values ​​before normalization, respectively.

[0064] A6: Automatically locate and crop the orbital region in the standardized image;

[0065] In this embodiment, a YOLOv5-based object detection model can be used to automatically detect the left and right eye sockets in the standardized image and output bounding box coordinates. Using the bounding box center as a reference, left and right eye socket image blocks of a set size (256×256 pixels) are cropped for subsequent independent processing of the left and right eyes, respectively. This operation significantly reduces the interference of irrelevant background information on the downstream segmentation model, ensuring that the region of interest is focused on the location of the extraocular muscle anatomy, thereby improving segmentation accuracy and model generalization.

[0066] In this embodiment, considering the subsequent training of the segmentation model and the classification model, step A7 may also be included;

[0067] A7: Based on task requirements, the standardized and cropped image data is divided into training, validation, and test sets, which are used for local training and performance evaluation of downstream segmentation and classification models, respectively.

[0068] B: Structural processing of the orbital region in the CT image is performed to automatically extract the preliminary segmentation masks of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles, which serve as the structural prior masks for the subsequent fine segmentation network;

[0069] In the present invention, the seed points of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles in the extraocular muscles are determined by morphological matching, and then a dynamic region growing algorithm based on grayscale similarity and position constraints is used to generate preliminary segmentation regions of each rectus muscle in the extraocular muscles. The preliminary segmentation mask of the extraocular muscles is obtained based on the preliminary segmentation regions, which serves as the structural prior mask for the subsequent fine segmentation network.

[0070] In the present invention, step B comprises the following specific steps:

[0071] B1: Anisotropic diffusion filtering is performed on the orbital region CT image to reduce background noise and enhance anatomical edges;

[0072] The diffusion coefficient c(x,y) is calculated as:

[0073]

[0074] in, is the grayscale gradient of point (x, y), H is the edge strength adjustment parameter, which can be 15, and the number of iterations is set to 10.

[0075] B2: The orbital region is extracted using an adaptive threshold segmentation method based on the grayscale histogram, and morphological opening and closing operations are used to remove isolated areas and fill edge holes to finally generate an orbital mask.

[0076] In this embodiment, the filtered image can be binarized by a dynamic threshold T to separate the eye socket area from the background and extract the eye socket area;

[0077] The dynamic threshold T is calculated as follows:

[0078]

[0079] Among them, I max and I min Represent the maximum grayscale value and minimum grayscale value of the image respectively.

[0080] B3: Calculate the center of mass of the orbital mask and divide the candidate regions of the superior rectus muscle, inferior rectus muscle, medial rectus muscle and lateral rectus muscle based on the center of mass;

[0081] Center of mass coordinates (x c ,y c ) is calculated as:

[0082]

[0083] Among them, N is the number of mask foreground pixels, (x i ,y i ) is the coordinate of the i-th foreground pixel;

[0084] Four candidate regions were divided along the horizontal and vertical lines of the centroid, and 20% of the orbital height and width were taken as buffer zones, respectively. Finally, candidate regions of the superior rectus muscle, inferior rectus muscle, medial rectus muscle, and lateral rectus muscle were constructed.

[0085] In this example, vertically, the area above the vertical centroid is selected as the superior rectus muscle candidate, and the area below the vertical centroid is selected as the inferior rectus muscle candidate. Horizontally, the area to the left of the centroid is selected as the medial rectus muscle candidate, and the area to the right of the centroid is selected as the lateral rectus muscle candidate. After dividing the candidate regions, based on the boundary range of the orbital mask, 20% of the orbital height and width are taken as buffer areas along the vertical and horizontal centroid directions, respectively. Ultimately, candidate regions for the superior, inferior, medial, and lateral rectus muscles are constructed.

[0086] B4: Using morphological matching method to locate the rectus muscle seed point in each candidate region;

[0087] Within the four candidate regions, we combined statistical anatomical information with edge features to locate seed points for the superior, inferior, medial, and lateral rectus muscles. To accurately estimate the center of the rectus muscles, we employed a template matching method based on the Active Shape Model (ASM).

[0088] Firstly, based on manually annotated samples, standard shape templates of four types of rectus muscles (superior rectus, inferior rectus, medial rectus and lateral rectus) were constructed. Each standard shape template consists of 50 equally spaced contour points, describing the average anatomical structure of the rectus muscle.

[0089] Canny edge detection (low threshold 30, high threshold 100) is used to extract edges within the candidate region. Combined with the ShapeContext matching method, the matching degree between each candidate edge in the image and the template point set is calculated. The edge-connected region with the highest matching score is selected and its centroid is used as the seed point for the rectus muscle.

[0090] The matching score is calculated as follows:

[0091]

[0092] Among them, SC(d i ) is the candidate edge point d i With the template point set T j The matching score is , M is the total number of template points, and σ is the matching sensitivity control parameter, which is defined as 25% of the average distance between points.

[0093] B5: Based on grayscale similarity and spatial position constraints, a dynamic region growing algorithm is used to generate a preliminary segmentation mask for each rectus muscle, which serves as the structural prior mask for the subsequent fine segmentation network.

[0094] In step B5, in each seed point neighborhood, the grayscale mean μ is calculated m and standard deviation σ m , let the dynamic grayscale similarity threshold of the mth rectus muscle be λ m :

[0095] λ m =o·σ m

[0096] Among them, the empirical coefficient o is 1.5.

[0097] During the region growing process, for each pixel p to be judged, if its gray value is equal to the gray mean μ of the current region, m The difference satisfies |p-μ m |<λ m , then the pixel is included in the segmentation area. Whenever a new pixel is included in the segmentation area, the grayscale mean μ of the area is dynamically updatedm and standard deviation σ m , and adjust the dynamic grayscale similarity threshold λ synchronously i .

[0098] To avoid expansion into non-target areas, a maximum expansion radius threshold is set (set to 30 pixels in this embodiment). If the aspect ratio of the segmented area exceeds the set ratio (which can be 1:4) or the length and width exceed the maximum expansion radius threshold, the area growth is terminated.

[0099] C: Fusion of structural prior masks to achieve multimodal collaborative segmentation of SPECT and CT images and obtain fine segmentation masks of extraocular muscles;

[0100] In the present invention, the constructed multimodal fine segmentation and prior fusion network can be used to fuse the structural prior mask to realize multimodal collaborative segmentation of SPECT images and CT images, automatically output the extraocular muscle fine segmentation mask, and provide structured high-quality segmentation results for subsequent intelligent diagnosis and analysis tasks.

[0101] The multimodal fine segmentation and prior fusion network consists of two independent encoder branches and a shared decoder module. The two encoder branches are used to process CT images and SPECT images respectively to extract anatomical structure information and metabolic function characteristics;

[0102] To enhance the network's ability to focus on the target region, the encoder input concatenates the CT and SPECT images with the preliminary segmentation masks for each extraocular muscle obtained in step B, along the channel dimension, to form an (image + mask) input. This structural design leverages prior information, suppresses background noise interference on feature extraction, and guides the model to focus on the edges and internal morphology of the extraocular muscle region, thereby improving downstream segmentation performance.

[0103] Each encoder branch contains four levels of convolutional modules; each convolutional module consists of two 3×3 convolutional layers, a batch normalization (BN) layer, and a Reluctant Unit (ReLU) activation function. The shallow layers extract low-level features such as edges, texture, and grayscale variations, particularly sensitive to details in muscle tissue. The deeper layers progressively extract high-level semantic features, including shape structure, spatial relationships, and cross-modal feature mapping. To effectively fuse multimodal features, after each convolutional block, the feature maps extracted from the CT and SPECT branches are concatenated along the channel dimension, and a lightweight gated attention mechanism is introduced to adaptively model the importance of features from different modalities. The fusion module consists of a 3×3 convolutional layer with a sigmoid activation function, generating a channel-level weight map that is used to weight the fusion ratio of CT and SPECT feature maps. This fusion is further fused through a 3×3 convolution operation, guiding the network to learn a more coordinated multimodal joint representation in the feature space, improving the semantic modeling of the target region. The fusion features include both the anatomical boundary information of CT images and the functional metabolic features of SPECT images, which helps to improve the accuracy of extraocular muscle structure recognition.

[0104] The decoder consists of four deconvolution layers, each followed by a convolution-BN-ReLU module to restore image spatial resolution and fine edge details. Each decoder layer receives high-resolution fused features from the corresponding encoder layer via skip connections, preserving local information from lower layers and enhancing the detail and accuracy of segmentation boundaries. Finally, a sigmoid activation function is used to output a binary segmentation mask.

[0105] Step C of the present invention comprises the following steps:

[0106] C1: Concatenate the CT image and SPECT image with the structural prior mask (i.e., the preliminary segmentation mask of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles obtained in step B) in the channel dimension to construct a multi-channel input containing the original image and structural prior information;

[0107] The prior guidance mechanism can enhance the network's ability to focus on the target area, suppress irrelevant noise, and improve the pertinence of feature extraction through structural prior.

[0108] C2: Extract anatomical and metabolic features of CT and SPECT images respectively;

[0109] In this embodiment, two independent encoder branches are used to extract the anatomical and metabolic features of CT and SPECT images respectively to obtain corresponding CT and SPECT feature maps; each encoder branch consists of four levels of convolutional modules (each convolutional module contains two layers of 3×3 convolution, batch normalization and ReLU activation), with the shallow layer focusing on edge and texture features and the deep layer focusing on high-level structure and semantic representation.

[0110] C3: Fusion of anatomical and metabolic features of CT and SPECT to obtain a fusion feature map;

[0111] After each convolutional module, the CT and SPECT feature maps are spliced ​​in the channel dimension and adaptively weighted fusion is performed using the input gated attention mechanism.

[0112] The gated attention mechanism includes a 3×3 convolutional layer with Sigmoid activation, which automatically generates channel-level weights and assigns different importance to feature maps of different modalities. After weighted adjustment, the features are fused through a layer of 3×3 convolution to achieve multimodal collaborative modeling, fully utilize anatomical and metabolic information, and improve the segmentation accuracy of the target area.

[0113] C4: Upsample and reconstruct the fused feature map layer by layer to restore the spatial resolution and detailed structure of the segmentation mask, and finally obtain a fine segmentation mask of the extraocular muscles.

[0114] The decoder consists of four deconvolutional layers, each of which includes deconvolution, 3×3 convolution, batch normalization, and activation operations. Each layer receives high-resolution features from the corresponding encoder layer via skip connections, ensuring the effective transfer of local structural details. Early decoding reconstructs overall muscle morphology, while late decoding enhances boundary continuity and internal structure. Finally, a sigmoid activation is applied to output a binary extraocular muscle segmentation mask.

[0115] In this invention, in order to balance the regional reconstruction accuracy and pixel-level classification accuracy and comprehensively evaluate the network segmentation performance, a total loss function L consisting of Dice coefficient loss, cross entropy loss and prior constraint loss is specially designed. seg :

[0116] L seg =L Dice +L CE +θ·L Prior

[0117] Among them, L Dice It is used to measure the degree of overlap between the predicted segmentation area and the true area, and is suitable for dealing with the problem of category imbalance in segmentation tasks; L CE is the pixel-level cross entropy loss, which improves the classification accuracy, θ is the weighted coefficient of the prior constraint, L Prioris the prior constraint loss, which is defined as the L2 distance between the predicted mask and the preliminary segmentation mask:

[0118]

[0119] in, is the network prediction value of the n-th pixel, is the structural prior mask of the nth pixel. By introducing the prior constraint, the network segmentation result can be closer to the structural prior, improving the accuracy and stability of the segmentation.

[0120] In the present invention, in order to ensure the efficient training and generalization ability of the segmentation network, the Adam optimizer can be used for parameter learning, where β1 is set to 0.9, β2 is set to 0.999, the prior loss weighting coefficient θ is set to 0.3, the initial learning rate is set to 0.001, the batch size is set to 4, and the training cycle is set to 100 rounds.

[0121] During training, we use the validation set to dynamically monitor loss trends, automatically adjusting the learning rate and terminating training early based on changes in model performance. Training is automatically terminated when validation set loss fails to decrease significantly over 15 consecutive epochs, effectively preventing overfitting and improving the model's generalization and training stability.

[0122] D: Utilizing fine segmentation masks of extraocular muscles, SPECT images, and CT images, the system fuses segmentation priors with multimodal features for deep discrimination, enabling automatic prediction of the activity stage of thyroid-related eye diseases and outputting prediction probabilities. This provides auxiliary verification data support for the diagnosis of thyroid-related eye diseases and improves diagnostic accuracy.

[0123] In the present invention, the constructed TAO activity staging multimodal 3D neural network can be used to fuse the extraocular muscle fine segmentation mask, SPECT image and CT image to realize the automatic discrimination and probability output of thyroid-related eye disease activity staging based on multimodal features and segmentation priors.

[0124] The TAO activity staging multimodal 3D neural network includes a multimodal input module, a multi-scale feature encoding module, a gated fusion module, a three-dimensional residual learning module, and a prediction output module, and has strong structural sensitivity and pathology discrimination capabilities;

[0125] The multimodal input module is used to combine the extraocular muscle fine segmentation mask, SPECT images, and CT images in the channel dimension to construct a three-channel input containing anatomical structure information, metabolic function information, and segmentation prior information. The multi-scale feature encoding module is used to extract multi-level features in three-dimensional space. The gated fusion module is used to fuse multi-scale fine-grained features. The three-dimensional residual learning module and the prediction output module are used to perform deep semantic extraction on the fused multi-scale features and ultimately output the TAO activity staging probability.

[0126] In the present invention, step D comprises the following steps:

[0127] D1: Using a multimodal input module, the fine segmentation mask of the extraocular muscles obtained in step C, the SPECT image, and the CT image are concatenated channel-wise to construct a three-channel input containing anatomical structure information, metabolic function information, and segmentation prior information. This multimodal input approach explicitly introduces structural guidance while preserving the original multimodal features, enhancing the network's ability to focus on the extraocular muscle region and express activity characteristics.

[0128] D2: Utilizes a multi-scale feature encoding module to extract multi-level features in three-dimensional space. This module consists of three parallel convolution paths with kernel sizes of 3×3×3, 5×5×5, and 7×7×7, respectively, to simultaneously extract local texture and global morphological features. Small-scale branches enhance the detection of subtle density and muscle fiber variations, while large-scale branches improve the ability to control the overall morphology of the extraocular muscles. Through multi-scale parallel branches, detailed and global contextual information can be simultaneously acquired, enhancing the ability to discriminate between different activity patterns.

[0129] D3: Use the gated fusion module to perform multi-scale fine-grained feature fusion.

[0130] After each convolution branch, a weight prediction subnetwork consisting of a 3×3×3 convolution operation and a Softmax activation function is connected to assign dynamic weights to feature maps of different scales, realize adaptive weighted integration of features at each scale, and improve the ability to discriminate different activity patterns.

[0131] D4: Using the 3D residual learning module and the prediction output module, deep semantic extraction is performed on the fused multi-scale features, and the final output is the TAO activity stage probability;

[0132] The 3D residual learning module and prediction output module consist of three 3D residual units connected in series. Each unit includes two 3×3×3 convolutional layers and batch normalization. Short connections are used to directly add input and output features, effectively improving gradient fluidity with network depth and preventing feature degradation. After residual stacking, the feature map undergoes dimensionality reduction using a global average pooling layer to extract overall statistical features. The output is then passed through a fully connected layer to generate the TAO activity stage probability, which is finally normalized to the [0, 1] range using a sigmoid activation function.

[0133] In this paper, in order to optimize the classification performance of 3D neural network in TAO activity staging task, a combined loss function L consisting of binary cross entropy loss and L2 regularization term is adopted. cls , which is expressed as:

[0134]

[0135] Among them, L BCE represents the binary cross entropy loss, which is used to measure the consistency between the predicted probability and the true label; λ is the regularization coefficient, which is used to balance the contribution between loss terms; Represents the L2 norm of all trainable weights of the network, which is used to suppress overfitting caused by excessive parameters;

[0136] To ensure the stability and generalization of network training, the Adam optimizer was used for parameter learning. The hyperparameters β1 and β2 were set to 0.9 and 0.999, the initial learning rate was set to 1e-4, the regularization coefficient λ was set to 1e-5, the batch size was set to 4, and the training cycle was set to 150 rounds. An early stopping mechanism was used during training to terminate training early if the validation set loss did not improve within 15 consecutive rounds, effectively preventing overfitting and improving model performance.

[0137] E: A dynamic model combination strategy based on style similarity and uncertainty quantification is used to obtain a dynamic combination model of segmentation and classification that adapts to the feature distribution of the new domain, accurately segmenting the extraocular muscle structure and predicting TAO activity in an invisible data center;

[0138] In this paper, to address the problem of feature distribution drift of invisible new centers in actual multi-center scenarios, a dynamic model fusion method combining style similarity and uncertainty quantification is proposed. This method can adaptively combine the segmentation and classification model parameters of multi-source centers without accessing the new center label data, and realize cross-center generalization prediction of extraocular muscle structure segmentation and TAO activity staging.

[0139] The step E comprises the following steps:

[0140] E1: During the federated learning training phase, each center independently trains the local extraocular muscle segmentation model (i.e., the multimodal fine segmentation and prior fusion network constructed in step C) and the TAO activity staging model (i.e., the TAO activity staging multimodal 3D neural network constructed in step D). During the federated training, all centers regularly upload local model parameters using the FedAvg method, and the server-side aggregates them to form a global model. After the training is completed, each center caches its own last round of local segmentation and classification model parameters, which are recorded as segmentation model parameters respectively. and classification model parameters Where (s=1, 2, ... S) is the center number, and S is the total number of centers participating in federated learning.

[0141] E2: Perform style feature extraction and uncertainty score calculation;

[0142] For all training samples of each center, the auxiliary pre-trained 3D ResNet-18 network is used to extract the style features of medical images. For each training sample, the 16-dimensional style vector after the global average pooling layer is extracted. Take the average of all sample style vectors at each center to get the global style feature μ s , calculated as follows:

[0143]

[0144] Among them, N s is the number of training samples for the sth center, is the 16-dimensional style feature vector of the p-th sample at the s-th center. Then, the variance vector between samples is calculated

[0145]

[0146] Among them, μ s is the global style feature of the s-th center. Then the variance vector The uncertainty score is obtained by taking the average value to characterize the consistency of the central data style. The lower the score, the more stable the style. s The calculation is as follows:

[0147]

[0148] in, is the style feature variance of the s-th center in the q-th dimension.

[0149] E3: Based on style similarity and uncertainty, a bilateral weighting strategy is used to calculate the combination weight α s And distribute the weights.

[0150] When encountering an invisible new center u, the above step E2 is used to extract the global style feature μ of the new center u and the uncertainty score u u Then, the style similarity s between the new center and each known center is calculated s , calculated as follows:

[0151] s s =cos(μ u ,μ s )

[0152] Where cos(·,·) is the cosine similarity. Combining style similarity and uncertainty, a bilateral weighting strategy is used to calculate the combined weight α s , calculated as follows:

[0153]

[0154] where s s is the style similarity between the new center u and the s-th source center, s t represents the style similarity between the new center u and the t-th source center, u t is the uncertainty score of the t-th source center. The weight allocation mechanism prioritizes centers with similar styles and stable distributions, weakens the influence of centers with large style deviations and high uncertainty, and improves the adaptability of the portfolio model.

[0155] E4: Combination weight α obtained based on E3 s Local segmentation model parameters for all source centers and the stage model parameters By performing weighted fusion, a dynamic combined segmentation and classification model is derived that adapts to the feature distribution of the new domain. Ultimately, this adaptive model can be directly applied to extraocular muscle structure segmentation and TAO activity stage inference in the new domain data. Ultimately, this enables accurate extraocular muscle structure segmentation and TAO activity prediction in an invisible data center, providing auxiliary verification data support for the diagnosis of thyroid-related eye diseases and improving diagnostic accuracy.

[0156] in,

[0157]

[0158] Finally, it should be noted that the embodiments described above are intended only to illustrate the technical solutions of the present invention and do not constitute limitations thereon. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art may modify or substitute equivalents for these embodiments without departing from the spirit and substance of the present invention, and such modifications and substitutions should be deemed to fall within the scope of protection claimed by the present invention.

Claims

1. A method for auxiliary verification of thyroid-related eye diseases based on SPECT / CT and federated learning, characterized in that: The steps include: A: Perform image data standardization and orbital region extraction; B: Structural processing is performed on the orbital region in the CT image, and preliminary segmentation masks of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles are automatically extracted as structural prior masks; C: Fusion of structural prior masks to achieve multimodal collaborative segmentation of SPECT and CT images and obtain fine segmentation masks of extraocular muscles; D: Using the fine segmentation mask of extraocular muscles, SPECT images and CT images, the segmentation prior is integrated with multimodal features for deep discrimination, automatic prediction of the activity stage of thyroid-related eye disease and output of prediction probability; E: A dynamic model combination strategy based on style similarity and uncertainty quantification is used to obtain a dynamic combination model of segmentation and classification that adapts to the feature distribution of the new domain, and accurately segment the extraocular muscle structure and predict TAO activity in an invisible data center.

2. The method according to claim 1, wherein: In step A, the object detection model is used to automatically detect the left and right eye sockets in the standardized image and output the bounding box coordinates; and the left and right eye socket image blocks of a set size are cropped based on the center of the bounding box.

3. The method according to claim 1, wherein: In step B, the seed points of the superior rectus, inferior rectus, medial rectus, and lateral rectus muscles in the extraocular muscles are determined by morphological matching. Then, a dynamic region growing algorithm based on grayscale similarity and position constraints is used to generate preliminary segmentation regions of each rectus muscle in the extraocular muscles. Based on the preliminary segmentation regions, a preliminary segmentation mask of the extraocular muscles is obtained as a structural prior mask.

4. The method according to claim 1, wherein Step B includes the following specific steps: B1: Anisotropic diffusion filtering is performed on the orbital region CT image to reduce background noise and enhance anatomical edges; B2: The orbital region is extracted using an adaptive threshold segmentation method based on the grayscale histogram. Morphological opening and closing operations are then used to remove isolated regions and fill in edge holes to generate an orbital mask. B3: Calculate the center of mass of the orbital mask and divide the candidate regions of the superior rectus muscle, inferior rectus muscle, medial rectus muscle and lateral rectus muscle based on the center of mass; B4: Using morphological matching method to locate the rectus muscle seed point in each candidate region; B5: Based on grayscale similarity and spatial position constraints, a dynamic region growing algorithm is used to generate the preliminary segmentation mask of each rectus muscle.

5. The method according to claim 1, wherein: In step C, a multimodal fine segmentation and prior fusion network is used to achieve multimodal collaborative segmentation of SPECT and CT images. The multimodal fine segmentation and prior fusion network consists of two independent encoder branches and a shared decoder module. The two encoder branches are used to process CT images and SPECT images, respectively, to extract anatomical structure information and metabolic function features. Each encoder branch contains multiple levels of convolutional modules. The shallow structure in the convolutional module is used to extract low-level features, and the deep structure is used to extract high-level semantic features. The fusion module is used to splice the feature maps generated by the convolutional module in the channel dimension and weightedly adjust the fusion ratio of CT and SPECT feature maps. The decoder consists of multiple deconvolution layers to restore the spatial resolution and fine edge details of the image; each layer of the decoder receives high-resolution fusion features from the corresponding layer of the encoder through jump connections, and finally outputs a preliminary segmentation mask of the extraocular muscles.

6. The method according to claim 1, wherein Step C includes the following steps: C1: Concatenate the CT image and SPECT image with the structural prior mask in the channel dimension to construct a multi-channel input containing the original image and structural prior information; C2: Extract anatomical and metabolic features of CT and SPECT images respectively; C3: Fusion of anatomical and metabolic features of CT and SPECT to obtain a fusion feature map; C4: Upsample and reconstruct the fused feature map layer by layer to restore the spatial resolution and detailed structure of the segmentation mask, and finally obtain a fine segmentation mask of the extraocular muscles.

7. The method according to claim 1, wherein: In step D, the TAO activity staging multimodal 3D neural network is used to automatically identify and output the activity staging of thyroid-associated eye disease based on the TAO activity staging model. TAO activity staging multimodal 3D neural network, including a multimodal input module, a multi-scale feature encoding module, a gated fusion module, a 3D residual learning module, and a prediction output module; The multimodal input module is used to stitch the extraocular muscle fine segmentation mask, SPECT image and CT image in the channel dimension to construct a three-channel input containing anatomical structure information, metabolic function information and segmentation prior information; the multi-scale feature encoding module is used to extract multi-level features in three-dimensional space; the gated fusion module is used to fuse multi-scale fine-grained features; The three-dimensional residual learning module and the prediction output module are used to perform deep semantic extraction on the fused multi-scale features and finally output the TAO activity stage probability.

8. The method according to claim 1, wherein Step D includes the following steps: D1: Using the multimodal input module, the extraocular muscle fine segmentation mask, SPECT image, and CT image are spliced ​​in the channel dimension to construct a three-channel input containing anatomical structure information, metabolic function information, and segmentation prior information; D2: Use the multi-scale feature encoding module to extract multi-level features in three-dimensional space; D3: Use the gated fusion module to perform multi-scale fine-grained feature fusion; D4: Utilize the three-dimensional residual learning module and the prediction output module to perform deep semantic extraction on the fused multi-scale features and ultimately output the TAO activity stage probability.

9. The method according to claim 1, wherein Step E includes the following steps: E1: During the federated learning training phase, each center independently trains a local extraocular muscle segmentation model and TAO activity staging model. All centers periodically upload local model parameters, which are aggregated on the server to form a global model. After training, the last round of local segmentation and classification model parameters are cached. E2: Perform style feature extraction and uncertainty score calculation; the global style feature is the mean of the style vectors of all samples at each center, and the uncertainty score is the mean of the variance vectors between samples; E3: Based on style similarity and uncertainty, a bilateral weighting strategy is used to calculate the portfolio weight and perform weight distribution; E4: Based on the combined weights, the local segmentation model parameters and staging model parameters of all source centers are weightedly fused to obtain a dynamic combination model of segmentation and classification that adapts to the feature distribution of the new domain and is directly used for the extraocular muscle structure segmentation and TAO activity staging inference of the new domain data, completing the accurate segmentation of the extraocular muscle structure and the prediction of TAO activity in the invisible data center.

10. The method according to claim 1, wherein Step A includes the following steps: A1: Acquire and import original SPECT and CT images from each data center; A2: Resample the original SPECT image and CT image; A3: Map the resampled image to the same 3D coordinate system to obtain a standardized 3D image; A4: Cut the standardized 3D image longitudinally to obtain the coronal image; A5: Perform grayscale normalization on SPECT and CT images respectively; A6: In the normalized image, locate and crop the orbital region.

Images