Deep learning prediction system and method based on multi-mode thyroid cancer lymph node metastasis

Abstract

The invention relates to the field of medical image analysis, in particular to a deep learning prediction system and method based on multi-modal thyroid cancer lymph node metastasis, and the system comprises a data collection module, a preprocessing module, a nodule segmentation module, a feature extraction module, a feature fusion module, a metastasis prediction module, an interpretability analysis module and a result display module. An ultrasonic image, an elastic imaging image, an ultra-micro blood flow image and clinical index data of a patient are integrated, an improved U-Net algorithm is used for precise segmentation of a thyroid nodule region, a multi-branch deep network is used for extracting multi-modal features, a dynamic weight fusion algorithm is used for integrating the features, and the accuracy of the thyroid nodule region is improved. According to the method, the thyroid cancer lymph node metastasis state (non-metastasis, central region metastasis or lateral neck metastasis) is predicted, meanwhile, a two-dimensional interpretability framework of Grad-CAM activation diagram and SHAP value contribution degree analysis is introduced, an intuitive prediction basis is provided for doctors, and the thyroid cancer lymph node metastasis prediction accuracy is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis, and in particular to a deep learning-based prediction system and method for lymph node metastasis in multimodal thyroid cancer, which can be applied to medical image diagnosis and clinical decision support systems. Background Technology

[0002] Thyroid cancer is one of the most common malignant tumors of the endocrine system, and its incidence is increasing year by year, showing a trend towards affecting younger people. Clinical practice shows that early detection and accurate diagnosis are key to improving the cure rate of thyroid cancer.

[0003] Traditional diagnosis of lymph node metastasis in thyroid cancer primarily relies on ultrasound imaging and fine-needle aspiration biopsy. However, these methods have several limitations: First, a single modality of imaging cannot comprehensively reflect the characteristics of the lesion, increasing the risk of misdiagnosis and missed diagnosis; second, the diagnostic results are highly dependent on the physician's experience and subjective judgment, with low consistency among different physicians in diagnosing the same images; third, for tiny nodules less than 5 mm in diameter, the sensitivity of traditional ultrasound examination is only 65%–78%; furthermore, the accuracy in differentiating lymph node metastasis types (such as central metastasis versus lateral cervical metastasis) is insufficient, directly affecting the determination of the subsequent surgical scope.

[0004] In recent years, deep learning technology has made significant breakthroughs in the field of medical image analysis. Previous studies have applied algorithms such as ResNet50 to classify benign and malignant thyroid nodules, achieving some success. However, existing methods still suffer from three main problems: first, single-modal data is insufficient to fully represent lesion characteristics; second, data imbalance leads to insufficient predictive ability for a few categories; and third, the black-box nature of deep learning models makes doctors lack confidence in the prediction results.

[0005] Therefore, there is an urgent need to develop a thyroid cancer lymph node metastasis prediction system that can integrate multimodal imaging data, improve the accuracy of metastasis prediction, and has interpretability, so as to provide reliable auxiliary decision support for clinical diagnosis and treatment. Summary of the Invention

[0006] The purpose of this invention is to provide a deep learning prediction system and method for thyroid cancer lymph node metastasis based on multimodal approaches, aiming to solve the technical problems existing in the prior art, such as the limitations of single-modal data, insufficient accuracy in metastasis type identification, and insufficient model interpretability.

[0007] This invention proposes a deep learning-based prediction system for lymph node metastasis in multimodal thyroid cancer, comprising:

[0008] The data acquisition module is used to acquire multimodal imaging data, including ultrasound images, elastography images, and ultramicrovascular flow images, as well as patient clinical indicator data.

[0009] A preprocessing module, connected to the data acquisition module, is used to perform standardization and data enhancement on the multimodal image data;

[0010] The nodule segmentation module, connected to the preprocessing module, is used to segment the thyroid nodule region of the multimodal image data based on the improved U-Net algorithm;

[0011] The feature extraction module, connected to the nodule segmentation module, is used to extract features from the ultrasound image, the elastography image, the ultramicro blood flow image, and the patient's clinical index data through a multi-branch deep network.

[0012] A feature fusion module, connected to the feature extraction module, is used to fuse the multimodal features extracted by the feature extraction module using a dynamic weight fusion algorithm;

[0013] The metastasis prediction module, connected to the feature fusion module, is used to classify the lymph node metastasis status of patients with thyroid cancer into non-metastasis, with central region metastasis, or with lateral neck metastasis based on the fused multimodal features.

[0014] An interpretability analysis module, connected to the transition prediction module, is used to generate class activation maps and feature contribution analysis, providing a visual interpretation of the prediction results;

[0015] The results display module, connected to the transfer prediction module and the interpretability analysis module, is used to display classification results, class activation maps and feature contribution analysis, and generate diagnostic reports.

[0016] Preferably, the preprocessing module includes:

[0017] The image normalization unit is used to uniformly adjust the resolution of the multimodal image data to 224×224 pixels and perform grayscale normalization processing.

[0018] The data enhancement unit is used to perform spatial transformation, elastic deformation, and contrast adjustment on the multimodal image data;

[0019] The missing data processing unit is used to fill in the missing items in the patient's clinical indicator data.

[0020] Preferably, the nodule segmentation module includes:

[0021] The encoder contains multi-scale dilated convolutional units to extract multi-scale features and expand the receptive field;

[0022] The decoder includes a cross-layer attention gating mechanism for dynamically fusing features from different layers;

[0023] A multi-task learning unit is used to simultaneously output nodule probability masks and microcalcification boundary curves.

[0024] Preferably, the feature extraction module includes:

[0025] The ultrasound image feature extraction unit, based on the EfficientNet-V2 network, is used to extract texture and acoustic features from ultrasound images.

[0026] The elastic imaging feature extraction unit includes a channel attention mechanism for focusing on regions of abnormal hardness.

[0027] The ultra-micro blood flow feature extraction unit is used to extract blood vessel distribution density and morphological features;

[0028] The clinical data feature extraction unit, based on a fully connected network, is used to mine nonlinear correlations between clinical indicators.

[0029] Preferably, the feature fusion module includes:

[0030] Feature importance assessment unit, used to calculate the weight coefficients of each modal feature;

[0031] A dynamic weight allocation unit is used to optimize the fusion weights of different modal features based on KL divergence.

[0032] The subtype-sensitive adjustment unit is used to dynamically adjust feature weights for different pathological subtypes.

[0033] Preferably, the transfer prediction module includes:

[0034] The multi-model integration unit integrates three models: ResNet50, EfficientNet-V2, and VisionTransformer.

[0035] The pathological subtype stratification unit is used to implement a three-stage classification strategy, which distinguishes between benign and malignant, metastatic and non-metastatic, and central metastasis and lateral neck metastasis in sequence.

[0036] The sample balancing unit is used to solve the class imbalance problem;

[0037] The decision threshold optimization unit is used to adjust the sensitivity and specificity of the model according to clinical needs.

[0038] Preferably, the interpretability analysis module includes:

[0039] Grad-CAM generation unit is used to generate heatmaps based on the feature map weights of convolutional layers to display the image regions of interest to the model;

[0040] The SHAP value calculation unit is used to quantify the contribution of each clinical indicator and imaging feature to the prediction results.

[0041] The dual-dimensional interpretation integration unit is used to correlate heatmaps with feature contribution analysis results, providing a comprehensive interpretation.

[0042] Preferably, the result display module includes:

[0043] The probability distribution display unit is used to show the predicted probabilities of the three transition states;

[0044] The activation map visualization unit is used to display the image regions that the model focuses on;

[0045] The feature contribution display unit is used to display key predictive factors and their contributions;

[0046] The diagnostic report generation unit is used to generate structured reports that conform to the TI-RADS classification standards.

[0047] As a preferred option, it also includes:

[0048] The system validation module is used to evaluate system performance through 10-fold cross-validation.

[0049] The clinical feedback module is used to collect feedback from doctors and optimize the system.

[0050] Deep learning-based prediction methods for lymph node metastasis in multimodal thyroid cancer include:

[0051] Acquire multimodal imaging data, including ultrasound images, elastography images, and ultramicrovascular flow images, as well as patient clinical indicator data;

[0052] The multimodal image data is standardized and augmented.

[0053] The improved U-Net algorithm is used to segment the thyroid nodule region in the multimodal image data;

[0054] Features were extracted from the ultrasound images, elastography images, ultramicrovascular blood flow images, and patient clinical index data using a multi-branch deep network.

[0055] The extracted multimodal features are fused using a dynamic weight fusion algorithm;

[0056] Based on the fused multimodal features, the lymph node metastasis status of patients with thyroid cancer is classified as non-metastasis, with central region metastasis, or with lateral neck metastasis.

[0057] Generate class activation maps and feature contribution analysis to provide a visual interpretation of the prediction results;

[0058] Displays classification results, class activation graphs, and feature contribution analysis, and generates a diagnostic report.

[0059] The beneficial effects of this invention include:

[0060] 1. By integrating multi-source information such as ultrasound images, elastography images, ultra-micro blood flow images, and patient clinical index data, a multimodal deep learning network was established, which significantly improved the accuracy of predicting lymph node metastasis in thyroid cancer, especially for the accurate classification of different metastasis types (no metastasis, with central metastasis, with lateral neck metastasis);

[0061] 2. An improved U-Net algorithm was used to achieve accurate segmentation of thyroid nodules, which improved the ability to capture key features such as microcalcifications and marginal spiculations, laying the foundation for subsequent metastasis prediction.

[0062] 3. By introducing a dynamic weight fusion strategy and a pathological subtype hierarchical modeling mechanism, the contribution assessment of different modal features was optimized, and the adaptability of the model to complex cases was improved.

[0063] 4. A two-dimensional interpretability analysis framework based on Grad-CAM and SHAP values ​​was constructed, enabling doctors to intuitively understand the basis of model decision-making and enhancing clinicians' trust in AI prediction results;

[0064] 5. The system has a complete clinical application interface and workflow, supporting the entire process from image uploading and automatic analysis to structured report generation, effectively improving clinical work efficiency. Attached Figure Description

[0065] Figure 1 This is a diagram illustrating the overall architecture of the deep learning-based prediction system for lymph node metastasis in multimodal thyroid cancer according to the present invention.

[0066] Figure 2 This is a flowchart of the data preprocessing module in the system of the present invention.

[0067] Figure 3 This is a structural diagram of the improved U-Net nodule segmentation module in the system of this invention.

[0068] Figure 4 This is a structural diagram of the multimodal feature extraction module in the system of the present invention.

[0069] Figure 5 This is a schematic diagram of the dynamic weight fusion strategy in the system of the present invention.

[0070] Figure 6 This is a flowchart of the thyroid cancer lymph node metastasis prediction module in the system of the present invention.

[0071] Figure 7 This is a structural diagram of the interpretability analysis module in the system of this invention. Detailed Implementation

[0072] Please refer to Figure 1 - Figure 7 The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, the present invention provides a deep learning prediction system for lymph node metastasis of thyroid cancer based on multimodal thyroid cancer, which mainly includes a data acquisition module 10, a preprocessing module 20, a nodule segmentation module 30, a feature extraction module 40, a feature fusion module 50, a metastasis prediction module 60, an interpretability analysis module 70, and a result display module 80.

[0074] The data acquisition module 10 acquires multimodal image data, including ultrasound images, elastography images, and ultra-micro blood flow images, as well as patient clinical indicator data. The preprocessing module 20 standardizes and augments the acquired multimodal image data. The nodule segmentation module 30 uses an improved U-Net algorithm to segment thyroid nodule regions from the preprocessed multimodal image data. The feature extraction module 40 extracts features from ultrasound images, elastography images, ultra-micro blood flow images, and patient clinical indicator data using a multi-branch deep network. The feature fusion module 50 fuses the multimodal features extracted by the feature extraction module using a dynamic weight fusion algorithm. The metastasis prediction module 60 classifies the lymph node metastasis status of the patient's thyroid cancer based on the fused multimodal features: non-metastasis, with central metastasis, or with lateral neck metastasis. The interpretability analysis module 70 generates class activation maps and feature contribution analysis, providing a visual interpretation of the prediction results. The results display module 80 displays the classification results, class activation maps, and feature contribution analysis, and generates a diagnostic report.

[0075] In a preferred embodiment of the present invention, the modules are connected through standardized data interfaces to ensure the consistency and integrity of data flow. The system adopts a layered design and can be deployed on the servers or cloud platforms of medical institutions, supporting concurrent access by multiple users.

[0076] The data acquisition module 10 includes an image acquisition unit 11 and a clinical data acquisition unit 12. The image acquisition unit 11 is used to acquire ultrasound images, elastography images, and ultra-micro blood flow images from the ultrasound equipment, supporting the DICOM standard format. The clinical data acquisition unit 12 is used to collect basic patient information (such as age and gender), laboratory test data (such as TSH value and BRAF gene mutation status), and other clinical indicators.

[0077] In one embodiment of the present invention, the system supports direct retrieval of patient image data from the hospital PACS system, and also supports manual uploading of local image files. For clinical data, the system can be integrated with the hospital information system (HIS) to automatically retrieve relevant clinical indicators, and also supports manual data entry.

[0078] like Figure 2 As shown, the preprocessing module 20 includes an image normalization unit 21, a data augmentation unit 22, and a missing data processing unit 23.

[0079] Image standardization unit 21 is used to uniformly process multimodal image data acquired from different sources and devices into a standard format. Specific implementation methods include: uniformly adjusting the resolution of all images to 224×224 pixels; performing grayscale normalization on the images to ensure pixel values ​​are distributed within the range of [0,1]; and adjusting the contrast to optimize the dynamic range to the standard level required for medical image diagnosis (typically 60-80dB).

[0080] Image normalization can be achieved using the following formula:

[0081] ,

[0082] in, The image after normalization. For the original image, The mean of the ImageNet pre-trained dataset (typically [0.485, 0.456, 0.406]). is the standard deviation of the ImageNet pre-trained dataset (typically [0.229, 0.224, 0.225]).

[0083] The data augmentation unit 22 is used to expand the training dataset through a series of transformation operations to improve the generalization ability of the model. In this embodiment, the data augmentation strategies employed include: random rotation (within ±15°), random translation (within ±10% of the image size), random scaling (within 0.9-1.1 times), elastic deformation (parameters set to alpha=1200, sigma=80, simulating tissue deformation), and random adjustment of brightness and contrast (within ±15%).

[0084] For special processing of thyroid ultrasound images, ultrasound-specific enhancement strategies were added, such as adding speckle noise to simulate the noise characteristics in real ultrasound imaging. Preferably, the data augmentation strategy employs a dynamic scheduling mechanism, using mild enhancement in the early stages of training and gradually increasing the enhancement intensity as training progresses, effectively preventing overfitting.

[0085] The missing data processing unit 23 is used to handle potential missing values ​​in clinical data. For continuous variables (such as maximum diameter), multiple imputation is used to estimate based on other complete variables; for categorical variables (such as metastasis), mode imputation is used. To improve estimation accuracy, a dedicated regression model is constructed to predict missing values ​​based on relevant clinical parameters, ensuring data integrity.

[0086] like Figure 3 As shown, the nodule segmentation module 30 adopts an improved U-Net algorithm, including an encoder 31, a decoder 32, and a multi-task learning unit 33.

[0087] Encoder 31 includes multi-scale dilated convolutional units for extracting multi-scale features and expanding the receptive field. The ordinary convolutional layers in the traditional U-Net architecture are replaced with multi-scale dilated convolutional groups containing convolutional operations with different dilation rates, preferably set to 1, 2, and 4, applied in parallel and then concatenated. This design expands the receptive field from the standard 7×7 pixels to 15×15 pixels, significantly enhancing the ability to capture subtle features such as microcalcifications and edge burrs.

[0088] The mathematical expression for multi-scale dilated convolution is:

[0089] ,

[0090] in, To output feature map at position The value at that location, For the input feature map, For convolution kernel weights, Half the size of the convolution kernel. This represents the dilation rate. Convolution results with different dilation rates are concatenated along the channel dimension.

[0091] ,

[0092] in, This is the spliced ​​feature map. , , These are the convolutional output feature maps with dilation rates of 1, 2, and 4, respectively. This indicates a splicing operation at the channel dimension.

[0093] Decoder 32 incorporates a cross-layer attention gating mechanism for dynamically fusing features from different layers. The simple skip connections in traditional U-Net are replaced with an attention gating mechanism, which dynamically adjusts weights based on feature relevance, highlighting important features and suppressing irrelevant information. The computation process of the attention gating mechanism is as follows:

[0094] ,

[0095] ,

[0096] ,

[0097] in, The features of skip connections are of dimension . , For upsampling features, the dimension is also [missing information]. , The sigmoid activation function is used. This is the gated weight matrix, with dimension 1. , The attention weight matrix has dimensions of . , and For the corresponding bias term, This indicates a feature concatenation operation, which concatenates features along the channel dimension. and connect, This is an attention weight graph with dimensions [missing information]. , The output feature has a dimension of . Indicates the feature map height. Indicates the width of the feature map. This indicates the number of channels in the feature map.

[0098] The multi-task learning unit 33 is used to simultaneously output nodule probability masks and microcalcification boundary curves. By designing a multi-task joint loss function, the Dice loss function is used to optimize region segmentation on one hand, and the boundary constraint loss function is used to accurately locate microcalcification boundaries on the other. These two tasks mutually promote each other, improving overall segmentation performance. The multi-task joint loss function is defined as follows:

[0099] ,

[0100] in, For the total loss, The Dice loss is used to evaluate the accuracy of region segmentation. Boundary constraint loss is used to accurately locate nodule boundaries; and As balancing factors, the weights of the two types of loss in the total loss are controlled separately, and are set to 0.7 and 0.3 based on experience.

[0101] The formula for calculating the Dice loss function is:

[0102] ,

[0103] in, Let be the predicted probability value for the i-th pixel position, with a value ranging from [0,1]. This represents the true label value at the i-th pixel position, which is either 0 or 1, indicating whether the pixel belongs to a nodule region. Summation operation. Iterate through all pixels in the image.

[0104] The formula for calculating boundary constraint loss is:

[0105] ,

[0106] in, This represents the gradient operator, used to calculate the edge gradients between the predicted mask and the ground truth mask. This represents the gradient vector of the predicted mask at the i-th pixel position. This represents the gradient vector of the real mask at the i-th pixel position. This represents the L2 norm (Euclidean norm). This represents the total number of pixels, used to normalize the loss value.

[0107] In practical applications, the nodule segmentation module 30 employs a mini-batch training strategy with a batch size of 8, an initial learning rate of 0.001, and dynamic adjustment using a cosine annealing strategy. For difficult samples with blurred edges or indistinct microcalcifications, an additional hard-case mining strategy is used to increase their sampling frequency during training and improve segmentation accuracy. After 10-fold cross-validation, this improved U-Net segmentation algorithm achieves a Dice coefficient of 93.2% on the thyroid nodule segmentation task, providing high-quality segmentation results for subsequent metastasis prediction.

[0108] like Figure 4 As shown, the feature extraction module 40 adopts a multi-branch architecture, including an ultrasound image feature extraction unit 41, an elastography feature extraction unit 42, an ultra-micro blood flow feature extraction unit 43, and a clinical data feature extraction unit 44.

[0109] The ultrasound image feature extraction unit 41, based on the EfficientNet-V2 network, is used to extract texture and sonographic features from ultrasound images. This network employs a composite scaling strategy to balance the depth, width, and resolution dimensions, achieving efficient feature extraction. The network input is a 224×224×1 grayscale image, which, through 7 MBConv blocks and 1 SE (Squeeze-and-Excitation) module, finally outputs a 1280-dimensional feature vector. Preferably, in ultrasound feature extraction, special attention is paid to typical features of thyroid malignancies such as microcalcifications, edge spiculations, and echo heterogeneity. Adaptive receptive field convolutions (convolutions with variable dilation rates) are added to enhance the capture of these features.

[0110] The elastography feature extraction unit 42 includes a channel attention mechanism for focusing on areas of abnormal stiffness. Elastography reflects the distribution of tissue stiffness and is an important basis for assessing the benign or malignant nature of thyroid nodules. This unit adopts a similar EfficientNet-V2 architecture to the ultrasound image feature extraction unit, but adds a channel attention module in the intermediate layer, calculated as follows:

[0111] ,

[0112] ,

[0113] in, The input feature map has a dimension of . , For global average pooling operation, Dimensionality reduced to 1 The vector, The weights of the first fully connected layer have a dimension of . , To reduce the dimensionality (usually set to 16), The weights for the second fully connected layer have a dimension of [value missing]. ReLU is the modified linear unit activation function. It is the sigmoid activation function. Multiplication by the channel dimension indicates that... (dimension is) )and (dimension is) Perform channel-by-channel multiplication. The weighted output features have the same dimensions as... Same, for . Indicates the feature map height. Indicates the width of the feature map. This indicates the number of channels in the feature map.

[0114] Through this mechanism, the model can automatically focus on areas of abnormal hardness, such as high-hardness areas in thyroid nodules with an elasticity score of 4-6 (BRAF scoring system), which are usually highly associated with malignant lesions.

[0115] The ultra-micro blood flow feature extraction unit 43 is used to extract the distribution density and morphological features of blood vessels. Ultra-micro blood flow imaging can display the distribution of tiny blood vessels and has special diagnostic value for lymph node metastasis in thyroid cancer. This unit adopts a variant ResNet structure with dilated convolutions, which can effectively capture morphological features of blood vessels at different scales. The network output is a 512-dimensional feature vector, representing the intensity, direction, and distribution pattern of the blood flow signal.

[0116] The clinical data feature extraction unit 44 is based on a fully connected network and is used to mine nonlinear correlations between clinical indicators. This unit receives standardized clinical indicators, including basic patient information (age, gender), laboratory test data (TSH value, anti-thyroid antibody level), and gene testing results (BRAF gene mutation status). The network structure consists of three fully connected layers, with batch normalization and Dropout (with a ratio of 0.3) added between layers to prevent overfitting, and finally outputs a 256-dimensional feature vector.

[0117] Each feature extraction unit operates independently, extracting the most discriminative features from its respective modality, and then passing these features to the feature fusion module for integration. Preferably, each feature extraction unit employs a pre-training-fine-tuning strategy, first pre-training on a large-scale public dataset and then fine-tuning on the target task data, effectively addressing the problem of limited sample size in medical data.

[0118] like Figure 5 As shown, the feature fusion module 50 includes a feature importance evaluation unit 51, a dynamic weight allocation unit 52, and a subtype sensitivity adjustment unit 53.

[0119] Feature importance assessment unit 51 is used to calculate the weight coefficients of each modality feature. In medical diagnosis, the importance of different modalities is usually different and may vary with the characteristics of the case. This unit uses a feature importance assessment method based on an attention mechanism, and the calculation formula is as follows:

[0120] ,

[0121] in, For the first The weight coefficients of each mode take values ​​in the range [0,1] and satisfy the following conditions: , This is the feature vector of this mode. It is a multilayer perceptron, consisting of two fully connected layers, used to map feature vectors to scalar scores. This represents the total number of modalities (4 in this system, corresponding to ultrasound, elastography, micro-blood flow, and clinical data). It is a natural exponential function.

[0122] The dynamic weight allocation unit 52 is used to optimize the fusion weights of different modal features based on KL divergence. To further improve the accuracy of weight allocation, this unit introduces a weight optimization mechanism based on KL divergence, with the optimization objective being:

[0123] ,

[0124] in, For KL divergence loss, For the first The probability distribution of individual predictions for each modality feature is a sequence of length . vectors ( The number of categories is 3 in this system, corresponding to three states: no metastasis, metastasis with central region, and metastasis with lateral neck. The probability distribution predicted after fusion is also of length . The vector, KL divergence is used to measure the difference between two probability distributions, and its calculation formula is:

[0125] ,

[0126] in, For the first The first modality prediction The probability of a class To predict the fusion of the first The probability of each class. By minimizing this loss, the system can find the weight combination that maximizes the contribution of each modality feature.

[0127] The subtype-sensitive adjustment unit 53 is used to dynamically adjust feature weights for different pathological subtypes. Different pathological subtypes of thyroid cancer (such as papillary carcinoma and follicular carcinoma) differ in their imaging manifestations, requiring adjustment of the weights of different modalities. This unit introduces prior knowledge of pathological subtypes to design specific weight adjustment strategies for different subtypes. For example, for papillary microcarcinoma (PTMC), ultrasound features are more sensitive to the detection of microcalcifications, thus increasing its weight in calcification detection; for follicular carcinoma, elastography is more effective in distinguishing between benign and malignant cases, thus increasing its contribution to benign / malignant differentiation; for lateral cervical lymph node metastasis, the vascular invasion features displayed by ultra-micro blood flow have higher predictive value, thus enhancing its influence in metastasis analysis.

[0128] The final feature fusion formula is:

[0129] ,

[0130] in, The fused feature vector This is a subtype-sensitive adjustment factor, a parameter dynamically adjusted based on the pathological subtype prediction results, with a value range of [0.5, 1.5]. The weighting coefficients are dynamically allocated. These are the feature vectors for each modality. All feature vectors have been unified in dimension before fusion to ensure dimension matching.

[0131] Through the collaborative work of the three units mentioned above, the feature fusion module 50 can achieve intelligent fusion of multimodal features, which not only preserves the unique information of each modality, but also makes full use of the complementarity between modalities, providing a comprehensive and accurate feature representation for subsequent transition prediction.

[0132] like Figure 6 As shown, the transfer prediction module 60 includes a multi-model integration unit 61, a pathological subtype stratification unit 62, a sample balancing processing unit 63, and a decision threshold optimization unit 64.

[0133] The multi-model ensemble unit 61 integrates three models: ResNet50, EfficientNet-V2, and VisionTransformer. In medical image analysis, different model architectures have different advantages: ResNet50 excels at capturing local texture features, EfficientNet-V2 achieves a good balance between resource efficiency and accuracy, and VisionTransformer is adept at modeling global dependencies. By integrating these three complementary model architectures, the system can combine the strengths of each model to improve prediction accuracy. Model ensemble employs a soft voting mechanism, calculated using the following formula:

[0134] ,

[0135] in, For the ensemble model to sample Prediction Category The probability, For the first The predicted probabilities of the basic models The weight coefficients for this model are determined by the performance of the validation set, and are preferably set to 0.3, 0.4, and 0.3, corresponding to ResNet50, EfficientNet-V2, and VisionTransformer, respectively. The number of basic models (3 in this system).

[0136] The pathological subtype stratification unit 62 is used to implement a three-stage classification strategy, successively distinguishing between benign and malignant nodules, metastatic and non-metastatic nodules, and central and lateral neck metastases. To address the complexity of predicting lymph node metastasis in thyroid cancer, this system employs a stratified classification strategy: the first stage distinguishes between benign and malignant nodules; the second stage further differentiates malignant nodules by the presence or absence of lymph node metastasis; and the third stage subdivides cases with metastasis into central and lateral neck metastases. This stratification strategy aligns with clinical diagnostic thinking, progressively narrowing the predictive range and improving accuracy.

[0137] The sample balancing unit 63 addresses the class imbalance problem. In clinical practice, the number of cases with different metastatic types is often imbalanced; for example, non-metastatic cases far outnumber lateral cervical metastasis cases, leading to insufficient predictive power for the minority class. This unit employs several strategies to handle sample imbalance: using the Synthetic Minority Over-sampling Technique (SMOTE) to synthesize samples for the minority class; applying a weighted sampling strategy during training to increase the sampling frequency of the minority class; and using a focal loss function to enhance learning on hard-to-classify samples. The focal loss function is defined as:

[0138] ,

[0139] in, This represents the model's predicted probability for the correct class, with a value ranging from [0,1]. This represents the category weight coefficient, set according to the category frequency. Minority classes have higher weights. For example, for the three categories of no migration, central region migration, and lateral neck migration, the weights can be set to [0.2, 0.35, ...]. This is the focusing parameter, used to adjust the loss weight ratio between easy and hard samples; it is typically set to 2. It is the natural logarithm function.

[0140] The decision threshold optimization unit 64 is used to adjust the sensitivity and specificity of the model according to clinical needs. In thyroid cancer diagnosis, different clinical scenarios may have different requirements for sensitivity and specificity. For example, in the initial screening stage, more attention may be paid to sensitivity to avoid missed diagnoses; while in the surgical decision-making stage, more attention may be paid to specificity to avoid overtreatment. This unit sets the optimal decision threshold for different application scenarios by analyzing the ROC curve. Preferably, the system provides three preset threshold schemes: a balanced mode (sensitivity and specificity are balanced, and the threshold is usually 0.5), a high-sensitivity mode (the threshold is reduced to 0.3 to increase sensitivity), and a high-specificity mode (the threshold is increased to 0.7 to increase specificity). Doctors can choose the appropriate mode according to the specific case.

[0141] Through the collaborative work of the above four units, the metastasis prediction module 60 can accurately predict the lymph node metastasis status of thyroid cancer, classifying patients into three states: no metastasis, with central metastasis, or with lateral cervical metastasis, providing reliable support for clinical treatment decisions.

[0142] The interpretability analysis module 70 includes a Grad-CAM generation unit 71, a SHAP value calculation unit 72, and a two-dimensional interpretation integration unit 73.

[0143] Grad-CAM generation unit 71 is used to generate heatmaps based on the feature map weights of convolutional layers, displaying the image regions of interest to the model. Grad-CAM (Gradient-weighted Class Activation Mapping) is a classic deep learning visualization technique that can generate intuitive heatmaps to show the image regions that the model focuses on when making decisions. The computation process of this unit is as follows:

[0144] 1. Calculate the gradient of the target category score with respect to the feature map of the last convolutional layer:

[0145] ,

[0146] in, The predicted score for target category c is a scalar value. This is the k-th feature map of the last convolutional layer, with dimension . , express right The gradient, also of dimension 1. .

[0147] 2. Calculate the importance weight for each channel of the feature map:

[0148] ,

[0149] in, The importance weight of feature map channel k to category c is a scalar value. Representation of feature map In position The values ​​at the specified positions, i and j, represent the row and column indices of the feature map, respectively. This represents the summation over all positions in the entire feature map, where Z is the normalization factor, equal to the total number of pixels in the feature map. .

[0150] 3. Generate a weighted feature map and activate it using ReLU:

[0151] ,

[0152] in, This is the Grad-CAM heatmap for category c, with dimensions [missing information]. , ReLU represents the summation over all feature map channels. ReLU is the modified linear unit activation function, defined as follows: This is used to retain only regions with positive contributions.

[0153] 4. Upsample the heatmap to the original image size and overlay it with the original image for display.

[0154] In predicting thyroid cancer metastasis, Grad-CAM heatmaps can visually display key areas of interest to the model, such as microcalcifications and marginal spiculations in ultrasound images, areas of abnormal stiffness in elastography, and areas of abnormal vascular signals in ultra-microvascular flow images. Physicians can use heatmaps to verify whether the model's focus aligns with clinical experience, thus increasing their confidence in the prediction results.

[0155] The SHAP value calculation unit 72 is used to quantify the contribution of each clinical indicator and imaging feature to the prediction result. SHAP (SHapley Additiveex Planations) is a game theory-based feature contribution analysis method that can fairly allocate the contribution of each feature to the prediction result. The calculation process of this unit is based on the KernelSHAP algorithm, and the main steps include:

[0156] 1. Generate feature subset samples (through combinations of feature presence / missing features);

[0157] 2. Make predictions for each subset;

[0158] 3. Solve for the Shapley value using weighted linear regression:

[0159] ,

[0160] in, Features The SHAP value represents the contribution of this feature to the prediction result. For the set of all features, Represents the total number of features. For features not included a subset of Representing a subset The number of features in For using only subsets Model predictions of features in the middle. For using subsets Add features to the middle features The model's predicted values, To combine the weights, ensure fair weighting of all possible combinations of features.

[0161] SHAP analysis revealed key predictive factors for different metastatic types. For example, the non-metastatic group was typically associated with smooth margins (+0.18) and normal TSH levels (+0.15); the central metastatic group was highly associated with an elasticity score >4.2 (+0.32) and BRAF mutation (+0.28); and the lateral neck metastatic group was closely associated with ultra-microflow (VFI) >6.5 (+0.41) and tumor diameter >2 cm (+0.35). These quantitative analysis results provide physicians with precise decision-making support.

[0162] The dual-dimensional interpretation integration unit 73 is used to correlate heatmaps with feature contribution analysis results, providing a comprehensive interpretation. This unit combines the visual interpretation of Grad-CAM with the numerical interpretation of SHAP to construct a complete interpretive framework. For example, for cases predicted to have central metastasis, the system not only displays the microcalcification areas of interest through heatmaps but also quantifies the contribution of elasticity scores and BRAF mutation status to this prediction through SHAP values, enabling physicians to comprehensively understand the predictive basis from both imaging and clinical perspectives.

[0163] Through this two-dimensional interpretability analysis, the system effectively solves the black-box problem of deep learning models, significantly enhancing clinicians' understanding and trust in AI predictions. In clinical validation, after adopting the two-dimensional interpretation, doctors' acceptance of the system's recommendations increased from 40% to 77%, and diagnostic efficiency improved by an average of 24%.

[0164] The results display module 80 includes a probability distribution display unit 81, a class activation graph visualization unit 82, a feature contribution display unit 83, and a diagnostic report generation unit 84.

[0165] The probability distribution display unit 81 is used to display the predicted probabilities of three metastatic states (no metastasis, metastasis with central region, and metastasis with lateral neck). This unit uses intuitive bar charts or pie charts to display the predicted probabilities of each category, enabling doctors to quickly understand the model's prediction confidence. Preferably, while displaying the probabilities, the system also marks high-confidence predictions that exceed a preset threshold (e.g., 0.7), facilitating doctors' rapid identification of reliable predictions.

[0166] The activation map visualization unit 82 is used to display the image regions of interest to the model. This unit overlays the heatmap generated by Grad-CAM onto the original medical image, using a gradient color scale from blue to red to indicate the level of interest, with red areas representing regions of high model interest. To improve clinical usability, the system provides a heatmap transparency adjustment function, allowing doctors to adjust the overlay ratio between the heatmap and the original image as needed.

[0167] Feature Contribution Display Unit 83 is used to display key predictive factors and their contribution. This unit presents the SHAP analysis results in the form of a waterfall chart or bar chart, intuitively showing the positive or negative contribution of each feature to the prediction results. The system sorts and displays features according to their contribution, using red and blue to represent positive and negative contributions respectively, making it easy for doctors to quickly identify key features.

[0168] The diagnostic report generation unit 84 is used to generate structured reports conforming to the TI-RADS classification standards. Based on system analysis results, this unit automatically generates standardized diagnostic reports, including basic patient information, examination images, nodule descriptions, metastasis prediction results, prediction basis, and treatment recommendations. The reports use a standard medical format and support export to PDF or DOCX formats for easy review and archiving by doctors. Preferably, the content of the system-generated reports can be modified and supplemented by doctors, and the final version can be automatically synchronized to the hospital information system.

[0169] The results display module 80 adopts a responsive design, compatible with display devices of different resolutions, ensuring a good user experience on desktop computers, tablets, and even mobile devices. The system interface design follows the human-computer interaction principles of medical software, with simplicity, efficiency, and professionalism as its design philosophy, providing intuitive operation processes and clear results presentation.

[0170] To verify the performance of the system of this invention, a ten-fold cross-validation method was used for evaluation. Specific steps included:

[0171] 1. Randomly divide the dataset into 10 subsets of similar size;

[0172] 2. Each time, select 9 subsets as the training set and the remaining 1 as the test set;

[0173] 3. Repeat the above process 10 times, so that each subset is used as the test set once;

[0174] 4. Calculate the average performance metrics for 10 tests, including accuracy, sensitivity, specificity, and AUC.

[0175] Validation results show that this system achieves excellent performance in predicting lymph node metastasis in thyroid cancer: the overall classification accuracy is 91.6%, the sensitivity is 88.5%, the specificity is 92.3%, and the AUC is 0.944. In particular, for lateral neck metastasis, the system's identification accuracy has increased from 75% of the traditional method to 89.7%, significantly improving the predictive ability for high-risk metastasis types.

[0176] In clinical application validation, the system was prospectively tested on 326 patients. The results showed that the average diagnosis time was shortened by 37%, the detection rate of papillary microcarcinoma (PTMC) was increased by 22.4%, and the diagnostic consistency among doctors was improved by 24%, which fully demonstrates the significant advantages of this system in improving the efficiency and accuracy of thyroid cancer lymph node metastasis diagnosis.

[0177] This invention's system can be widely applied in the diagnostic process of thyroid cancer in medical institutions, suitable for multiple stages such as preoperative assessment, treatment plan development, and postoperative follow-up. The system supports integration with existing hospital information systems, enabling data exchange and workflow optimization.

[0178] During application, the system can be expanded and optimized according to actual needs. For example, for the detection of tiny nodules with a diameter of <3mm, an adaptive ROI cropping algorithm (combined with cv2.HoughCircles detection) can be introduced to improve sensitivity; for complex cases requiring comprehensive evaluation, the 3D ultrasound analysis function can be expanded, and a three-dimensional convolutional network can be built based on PyTorch3D to analyze the rate of change of tumor volume and the spatiotemporal characteristics of blood flow signals; for the optimization of clinical workflow, intelligent measurement tools can be added to achieve automatic annotation of the maximum diameter of the tumor and quantitative analysis of elastography.

[0179] In addition, the system can be expanded into a cloud-based collaboration platform, developing a web interface based on the Flask framework to support multi-center data sharing and remote consultations, providing high-quality diagnostic services for areas with uneven distribution of medical resources.

[0180] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A deep learning-based prediction system for lymph node metastasis in multimodal thyroid cancer, characterized in that, include: The data acquisition module is used to acquire multimodal imaging data, including ultrasound images, elastography images, and ultramicrovascular flow images, as well as patient clinical indicator data. A preprocessing module, connected to the data acquisition module, is used to perform standardization and data enhancement on the multimodal image data; The nodule segmentation module, connected to the preprocessing module, is used to segment the thyroid nodule region of the multimodal image data based on the improved U-Net algorithm; The feature extraction module, connected to the nodule segmentation module, is used to extract features from the ultrasound image, the elastography image, the ultramicro blood flow image, and the patient's clinical index data through a multi-branch deep network. A feature fusion module, connected to the feature extraction module, is used to fuse the multimodal features extracted by the feature extraction module using a dynamic weight fusion algorithm; The metastasis prediction module, connected to the feature fusion module, is used to classify the lymph node metastasis status of patients with thyroid cancer into non-metastasis, with central region metastasis, or with lateral neck metastasis based on the fused multimodal features. An interpretability analysis module, connected to the transition prediction module, is used to generate class activation maps and feature contribution analysis, providing a visual interpretation of the prediction results; The results display module, connected to the transfer prediction module and the interpretability analysis module, is used to display classification results, class activation maps and feature contribution analysis, and generate diagnostic reports.

2. The system according to claim 1, characterized in that, The preprocessing module includes: The image normalization unit is used to uniformly adjust the resolution of the multimodal image data to 224×224 pixels and perform grayscale normalization processing. The data enhancement unit is used to perform spatial transformation, elastic deformation, and contrast adjustment on the multimodal image data; The missing data processing unit is used to fill in the missing items in the patient's clinical indicator data.

3. The system according to claim 1, characterized in that, The nodule segmentation module includes: The encoder contains multi-scale dilated convolutional units to extract multi-scale features and expand the receptive field; The decoder includes a cross-layer attention gating mechanism for dynamically fusing features from different layers; A multi-task learning unit is used to simultaneously output nodule probability masks and microcalcification boundary curves.

4. The system according to claim 1, characterized in that, The feature extraction module includes: The ultrasound image feature extraction unit, based on the EfficientNet-V2 network, is used to extract texture and acoustic features from ultrasound images. The elastic imaging feature extraction unit includes a channel attention mechanism for focusing on regions of abnormal hardness. The ultra-micro blood flow feature extraction unit is used to extract blood vessel distribution density and morphological features; The clinical data feature extraction unit, based on a fully connected network, is used to mine nonlinear correlations between clinical indicators.

5. The system according to claim 1, characterized in that, The feature fusion module includes: Feature importance assessment unit, used to calculate the weight coefficients of each modal feature; A dynamic weight allocation unit is used to optimize the fusion weights of different modal features based on KL divergence. The subtype-sensitive adjustment unit is used to dynamically adjust feature weights for different pathological subtypes.

6. The system according to claim 1, characterized in that, The transfer prediction module includes: The multi-model integration unit integrates three models: ResNet50, EfficientNet-V2, and VisionTransformer. The pathological subtype stratification unit is used to implement a three-stage classification strategy, which distinguishes between benign and malignant, metastatic and non-metastatic, and central metastasis and lateral neck metastasis in sequence. The sample balancing unit is used to solve the class imbalance problem; The decision threshold optimization unit is used to adjust the sensitivity and specificity of the model according to clinical needs.

7. The system according to claim 1, characterized in that, The interpretability analysis module includes: Grad-CAM generation unit is used to generate heatmaps based on the feature map weights of convolutional layers to display the image regions of interest to the model; The SHAP value calculation unit is used to quantify the contribution of each clinical indicator and imaging feature to the prediction results. The dual-dimensional interpretation integration unit is used to correlate heatmaps with feature contribution analysis results, providing a comprehensive interpretation.

8. The system according to claim 1, characterized in that, The results display module includes: The probability distribution display unit is used to show the predicted probabilities of the three transition states; The activation map visualization unit is used to display the image regions that the model focuses on; The feature contribution display unit is used to display key predictive factors and their contributions; The diagnostic report generation unit is used to generate structured reports that conform to the TI-RADS classification standards.

9. The system according to claim 1, characterized in that, Also includes: The system validation module is used to evaluate system performance through 10-fold cross-validation. The clinical feedback module is used to collect feedback from doctors and optimize the system.

10. A deep learning-based prediction method for lymph node metastasis in multimodal thyroid cancer, using the system described in any one of claims 1-9, characterized in that, include: Acquire multimodal imaging data, including ultrasound images, elastography images, and ultramicrovascular flow images, as well as patient clinical indicator data; The multimodal image data is standardized and augmented. The improved U-Net algorithm is used to segment the thyroid nodule region in the multimodal image data; Features were extracted from the ultrasound images, elastography images, ultramicrovascular blood flow images, and patient clinical index data using a multi-branch deep network. The extracted multimodal features are fused using a dynamic weight fusion algorithm; Based on the fused multimodal features, the lymph node metastasis status of patients with thyroid cancer is classified as non-metastasis, with central region metastasis, or with lateral neck metastasis. Generate class activation maps and feature contribution analysis to provide a visual interpretation of the prediction results; Displays classification results, class activation graphs, and feature contribution analysis, and generates a diagnostic report.

Images