Thyroid tumor classification and recurrence risk prediction method based on deep learning

Abstract

The invention discloses a thyroid tumor classification and recurrence risk prediction method based on deep learning, and belongs to the field of medical image processing, and the method comprises the following steps: collecting and preprocessing multi-modal data related to thyroid follicular carcinoma FTC or follicular adenoma FTA; performing feature extraction on the preprocessed multi-modal data to obtain image features and pathological features; respectively adopting different fusion and optimization methods for the combination of the image features and the pathological features and the image features to obtain a first optimized fusion feature and a second optimized fusion feature; and inputting the first optimized fusion feature into a preoperative classification model to obtain a thyroid tumor type classification probability, and inputting the second optimized fusion feature and the pathological feature into a postoperative recurrence risk prediction model to obtain a recurrence risk score. By means of the multi-mode image classification model and the recurrence risk prediction model which are specially constructed, the sensitivity and specificity of preoperative classification are improved, and accurate individualized management is provided for postoperative patients.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and more specifically, to a method for thyroid tumor classification and recurrence risk prediction based on deep learning. Background Art

[0002] Follicular thyroid carcinoma (FTC) is a highly aggressive and clinically insidious subtype of thyroid cancer. Traditional diagnostic imaging methods, such as ultrasound and enhanced CT, are limited by low specificity and an inability to effectively capture dynamic microvascular characteristics, making early diagnosis of FTC difficult. Existing pathological diagnostic methods, such as fine needle aspiration (FNA), also face inaccuracies, particularly in differentiating FTC from follicular adenoma (FTA). The TNM staging system, which describes the primary tumor (T), regional lymph nodes (N), and distant metastasis (M) of a malignant tumor, is widely used for prognostic assessment. However, it is primarily based on the anatomical characteristics of the tumor. Approximately 20% of FTC patients already have occult metastases at initial diagnosis. However, TNM staging relies on imaging detection and lacks accurate prediction of recurrence risk.

[0003] Existing methods for classifying thyroid tumor types and predicting postoperative recurrence risk mainly include the following: (1) Deep learning diagnostic models based on single-modality images. Some studies have developed deep learning models based on ultrasound images to classify benign and malignant thyroid nodules using convolutional neural networks. However, this type of method only uses single-modality ultrasound images and cannot comprehensively analyze the multidimensional characteristics of the tumor. In addition, enhanced CT images have also been used to develop risk prediction models, but due to the lack of dynamic microblood flow in tumors, the diagnostic performance is limited. (2) Models based on simple fusion of multimodality images. Some studies have begun to try to fuse contrast-enhanced ultrasound (CEUS) and enhanced CT image features in malignant tumors such as the liver, and use simple feature splicing methods to construct multimodality models. However, the fusion methods in these studies have failed to fully explore the complementarity of different modality features, and it is difficult to dynamically adjust the focus of the model according to the importance of the features, resulting in limited improvement in diagnostic performance. (3) Separate analysis of imaging and pathological data. Most studies on FTC prognosis only analyze postoperative pathological data and fail to establish an effective connection between preoperative imaging and postoperative pathological data. The separate use of preoperative imaging features and postoperative pathological features limits the depth and breadth of joint analysis. In addition, existing methods for classifying thyroid tumor types and predicting postoperative recurrence risk lack interpretability support, and the application of deep learning models in medical image analysis suffers from a "black box" problem, which means that the model's decision-making process lacks transparency for doctors. Although these models can make accurate diagnoses or risk predictions through training with large amounts of data, doctors often find it difficult to understand why the model makes a specific decision. Especially in complex clinical cases, doctors tend to rely more on their own expertise rather than relying entirely on the model's predictions, making it difficult for the model to truly play a supporting role in clinical decision-making. Summary of the Invention

[0004] In the existing technology, the thyroid tumor classification model based on deep learning only uses single-modal ultrasound images and cannot comprehensively analyze the multidimensional characteristics of the tumor, or uses a simple feature splicing method to construct a multimodal model, failing to fully explore the complementarity of different modal features, resulting in limited improvement in diagnostic performance; the existing postoperative recurrence risk prediction model only analyzes postoperative pathological data and fails to establish an effective connection between preoperative images and postoperative pathological data. In response to the above technical problems, the present invention proposes a thyroid tumor classification and recurrence risk prediction method based on deep learning.

[0005] In one aspect of the present invention, a method for thyroid tumor classification and recurrence risk prediction based on deep learning is proposed, comprising the following steps: S1, collecting and preprocessing multimodal data related to thyroid follicular carcinoma FTC or follicular adenoma FTA, wherein the multimodal data includes dynamic microblood flow data, anatomical data and whole-slice pathological image WSI data; S2, extracting features from the preprocessed dynamic microblood flow data, anatomical data and whole-slice pathological image WSI data using different feature extraction methods, respectively, to obtain dynamic microblood flow features F_CEUS, anatomical features F_CT and pathological features F_path; S3, extracting dynamic microblood flow features F_CEUS, anatomical features F_CT and pathological features F_path The feature F_CEUS, the anatomical feature F_CT and the pathological feature F_path are fused and optimized to obtain the first optimized fusion feature F_fused_optimized; the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT are fused and optimized to obtain the second optimized fusion feature F_temporal; S4, the first optimized fusion feature F_fused_optimized is input into the preoperative classification model to obtain the thyroid tumor type classification probability, and the second optimized fusion feature F_temporal and the pathological feature F_path are input into the postoperative recurrence risk prediction model to obtain the recurrence risk score.

[0006] More specifically, in the above aspects, the acquisition of multimodal data in step S1 includes: obtaining dynamic microblood flow data through contrast-enhanced ultrasound CEUS using a time series acquisition technology; obtaining anatomical data through enhanced CT using multi-phase scanning and iterative reconstruction algorithm; obtaining whole-slice pathological image WSI data through a digital scanner; the preprocessing of multimodal data in step S1 includes: applying adaptive median filtering and wavelet transform to the dynamic microblood flow data to obtain first dynamic microblood flow data, and applying non-local mean denoising to the anatomical data to obtain first anatomical data; applying a mutual information maximization algorithm to the first dynamic microblood flow data and the first anatomical data to obtain sub-pixel spatially aligned dynamic microblood flow data and anatomical data; applying a pyramid blocking strategy to the whole-slice pathological image WSI data to extract pixel blocks of the whole-slice pathological image WSI at different magnifications, and applying multi-scale feature fusion to the pixel blocks to obtain fused whole-slice pathological image WSI data.

[0007] More specifically, in the above aspect, the feature extraction of the preprocessed multimodal data in step S2 includes: using the CNN-GRU hybrid architecture to perform spatial feature extraction and temporal feature modeling on the preprocessed dynamic microblood flow data to obtain the dynamic microblood flow feature F_CEUS associated with the dynamic microblood flow data; using the pre-trained ResNet50 to perform multi-level feature extraction on the preprocessed anatomical data, and fusing the multi-level scale features through the pyramid pooling module PPM to obtain the anatomical feature F_CT associated with the anatomical data; dividing the preprocessed full-slice pathology image WSI into N blocks, using the InceptionResNetV2 model to process each block to obtain the corresponding instance feature, and then using the attention weight network to calculate the attention weight of each block, and according to the calculated attention weight, weighted aggregation of all instance features to obtain the pathological feature F_path associated with the full-slice pathology image WSI data.

[0008] More specifically, in the above aspect, in step S3, the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT and the pathological feature F_path are fused and optimized to obtain a first optimized fused feature F_fused_optimized, including: for the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT and the pathological feature F_path, a multi-head self-attention mechanism is used to generate a first initial weight vector ; Combined with feature quality evaluation and task relevance analysis, the first initial weight vector Perform adaptive adjustment to obtain the first initial weight vector after the first adjustment ; Use the task-aware dynamic weight adjustment mechanism to adjust the first initial weight vector after the first adjustment Perform dynamic adjustment to obtain the first initial weight vector after the second adjustment ; The first initial weight vector after the second adjustment Normalization is performed and task-sensitive adaptive adjustment is adopted to obtain the first weight vector [α, β, γ], where α+β+γ=1. Based on the first weight vector [α, β, γ], the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT, and the pathological feature F_path are weightedly fused to obtain the first fused feature F_fused, as shown in the following formula: F_fused=α·F_CEUS+β·F_CT+γ·F_path The first fused feature F_fused is then optimized through the Transformer encoder to obtain the first optimized fused feature F_fused_optimized.

[0009] More specifically, in the above aspect, in step S3, the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT are fused and optimized to obtain a second optimized fusion feature F_temporal, including: for the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT, a multi-head attention mechanism is used to calculate the feature similarity matrix, and a second initial weight vector is generated. ; Use task-aware dynamic weight adjustment mechanism to adjust the second initial weight vector Adjust to get the adjusted second initial weight vector ; The adjusted second initial weight vector Normalize and obtain the second weight vector [α', β'], where α'+β' = 1. Based on the second weight vector [α', β'], perform weighted fusion of the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT to obtain the second fused feature F_fused', as shown in the following formula: F_fused'=α'·F_CEUS+β'·F_CT; Introduce residual connections to obtain enhanced features F_enhanced: F_enhanced = F_fused' + Dropout(FC(F_fused')) Among them, FC is the fully connected layer, Dropout is the random deactivation operation, and FC is the fully connected layer; Apply layer normalization to the enhanced feature F_enhanced to obtain the normalized enhanced feature F_normalized: F_normalized = LayerNorm(F_enhanced) The normalized enhanced feature F_normalized is input into the attention-enhanced GRU network to obtain the second optimized fusion feature F_temporal: F_temporal = GRU(F_normalized).

[0010] More specifically, in the above aspects, the preoperative classification model in step S4 is constructed in the following manner: a two-branch fully connected network structure is used to construct an initial preoperative classification model; multimodal data related to thyroid follicular carcinoma FTC or follicular adenoma FTA is collected as a data set, and stratified sampling is used to divide the data set into a first training set, a first validation set and a first test set according to a predetermined ratio; the first training set is used to train the initial preoperative classification model, and the first validation set is used to evaluate the performance of the initial preoperative classification model during the training process, and a multi-stage transfer learning strategy, a cross-entropy loss function and an L2 regularization term, a dynamic fine-tuning mechanism based on the characteristics of the CEUS device and the CT device, and an adaptive learning rate adjustment are used to adjust the parameters of the initial preoperative classification model. After the training is completed, the preoperative classification model is obtained, and the performance of the preoperative classification model is evaluated using the first test set.

[0011] More specifically, in the above aspects, the postoperative recurrence risk prediction model in S4 is constructed in the following manner: a multimodal contrastive learning framework is used to construct an initial postoperative recurrence risk prediction model; multimodal data related to thyroid follicular carcinoma (FTC) and follicular adenoma (FTA) are collected as a data set, and stratified sampling is used to divide the data set into a second training set, a second validation set, and a second test set according to a predetermined proportion; the second training set is used to train the initial postoperative recurrence risk prediction model, and the second validation set is used to evaluate the performance of the initial postoperative recurrence risk prediction model, and domain-invariant feature learning based on adversarial training and an adaptive early stopping mechanism based on validation set performance are used to adjust the parameters of the initial postoperative recurrence risk prediction model. After the training is completed, the postoperative recurrence risk prediction model is obtained, and the performance of the postoperative recurrence risk prediction model is evaluated using the first test set.

[0012] In a further embodiment of the above embodiment, a dual-branch contrastive learning network and a risk scoring mechanism are further employed to enhance the performance of the postoperative recurrence risk prediction model.

[0013] In the above aspects, in a further embodiment, based on the thyroid tumor type classification probability and recurrence risk score, Grad-CAM optimization is used to generate heat maps to improve the clarity and reliability of the visualization results, and SHAP feature contribution analysis is used to quantify the contribution of various features to the prognostic results.

[0014] In another aspect of the present invention, an electronic device is proposed, which includes a processor and a memory, and is characterized in that a computer program is stored in the memory, and when the computer program is executed by the processor, the processor will execute the deep learning-based thyroid tumor classification and recurrence risk prediction method of the above-mentioned aspect of the present invention.

[0015] The beneficial effects of the present invention are: 1. This invention does not only utilize imaging data of a single modality, but integrates data from three modalities: dynamic microblood flow data (CEUS), anatomical data (CT), and whole-slice pathology images (WSI), providing more comprehensive tumor characteristic information.

[0016] 2. For data of different modalities, the present invention adopts specific feature extraction methods: using a CNN-GRU hybrid architecture to extract dynamic microblood flow features of CEUS, using a pre-trained ResNet50 and a pyramid pooling module (PPM) to extract multi-level and multi-scale anatomical features of CT, and using InceptionResNetV2 and an attention weight network to extract pathological features of WSI.

[0017] 3. This paper proposes a self-attention-based dynamic weighted fusion mechanism for the fusion of imaging data (CEUS and CT) and pathology data. This mechanism adaptively adjusts the weights of each modality's features by combining feature quality assessment and task relevance analysis. Furthermore, the Transformer encoder is used to globally optimize the deep self-attention network, further enhancing the expressive power of the fused features and addressing the inability of traditional simple splicing and fusion approaches to exploit modal complementarity.

[0018] 4. For the fusion of imaging data (CEUS and CT), the present invention adopts an attention mechanism based on the feature similarity matrix to generate weights, combined with a GRU network with residual connections, layer normalization, and attention enhancement, to strengthen the interaction of spatiotemporal features, capture the synergistic effect of CEUS temporal evolution and CT spatial features, and improve the efficiency of feature fusion between imaging modalities.

[0019] 5. This invention constructs separate models for preoperative classification and postoperative recurrence risk prediction. The preoperative classification model utilizes a dual-branch fully connected network structure, combined with a multi-stage transfer learning strategy and dynamic fine-tuning mechanism to improve the model's adaptability across different device environments. Model parameters are dynamically adjusted based on device characteristics (resolution, frame rate, etc.), and a cyclic decay learning rate strategy is used to prevent training from becoming trapped in local optima, thereby improving convergence speed and cross-device performance stability. The postoperative recurrence risk prediction model, based on a multimodal contrastive learning framework, introduces a dual-branch contrastive learning network. This forces the imaging and pathological features of similar samples (recurrence / non-recurrence) to be close in feature space, while dissimilar samples are separated. This improves feature discriminability and modality semantic alignment, resulting in a 7-13% increase in the recurrence risk area under the curve (AUC). A weighted hybrid loss (binary cross entropy + contrastive loss) is used to output a continuous risk score. Combined with domain-invariant feature learning based on adversarial training, this reduces the impact of device differences on model performance, improving cross-device performance retention to 85-92%.

[0020] 6. The present invention adopts adaptive learning rate adjustment and early stopping strategy, which effectively avoids the overfitting problem in the model construction process, while reducing unnecessary training time and improving the training efficiency and performance of the model.

[0021] 7. The present invention uses Grad-CAM to optimize and generate heat maps, superimposes time dimension weights on CEUS heat maps, accurately locates areas of abnormal arterial perfusion, and combines CT and pathological features to achieve multimodal fusion visualization, thereby improving the transparency of the model decision-making process and helping doctors understand key diagnostic areas. Through SHAP feature contribution analysis, the contribution of each modality feature (such as CEUS peak flow velocity, CT boundary ambiguity, and pathological nuclear division count) to the prognostic results is quantified, and a downloadable explanatory report is generated to solve the "black box" problem of deep learning models and enhance clinicians' trust in model predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention and make other features, objects and advantages of the present invention more apparent. The accompanying drawings and descriptions of the exemplary embodiments of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0023] In the attached figure: Figure 1 This is a flowchart of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0024] Figure 2 This is a data collection and preprocessing flowchart of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0025] Figure 3 This is a feature extraction block diagram of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0026] Figure 4 This is a schematic diagram of the trimodal feature fusion and optimization principle of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0027] Figure 5 This is a schematic diagram of the dual-modal feature fusion and optimization principle of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0028] Figure 6 This is a classification and prediction flow chart of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0029] Figure 7 This is a flowchart of the interpretable analysis of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention. DETAILED DESCRIPTION

[0030] See Figure 1 , Figure 1 This is a flow chart of the method for thyroid tumor classification and recurrence risk prediction based on deep learning of the present invention. The method for thyroid tumor classification and recurrence risk prediction based on deep learning of the present invention includes: S1, collecting and preprocessing multimodal data related to thyroid follicular carcinoma FTC or follicular adenoma FTA, wherein the multimodal data includes dynamic microblood flow data, anatomical data and whole-slice pathological image WSI data; S2, using different feature extraction methods to extract features from the preprocessed dynamic microblood flow data, anatomical data and WSI data, respectively, to obtain dynamic microblood flow features F_CEUS, anatomical features F_CT and pathological features F_path; S3, using a multi-head self-attention mechanism to generate weight vectors for the dynamic microblood flow features F_CEUS, anatomical features F_CT and pathological features F_path, and using an adaptive weight adjustment mechanism that combines feature quality assessment with task correlation analysis to perform weighted fusion. , and optimized using the Transformer encoder to obtain the first optimized fusion feature F_fused_optimized; for the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT, the attention mechanism based on the inter-modal correlation matrix is ​​used to generate the weight vector and the task-aware dynamic weight adjustment mechanism is adopted for weighted fusion, and the residual connection, layer normalization and attention-enhanced GRU network are used for optimization to obtain the second optimized fusion feature F_temporal; S4, the first optimized fusion feature F_fused_optimized is input into the preoperative classification model to obtain the thyroid tumor type classification probability, and the second optimized fusion feature F_temporal and the pathological feature F_path are input into the postoperative recurrence risk prediction model to obtain the recurrence risk score.

[0031] The preprocessing, feature extraction, fusion and optimization, model training and construction, and interpretability enhancement techniques in the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention are all implemented in Python. In this specification, the dynamic microblood flow feature F_CEUS can also be referred to as a CEUS feature, and the anatomical feature F_CT can also be referred to as a CT feature.

[0032] See Figure 2 , which is a data collection and preprocessing flow chart of the present invention. It mainly includes the following steps: 1. Acquire CEUS data: Obtain CEUS time series images to capture the dynamic characteristics of the arterial and venous phases.

[0033] 2. Acquire CT data: Collect three-phase enhanced CT images, including arterial phase, venous phase, and delayed phase.

[0034] 3. Collect pathology slide data: Obtain H&E-stained full-slide images of pathology slides and perform image segmentation.

[0035] 4. Data cleaning: Remove invalid blank areas in the image and fill in missing or incomplete data.

[0036] 5. Format conversion: Save CEUS data in standard DICOM format and convert pathology slides into JPEG format to facilitate deep learning processing.

[0037] 6. Standardization: Standardize brightness and contrast, and unify resolution, for example, 640×480 for CEUS, 256×256 for CT, and 256×256 for pathology sections.

[0038] 7. Spatial Registration: Use Elastix to spatially align CT and CEUS images and register multimodal data to ensure image consistency in spatial position.

[0039] 8. Complete preprocessing: Output standardized and registered multimodal data.

[0040] More specifically, in one embodiment of the present invention, regarding the data collection part, the following standardized collection process is proposed based on the multimodal characteristics of FTC: (1) Capture of dynamic microblood flow characteristics (CEUS): Using time series acquisition technology, the arterial phase (30-60 seconds) and the venous phase (60-120 seconds) are accurately divided. The spatiotemporal dynamic characteristics of tumor microblood flow perfusion are captured through a high-frequency ultrasound probe (frequency ≥12MHz). The data are stored in DICOM format (resolution 640×480, frame rate 15fps).

[0041] (2) High-resolution anatomical feature extraction (enhanced CT): Multi-phase scanning (arterial phase, venous phase, delayed phase) with a layer thickness of 1.0-1.5 mm is used, combined with an iterative reconstruction algorithm to reduce noise and ensure the fine depiction of the three-dimensional anatomical structure.

[0042] (3) Digital processing of pathological data: Whole-slice pathological images (WSIs) were digitized using a digital scanner (resolution 0.25 μm / pixel) and processed in blocks (256 × 256 pixels) combined with a tissue region screening algorithm (based on Otsu threshold segmentation) to eliminate interference from blank areas.

[0043] (4) Data storage and cross-platform compatibility: Build a standardized data warehouse to uniformly map CEUS (DICOM), CT (DICOM), and pathology images (JPEG) to the same spatial coordinate system, supporting lossless transmission of multi-center data and cross-device analysis.

[0044] Through the above standardized acquisition process, the standardized collection of CEUS, CT and pathology data can be ensured, and the dynamic microblood flow characteristics, high-resolution anatomical characteristics and pathological characteristics can be accurately obtained. By building a standardized data warehouse, it supports the unified storage and cross-platform compatibility of multimodal data, and realizes the lossless transmission and analysis of multi-center data.

[0045] In one embodiment of the present invention, the following operations are used for data preprocessing: (1) Spatial registration and dynamic alignment: The Elastix registration framework (based on B-spline elastic transformation) is used to achieve sub-pixel spatial alignment of CEUS and enhanced CT images through the mutual information maximization algorithm, eliminating artifacts caused by respiratory motion and device offset.

[0046] (2) Multi-scale optimization of pathological data: For WSI data, a pyramid block strategy is proposed: 256×256 pixel blocks are extracted at 5×, 10×, and 20× magnifications, respectively. These pixel blocks are input into the multi-scale feature fusion module of InceptionResNetV2. The output is a 512-dimensional feature vector containing cross-scale information of cell morphology and tissue structure, realizing cross-scale representation of cell morphology and tissue structure.

[0047] (3) Noise suppression and enhancement processing: Adaptive median filtering (window size 3×3) combined with wavelet transform (Daubechies basis function) is used for CEUS data to effectively separate blood flow signals from background noise; non-local mean denoising (NL-Means) is used for CT data to retain anatomical details while suppressing metal artifacts.

[0048] Through the above data preprocessing operations, sub-pixel spatial alignment of CEUS and CT images is achieved, eliminating artifacts caused by respiratory motion and device offset; multi-scale optimization of pathological data is performed to extract cross-scale representations of cell morphology and tissue structure; through noise suppression and enhancement processing, CEUS blood flow signals and background noise are effectively separated, while retaining CT anatomical details.

[0049] See Figure 3 , which is a feature extraction block diagram of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention. It mainly includes the following parts: 1. CEUS dynamic microblood flow feature extraction: A convolutional neural network (CNN) is used to extract the spatial features of a single CEUS image. A gated recurrent unit (GRU) is used to capture time series features and generate a microblood flow feature vector (e.g., 128 dimensions).

[0050] 2. CT static anatomical feature extraction: ResNet50 is used to extract anatomical features such as tumor boundary fuzziness and capsule invasion. A pyramid pooling module (PPM) is used to enhance the combination of global and local features to generate a CT feature vector (e.g., 256 dimensions).

[0051] 3. Pathology Feature Extraction: Full-slide pathology images are segmented into small tiles and cleaned. Cell morphology features are extracted using InceptionResNetV2. Based on the Multiple Instance Learning (MIL) framework, the features of the small tiles are weighted and aggregated to generate a pathology feature vector (e.g., 512 dimensions).

[0052] More specifically, the present invention extracts dynamic microblood flow characteristics, anatomical characteristics, and pathological characteristics through the following operations: 1. Extraction of dynamic micro-blood flow characteristics F_CEUS of CEUS Adopting CNN-GRU hybrid architecture design, including: Spatial feature extraction: A three-layer convolutional network (Conv3×3, ReLU, BatchNorm) was constructed to output a 128-dimensional feature vector for each CEUS image frame. Temporal feature modeling: A bidirectional GRU (hidden layer with 64 units) is used to capture the hemodynamic changes from the arterial phase to the venous phase and output dynamic features (e.g., 128 dimensions).

[0053] CNN spatial feature extraction mathematical expression: F_spatial^i = CNN(I_i) (Where I_i represents the i-th CEUS image, and F_spatial^i represents the extracted 128-dimensional spatial feature vector.) GRU time series feature modeling: h_t = GRU(F_spatial^t, h_{t-1}) r_t = σ(W_r·[h_{t-1}, F_spatial^t]) z_t = σ(W_z·[h_{t-1}, F_spatial^t]) ñ_t = tanh(W·[r_t * h_{t-1}, F_spatial^t]) h_t = (1 - z_t) * h_{t-1} + z_t * ñ_t Where h_t is the hidden state at time t, r_t is the reset gate, z_t is the update gate, σ is the sigmoid activation function, and the final dynamic feature F_CEUS = h_T is obtained. T is the last time step of the sequence. ñ_t is the candidate hidden state in the GRU, which represents the new candidate state calculated at time step t based on the current input and the historical information after reset. W is the weight matrix for calculating the candidate hidden state in the GRU, which is used to map the historical state controlled by the reset gate and the current input to the candidate state space.

[0054] In summary, the CNN-GRU hybrid architecture is used to capture the hemodynamic changes from the arterial phase to the venous phase, and output 128-dimensional dynamic microblood flow features.

[0055] 2. Extraction of anatomical features of enhanced CT F_CT ResNet50-PPM joint optimization is adopted to extract multi-level features (Conv1 to Conv5) based on pre-trained ResNet50, and 4-level scale features (1×1, 2×2, 3×3, 6×6 pooling) are fused through the pyramid pooling module (PPM) to output 256-dimensional global-local joint features. Since the data scale of enhanced CT is relatively small, if a randomly initialized ResNet50 is directly used to extract features, it will perform extremely poorly in small data scenarios due to problems such as parameter redundancy, inefficient feature learning, and difficulty in optimization. Therefore, the present invention uses pre-trained ResNet50 to transfer the general feature representation learned in big data to a small data set through transfer learning, significantly reducing the model's dependence on the target data. Among them, ResNet50 feature extraction: F_conv_i = ResNet50_layer_i(I_CT), i ∈ {1,2,3,4,5} Among them, I_CT represents the CT input image, and F_conv_i represents the feature map of the convolution output of the i-th layer of ResNet50.

[0056] Pyramid Pooling Module (PPM): F_pool_j = Pool_j(F_conv_5), j ∈ {1×1, 2×2, 3×3, 6×6} F_ppm_j = Conv(Upsample(F_pool_j)) F_CT = Concat(F_ppm_1, F_ppm_2, F_ppm_3, F_ppm_4) Among them, F_pool_j is the multi-scale feature map obtained after the 5th layer feature map of ResNet50 is pooled at different scales (1×1, 2×2, 3×3, 6×6); F_ppm_j is the feature map of the pooled feature F_pool_j restored to its original size after upsampling and convolution, which is used for subsequent splicing to form the final 256-dimensional CT feature vector; F_CT is a CT feature vector with a dimension of 256, which contains multi-scale feature information.

[0057] In summary, based on the ResNet50-PPM joint optimization, multi-scale anatomical features are extracted and 256-dimensional global-local joint features are output.

[0058] 3. Extraction of pathological features F_path Weakly supervised feature learning (MIL) for pathology images was employed. Using multiple instance learning and an attention mechanism, the WSI was segmented into N tiles (256×256). 512-dimensional features were extracted using InceptionResNetV2, and an instance-level feature pool was constructed. An attention weighting network (fully connected layers + sigmoids) was designed to dynamically select key tiles related to FTC (such as vascular invasion areas) and output aggregated 512-dimensional pathology features.

[0059] Instance-level feature extraction: f_i = InceptionResNetV2(p_i), i ∈ {1,2,...,N} Among them, p_i represents the i-th pathological patch, and fi_i represents the corresponding 512-dimensional instance feature.

[0060] Attention weight calculation: w_i = sigmoid(W·f_i + b) Where W and b are the learnable weight matrix and bias, and w_i∈[0,1] represents the attention weight of the i-th patch.

[0061] Weighted aggregation: F_path = Σ(w_i·f_i) / Σ(w_i) Among them, F_path is the final 512-dimensional pathological feature, which represents the global representation of the pathological image.

[0062] In summary, through multi-instance learning and attention mechanism, key pathological patches related to FTC are screened and 512-dimensional pathological features are output.

[0063] See Figure 4 , Figure 4This is a schematic diagram of the trimodal feature fusion and optimization principle of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention. It mainly includes the following parts: 1. Adaptive weighting mechanism: Dynamically calculate the weights of the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT, and the pathological feature F_path, and adjust the contribution ratio of each modality feature according to task requirements.

[0064] 2. Deep Self-Attention Network: Captures global dependencies between modalities through a multi-head (e.g., 8-head) attention mechanism to optimize the fusion feature expression.

[0065] 3. Inter-modal feature optimization: Through feature interaction and redundancy removal, the performance of fused features is further improved.

[0066] 4. Fusion feature output: Output high-dimensional optimized fusion features for subsequent classification and prediction.

[0067] Specifically, the adaptive fusion and optimization of the trimodal features of the present invention includes the following operations: 1. Dynamic weighted fusion based on self-attention 1.1 Using the weight distribution algorithm, we input CEUS (128-dimensional), CT (256-dimensional), and pathology (512-dimensional) features, and calculate the inter-modality correlation matrix through the multi-head self-attention mechanism (number of heads = 8) to generate the first initial weight vector .

[0068] First we compute the query (Q), key (K), and value (V) matrices: Q = W_Q·F_concat K = W_K·F_concat V = W_V·F_concat Among them, F_concat = [F_CEUS, F_CT, F_path] is the concatenated feature vector.

[0069] Then calculate the attention score: Attention(Q, K, V) = softmax(QK^T / √d_k)V Among them, d_k is the dimension of the key vector in the attention mechanism, which is used to scale the attention scores to prevent gradient disappearance.

[0070] Then perform multi-head attention splicing (number of heads = 8): MultiHead(F_concat) = Concat(head_1, head_2, ..., head_8)·W_O head_i = Attention(Q_i, K_i, V_i) Among them, W_O is the output weight matrix of multi-head attention, which is used to map the spliced ​​multi-head attention results to the final output feature space.

[0071] Then generate the first initial weight vector from the multi-head attention result: = softmax(W_modal·MultiHead(F_concat)) Among them, W_modal is the modal weight generation matrix, which converts the output of multi-head attention into the weight coefficients of the three modalities .

[0072] 1.2 Adaptive adjustment of the first initial weight vector based on feature quality evaluation and task relevance analysis: Dynamically adjust the first initial weight vector based on feature quality evaluation and task relevance analysis : 1.2.1 Calculate the quality assessment score: quality_CEUS = w_q·Quality(F_CEUS) + w_r·Relevance(F_CEUS,task) quality_CT = w_q·Quality(F_CT) + w_r·Relevance(F_CT,task) quality_path = w_q·Quality(F_path) + w_r·Relevance(F_path,task) Among them, quality_CEUS is the quality score of CEUS features, which is calculated by a weighted combination of feature quality and task relevance; w_q is the quality weight coefficient; the Quality function evaluates the quality of the current feature; w_r is the relevance weight coefficient; the Relevance function evaluates the relevance of the feature to the current task; quality_CT is the quality score of CT features; quality_path is the quality score of pathology features 1.2.2 Normalized quality score: [q_α, q_β, q_γ] = softmax([quality_CEUS, quality_CT, quality_path]) where q_α, q_β, and q_γ are the quality scores of the normalized CEUS features, the quality scores of the normalized CT features, and the quality scores of the normalized pathological features, respectively.

[0073] 1.2.3 Adjust based on initial weights: α1_temp = λ·α0+ (1-λ)·q_α β1_temp = λ·β0+ (1-λ)·q_β γ1_temp = λ·γ0+ (1-λ)·q_γ Among them, λ = 0.7 is the initial weight retention coefficient.

[0074] 1.2.4 Renormalization: = softmax([α1_temp, β1_temp, γ1_temp]) in, is the first initial weight vector after the first adjustment.

[0075] 2. Task-aware weight optimization: L_weight = L_task + λ·R(α1, β1, γ1) Where L_task is the task loss (classification or regression) and R is the weight regularization term.

[0076] 2.1 Obtain the first initial weight vector after the second adjustment through gradient-based task adaptive update : α2= α1- η_α· L_weight / α1 β2 = β1 - η_β· L_weight / β1 γ2= γ1- η_γ· L_weight / γ1 Among them, η_α is the learning rate corresponding to the weight α1, which is used to control the update step size of the weight parameter α1; η_β is the learning rate corresponding to the weight β1, which is used to control the update step size of the weight parameter β1; η_γ is the learning rate corresponding to the weight γ1, which is used to control the update step size of the weight parameter γ1.

[0077] Finally, the first weight vector [α, β, γ] = softmax( ).

[0078] 2.2 Task-sensitive adaptive adjustment: When the diagnosis task is prioritized: increasing η_α will increase the weight of CEUS features. When the prediction task is prioritized: increasing η_γ increases the weight of pathological features Monitor the validation set performance in real time and dynamically adjust the final weight vector In practice, α≈0.4 is usually larger for diagnostic tasks (favoring CEUS), while γ≈0.5 is larger for recurrence prediction tasks (favoring pathological features). The weights are automatically adapted to the equipment characteristics and data quality of different institutions.

[0079] Finally, the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT and the pathological feature F_path are fused through the first weight vector to obtain the first fused feature F_fused: F_fused=α·F_CEUS+β·F_CT+γ·F_path, (α+β+γ=1) In summary, through the multi-head self-attention mechanism, the weights of CEUS, CT and pathological features are dynamically allocated to achieve accurate fusion of cross-modal features.

[0080] 2. Global Optimization of Deep Self-Attention Networks Using Transformer-based feature interaction, the fused features (512 dimensions) are fed into a 6-layer Transformer encoder (8 heads, 2048 feedforward layer dimensions), capturing cross-modal global dependencies through positional encoding and layer normalization. The specific process is as follows: X' = ​​LayerNorm(X + MultiHeadAttention(X)) X'' = LayerNorm(X' + FFN(X')) Among them, MultiHeadAttention(X) = Concat(head1,...,head8)W^O head_i = Attention(XW_i^Q, XW_i^K, XW_i^V) Attention(Q,K,V) = softmax(QK^T / √d_k)V Among them, X' is the intermediate feature representation after multi-head self-attention and residual connection and then layer normalization; X'' is the final output feature representation after feedforward network FFN processing and the second residual connection and layer normalization; W^O is the output projection matrix of the multi-head attention mechanism, which is used to map the spliced ​​multi-head attention results to the final output dimension; XW_i^Q is the query vector Q_i obtained by transforming the input feature X through the query weight matrix W_i^Q of the i-th head; XW_i^K is the key vector K_i obtained by transforming the input feature X through the key weight matrix W_i^K of the i-th head; XW_i^V is the value vector V_i obtained by transforming the input feature X through the value weight matrix W_i^V of the i-th head.

[0081] Positional encoding: PE(pos, 2i) = sin(pos / 10000^(2i / d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model)) Transformer encoder layer: x' = LayerNorm(x + MultiHeadAttention(x, x, x)) output = LayerNorm(x' + FFN(x')) Where FFN is a feed-forward network: FFN(x) = max(0, xW_1 + b_1)W_2 + b_2 By stacking 6 layers of Transformer encoders, the final feature representation is: F_fused_optimized = Transformer_6(Transformer_5(...Transformer_1(F_fused)...)) In summary, the Transformer encoder captures cross-modal global dependencies and outputs 512-dimensional optimized features.

[0082] See Figure 5 , Figure 5This is a schematic diagram of the bimodal feature fusion and optimization principles for the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention. It is worth noting that in trimodal fusion, due to the significant semantic differences between the modalities, a Transformer encoder and positional encoding are used to establish global contextual relationships and conduct deep cross-modal feature interaction. Bimodal fusion primarily processes relatively homogeneous imaging data, CEUS and CT, and focuses more on the temporal evolution of CEUS, thus employing residual connections and GRU networks to capture temporal dependencies.

[0083] Specifically, the dual-modal adaptive fusion and optimization of the present invention includes the following operations: 1. Attention-based weight distribution Input CEUS (128 dimensions) and CT (256 dimensions) features, calculate the feature similarity matrix similarity_matrix, and thus generate the second initial weight vector .

[0084] The specific process is as follows: Calculate the cross-modal similarity score cross_similarity: similarity_matrix = F_CEUS·F_CT^T / √d cross_similarity = mean(softmax(similarity_matrix)) Among them, F_CEUS·F_CT^T represents the feature dot product operation, d is the feature dimension, and softmax can normalize the similarity into a probability distribution Modal autocorrelation calculation: self_sim_CEUS = mean(F_CEUS·F_CEUS^T / √128) self_sim_CT = mean(F_CT·F_CT^T / √256) Weighted raw score calculation: w_CEUS_raw = cross_similarity * self_sim_CEUS w_CT_raw = (1 - cross_similarity) * self_sim_CT Normalize the weight vector: = softmax([w_CEUS_raw, w_CT_raw]) 2. Dynamic weight adjustment of task perception Weight optimization is based on gradient descent and task loss: L_weight' = L_task' + λ'·R' (α0', β0') Where L_task' is the main task loss function, λ' is the regularization coefficient, and R' (α0', β0') is the weight regularization term. Monitor the performance of the second validation set in real time and dynamically fine-tune the weight parameters.

[0085] 2.1 Gradient-based weight updates Gradient update formula: α1' = α0' - η_α'· L_weight / α0' β1' = β0' - η_β'· L_weight / β0' [α', β'] = softmax( ) in: η_α' and η_β' are the learning rate parameters of CEUS and CT modalities respectively; α1' and β1' are temporary weights after gradient update; α' and β' are normalized weights, i.e., the second weights.

[0086] The present invention adopts a task adaptive adjustment strategy in the fusion process of dynamic microblood flow features F_CEUS and anatomical features F_CT, as follows: When diagnostic tasks take priority: Increasing η_α' emphasizes the micro-blood flow characteristics of CEUS, with typical weight distribution: α'≈0.6, β'≈0.4.

[0087] When other analytical tasks take priority: Increasing η_β' emphasizes the anatomical details of CT. The typical weight distribution is: α'≈0.45, β'≈0.55.

[0088] It is also possible to monitor the validation set performance in real time, dynamically fine-tune the weight parameters, and automatically adjust the learning rates of η_α' and η_β' based on the validation set performance to ensure the optimal weight distribution of the model in different task scenarios.

[0089] The final fusion output is the second fusion feature: F_fused' = α'·F_CEUS + β'·F_CT where α' + β' = 1.

[0090] 3. Feature optimization and interaction enhancement 3.1 Residual Connection and Feature Enhancement Introducing residual connections to avoid gradient disappearance and enhance information flow: F_enhanced = F_fused' + Dropout(FC(F_fused')) Among them, F_enhanced is the enhanced feature, F_fused' is the second fusion feature, FC is the fully connected layer, and Dropout is a random deactivation operation to prevent overfitting.

[0091] Then, layer normalization is applied to unify the distribution of features of different modalities and obtain the normalized enhanced features F_normalized: F_normalized = LayerNorm(F_enhanced) 3.2. Spatiotemporal Dependency Modeling Input the normalized enhanced feature F_normalized into the attention-enhanced GRU network: F_temporal = GRU(F_normalized) Among them, GRU is a gated recurrent unit, and F_temporal is the feature after capturing temporal dependency, that is, the second optimized fusion feature.

[0092] In summary, the dual-modal optimization fusion method of the present invention captures the synergistic effect of CEUS temporal features and CT spatial features, and strengthens the interaction of spatiotemporal features through the self-attention mechanism (number of heads = 4).

[0093] Figure 6 This is a classification and prediction flow chart for the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention. The thyroid tumor classification method of the present invention is implemented using a preoperative classification model, while the recurrence risk prediction method of the present invention is implemented using a postoperative recurrence risk prediction model. The following describes the construction of the preoperative classification model and the postoperative recurrence risk prediction model.

[0094] 1. Construction of preoperative classification model 1.1 Data Collection and Preprocessing In one embodiment of the present invention, multimodal data from a total of 1,500 patients were used, including cases of follicular thyroid carcinoma (FTC) and follicular adenoma (FTA), from multiple tertiary hospitals using imaging equipment of different brands and models. The dataset was divided into training, validation, and test sets at a ratio of approximately 70% (1,050 cases), 15% (225 cases), and 15% (225 cases), respectively. Stratified sampling was used to ensure that the proportions of FTC and FTA in each subset were consistent. CEUS data, CT data, and pathology data were then standardized and processed: CEUS data: Using time series acquisition technology, it accurately divides the arterial phase (30-60 seconds) and the venous phase (60-120 seconds), and uses a high-frequency ultrasound probe to capture the spatiotemporal dynamic characteristics of tumor micro-blood perfusion; CT data: Multi-phase scanning with a slice thickness of 1.0-1.5 mm was used, combined with an iterative reconstruction algorithm to reduce noise; Pathological data: Whole-slide pathological images were digitized using a digital scanner (resolution 0.25 μm / pixel) and processed using a block-by-block method and a tissue region screening algorithm.

[0095] 1.2. Training of the Preoperative Classification Model The preoperative classification model of this invention is trained using a two-branch fully connected network structure. The input layer receives 512-dimensional fused features; a two-layer fully connected network (FC-128) with ReLU activation function performs feature dimensionality reduction; and the output layer outputs the classification probabilities of FTC and FTA using a Softmax function. This two-branch fully connected network achieves accurate classification of tumor types and outputs classification probabilities.

[0096] The preoperative classification model constructed in this paper employs a multi-stage transfer learning strategy, utilizing a three-stage progressive training framework consisting of pre-training, feature transfer, and fine-tuning. The input is source domain data (1050 multi-center training data), and the output is the domain-adapted model parameters θ, optimized for the specific medical device environment.

[0097] The adjustment method of the model parameter θ is as follows: Phase 1: Pre-training θ_pretrain = argmin_θ L_src(θ, D_src) Among them, argmin_θ is the optimization operator, which means finding the optimal parameter θ that minimizes the objective function; L_src is the source domain loss function, which is used to measure the prediction error of the model on the source domain dataset.

[0098] Phase 2: Feature Transfer θ_transfer = argmin_θ [L_src(θ, D_src) + λ_1·d_MMD(F_src, F_tgt)] Among them, d_MMD is the maximum mean difference metric, which is used to reduce the difference in feature distribution between the source domain and the target domain.

[0099] Phase 3: Fine-tuning θ_finetune = argmin_θ [L_tgt(θ, D_tgt) + λ_2·R(θ, θ_transfer)] Among them, R is the regularization term, which limits the parameters from deviating too much from the pre-training model.

[0100] Through progressive migration, the distribution difference between the source and target domains is reduced, the adaptability of the model on new devices is improved, and the accuracy on the validation set is improved by about 8-12%.

[0101] The preoperative classification model constructed in this paper also utilizes a dynamic fine-tuning mechanism and an adaptive parameter adjustment algorithm based on device characteristics. The input is the target device's CEUS and CT characteristic parameters, including imaging resolution, frame rate, and contrast, and the output is the network weights optimized for the specific device characteristics.

[0102] The network weights are adjusted as follows: Device Characterization: v_device = Encoder(device_params) Conditional parameter generation: α_device = MLP(v_device) Dynamic weight adjustment: W_adjusted = W_base * (1 + α_device * Tanh(v_device)) Where W_base is the base weight and α_device is the device-specific adjustment factor.

[0103] By automatically adjusting model parameters based on device characteristics, the model can adapt to the differences in imaging characteristics of devices from different manufacturers, reducing cross-device performance degradation by 20-35%.

[0104] The preoperative classification model of the present invention also uses adaptive learning rate adjustment, which is a cyclically decaying learning rate strategy based on validation set performance. The input is the performance indicators (accuracy, loss value) on the validation set, and the output is the dynamically adjusted learning rate value.

[0105] The learning rate is adjusted as follows: Base cycle learning rate: η_t = η_min + 0.5*(η_max - η_min)*(1 + cos(t / T_cycle * π)) Where η_t is the learning rate value at the t-th training step, which is dynamically calculated using the cosine annealing formula; η_min is the minimum learning rate, which prevents training from stagnating due to a too small learning rate; η_max is the maximum learning rate, which can be dynamically adjusted according to performance during training; T_cycle is the number of training steps in a complete learning rate cycle, which determines the length of the cosine annealing cycle.

[0106] Performance Monitoring: If val_loss_t>val_loss_{t-1} * (1 - ε): patience += 1 Else: patience = 0 Among them, val_loss_t is the loss value on the validation set at the t-th training step, which is used to monitor the model performance; patience is a patience counter that records how many consecutive times the validation set performance has not improved, which is used to trigger the learning rate adjustment.

[0107] Learning rate adjustment: If patience>patience_threshold: η_max = η_max * 0.5 patience = 0 By preventing training from falling into local optimality and automatically lowering the upper limit of the learning rate as training progresses, model convergence is promoted, and the average convergence speed is increased by about 30%.

[0108] The loss function of the preoperative classification model of the present invention adopts the cross entropy loss function and the L2 regularization term: L_diag = -Σ y_i log(P(y=i)) + λ||θ||^2 Among them, y_i is the true label, λ||θ||^2 is the L2 regularization term to prevent overfitting.

[0109] 2. Construction of a postoperative recurrence risk prediction model The postoperative recurrence risk prediction model of the present invention is based on a multimodal contrastive learning framework. The second optimized fusion feature is extracted through the gated recurrent unit (GRU) to extract temporal dependencies, the pathological features are self-attentioned to screen high-risk areas, and contrastive loss is used to bring similar samples (recurrence vs. non-recurrence) closer together, with a spacing threshold of δ=0.5.

[0110] The postoperative recurrence risk prediction model of the present invention employs a contrastive learning strategy, utilizing a dual-branch contrastive learning network combined with inter-sample similarity learning to learn discriminative features and align extended modal semantics. The dual-branch contrastive learning network takes as input the features F_temporal extracted from the imaging branch and F_path extracted from the pathology branch, and outputs a semantically aligned representation in the feature space. This process involves the following operations: Construction of positive and negative sample pairs: y_ij = 1 if samples i,j are both relapse or non-relapse y_ij = 0 otherwise Contrastive loss calculation: L_contra = Σ_i Σ_j y_ij max(0, δ - cos(F_temporal^i, F_path^j)) +(1-y_ij) max(0, cos(F_temporal^i, F_path^j) - m) Among them, δ=0.5 is the similarity threshold of the positive sample pair, indicating that the distance between the features of the same type of samples should be less than this value; m=0.2 is the boundary interval of the negative sample pair, indicating that the distance between the features of different types of samples should be greater than this value; cos(,) represents the cosine similarity, and y_ij is 1, indicating that samples i and j are both recurrent or non-recurrent.

[0111] By enabling the model to learn discriminative features between recurrent and non-recurrent cases while maintaining semantic consistency between imaging and pathological modalities, the recurrence risk AUC was improved by 7-13%.

[0112] The postoperative recurrence risk prediction model of the present invention adopts a risk scoring mechanism, which is a risk scoring regressor based on optimized fusion features. The input is the image feature F_temporal and the pathological feature F_path after contrast learning alignment, and the output is a continuous recurrence risk score (0-1). The process involves the following operations: 1. Risk score calculation Risk_score = sigmoid(W_risk·[F_temporal, F_path] + b_risk) W_risk is the risk weight matrix, a trainable weight matrix used to map the second optimized fusion feature F_temporal and the pathological feature F_path into the risk space. The dimension of W_risk depends on the dimension of the input features and the dimension of the risk score, typically (d_combined × 1), where d_combined is the concatenated dimension of F_temporal and F_path. This weight matrix is ​​automatically learned during training through backpropagation, assigning higher weights to features that are more important for recurrence prediction. The weights reflect the contribution of different features to recurrence risk prediction. b_risk is the risk bias term, a scalar value used to adjust the baseline level of the risk score and ensure that the model output has an appropriate bias. It is optimized along with W_risk during training to help the model cope with imbalanced feature distributions. It can be understood as the model's default risk prediction value in the absence of any feature input. W_risk and b_risk together constitute the linear portion of the risk score. The sigmoid function then maps the output to a range between 0 and 1, representing the probability of postoperative recurrence in the patient.

[0113] 2. Weighted Hybrid Loss L_pred = BCE(Risk_score, y_recur) + α·L_contra L_pred consists of two components: a weighted sum of binary cross-entropy loss (BCE) and contrastive loss (L_contra). BCE(risk_score, y_recur) represents the binary cross-entropy loss between the predicted recurrence risk value (risk_score) and the true recurrence label (y_recur), directly optimizing the accuracy of risk prediction. y_recur∈{0,1} represents whether the patient has recurred (1 for recurrence, 0 for non-recurrence). α is the weighting coefficient for the contrastive loss, balancing the contribution of direct prediction loss and feature contrastive loss. In actual training, α is typically set between 0.3 and 0.5 and can be adjusted based on validation set performance. L_contra is the contrastive loss component. As described above, it is used to bring the feature representations of samples of the same type (both recurrence and non-recurrence) closer together and push the feature representations of samples of different types further apart, thereby enhancing the model's ability to distinguish recurrence risk.

[0114] Through the risk scoring mechanism, the model can provide more fine-grained risk predictions rather than just binary classification results, improving clinical decision support capabilities by approximately 25%.

[0115] The postoperative recurrence risk prediction model of this invention utilizes cross-device adaptive optimization, a domain adaptation technique specifically characterized by domain-invariant feature learning based on adversarial training. The input is the feature representation of the source domain data (training device), and the output is a domain-invariant feature representation. This process involves the following operations: 1. Domain classifier training L_domain = -Σ[d_i log(D(f_i)) + (1-d_i)log(1-D(f_i))] 2. Feature Extractor Training (Adversarial Target) L_feature = L_task - λ_d·L_domain Among them, D is the domain classifier, d_i is the domain label (0 for source domain, 1 for target domain), and fi is the feature representation.

[0116] Through adversarial training, the model learns feature representations that are not affected by device differences, allowing the model to achieve performance close to that of the original device on new devices without requiring a large amount of data, and the cross-device performance retention rate is increased to 85-92%.

[0117] The postoperative recurrence risk prediction model constructed in this paper employs an early stopping strategy, an adaptive early stopping mechanism based on validation set performance. Inputs are risk prediction performance metrics on the validation set (AUC (Area Under the Curve) and Recall@HighRisk (recall rate for high-risk categories, measuring the model's ability to correctly identify high-risk samples)). Outputs are a training stop signal and the optimal model save point. This process involves the following operations: 1. Performance Monitoring metric_t = 0.7*AUC + 0.3*Recall@High Risk If metric_t>best_metric: best_metric = metric_t best_epoch = t save_model(θ_t) patience = 0 Else: patience += 1 Where metric_t is the comprehensive performance metric of the t-th training cycle, calculated by a weighted combination of AUC and high-risk recall; best_metric is the best performance metric value recorded so far, used to determine whether the current model has reached a new optimal performance; best_epoch is the number of training cycles corresponding to the best performance metric, used to record the training time of the optimal model; t is the current training cycle number, indicating how many epochs the model has been trained for; save_model(θ_t) is a model save operation, which saves the model parameters θ_t of the current cycle t as the best model checkpoint; patience is a patience counter, which records the number of consecutive cycles without performance improvement and is used to trigger the early stopping mechanism.

[0118] 2. Early Stopping If patience>patience_max or t - best_epoch>window_size: stop_training() restore_model(best_epoch) By adopting this early stopping strategy, we can effectively avoid overfitting problems and reduce unnecessary training time. The model training time is reduced by an average of 35% while maintaining or improving the test set performance.

[0119] The performance of the preoperative classification model and recurrence risk prediction model of the present invention on the test set (225 cases) is as follows: Preoperative classification model: FTC / FTA classification accuracy reached 91.3%, sensitivity 89.6%, and specificity 92.5%.

[0120] Recurrence risk prediction model: AUC reached 0.873, and the accuracy of predicting recurrence rate in the high-risk group reached 83.7%.

[0121] Through detailed training strategy design and optimization, the present invention can maintain stable performance in different medical institutions and equipment environments, providing reliable support for clinical thyroid cancer diagnosis and risk assessment.

[0122] Figure 7 This is a flowchart of the interpretability analysis of the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention. The flowchart includes the following parts: 1. Grad-CAM heat map generation: Input image features: Extract CEUS features and CT features used in classification and prediction models.

[0123] Generate key area heatmap: Based on Grad-CAM technology, generate heatmap to highlight the key areas that the model focuses on.

[0124] Marking dynamic abnormal areas: Highlight abnormal blood perfusion areas on the CEUS heat map.

[0125] Marking static abnormal areas: Marking the blurred boundary areas or calcification areas of the tumor on the CT thermal image.

[0126] 2. SHAP feature contribution analysis: Input fusion features: Input feature vectors for classification and prediction models.

[0127] Calculate feature importance: Based on SHAP technology, analyze the contribution of input features to classification results.

[0128] 2.1. Quantifying feature contributions: Contribution of CEUS features: such as dynamic perfusion rate and time series pattern.

[0129] Contribution of CT features: such as boundary fuzziness and anatomical morphology.

[0130] Contribution of pathological characteristics: such as cell morphology and histological structure.

[0131] 3. Presentation of explanatory analysis results: Display Grad-CAM heatmap: Displays key areas in the image on the user interface.

[0132] Display feature contribution ranking: Quantify and visualize the contribution weights of features from different modalities.

[0133] Generate explanatory reports: Integrate heatmaps and feature contribution rankings to generate downloadable reports.

[0134] Specifically, the Grad-CAM optimization and SHAP feature contribution analysis adopted by the interpretability enhancement technology of the present invention are specifically implemented as follows.

[0135] 1. Grad-CAM optimization: The optimization strategy is to superimpose the time dimension weight on the CEUS heat map to accurately locate the area of ​​abnormal perfusion in the arterial phase. At the same time, it combines CT images and pathological features (such as tumor boundary ambiguity and nuclear division count) to achieve multimodal data fusion, thereby improving the clarity and reliability of the visualization results.

[0136] The specific implementation process of Grad-CAM is as follows: For target category c, the feature map weight α_k^c is calculated as: α_k^c = (1 / Z) Σ_i Σ_j y^c / A_ij^k Where Z is the normalization factor, which is equal to the total number of pixels in the feature map and is used to normalize the weights; Σ_i Σ_j is a double summation symbol, indicating that the accumulation operation is performed on all spatial positions (i, j) of the feature map; y^c / A_ij^k is the partial derivative of the predicted value of the target category c with respect to the activation value of the kth feature map at position (i, j), indicating the degree of influence of this position on the classification result.

[0137] Time dimension weight integration (for CEUS sequence): α_k^c(t) = α_k^c·w_t Among them, w_t is the weight coefficient of time step t, which is related to the blood perfusion stage.

[0138] Grad-CAM heat map calculation: L_Grad-CAM^c = ReLU(Σ_k α_k^c A^k) Among them, the weights α_k^c of all channels are weighted and summed with the corresponding feature maps A^k, and then the Grad-CAM heat map is obtained through the ReLU activation function, which highlights the feature areas important to the target category c.

[0139] CEUS-specific heatmap: L_Grad-CAM^c(CEUS) = ReLU(Σ_k Σ_t α_k^c(t) A^k(t)) Among them, based on the previous Grad-CAM, we further consider the changes of α_k^c(t) and the feature map A^k(t) at different times t in the time dimension, so as to obtain a heat map that is more consistent with the characteristics of the CEUS sequence and is used to analyze the feature importance distribution of the target category in the time series.

[0140] The Grad-CAM optimization adopted in the present invention can accurately locate the abnormal perfusion area during the CEUS arterial phase and realize the fusion visualization of multimodal data by combining CT images and pathological features.

[0141] 2. SHAP feature contribution analysis: Through SHAP feature contribution analysis, the model can quantify the contribution of each feature to prognosis, helping to reveal the role of different factors in prediction. The SHAP value assigns an importance score to each feature. By measuring the impact of each feature on the prediction results, clinicians can intuitively understand how factors such as CEUS peak flow velocity, boundary ambiguity in CT images, and pathological mitotic counts affect prognosis.

[0142] The specific implementation process of SHAP feature contribution analysis is as follows: Calculate the SHAP value of feature i: φ_i = Σ_{S⊆N\{i}} |S|!(|N|-|S|-1)! / |N|! [f_x(S∪{i}) - f_x(S)] Where N is the set of all features, S is the subset that does not contain feature i, and f_x(S) is the prediction of the model for sample x when using feature set S.

[0143] SHAP value of inter-modal feature interaction: φ_ij = Σ_{S⊆N\{i,j}} |S|!(|N|-|S|-2)! / 2|N|! [f_x(S∪{i,j}) - f_x(S∪{i}) - f_x(S∪{j}) + f_x(S)] Feature Importance Quantification: Importance(i) = Σ_x |φ_i(x)| Modal Importance Quantification: Importance(CEUS) = Σ_{i∈CEUS_features} Importance(i) Importance(CT) = Σ_{i∈CT_features} Importance(i) Importance(Pathology) = Σ_{i∈Pathology_features} Importance(i) The SHAP feature contribution analysis used in the present invention can quantify the contribution of each feature to the prognostic result and reveal the role of different factors in prediction.

[0144] In summary, the present invention integrates the dynamic microblood flow characteristics of CEUS and the high-resolution anatomical information of enhanced CT through a deep learning method to establish a preoperative classification model based on multimodal imaging data, thereby comprehensively improving the sensitivity and specificity of preoperative classification of FTC. An adaptive weight dynamic adjustment mechanism is adopted to achieve the optimal fusion of multimodal imaging features through a deep self-attention network, fully mining the complementary information of each modality feature. By utilizing the joint analysis of imaging and pathological data, a recurrence risk prediction model based on molecular and cellular level features is constructed to provide accurate individualized management strategies for high-risk patients after surgery. By introducing Grad-CAM and SHAP technologies, the present invention integrates decision-making visualization into the model, thereby improving the model's decision-making support capabilities for doctors, achieving the interpretability of the diagnostic process, and providing intuitive and reliable auxiliary information for clinicians.

[0145] Furthermore, the present invention also proposes an electronic device, which mainly includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, the processor will execute the deep learning-based thyroid tumor classification and recurrence risk prediction method of the present invention.

[0146] The memory may be a random access memory, a read-only memory, a non-volatile memory, a programmable ROM, an erasable PROM, an electrically erasable memory, a flash memory, an optical memory, a register, and the like.

[0147] The processor can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0148] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or solid-state drive (SSD).

Claims

1. A method for thyroid tumor classification and recurrence risk prediction based on deep learning, characterized in that: The following steps are involved: S1. Acquiring and preprocessing multimodal data related to follicular thyroid carcinoma (FTC) or follicular adenoma (FTA), wherein the multimodal data includes dynamic microblood flow data, anatomical data, and whole-slice pathology image (WSI) data; S2. Different feature extraction methods are used to extract the preprocessed dynamic microblood flow data, anatomical data and whole-slice pathological image WSI data, respectively, to obtain dynamic microblood flow features F_CEUS, anatomical features F_CT and pathological features F_path; S3. Fusing and optimizing the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT, and the pathological feature F_path to obtain a first optimized fusion feature F_fused_optimized; fusing and optimizing the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT to obtain a second optimized fusion feature F_temporal; S4. Input the first optimized fusion feature F_fused_optimized into the preoperative classification model to obtain the thyroid tumor type classification probability, and input the second optimized fusion feature F_temporal and the pathological feature F_path into the postoperative recurrence risk prediction model to obtain the recurrence risk score.

2. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 1, characterized in that: The acquisition of multimodal data in step S1 includes: Dynamic microblood flow data were obtained by contrast-enhanced ultrasound (CEUS) using time series acquisition technology. Anatomical data were obtained by using enhanced CT with multi-phase scanning and iterative reconstruction algorithm; Whole-slice pathology image WSI data were obtained using a digital scanner; The preprocessing of multimodal data in step S1 includes: Applying adaptive median filtering and wavelet transform to the dynamic microblood flow data to obtain first dynamic microblood flow data, and applying non-local mean denoising to the anatomical data to obtain first anatomical data; Applying a mutual information maximization algorithm to the first dynamic microblood flow data and the first anatomical data to obtain sub-pixel spatially aligned dynamic microblood flow data and anatomical data; A pyramid block partitioning strategy is applied to the whole-slice pathology image WSI data. Pixel blocks of the whole-slice pathology image WSI are extracted at different magnifications. Multi-scale feature fusion is then applied to the pixel blocks to obtain the fused whole-slice pathology image WSI data.

3. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 1, characterized in that: The feature extraction of the pre-processed multimodal data in step S2 includes: The CNN-GRU hybrid architecture is used to extract spatial features and model temporal features of the preprocessed dynamic microblood flow data, and the dynamic microblood flow features F_CEUS associated with the dynamic microblood flow data are obtained; The pre-trained ResNet50 is used to extract multi-level features from the pre-processed anatomical data, and the multi-scale features are fused through the pyramid pooling module PPM to obtain the anatomical features F_CT associated with the anatomical data; The preprocessed whole-slice pathology image (WSI) is divided into N tiles. Each tile is processed using the InceptionResNetV2 model to obtain the corresponding instance features. The attention weight of each tile is then calculated using the attention weight network. Based on the calculated attention weight, all instance features are weightedly aggregated to obtain the pathology feature F_path associated with the whole-slice pathology image (WSI) data.

4. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 3, characterized in that: In step S3, the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT, and the pathological feature F_path are fused and optimized to obtain a first optimized fused feature F_fused_optimized, including: For the dynamic microblood flow feature F_CEUS, anatomical feature F_CT and pathological feature F_path, a multi-head self-attention mechanism is used to generate the first initial weight vector ; Combined with feature quality evaluation and task relevance analysis, the first initial weight vector Perform adaptive adjustment to obtain the first initial weight vector after the first adjustment ; The first initial weight vector after the first adjustment is adjusted using the task-aware dynamic weight adjustment mechanism Perform dynamic adjustment to obtain the first initial weight vector after the second adjustment ; The first initial weight vector after the second adjustment Normalize and use task-sensitive adaptive adjustment to obtain the first weight vector [α, β, γ], where α+β+γ=1; Based on the first weight vector [α, β, γ], the dynamic microblood flow feature F_CEUS, the anatomical feature F_CT, and the pathological feature F_path are weightedly fused to obtain the first fused feature F_fused, as shown in the following formula: F_fused=α·F_CEUS+β·F_CT+γ·F_path The first fused feature F_fused is then optimized through the Transformer encoder to obtain the first optimized fused feature F_fused_optimized.

5. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 3, characterized in that: In step S3, the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT are fused and optimized to obtain a second optimized fusion feature F_temporal, including: For the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT, the multi-head attention mechanism is used to calculate the feature similarity matrix and generate the second initial weight vector ; The second initial weight vector is adjusted using a task-aware dynamic weight adjustment mechanism Adjust to get the adjusted second initial weight vector ; The adjusted second initial weight vector Normalize to obtain the second weight vector [α', β'], where α'+β' = 1; Based on the second weight vector [α', β'], the dynamic microblood flow feature F_CEUS and the anatomical feature F_CT are weightedly fused to obtain the second fused feature F_fused', as shown in the following formula: F_fused'=α'·F_CEUS+β'·F_CT; Introduce residual connections to obtain enhanced features F_enhanced: F_enhanced = F_fused' + Dropout(FC(F_fused')) Among them, FC is the fully connected layer, Dropout is the random deactivation operation, and FC is the fully connected layer; Apply layer normalization to the enhanced feature F_enhanced to obtain the normalized enhanced feature F_normalized: F_normalized = LayerNorm(F_enhanced) The normalized enhanced feature F_normalized is input into the attention-enhanced GRU network to obtain the second optimized fusion feature F_temporal: F_temporal = GRU(F_normalized).

6. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 1, characterized in that: The preoperative classification model in step S4 is constructed in the following way: A dual-branch fully connected network structure was used to construct the initial preoperative classification model; Multimodal data related to follicular thyroid carcinoma (FTC) or follicular adenoma (FTA) are collected as a dataset, and stratified sampling is used to divide the dataset into a first training set, a first validation set, and a first test set according to a predetermined ratio; The first training set was used to train the initial preoperative classification model. During the training process, the first validation set was used to evaluate the performance of the initial preoperative classification model. A multi-stage transfer learning strategy, a cross-entropy loss function and an L2 regularization term, a dynamic fine-tuning mechanism based on the characteristics of CEUS and CT devices, and an adaptive learning rate adjustment were used to adjust the parameters of the initial preoperative classification model. After training, the preoperative classification model was obtained, and the performance of the preoperative classification model was evaluated using the first test set.

7. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 1, characterized in that: The postoperative recurrence risk prediction model in S4 was constructed in the following way: A multimodal contrastive learning framework was used to construct an initial postoperative recurrence risk prediction model; Multimodal data related to follicular thyroid carcinoma (FTC) and follicular adenoma (FTA) are collected as a dataset, and stratified sampling is used to divide the dataset into a second training set, a second validation set, and a second test set according to a predetermined proportion; The second training set was used to train the initial postoperative recurrence risk prediction model, and the second validation set was used to evaluate the performance of the initial postoperative recurrence risk prediction model. Domain-invariant feature learning based on adversarial training and an adaptive early stopping mechanism based on validation set performance were used to adjust the parameters of the initial postoperative recurrence risk prediction model. After training, the postoperative recurrence risk prediction model was obtained, and the performance of the postoperative recurrence risk prediction model was evaluated using the first test set.

8. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to claim 7, characterized in that: A dual-branch contrastive learning network and risk scoring mechanism were further adopted to enhance the performance of the postoperative recurrence risk prediction model.

9. The method for thyroid tumor classification and recurrence risk prediction based on deep learning according to any one of claims 1 to 8, characterized in that: Based on the thyroid tumor type classification probability and recurrence risk score, Grad-CAM was used to optimize the generation of heat maps to improve the clarity and reliability of the visualization results, and SHAP feature contribution analysis was used to quantify the contribution of various features to the prognostic results.

10. An electronic device comprising a processor and a memory, characterized in that: A computer program is stored in the memory, and when the computer program is executed by the processor, the processor will execute the deep learning-based thyroid tumor classification and recurrence risk prediction method according to any one of claims 1 to 9.

Images