Method and device for predicting thyroid cancer lymph node metastasis risk
Through a multimodal deep learning model combined with multiple data sources, the risk of lymph node metastasis in thyroid cancer is predicted, which solves the problem of insufficient subjectivity and accuracy of prediction methods in the prior art, and achieves efficient and accurate assessment of lymph node metastasis risk.
Patent Information
- Application Number
- CN202411060582.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
AI Technical Summary
The existing thyroid cancer lymph node metastasis risk prediction methods rely on imaging and pathological analysis, have subjectivity and misjudgment rates, and are difficult to process a large amount of image data, resulting in low accuracy and stability of prediction results, which cannot meet the rapid and rapid The need for accurate diagnosis.
Using a multimodal deep learning model, combining pathological slice images, genomic data and single-cell RNA sequencing data, the model is trained through the Adam optimizer, and the self-attention module is used to identify areas related to lymph node metastasis, and a comprehensive feature vector is generated for lymph node metastasis. risk assessment.
It improves the accuracy and real-time prediction of lymph node metastasis risk in thyroid cancer, enhances the reliability of diagnosis and the ability to formulate personalized treatment plans, and reduces the doctor's subjectivity and misjudgment rate.
Smart Images

Figure CN119028569A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of thyroid disease risk prediction, and specifically to a method and device for predicting the risk of lymph node metastasis of thyroid cancer. Background Art
[0002] Thyroid cancer is a common endocrine malignancy worldwide, and the occurrence of lymph node metastases (LNM) significantly affects the treatment of thyroid cancer. Therefore, accurately predicting the risk of thyroid cancer lymph node metastasis is of great significance for formulating personalized treatment plans.
[0003] However, the existing clinical methods for predicting the risk of lymph node metastasis (LNM) in PTC patients mainly rely on imaging examinations and pathological analysis, which have certain limitations: imaging and pathological analysis rely on the experience and skills of doctors, and have certain subjectivity and misjudgment rates. Secondly, single modality data such as imaging or pathological images are difficult to fully reflect the diversity and complexity of the disease, resulting in low accuracy and stability of the prediction results. In addition, the existing methods are inefficient in processing large amounts of image data, and the degree of automation is insufficient, making it difficult to meet the needs of clinical rapid and accurate diagnosis.
[0004] Therefore, there is an urgent need for a new technical method that can combine multimodal data and use artificial intelligence algorithms to achieve efficient and accurate prediction of lymph node metastasis risk. Summary of the invention
[0005] The present invention aims to provide a method for comprehensive analysis by combining multimodal data and utilizing artificial intelligence technology to improve the accuracy and real-time performance of risk prediction of lymph node metastasis of thyroid cancer.
[0006] According to one aspect of the present invention, a method for predicting the risk of lymph node metastasis of thyroid cancer is provided, comprising: collecting data of thyroid cancer patients; constructing a multimodal deep learning model based on the data of the thyroid cancer patients; training the multimodal deep learning model using an Adam optimizer to obtain a trained multimodal deep learning model; and integrating the trained multimodal deep learning model into clinical diagnosis to predict the risk of lymph node metastasis of thyroid cancer.
[0007] Preferably, the data of the thyroid cancer patient includes: pathological section images, genome data and single-cell RNA sequencing data.
[0008] Preferably, based on the data of the thyroid cancer patient, constructing a multimodal deep learning model includes: using a pre-trained ResNet50 model to extract the features of the pathological section image to obtain pathological section image features; introducing a self-attention module to analyze the pathological section image features and identify areas related to lymph node metastasis; fusing the pathological section image features with the genome data and the single-cell RNA sequencing data to obtain fused multimodal data; and constructing a multimodal deep learning model suitable for processing the multimodal data within the area identified by the self-attention module.
[0009] Preferably, within the area identified by the self-attention module, constructing a multimodal deep learning model suitable for processing the multimodal data includes: inputting the multimodal data; generating attention weights for the multimodal data through linear transformation and activation functions; performing Softmax normalization on the attention weights and applying them to the multimodal data to generate a comprehensive feature vector; passing the comprehensive feature vector to the classifier; and the classifier outputs the scoring result of the lymph node metastasis risk.
[0010] Preferably, before the comprehensive feature vector is passed to the classifier, the method further includes: detecting whether the multimodal data contains a gene feature; if the gene feature is contained, concatenating the gene feature with the comprehensive feature vector.
[0011] Preferably, training the multimodal deep learning model using an Adam optimizer includes: setting a learning rate and weight decay as hyperparameters of the Adam optimizer.
[0012] Preferably, after obtaining the trained multimodal deep learning model, the method includes: evaluating the prediction performance of the trained multimodal deep learning model on a validation set, wherein the evaluation indicators include an ROC curve and an AUC value.
[0013] Preferably, after evaluating the prediction performance of the trained multimodal deep learning model on the validation set, the method further includes: generating a heat map using an attention mechanism to display high-risk areas predicted by the trained multimodal deep learning model.
[0014] According to another aspect of the present invention, a device for predicting the risk of lymph node metastasis of thyroid cancer is provided, comprising: a data collection module for collecting data of thyroid cancer patients, including pathological section images, genomic data and single-cell RNA sequencing data; a model construction module for constructing a multimodal deep learning model based on the data of the thyroid cancer patients; a model training module for training the multimodal deep learning model using an Adam optimizer to obtain a trained multimodal deep learning model; and a model application module for integrating the trained multimodal deep learning model into clinical diagnosis to predict the risk of lymph node metastasis of thyroid cancer.
[0015] Preferably, the model construction module includes: an extraction unit, which is used to extract the features of the pathological section image using a pre-trained ResNet50 model to obtain pathological section image features; an analysis unit, which is used to introduce a self-attention module to analyze the pathological section image features and identify areas related to lymph node metastasis; a fusion unit, which fuses the pathological section image features with the genome data and the single-cell RNA sequencing data to obtain fused multimodal data; and a construction unit, which constructs a multimodal deep learning model suitable for processing the multimodal data in the area identified by the self-attention module.
[0016] The present invention discloses a method and device for predicting the risk of lymph node metastasis of thyroid cancer, the method comprising: collecting data of thyroid cancer patients; constructing a multimodal deep learning model according to the data of the thyroid cancer patients; training the multimodal deep learning model using an Adam optimizer to obtain a trained multimodal deep learning model; integrating the trained multimodal deep learning model into clinical diagnosis to predict the risk of lymph node metastasis of thyroid cancer. The present invention can combine multimodal data and realize efficient and accurate prediction of lymph node metastasis risk through an artificial intelligence algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0018] Figure 1 A flowchart of a method for predicting the risk of lymph node metastasis of thyroid cancer according to an embodiment of the present invention;
[0019] Figure 2 A flowchart of a technical solution according to an embodiment of the present invention;
[0020] Figure 3 A model architecture diagram according to an embodiment of the present invention; and
[0021] Figure 4 4 is a schematic diagram of a device for predicting the risk of lymph node metastasis of thyroid cancer according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0023] The present invention provides a method for predicting the risk of lymph node metastasis of thyroid cancer. Figure 1As shown, the method includes: collecting data of thyroid cancer patients; constructing a multimodal deep learning model based on the data of the thyroid cancer patients; training the multimodal deep learning model using the Adam optimizer to obtain a trained multimodal deep learning model; integrating the trained multimodal deep learning model into clinical diagnosis to predict the metastasis risk of the thyroid cancer lymph nodes.
[0024] In the related art, pathological image analysis relies on the experience and skills of pathologists, has certain subjectivity and misjudgment rate, and is inefficient when processing large amounts of image data. The method proposed in the present invention, by constructing an artificial intelligence multimodal analysis system, uses a deep learning algorithm to automatically extract and analyze image features, thereby greatly improving the analysis efficiency and accuracy of pathological images. By comprehensively utilizing histopathological images, genomes, transcriptomes, and immune cell data, the multimodal model of the present invention can more accurately predict the risk of preoperative lymph node metastasis in PTC patients. The integrated analysis of multidimensional data improves the accuracy and reliability of diagnosis, enabling doctors to develop more precise and personalized treatment plans.
[0025] According to an embodiment of the present invention, the data of the thyroid cancer patient includes: pathological section images, genome data and single-cell RNA sequencing data.
[0026] In an embodiment of the present invention, by integrating pathological section images, genomic data and single-cell RNA sequencing data, multi-dimensional information of thyroid cancer patients can be captured more comprehensively.
[0027] According to an embodiment of the present invention, based on the data of the thyroid cancer patient, constructing a multimodal deep learning model includes: using a pre-trained ResNet50 model to extract the features of the pathological section image to obtain pathological section image features; introducing a self-attention module to analyze the pathological section image features and identify areas related to lymph node metastasis; fusing the pathological section image features with the genome data and the single-cell RNA sequencing data to obtain fused multimodal data; and constructing a multimodal deep learning model suitable for processing the multimodal data in the area identified by the self-attention module.
[0028] In the embodiment of the present invention, by constructing a multimodal deep learning model, the analysis efficiency of pathological images is improved, and the comprehensive analysis capability of histopathological images, genome, transcriptome and immune cell data is improved, thereby enhancing the accuracy and reliability of predicting preoperative lymph node metastasis in PTC patients. The model can effectively utilize the information of pathological slice image features, genome data and single-cell RNA sequencing data to predict the risk of lymph node metastasis in patients with thyroid cancer.
[0029] According to an embodiment of the present invention, within the area identified by the self-attention module, a multimodal deep learning model suitable for processing the multimodal data is constructed, including: inputting the multimodal data; generating attention weights for the multimodal data through linear transformation and activation functions; performing Softmax normalization on the attention weights and applying them to the multimodal data to generate a comprehensive feature vector; passing the comprehensive feature vector to the classifier; and the classifier outputs the scoring result of the lymph node metastasis risk.
[0030] In an embodiment of the present invention, a multimodal multitask analysis model based on artificial intelligence is provided, which can comprehensively analyze different data sources, including histopathological images, genomes, transcriptomes, and immune cell data, to predict preoperative lymph node metastasis LNM. The model includes multiple sub-models, which extract and analyze features for different types of data respectively, and finally integrate the prediction results of different sub-models through data fusion technology, thereby improving the breadth of prediction content and the accuracy of prediction results. At the same time, the multimodal deep learning model learns the mapping relationship from input features to lymph node metastasis risk, and can perform risk assessment on new patient data, thereby assisting doctors in making more accurate diagnosis and treatment decisions.
[0031] According to an embodiment of the present invention, before the comprehensive feature vector is passed to the classifier, it also includes: detecting whether the multimodal data contains gene features; if the gene features are contained, concatenating the gene features with the comprehensive feature vector.
[0032] In an embodiment of the present invention, by splicing gene features with comprehensive feature vectors, the accuracy and robustness of the multimodal deep learning model are improved, thereby improving the reliability and effectiveness of the model in clinical applications.
[0033] According to an embodiment of the present invention, using the Adam optimizer to train the multimodal deep learning model includes: setting a learning rate and weight decay as hyperparameters of the Adam optimizer.
[0034] In the embodiment of the present invention, the Adam optimizer combines the advantages of momentum and RMSProp, and can adaptively adjust the learning rate of each parameter, thereby accelerating the convergence speed of the model. At the same time, by reasonably setting the learning rate and weight decay, the learning rate determines the step size each time the model parameters are updated, and the weight decay is used to control the complexity of the model to prevent overfitting. This can effectively prevent overfitting, improve the generalization ability of the model, and thus improve the accuracy and reliability of the prediction of the risk of lymph node metastasis of thyroid cancer.
[0035] According to an embodiment of the present invention, after obtaining the trained multimodal deep learning model, the method includes: evaluating the prediction performance of the trained multimodal deep learning model on a validation set, wherein the evaluation indicators include an ROC curve and an AUC value.
[0036] In the embodiment of the present invention, the ROC curve can fully reflect the discrimination ability of the model by depicting the true positive rate and false positive rate under different thresholds. The AUC value, as the area under the ROC curve, can quantify the overall prediction performance of the model. By using these evaluation indicators, the generalization ability and robustness of the model can be more accurately evaluated to ensure the reliability and effectiveness of the model in clinical applications.
[0037] According to an embodiment of the present invention, after evaluating the prediction performance of the trained multimodal deep learning model on the validation set, the method includes: generating a heat map using an attention mechanism to display high-risk areas predicted by the trained multimodal deep learning model.
[0038] In the embodiment of the present invention, the generated heat map can intuitively display the location of the high-risk areas identified by the model, which not only improves the transparency and interpretability of the model prediction, but also provides a visual diagnostic reference for medical staff, thereby enhancing the practicality and credibility of the model in clinical applications.
[0039] Another embodiment of the present invention provides a device for predicting the risk of lymph node metastasis of thyroid cancer, such as Figure 4 As shown, it includes: a data collection module, which is used to collect data of thyroid cancer patients, including pathological section images, genomic data and single-cell RNA sequencing data; a model construction module, which is used to construct a multimodal deep learning model according to the data of the thyroid cancer patients; a model training module, which is used to train the multimodal deep learning model using the Adam optimizer to obtain a trained multimodal deep learning model; and a model application module, which is used to integrate the trained multimodal deep learning model into clinical diagnosis to predict the metastasis risk of the thyroid cancer lymph nodes.
[0040] In the related art, pathological image analysis relies on the experience and skills of pathologists, has certain subjectivity and misjudgment rate, and is inefficient when processing large amounts of image data. The device proposed in the present invention, by constructing an artificial intelligence multimodal analysis system, uses a deep learning algorithm to automatically extract and analyze image features, thereby greatly improving the analysis efficiency and accuracy of pathological images. By comprehensively utilizing histopathological images, genomes, transcriptomes, and immune cell data, the multimodal model of the present invention can more accurately predict the risk of preoperative lymph node metastasis in PTC patients. The integrated analysis of multidimensional data improves the accuracy and reliability of diagnosis, enabling doctors to develop more precise and personalized treatment plans.
[0041] According to an embodiment of the present invention, the model construction module includes: an extraction unit, which is used to use a pre-trained ResNet50 model to extract the features of the pathological section image to obtain the features of the pathological section image; an analysis unit, which is used to introduce a self-attention module to analyze the features of the pathological section image and identify areas related to lymph node metastasis; a fusion unit, which fuses the features of the pathological section image with the genome data and the single-cell RNA sequencing data to obtain fused multimodal data; and a construction unit, which constructs a multimodal deep learning model suitable for processing the multimodal data in the area identified by the self-attention module.
[0042] In an embodiment of the present invention, by constructing a multimodal deep learning model, the analysis efficiency of pathological images is improved, and the comprehensive analysis capability of tissue pathology images, genome, transcriptome and immune cell data is improved, thereby enhancing the accuracy and reliability of predicting preoperative lymph node metastasis in PTC patients.
[0043] The present invention is further described in detail below in conjunction with specific implementation modes and accompanying drawings.
[0044] Figure 2 is a flow chart of the technical solution according to an embodiment of the present invention, such as Figure 2 As shown, the technical solution process of the present invention includes:
[0045] 1. Dataset Collection and Processing
[0046] (1) Collect pathological slice images (Whole Slide Imagings, WSIs) of PTC patients.
[0047] ① Preparation of pathological sections: Preparation of pathological sections from tissue samples removed during surgery of PTC patients. This involves steps such as tissue fixation, embedding, slicing, and staining to facilitate observation of cells and tissue structures under a microscope.
[0048] ②Digital scanning: Use professional scanning equipment to convert pathological sections into digital images, namely WSIs. These images preserve all the details of the tissue at high resolution, allowing magnification and detailed observation.
[0049] ③ Image format: Make sure the image is stored in a format suitable for CLAM processing, such as TIFF or JPEG. CLAM (Clustering-constrained Attention Multiple Instance Learning) is a computational learning tool for medical image analysis, which plays an important role in the preprocessing of pathology slide images. The following are the detailed steps for preprocessing pathology slide images using the CLAM tool.
[0050] ④CLAM uses computer vision algorithms to identify and segment tissue regions in pathological sections and distinguish tissues and boundaries. The segmented tissue regions are cropped into multiple smaller image patches, which will be used for subsequent deep learning model training. The cropped image patches are adjusted to a uniform size of 256x256 pixels to adjust the input requirements of the deep learning model.
[0051] ⑤ Image quality control: Ensure that the quality of the digitized image meets the requirements of subsequent analysis, including color correction, contrast adjustment and resolution confirmation.
[0052] (2) Obtain genomic data through DNA-based next-generation sequencing (NGS) technology.
[0053] ① Sample preparation: DNA was extracted from FFPE (formaldehyde-fixed paraffin-embedded) tumor tissue.
[0054] ②DNA library construction: The extracted DNA is end-repaired, phosphorylated and adapter-ligated, and then DNA fragments of a specific size are selected for enrichment.
[0055] ③ Target region capture: Use specific probes to capture genomic regions associated with PTC, such as regions containing gene mutations such as BRAF, RAS, and RET.
[0056] ④ Sequencing: Use the NGS platform to perform high-throughput sequencing on the captured DNA fragments to generate a large number of short sequence reads.
[0057] ⑤Data analysis: Through bioinformatics analysis, the sequencing reads are aligned with the human genome reference sequence to identify gene mutations, copy number variations, etc.
[0058] (3) Use single-cell RNA sequencing (scRNA-seq) technology to obtain cellular heterogeneity data in the tumor microenvironment.
[0059] ① Single cell isolation: Isolate single cells from fresh PTC tissue samples to avoid the influence of cell-cell interactions on the results.
[0060] ②Library construction: Construct an RNA sequencing library for each individual cell, which usually includes reverse transcription and PCR amplification steps.
[0061] ③Sequencing: Use high-throughput sequencing technology to sequence the RNA library of a single cell to obtain the gene expression profile of each cell.
[0062] ④Data analysis: Through computational analysis, we can identify specific gene expression patterns of different cell types and reveal the cellular heterogeneity in the tumor microenvironment.
[0063] 2. Multimodal deep learning model construction:
[0064] (1) Use the pre-trained ResNet50 model as the basis for feature extraction and perform feature extraction on pathological slice images.
[0065] ① Model selection: ResNet50 is a deep residual network that has been pre-trained in multiple image recognition tasks and has proven its effectiveness. It is able to capture the complex features of images.
[0066] ② Feature extraction: Input pathological section images (WSIs) into the ResNet50 network and use the network’s convolutional layer to automatically learn image features. These features include but are not limited to cell morphology, tissue structure, and staining pattern.
[0067] ③ Network adjustment: According to the characteristics of pathological images, necessary adjustments are made to the ResNet50 network, such as modifying the input layer of the network to adjust the image resolution.
[0068] (2) A self-attention module is introduced to automatically extract the effective areas related to LNM.
[0069] ① Attention mechanism: The self-attention module (such as the mechanism in Transformer) enables the model to focus on the more important parts when processing data. In pathological image analysis, this helps the model identify areas related to LNM.
[0070] ② Effective region extraction: Through the self-attention module, the model can identify and highlight areas in pathological sections that may be related to lymph node metastasis, such as tumor margins, high-grade cell proliferation areas, etc.
[0071] ③ Feature enhancement: The output of the self-attention module is combined with the features extracted by ResNet50 to enhance the model's ability to recognize key features in pathological images.
[0072] 3. Feature fusion and risk assessment:
[0073] (1) Integrate pathological slice image features, genomic data, transcriptome data, and immune cell data.
[0074] ① Data integration: Integrate the features extracted from pathological slide images with genomic data, transcriptome data, and immune cell data. These data may include mutation information, gene expression levels, distribution of immune cell types, etc.
[0075] ② Feature alignment: Ensure that the features of different data sources are comparable, and pre-process the features through normalization and dimensionality reduction techniques to facilitate fusion.
[0076] ③ Information complementarity: Utilize complementary information from different data sources to improve the accuracy and comprehensiveness of the model’s LNM risk assessment.
[0077] (2) Use deep learning models to evaluate LNM risks.
[0078] ① Model construction: Build a deep learning model that can accept the fused features as input and output the risk score of LNM.
[0079] ②Task definition: Define the model’s task as binary classification (with or without LNM) and select a suitable loss function for training, such as cross entropy loss or mean square error loss.
[0080] ③ Risk prediction: The model learns the mapping relationship from input features to LNM risk, performs risk assessment on new patient data, and assists doctors in making more accurate diagnosis and treatment decisions.
[0081] 4. Model training and optimization:
[0082] (1) Use the Adam optimizer for model training and set appropriate learning rate and weight decay.
[0083] (2) Use early stopping method to avoid overfitting.
[0084] 5. Prediction and verification:
[0085] (1) Train the model on the training set and evaluate the prediction accuracy of the model on the validation set.
[0086] (2) The receiver operating characteristic curve (ROC) and area under the curve (AUC) were used as evaluation indicators.
[0087] 6. Visualization and interpretation:
[0088] (1) Generate a heat map using the attention score algorithm to visualize the high-risk areas predicted by the model.
[0089] 7. Clinical application:
[0090] (1) Integrate the model into the clinical diagnostic process to assist doctors in LNM risk assessment and treatment decision-making.
[0091] Figure 3 is a model architecture diagram according to an embodiment of the present invention, such as Figure 3 As shown, the model framework of the embodiment of the present invention includes:
[0092] 1. Input module:
[0093] Input feature matrix h and optional gene features gene. These features can come from different data sources (e.g., image feature extractor and gene data).
[0094] 2. Feature Extractor:
[0095] Perform preliminary processing on the input feature h. This includes linear transformation and nonlinear activation function (ReLU), and if enabled, a Dropout layer. This step enhances the expressiveness of the features and the generalization ability of the model.
[0096] 3. Attention Network:
[0097] It includes three parts: attention weight generation, gating mechanism and weight combination.
[0098] Attention weight generation: Generate preliminary attention weights through a series of linear transformations and activation functions.
[0099] Gating mechanism: The weights are adjusted through Tanh and Sigmoid functions to make them more accurate.
[0100] Weight combination: Apply the adjusted weights to the input features to generate the final attention-weighted features.
[0101] 4. Attention Weights and Weighted Sum:
[0102] The generated attention weights are Softmax normalized to ensure that the sum of the weights is 1. These weights are then applied to the input features through matrix multiplication to generate a comprehensive feature vector.
[0103] 5. Concatenate with gene:
[0104] If gene signatures are included, they are concatenated with the comprehensive signature vector to provide additional information.
[0105] 6. Classifier:
[0106] Classify the comprehensive feature vector. The classifier consists of a series of linear layers and finally outputs the classification results, including logits (unnormalized predicted values), predicted probabilities (Y_prob) and predicted labels (Y_hat).
[0107] 7. Output module:
[0108] Provides the final classification results, including logits, which are used to further calculate the loss; Y_prob, which is used to represent the predicted probability of each category; and Y_hat, which is used to represent the final predicted category.
[0109] The above embodiments are only examples for clearly explaining the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications may be made based on the following description, and these various changes, modifications, substitutions and deformations derived from the principles and spirit of the present invention still fall within the scope of protection of the present invention.
Claims
1. A method for predicting the risk of lymph node metastasis of thyroid cancer, characterized in that: include: Data from patients with thyroid cancer were collected; constructing a multimodal deep learning model based on the data of the thyroid cancer patients; Using the Adam optimizer to train the multimodal deep learning model to obtain a trained multimodal deep learning model; The trained multimodal deep learning model is integrated into clinical diagnosis to predict the metastasis risk of the thyroid cancer lymph nodes.
2. The method according to claim 1, characterized in that: The data of the thyroid cancer patients include: pathological section images, genome data and single-cell RNA sequencing data.
3. The method according to claim 2, characterized in that Based on the data of the thyroid cancer patients, a multimodal deep learning model is constructed including: Extracting features of the pathological slice image using a pre-trained ResNet50 model to obtain pathological slice image features; A self-attention module is introduced to analyze the features of the pathological slice image and identify areas related to lymph node metastasis; fusing the pathological section image features with the genome data and the single-cell RNA sequencing data to obtain fused multimodal data; Within the region identified by the self-attention module, a multimodal deep learning model suitable for processing the multimodal data is constructed.
4. The method according to claim 3, characterized in that In the region identified by the self-attention module, constructing a multimodal deep learning model suitable for processing the multimodal data includes: inputting the multimodal data; Generating attention weights for the multimodal data through linear transformation and activation function; Performing Softmax normalization on the attention weights and applying them to the multimodal data to generate a comprehensive feature vector; passing the integrated feature vector to the classifier; The classifier outputs a score result of the lymph node metastasis risk.
5. The method according to claim 4, characterized in that Before passing the comprehensive feature vector to the classifier, the method further comprises: Detecting whether the multimodal data contains gene features; If the gene feature is contained, the gene feature is concatenated with the comprehensive feature vector.
6. The method according to claim 1, characterized in that Training the multimodal deep learning model using the Adam optimizer includes: Set the learning rate and weight decay as hyperparameters of the Adam optimizer.
7. The method according to claim 1, characterized in that After obtaining the trained multimodal deep learning model, it includes: The prediction performance of the trained multimodal deep learning model is evaluated on the validation set, wherein the evaluation indicators include the ROC curve and the AUC value.
8. The method according to claim 7, characterized in that After evaluating the predictive performance of the trained multimodal deep learning model on the validation set, the following steps are included: The attention mechanism is used to generate a heat map to display the high-risk areas predicted by the trained multimodal deep learning model.
9. A device for predicting the risk of lymph node metastasis of thyroid cancer, characterized in that: include: The data collection module is used to collect data from thyroid cancer patients, including pathological slide images, genomic data, and single-cell RNA sequencing data; A model building module, used to build a multimodal deep learning model based on the data of the thyroid cancer patient; A model training module, used to train the multimodal deep learning model using an Adam optimizer to obtain a trained multimodal deep learning model; A model application module is used to integrate the trained multimodal deep learning model into clinical diagnosis to predict the metastasis risk of the thyroid cancer lymph nodes.
10. The device according to claim 9, characterized in that The model building module includes: An extraction unit, used to extract features of the pathological slice image using a pre-trained ResNet50 model to obtain pathological slice image features; An analysis unit, used for introducing a self-attention module to analyze the features of the pathological slice image and identify areas related to lymph node metastasis; A fusion unit, which fuses the pathological section image features with the genome data and the single-cell RNA sequencing data to obtain fused multimodal data; A construction unit constructs a multimodal deep learning model suitable for processing the multimodal data within the area identified by the self-attention module.
Citation Information
Cited By
Disease diagnosis method and system based on multi-mode space-frequency domain adaptive fusion
CN121601221A
