Thyroid cancer risk assessment system and device based on multiple modes

Abstract

The invention relates to the technical field of medical risk assessment, in particular to a thyroid cancer risk assessment system and device based on multiple modes. The system comprises a data preprocessing module used for unifying formats and scales of radiomics images, transcriptomics data and clinical medicine texts; the multi-modal feature extraction module is used for extracting image data features by adopting 3D ResNet-34 as a bottom feature extractor and combining a deformable convolution technology; performing feature selection on the exosome miRNA data through a machine learning model, and performing importance sorting on features through an SHAP method; performing feature extraction on the clinical medicine text data through a multi-granularity semantic fusion network architecture; the cross-modal dynamic fusion module is used for aligning and fusing features; and the decision and interpretability module is used for generating a diagnosis result and providing interpretability analysis. According to the technical scheme, the accuracy and efficiency of thyroid cancer risk assessment can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical risk assessment, and in particular to a multimodal thyroid cancer risk assessment system and device. Background Art

[0002] As one of the malignant tumors with the fastest growing incidence worldwide, early and accurate diagnosis of thyroid cancer is crucial for optimizing clinical diagnosis and treatment strategies and improving patient prognosis. Traditional diagnostic methods mainly rely on ultrasound imaging feature analysis and fine needle aspiration (FNA). However, these methods have significant limitations in practical application. Ultrasound diagnosis is highly dependent on the operator's experience and subjective judgment, resulting in limited consistency and accuracy of diagnostic results. Although FNA is the gold standard, its uncertainty diagnosis rate is as high as 10% to 25%. As an invasive examination, it may be accompanied by the risk of complications such as bleeding and infection, which increases the physical and mental burden on patients. Therefore, exploring more efficient, accurate, and non-invasive diagnostic technologies has become an urgent need in the current field of thyroid cancer research.

[0003] In recent years, with the rapid development of biomedical information technology, emerging technologies such as radiomics, transcriptomics, and clinical medical parameter analysis have provided new perspectives for the precise diagnosis of thyroid cancer. Radiomics can deeply explore tumor heterogeneity information by extracting quantitative features from medical images at high throughput, providing a basis for personalized diagnosis and treatment; transcriptomics technology reveals the association mechanism between gene expression profiles and tumor malignant progression at the molecular level, which helps to understand the nature of tumor occurrence and development; and in-depth mining of clinical medical parameters, such as serological indicators and medical history characteristics, can supplement individualized risk assessment information and improve the comprehensiveness of diagnosis. However, existing studies have mostly focused on the analysis of single-modality data and lack an effective fusion mechanism for cross-modal information, resulting in obvious deficiencies in the generalization ability and clinical applicability of prediction models. Single-modality data can generally only reflect the characteristics of a certain aspect of the tumor and it is difficult to fully capture the complex biological behavior of the tumor, thus limiting further improvement in diagnostic accuracy. Summary of the Invention

[0004] The purpose of the present invention is to propose a multimodal thyroid cancer risk assessment system and device, which can improve the accuracy and efficiency of thyroid cancer risk assessment.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a multimodal thyroid cancer risk assessment system, comprising:

[0006] Data preprocessing module, used to unify the format and scale of different modal data, including imaging data, plasma exosome miRNA data and clinical text data;

[0007] The multimodal feature extraction module is used to extract features from the standardized data. It uses 3D ResNet-34 as the underlying feature extractor, extracts image data features through the initial convolution layer, residual block, and 3D Swin Transformer modules, and introduces a deformable convolution layer and artifact-aware attention module to output features. The plasma exosome data feature extraction uses a machine learning model to select features from plasma exosome miRNA data and uses the SHAP method to interpret model features. ClinicalBERT is used to process the original clinical text, outputting a CLS token vector to represent the overall diagnostic tendency, and constructing an entity relationship graph to reveal the mutual relationships and semantic connections between entities in the text, ultimately generating a graph feature vector. The cross-modal dynamic fusion module is used to align and fuse features from different modalities.

[0008] Cross-modal dynamic fusion module, used to align and fuse features of different modalities;

[0009] The decision and explainability module is used to generate diagnostic results and provide explainable analysis.

[0010] Beneficial effects of the basic scheme: This scheme integrates data from three dimensions: radiomics, transcriptomics, and clinical medicine. It predicts the diagnosis of thyroid cancer from three dimensions and integrates the prediction probabilities of different dimensions through multimodality, thereby improving the accuracy and robustness of the prediction.

[0011] For CT images, a 3D hybrid convolutional-Transformer architecture (3D ResNet-34 as the underlying feature extractor, combined with a 3D Swin Transformer module) is used. This architecture combines the advantages of ResNet with the characteristics of the Swin Transformer. ResNet's residual blocks effectively address the vanishing gradient problem during deep network training, enabling deeper networks to extract more complex and abstract image features, such as better capturing subtle texture features at the edges of thyroid nodules.

[0012] The Swin Transformer features a local attention mechanism that can calculate attention within different windows, more effectively processing local information in the image and improving the ability to recognize CT images. It also introduces a deformable convolution layer and an artifact-aware attention module. The deformable convolution layer can adaptively adjust the shape of the convolution kernel based on the shape of the object in the image, better adapting to the irregular shapes of organs such as the thyroid gland; the artifact-aware attention module can focus on the real information in the image and ignore artifacts caused by factors such as equipment or patient movement, further improving the quality of image feature extraction.

[0013] Exosomal miRNA data was predicted using a machine learning model, and model interpretation was performed using SHAP (SHapley Additive ExPlanations). The machine learning model was able to uncover complex patterns in exosomal miRNA data, such as the potential relationship between the expression levels of different miRNAs and the risk of thyroid cancer. SHAP was then used to interpret the model's predictions, determining the contribution of each feature (such as the expression level of a specific miRNA) to the prediction. This not only improved prediction accuracy but also made the model's decision-making process more transparent, making it easier for researchers and clinicians to understand the model's rationale.

[0014] The decision-making and interpretable modules improve the model's confidence and generalization capabilities.

[0015] The present invention proposes a multimodal fusion framework based on deep learning. By constructing a joint representation space of radiomics, transcriptomics and clinical medicine, it realizes the collaborative optimization of feature level and decision level of multi-source heterogeneous medical data, and then develops a benign and malignant prediction model and supporting devices with higher diagnostic efficiency, providing an innovative solution for thyroid cancer risk assessment.

[0016] As an implementable preferred solution, the data preprocessing module processes thyroid CT image data, including the following contents:

[0017] CycleGAN technology is used to remove metal artifacts to reduce image distortion caused by metal objects in the body; the CT layer thickness is adjusted to 1mm 3 Voxel size and uniform voxel spacing of 1mm 3 CT values were mapped to the standard HU range and the window width / window level were adjusted to thyroid-specific settings. 3D U-Net was used to automatically segment the region of interest (ROI) and output an ROI with a size of 128 × 128 × 128 voxels.

[0018] As an implementable preferred solution, the data preprocessing module processes plasma exosome miRNA data, including the following contents:

[0019] Filter low-quality miRNA data; de-batch data from different batches; standardize the miRNA expression matrix and output miRNA feature vectors.

[0020] As an implementable preferred solution, the data preprocessing module processes text data, including the following contents:

[0021] Remove noise, delete irrelevant characters, and correct spelling errors; map medical terms using a standard terminology set; analyze the sentence structure and logical relationships in the text, identify and extract the association information between entities, use BioBERT for medical entity recognition, and use dependency syntactic analysis technology to construct an entity relationship graph.

[0022] As an implementable optimal solution, SHAP is used to interpret the model, sort the features according to their importance, and weight the expression values according to their importance. The formula is as follows:

[0023] Weighted_Feature i =SHAP_Weight i ×Exp i

[0024] Among them, Weighted_Featurei represents the weighted feature value of the i-th miRNA, reflecting the comprehensive contribution of the miRNA to the model prediction; SHAP_Weighti represents the SHAP global importance weight of the i-th miRNA; Expi represents the expression value of the i-th miRNA.

[0025] As an implementable preferred solution, the cross-modal dynamic fusion module aligns and concatenates CT image features, miRNA features, and clinical text features to form a 784-dimensional vector, which is expressed as follows:

[0026] f concat =[f CT ;f miRNA； f text ]∈R 512+16+256 =R 784

[0027] Through nonlinear mapping (ReLU activation function), it is reduced to 256 dimensions. The formula is as follows:

[0028] f fusion =ReLU(W1·f concat +b1)(W1∈R 256×784 ,b1∈R 256 )

[0029] Enforce feature interactions between modalities.

[0030] As an implementable preferred solution, the cross-modal dynamic fusion module generates preliminary prediction results for each single modality data through independent classifiers, including CT image classifiers, miRNA classifiers, and clinical text classifiers; calculates adaptive weights based on the quality indicators of each modality; and uses a weighted voting method to obtain a comprehensive prediction result, which is expressed as follows:

[0031] P=w1·P_CT+w2·P_miRNA+w3·P_text.

[0032] As an implementable preferred solution, the decision and interpretability module predicts the malignancy probability based on the fusion features and dynamic weight distribution results; inputs the results of the feature-level fusion submodule and the decision-level fusion submodule, and uses the classifier to calculate the final malignancy probability P final ; This classifier uses the Sigmoid function σ to output a probability value between 0 and 1. The confidence calculation formula is as follows:

[0033] Confidence = 1-|P final -0.5|×2

[0034] The range is [0,1] and is used to evaluate the level of certainty of the prediction; the initial threshold is determined based on the ROC curve of the validation set.

[0035] As an implementable optimal solution, the decision and interpretability module uses 3D Grad-CAM, SHAP value analysis, and LIME methods to perform interpretability analysis on CT images, miRNA, and text modalities, respectively; evaluates the consistency between the prediction results of each modality, and triggers a manual review alarm when the prediction probability difference exceeds a threshold or a key feature contradiction is found.

[0036] In a second aspect, the present invention provides a multimodal thyroid cancer risk assessment device, which utilizes the above-mentioned multimodal thyroid cancer risk assessment system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure and logic of a multimodal thyroid cancer risk assessment.

[0038] Figure 2 FIG. 2 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical solution and advantages of the present application clearer, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It will be understood that the specific embodiments described herein are only partial embodiments of the present invention, which are only used to explain the present application, rather than to limit the present application. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered to be isolated, and they can be combined with each other to achieve better technical effects. The same reference numerals appearing in the drawings of the following embodiments represent the same features or components, which can be applied to different embodiments.

[0040] In addition, unless otherwise defined, technical or scientific terms used in the description of the present invention should have the common meanings understood by those skilled in the art in the art to which the present invention belongs.

[0041] The present invention will be further described in detail below with reference to the accompanying drawings:

[0042] Reference numerals: electronic device 500 , processor 501 , communication interface 502 , memory 503 , bus 504 .

[0043] Reference Figure 1 The embodiment of the present disclosure provides a multimodal thyroid cancer risk assessment system, including a data preprocessing module, a multimodal feature extraction module, a cross-modal dynamic fusion module, and a decision and explainability module.

[0044] The data preprocessing module is used to unify the format and scale of data of different modalities, eliminate noise and redundant information, and provide a standardized data stream for model input. It includes an image processing submodule, a transcriptome processing submodule, and a text processing submodule.

[0045] Ultrasound and CT scans are commonly used medical imaging tests for thyroid cancer. Ultrasound is the most common and preferred imaging method for evaluating thyroid nodules. High-resolution ultrasound can clearly demonstrate the size, number, location, cysticity, shape, boundaries, calcification, and blood flow of thyroid nodules, and can also assess cervical lymph nodes. CT scans are valuable in evaluating the extent of thyroid tumors, their relationship to surrounding vital structures, and the presence of lymph node metastasis. Enhanced scans after injection of contrast agent can provide even clearer images.

[0046] The image processing submodule is used to standardize the image, including resizing, enhancement, and denoising. The pixel values are adjusted to a specific range, such as between 0 and 1, or normalized according to the overall mean and standard deviation of the data set. Resizing is to adapt to the requirements of the neural network input layer, and the image is scaled or cropped to a uniform size to improve training efficiency and ensure that all input images have the same resolution. Enhancement is to increase the diversity of the data set through rotation, flipping, scaling, cropping, etc., thereby improving the generalization ability of the model. Denoising is to remove random noise in the image to improve image quality and the accuracy of feature extraction. To define the lesion region (ROI), the tumor boundary needs to be accurately marked so that feature extraction within the area can be focused.

[0047] In this embodiment, the image processing submodule mainly analyzes CT images. The input is a 3D CT sequence in DICOM format, with a slice thickness of 0.5-5 mm and a matrix size of 512×512×number of slices, covering the thyroid gland and surrounding neck area. Data optimization is performed through a series of core processing steps.

[0048] Specifically, CycleGAN technology is first used to remove metal artifacts to reduce image distortion caused by metal objects in the body. The generator network in the model is a U-Net architecture, which learns the mapping from CT with metal artifacts to clean CT, reducing the impact of radiation hardening artifacts caused by metal objects in the thyroid adjacent area on lesion segmentation; then, a layer thickness standardization operation is performed to adjust all CT layers to 1mm. 3 Voxel size, unified voxel spacing is 1mm through trilinear interpolation 3 The original CT values were mapped to a standard HU range (-1024 to 3071), and the window width and window level were adjusted to thyroid-specific settings (window width 400HU, window level 60HU) to ensure image consistency and comparability. Finally, a trained 3D U-Net was used to automatically segment the region of interest (ROI), accurately locating key areas that may contain lesion information. After this processing, the output was a 128×128×128 voxel ROI region, providing high-quality, standardized input for subsequent feature extraction and multimodal data analysis, helping to improve the accuracy and reliability of thyroid cancer diagnosis.

[0049] RNA-seq (RNA sequencing) and microarrays are two commonly used high-throughput technologies for measuring gene expression levels in cell or tissue samples. Preprocessing transcriptional data, whether RNA-seq or microarray data, requires quality control and normalization. Quality control filters out data that does not meet expectations and contaminated data, while normalization eliminates systematic biases such as experimental batches and sample concentration variations. Data normalization is essential. Depending on the type of data used, additional data preprocessing is also required to ensure that the data conforms to the standard input format of the model.

[0050] The input data of the transcriptome processing submodule is the miRNA counts matrix or miRNA TPM matrix. First, the miRNA expression matrix is filtered to filter out low-quality miRNA data. If the data comes from different batches, the data from different batches need to be de-batched. Finally, the data is standardized and output as a miRNA expression matrix, which usually needs to contain hundreds of features.

[0051] The text processing submodule processes patient admission records, medical histories, pathology reports, medical imaging data, surgical consent forms, anesthesia consent forms, informed consent for blood transfusion treatment, consent forms for special examinations, critical / serious illness notifications, medical orders, auxiliary examination reports, and other related text materials (unstructured free text). This text material undergoes preprocessing operations such as data cleaning, terminology standardization, context understanding and relationship extraction, and data vectorization. Data cleaning involves removing noise, deleting irrelevant characters (such as HTML tags and special symbols), correcting spelling errors, and standardizing the extracted data. Term standardization involves mapping medical terms using standard terminology sets (such as SNOMED CT and the ICD coding system) to ensure consistent interpretation of data from different sources. Context understanding and relationship extraction analyze the sentence structure and logical relationships within the text to identify and extract associations between entities, such as causal relationships and the relationship between treatment plans and outcomes. Data vectorization converts the processed text into a form that can be understood by machine learning algorithms. Common methods include TF-IDF, Word2Vec, and GloVe.

[0052] The text processing submodule takes unstructured medical text as input (including ultrasound / pathology reports, electronic medical records, etc.). By converting this unstructured data into a structured form suitable for analysis, BioBERT is first used to perform medical entity recognition, accurately annotating and classifying key medical concepts in the text, such as diseases, symptoms, and treatment plans. Next, dependency parsing techniques are used to construct an entity-relationship graph, revealing the relationships and semantic connections between entities in the text. The final output includes both the entity-relationship graph enhanced by the knowledge graph and the original text sequence.

[0053] The multimodal feature extraction module includes the radiomics extraction submodule, the transcriptomics extraction submodule, and the clinical medicine extraction submodule.

[0054] The feature extraction submodule of the radiomics extraction module takes a 128×128×128×1 grayscale single-channel CT image as input and uses a 3D ResNet-34 as the underlying feature extractor. An initial 7×7×7 convolutional layer (stride 2) outputs a 64-channel feature map, which is then processed by a 3D BatchNorm and ReLU activation function. This is followed by four residual blocks (each containing three Bottleneck units) with channel sizes of [64, 128, 256, 512] and a downsampling stride of [1, 2, 2, 2], outputting a feature map of size 8×8×8×512. Residual connections are used to mitigate the vanishing gradient problem, and small convolution kernels capture local texture. The 3D Swin Transformer module then partitions the 8×8×8×512 feature map at the end of the ResNet into 2×2×2 non-overlapping windows. A windowed self-attention mechanism is applied to calculate inter-voxel QKV attention weights. Shifted window self-attention is used to enhance cross-window interactions. Adjacent windows are then gradually merged via patch merging to generate a multi-scale feature pyramid. The resulting 8×8×8×512 enhanced feature map captures the overall morphology of the nodule and its relationship to surrounding tissue, reducing computational complexity. A deformable convolutional layer (Deformable Conv3D) is introduced after the third residual block of the ResNet. This layer generates an offset field through standard 3×3×3 convolutions. Bilinear interpolation sampling is performed based on the offsets, enabling the convolution kernel to adaptively conform to irregular edge features. An artifact-aware attention module processes the raw CT volume data and the intermediate ResNet feature map. A 1×1×1 convolution is used to generate an artifact probability map. The feature map channels are weighted to suppress feature responses in artifact regions, ultimately outputting a 512-dimensional feature map.

[0055] The feature extraction of the transcriptomics extraction submodule is carried out by using the miRNA expression spectrum output by the transcriptomics processing submodule. The data set is divided into a training set and a test set in a 1:1 ratio using the CreateDataPartition function to ensure that the ratio of positive samples and negative samples in the training set and the test set are consistent. The training set is trained using machine learning models such as random forest (RF), support vector machine (SVM), XGBoost, and Lasso. Taking the Lasso model as an example, hundreds of miRNA features are reduced to dozens of features by adjusting the λ value, and then the model is interpreted using SHAP and sorted according to the importance of the features. In this embodiment, the top 16 feature miRNAs in importance are selected, and the expression values are weighted according to their importance weights (normalized to [0,1]). The formula is as follows:

[0056] Weighted_Feature i =SHAP_Weight i ×Exp i

[0057] Among them, Weighted_Featurei represents the weighted feature value of the i-th miRNA, reflecting the comprehensive contribution of the miRNA to the model prediction; SHAP_Weighti represents the SHAP global importance weight of the i-th miRNA; Expi represents the expression value of the i-th miRNA.

[0058] Output 16-dimensional importance-weighted miRNA feature vectors. Different machine learning models can avoid the problem of too few features to a certain extent.

[0059] It is important to note that the sample size for this module is preferably no less than 100. Too few samples may result in too few miRNAs being obtained, and insufficient features may be extracted through machine learning for subsequent feature importance ranking. Choosing different machine learning methods can mitigate this problem to some extent, but it cannot be completely avoided when too few miRNAs are obtained through preprocessing.

[0060] The input to the clinical medicine extraction submodule consists of an entity-relationship graph (ERG) enhanced by the knowledge graph generated by the preprocessing module and the original text sequence. A multi-granular semantic encoding strategy is employed, comprising local semantic encoding and global context encoding. Local semantic encoding utilizes a bidirectional LSTM (hidden layer dimension = 128) to process sentence-by-sentence text with 768-dimensional word embeddings generated by BioBERT. Max pooling is used to extract salient features from each sentence, identify negation words, and capture local context information. Global context encoding utilizes ClinicalBERT to process the entire original text (truncated to 512 tokens). Domain-adaptive pre-training is performed on 100,000 thyroid medical records. Model parameters are then fine-tuned to optimize the task of benign / malignant classification. The model outputs a 768-dimensional vector of CLS tokens to represent the overall diagnostic tendency. Knowledge graph features are integrated with node embeddings in the entity-relationship graph (pre-trained using the TransE algorithm). A graph attention network (GAT) aggregates neighborhood information, and pooling is performed to generate a 64-dimensional graph feature vector. The BiLSTM output (128D), ClinicalBERT output (768D) and graph features (64D) are concatenated into a 960-dimensional vector, and then dimensionality reduction is achieved through a fully connected layer (960D→256D) plus Layer Normalization, and finally a 256-dimensional text feature vector is output.

[0061] The cross-modal dynamic fusion module includes a feature-level fusion sub-module and a decision-level fusion sub-module.

[0062] The feature-level fusion submodule reduces redundant information and promotes feature interaction between modalities by aligning and concatenating feature vectors from CT images, miRNA expression values, and clinical text. Specifically, a joint embedding technique is used to align and concatenate data from different modalities, including 512-dimensional CT image features (extracted and reduced using 3D ResNet-34), 16-dimensional miRNA features (SHAP-weighted key miRNA expression values), and 256-dimensional clinical text features (enhanced by ClinicalBERT and knowledge graphs), to form a 784-dimensional vector. The formula is as follows:

[0063] f concat =[f CT ;f miRNA； f text ]∈R 512+16+256 =R 784

[0064] Through nonlinear mapping (ReLU activation function), it is reduced to 256 dimensions. The formula is as follows:

[0065] f fusion =ReLU(W1·f concat +b1)(W1∈R 256×784 ,b1∈R 256 )

[0066] To force feature interaction between modalities and reduce redundant information.

[0067] The decision-level fusion submodule generates preliminary predictions for each modality using independent classifiers. Specifically, the CT image classifier uses a sigmoid function to output the probability of malignancy; the miRNA classifier uses a pretrained Lasso model to process 16-dimensional features and output a probability value; and the clinical text classifier also uses a sigmoid function to output the probability of malignancy. It also calculates adaptive weights using a dynamic gating network based on quality metrics for each modality, such as CT image clarity (SSIM), miRNA detection batch coefficient of variation (CV), and clinical text uncertainty scores. For example, a blurry CT image will result in a lower weight, while a stable miRNA detection will result in an increased weight.

[0068] Finally, the weighted voting method is used to obtain the comprehensive prediction result P, which is expressed as follows:

[0069] P=w1·P_CT+w2·P_miRNA+w3·P_text.

[0070] This will enable accurate judgment of whether thyroid cancer is benign or malignant.

[0071] The decision-making and interpretability module includes a malicious probability prediction submodule, a multimodal interpretability submodule, and a conflict detection mechanism submodule.

[0072] The malignancy probability prediction submodule predicts the malignancy probability based on the output results of the fusion-level feature submodule and the decision-level feature submodule to quantify the possibility that the tumor is malignant. Specifically, the 256-dimensional fusion feature (from the feature-level fusion submodule) and the dynamic weight allocation result (from the decision-level fusion submodule) are input, and the final malignancy probability P is calculated using the classifier. final , and its calculation formula is:

[0073] P final =σ(W final ·f fusion +b final )

[0074] Among them, P fina l refers to the final malignancy probability output by the model, which ranges from [0, 1]; σ is the Sigmoid function that maps the output to the interval [0, 1]; W final ∈R 1×256 is the weight matrix of the classifier, with a dimension of 1×256, W final [i] represents the contribution weight of the i-th dimension in the fusion feature to the final prediction; f fusion ∈R 256 Refers to the joint feature vector after feature-level fusion, with a dimension of 256; b final ∈R refers to the bias term of the classifier, which is used to adjust the baseline prediction probability of the model, independent of the input features. final ∈R 1×256 and b final ∈R is a learnable parameter.

[0075] The formula for calculating confidence is as follows:

[0076] Confidence = 1-|P final -0.5|×2

[0077] It ranges from [0,1] and is used to assess the level of certainty in the prediction.

[0078] The threshold adaptive mechanism determines the initial threshold based on the validation set ROC curve and modifies it based on clinical feedback.

[0079] The multimodal interpretability submodule uses 3D Grad-CAM to generate heat maps for CT image data, marking high-activation areas such as burr areas at the edge of nodules. Its biological significance lies in the spatial consistency verification with the pathological gold standard (Dice coefficient>0.85). The miRNA modality uses SHAP value analysis to identify and sort key driver miRNAs. In this embodiment, the top 16 most influential driver miRNAs are selected to visualize the relationship between their expression and prediction, and cross-validate their biological validity with differentially expressed genes in the TCGA database. The text modality uses the LIME method to perturb samples and train a surrogate linear model to extract keywords that have the greatest impact on prediction and highlight them, such as descriptors such as "aspect ratio>1", while quantifying the uncertainty score at the sentence level.

[0080] The conflict detection mechanism submodule is used to evaluate the consistency between the prediction results of each modality. When the prediction probability difference between any two modalities exceeds a set threshold (in this embodiment, the prediction probability difference threshold is set to 0.4) or a contradiction in key features is found (such as CT showing "smooth edges" but the text describes "burr sign"), the system will trigger a manual review alarm, prompting further professional review to ensure the accuracy and reliability of the final diagnosis result.

[0081] The multimodal interpretability submodule not only improves the predictive performance of the model, but also greatly enhances its practicality and trustworthiness in actual medical applications.

[0082] The present disclosure also provides a multimodal thyroid cancer risk assessment method, which uses the multimodal thyroid cancer risk assessment system. The specific steps and logic can be referred to Figure 1 Medium S1-S4.

[0083] Those skilled in the art will understand that the implementation of all or part of the process in a multimodal thyroid cancer risk assessment method can be accomplished by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the process of each embodiment of a multimodal thyroid cancer risk assessment method. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] The present application also provides a multimodal thyroid cancer risk assessment device that utilizes the aforementioned multimodal thyroid cancer risk assessment system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the aforementioned multimodal thyroid cancer risk assessment method are implemented. In the present application, the processor is the control center of the computer method and can be a processor of a physical machine or a processor of a virtual machine.

[0085] Reference Figure 2 The electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503, and at least one bus 504. Bus 504 is used to implement communication between these components, communication interface 502 is used to communicate signaling or data with other node devices, and memory 503 stores machine-readable instructions executable by processor 501. When the electronic device 500 is running, processor 501 communicates with memory 503 via bus 504, and when the machine-readable instructions are called by processor 501, the steps of the multimodal thyroid cancer risk assessment method described above are executed.

[0086] The above contents are merely embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. A person of ordinary skill in the art is aware of all common technical knowledge in the technical field to which the invention belongs before the filing date or priority date, is able to obtain all existing technologies in the field, and has the ability to apply conventional experimental means before that date. A person of ordinary skill in the art can, under the guidance of this application, improve and implement this scheme in combination with his or her own abilities. Some typical known structures or known methods should not become an obstacle for a person of ordinary skill in the art to implement this application. It should be pointed out that for a person of ordinary skill in the art, several variations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection claimed in this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A multimodal thyroid cancer risk assessment system, characterized in that: include: Data preprocessing module, used to unify the format and scale of different modal data, including imaging data, plasma exosome miRNA data and clinical text data; The multimodal feature extraction module extracts features from the standardized data. It uses 3D ResNet-34 as the underlying feature extractor, extracts image data features through an initial convolutional layer, residual blocks, and a 3D Swin Transformer module. It also introduces a deformable convolutional layer and an artifact-aware attention module to output features. For plasma exosome data feature extraction, a machine learning model is used to select features from plasma exosome miRNA data and interpret the model features using the SHAP method. ClinicalBERT is used to process the original clinical text, outputting a CLS token vector to represent the overall diagnostic tendency. An entity relationship diagram is then constructed to reveal the relationships and semantic connections between entities in the text, ultimately generating a graph feature vector. Cross-modal dynamic fusion module, used to align and fuse features of different modalities; The decision and explainability module is used to generate diagnostic results and provide explainable analysis.

2. A multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The data preprocessing module processes thyroid CT image data, including the following contents: CycleGAN technology is used to remove metal artifacts to reduce image distortion caused by metal objects in the body; the CT layer thickness is adjusted to 1mm 3 Voxel size and uniform voxel spacing of 1mm 3 CT values were mapped to the standard HU range and the window width / window level were adjusted to thyroid-specific settings. 3D U-Net was used to automatically segment the region of interest (ROI) and output an ROI with a size of 128 × 128 × 128 voxels.

3. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The data preprocessing module processes exosomal miRNA data and includes the following contents: Filter low-quality miRNA data; de-batch data from different batches; standardize the miRNA expression matrix and output miRNA feature vectors.

4. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The data preprocessing module processes text data, including the following contents: Remove noise, delete irrelevant characters, and correct spelling errors; Use standard terminology sets to map medical terms; analyze the sentence structure and logical relationships in the text, identify and extract the association information between entities, use BioBERT for medical entity recognition, and use dependency syntactic analysis technology to construct an entity relationship graph.

5. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: SHAP is used to interpret the model, sort the features according to their importance, and weight the expression values according to their importance. The formula is as follows: Weighted_Feature i =SHAP_Weight i ×Exp i Among them, Weighted_Featurei represents the weighted feature value of the i-th miRNA, reflecting the comprehensive contribution of the miRNA to the model prediction; SHAP_Weighti represents the SHAP global importance weight of the i-th miRNA; Expi represents the expression value of the i-th miRNA.

6. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The cross-modal dynamic fusion module aligns and concatenates CT image features, miRNA features, and clinical text features to form a 784-dimensional vector, which is expressed as follows: f concat =[f CT ;f miRNA； f text ]∈R 512+16+256 =R 784 Through nonlinear mapping (ReLU activation function), it is reduced to 256 dimensions. The formula is as follows: f fusion =ReLU(W1 f concat +b1)(W1∈R 256×784 ,b1∈R 256 ) Enforce feature interactions between modalities.

7. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The cross-modal dynamic fusion module generates preliminary prediction results for each single modality data through independent classifiers, including CT image classifiers, miRNA classifiers, and clinical text classifiers; calculates adaptive weights based on the quality indicators of each modality; and uses weighted voting to obtain a comprehensive prediction result, which is expressed as follows: P=w1·P_CT+w2·P_miRNA+w3·P_text.

8. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The decision and interpretability module predicts the malignancy probability based on the fusion features and dynamic weight distribution results; inputs the results of the feature-level fusion submodule and the decision-level fusion submodule, and uses the classifier to calculate the final malignancy probability P final , the formula is as follows: P final =σ(W final ·f fusion +b final ) Among them, P fina l refers to the final malignancy probability output by the model, which ranges from [0, 1]; σ is the Sigmoid function that maps the output to the interval [0, 1]; W final ∈R 1×256 is the weight matrix of the classifier, with a dimension of 1×256, W final [i] represents the contribution weight of the i-th dimension in the fusion feature to the final prediction; f fusion ∈R 256 Refers to the joint feature vector after feature-level fusion, with a dimension of 256; b final ∈R refers to the bias term of the classifier, which is used to adjust the baseline prediction probability of the model, independent of the input features; The formula for calculating confidence is as follows: Confidence=1-|P final -0.5|×2 The range is [0,1] and is used to evaluate the level of certainty of the prediction; the initial threshold is determined based on the ROC curve of the validation set.

9. The multimodal thyroid cancer risk assessment system according to claim 1, characterized in that: The decision and interpretability module uses 3D Grad-CAM, SHAP value analysis, and LIME methods to perform interpretability analysis on CT images, miRNA, and text modalities, respectively; evaluates the consistency between the prediction results of each modality, and triggers a manual review alert when the prediction probability difference exceeds a threshold or a contradiction in key features is found.

10. A multimodal thyroid cancer risk assessment device, characterized in that: A multimodal thyroid cancer risk assessment system according to any one of claims 1 to 9 is used, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor.

Images