Thyroid cancer electronic medical record system based on multi-modal data fusion

Abstract

The invention relates to the field of medical informatization. The invention discloses a thyroid cancer electronic medical record system based on multi-modal data fusion. The thyroid cancer electronic medical record system comprises a multi-modal data acquisition module which acquires patient texts, ultrasonic images, genes, biochemical indexes and clinical data and performs standardized calibration to generate standard data; the multi-modal feature extraction module extracts semantic, structure, mutation, change and fluctuation features of each standard data through multiple technologies; the single-mode prediction model construction module constructs single-mode prediction models of texts, images and the like based on the features and outputs results; and the multi-modal fusion prediction module fuses the single-modal model based on the deep learning framework to output a multi-modal fusion prediction result. According to the invention, multi-modal data are integrated, and the accuracy and comprehensiveness of thyroid cancer diagnosis are improved. The system ensures consistency through standardized data processing, and assists doctors to accurately judge pathological types, recommend therapeutic schedules and evaluate prognosis by means of a multi-modal feature extraction and fusion mechanism.

Description

Technical Field

[0001] The present invention relates to the field of medical informationization, and in particular to a thyroid cancer electronic medical record system based on multimodal data fusion. Background Art

[0002] In the current diagnosis and treatment of thyroid cancer, traditional electronic medical record systems mostly rely on single-modal data for diagnosis, making it difficult to integrate multi-dimensional information such as gene sequencing and biochemical indicators, resulting in limitations in the comprehensive assessment of the condition. In addition, clinical data often suffer from noise interference and insufficient standardization. The existing system lacks a mechanism for quantifying and robustly processing defects in multi-source data, which can easily affect diagnostic accuracy due to fluctuations in data quality. At the same time, traditional systems mostly remain at the data recording level and lack intelligent prediction models based on multi-modal features. They are unable to provide doctors with precise decision-making support such as pathological type judgment and treatment plan recommendations, making it difficult to meet the clinical demand for intelligent and precise diagnosis and treatment of thyroid cancer.

[0003] The existing system is mainly based on structured text data records, and its ability to integrate and process multimodal data such as ultrasound images, genetic testing data, and digital images of pathological sections is obviously insufficient. Multimodal data is stored in independent systems such as imaging departments and laboratory departments, forming data silos. Doctors need to frequently switch between different systems to retrieve information when making diagnoses, making it difficult to quickly build a comprehensive portrait of the patient's condition. For example, when assessing the risk of malignancy of thyroid nodules, doctors cannot correlate the cell morphological characteristics in the pathology report with the structural characteristics of the ultrasound image, such as blood flow distribution and boundary clarity, in real time. This results in a single basis for diagnosis, which is easily affected by subjective judgment and cannot meet the needs of precise diagnosis and treatment.

[0004] The existing system has significant shortcomings in medical record management and data application. Multi-source data, including outpatient, inpatient, and imaging, lacks unified structured management, leading to significant information fragmentation. Medical staff must repeatedly enter basic information, and key diagnosis and treatment processes rely on manual reporting, which carries the risk of omissions and errors, increasing workload and impacting medical record quality. Furthermore, the system's ability to analyze semantic associations and deep analysis of unstructured data is weak, making it unable to uncover the underlying patterns of disease development within multimodal data. This makes it difficult to support clinical decision support and scientific research-level data governance, severely hindering the intelligent and personalized development of thyroid cancer diagnosis and treatment.

[0005] With the development of medical big data and artificial intelligence technologies, the value of multimodal data fusion in disease diagnosis is becoming increasingly prominent. However, existing thyroid cancer electronic medical record systems face technical bottlenecks in cross-modal feature extraction, defective data processing, and multi-model fusion reasoning. The semantic gap between different modal data is difficult to bridge, the impact of defective data is not quantitatively assessed, and fusion models are insufficiently integrated with clinical guidelines, limiting the effectiveness of the comprehensive application of multimodal data. Summary of the Invention

[0006] In order to address the deficiencies of the prior art, the present invention discloses a thyroid cancer electronic medical record system based on multimodal data fusion, which can integrate multi-source data, process data defects and realize intelligent fusion prediction.

[0007] The present invention discloses a thyroid cancer electronic medical record system based on multimodal data fusion, which includes:

[0008] The multimodal data acquisition module is used to collect multimodal data from thyroid cancer patients. The multimodal data includes text data, ultrasound image data, genetic data, biochemical indicator data, and clinical data, and standardize and calibrate the data to generate standard data.

[0009] The multimodal feature extraction module is connected to the multimodal data acquisition module. Through natural language processing, computer vision, genetic analysis, biostatistics analysis and time series analysis, it extracts semantic, structural, mutation, change and fluctuation features from each standard data.

[0010] A unimodal prediction model construction module is connected to the multimodal feature extraction module, which constructs a text prediction model based on semantic features, an image prediction model based on structural features, a molecule prediction model based on mutation features, a metabolism prediction model based on change features, and a physical sign prediction model based on fluctuation features, and outputs unimodal prediction results with text, images, molecules, metabolism, and physical signs respectively;

[0011] The multimodal fusion prediction module is connected to the single-modal prediction model construction module, fuses the single-modal prediction model based on the deep learning framework, inputs the single-modal prediction result, and outputs the multimodal fusion prediction result.

[0012] Furthermore, the multimodal data acquisition module includes:

[0013] The defect feature quantification unit is used to collect statistics on defect quantification indicators of each modal data, including:

[0014] The ambiguity of the medical history description and the coverage of symptom keywords in the text;

[0015] Noise intensity and edge clarity of ultrasound images;

[0016] Sequencing error rate and mutation site coverage of genetic data;

[0017] The degree of dispersion of test values ​​of biochemical indicators and the deviation from the reference range;

[0018] The consistency of palpation results and completeness of vital signs records in clinical data;

[0019] Based on the above quantitative indicators, a simulated defect feature library is constructed. According to the defect distribution statistics of real clinical data, multi-dimensional defects are injected into the calibration data at a ratio of 5%-30% to generate enhanced training samples.

[0020] Furthermore, when the multimodal fusion prediction module does not detect defective data, it is built based on the Transformer architecture and includes:

[0021] Cross-modal attention mechanism layer: calculates the interaction weights of each modal feature;

[0022] Multi-scale feature fusion layer: stratifies and fuses single-modal results by time series, spatial location, and biological association;

[0023] Fusion decision layer: Combined with clinical guidelines, it outputs fusion results including diagnosis probability, treatment recommendations and prognosis assessment.

[0024] Furthermore, the multimodal fusion prediction module includes a defect feature alignment unit, which is used to align the defect features through the defect feature alignment unit and fuse the defective data through the hierarchical fusion unit when the defect feature quantization unit detects that the multimodal data has defects;

[0025] The defect feature alignment unit aligns the single-modal prediction results with the simulated defect feature library in feature space, calculates the defect matching degree of multimodal data in semantic, structural, mutation, metabolic and physical dimensions, and generates defect attention weights based on the defect matching degree;

[0026] Hierarchical fusion unit, including bottom-level denoising unit and top-level prediction unit;

[0027] The noise reduction unit is used to filter defects in each modal data and adjust the input weight of each modal based on the defect matching degree;

[0028] The prediction unit uses the cross-modal attention mechanism layer of the Transformer architecture based on the aligned defect features and denoised data, and superimposes the defect attention weights for interactive calculation to generate a fused prediction result.

[0029] Furthermore, the training method of the hierarchical fusion unit includes:

[0030] Defects are injected into the labeled data based on the simulated defect feature library to generate multimodal sample data containing semantic loss, structural ambiguity, false negative mutations, metabolic index deviations, and incomplete vital sign records;

[0031] Multimodal sample data is used to train the hierarchical fusion unit. By calculating the difference between the defect repair results and the label data, the denoising unit weights and prediction unit parameters of the hierarchical fusion unit are iteratively adjusted until the hierarchical fusion unit's repair accuracy for defect features reaches a preset threshold.

[0032] Labeled data refers to sample data that has been annotated with a gold standard, which is a diagnosis result or clinical indicator confirmed by a doctor;

[0033] The repair accuracy is the consistency ratio between the prediction results after defect data processing and the gold standard.

[0034] Furthermore, the defect feature alignment unit generates attention weights based on the defect matching degree, including:

[0035] Defect severity assessment, real-time calculation of each modal defect index;

[0036] When the defect index of a certain modality exceeds the preset threshold, the fusion weight of other modalities will be automatically increased by at least 10-30%.

[0037] Furthermore, the multimodal feature extraction module is also used to construct a cross-modal defect correlation map;

[0038] Based on the simulated defect feature library, the Apriori algorithm or equivalent algorithm is used to discover strong association rules between defects in each modality, and an association rule library is established to record the association confidence, support, and improvement indicators;

[0039] Based on the Pearl causal graph model or equivalent methods, construct a causal relationship network between defects, calculate the average causal effect and natural direct effect, and establish a causal effect knowledge base to quantify the impact of each defect on the prediction results;

[0040] The association rule base and the causal effect knowledge base are integrated into a cross-modal defect association graph, where:

[0041] The nodes represent each modal defect type;

[0042] Edges represent the strength of association or causal effect value between defects;

[0043] Regularly update cross-modal defect association map parameters based on new sample data, and use incremental learning algorithms to ensure the timeliness of cross-modal defect association maps;

[0044] The correlation strength value in the graph is used as a correction factor for the cross-modal attention mechanism;

[0045] The defect feature alignment unit adjusts the attention weights according to the correction factors in the association map.

[0046] Furthermore, the multimodal fusion prediction module includes:

[0047] Defect perception reasoning engine, which infers and completes defect features based on cross-modal defect association graphs;

[0048] The uncertainty quantification layer calculates the confidence interval of the prediction results through Monte Carlo dropout or Bayesian neural network, and outputs probabilistic diagnostic recommendations containing defect influencing factors that are used to quantify the impact of data defects on diagnostic results, providing credibility assessment and defect attribution reference for clinical decision-making.

[0049] Furthermore, the standardization calibration of the data by the multimodal data acquisition module includes:

[0050] Perform word segmentation, entity recognition, and medical terminology standardization on text;

[0051] Perform grayscale normalization and lesion labeling on ultrasound images;

[0052] Perform sequencing quality filtering and mutation annotation on gene data;

[0053] Perform outlier detection and normalization on biochemical indicators;

[0054] Clinical data were time aligned and missing values ​​interpolated.

[0055] Furthermore, in the multimodal feature extraction module:

[0056] Natural language processing uses a bidirectional long short-term memory network to extract semantic features of lesion descriptions, pathological types, and treatment plans from texts;

[0057] Computer vision uses convolutional neural networks to extract structural features of nodule boundaries, echo intensity, and blood flow distribution in ultrasound images;

[0058] Genetic analysis uses variant detection algorithms to identify mutational signatures of single nucleotide polymorphisms, indels, and copy number variations in genetic data;

[0059] Biostatistical analysis used principal component analysis and cluster analysis to extract the dynamic change characteristics of thyroid function indicators and tumor markers in biochemical index data;

[0060] Time series analysis uses the autoregressive moving average model to analyze the temporal fluctuation characteristics of vital signs and medication records in clinical data.

[0061] Beneficial effects of the present invention:

[0062] The electronic medical record system for thyroid cancer based on multimodal data fusion provided by the present invention breaks through the diagnostic limitations of single-modal data by integrating multi-dimensional information such as text, ultrasound images, genes, biochemical indicators and clinical data, realizes a multi-dimensional comprehensive analysis of the disease, and significantly improves the accuracy and comprehensiveness of thyroid cancer diagnosis. The electronic medical record system for thyroid cancer uses a standardized data processing process to perform word segmentation, grayscale normalization, quality filtering and other calibrations on various types of data to ensure the consistency of data from different sources, laying a reliable foundation for subsequent analysis. The multimodal feature extraction and fusion mechanism uses natural language processing, computer vision and other technologies to deeply mine clinical information, construct single-modal prediction models such as text and images, and fuse them through a deep learning framework. It can assist doctors in obtaining more accurate pathological type judgments, treatment plan recommendations and prognosis assessments, effectively support clinical decision-making, and promote the upgrade of the electronic medical record system for thyroid cancer to intelligence and precision.

[0063] The thyroid cancer electronic medical record system introduces a defect feature quantification and processing mechanism. By constructing a simulated defect feature library and injecting multi-dimensional defects to generate enhanced training samples, the system improves its ability to handle noisy data in real clinical scenarios. When data defects are detected, the defect feature alignment unit and the hierarchical fusion unit dynamically adjust modal weights and filter noisy data. Combined with the cross-modal attention mechanism of the Transformer architecture, they mitigate the impact of data quality fluctuations on diagnosis. The cross-modal defect association graph uses the Apriori algorithm and the Pearl causal graph model to mine association rules and causal effects between defects, providing a basis for defect reasoning and weight correction for the thyroid cancer electronic medical record system while enhancing the interpretability of diagnostic results. The uncertainty quantification layer outputs probabilistic diagnostic recommendations that include defect influencing factors, quantifying the impact of data defects on diagnostic results. This provides credibility assessment and defect attribution for clinical decision-making, improving the clinical practicality of the thyroid cancer electronic medical record system and physicians' trust in diagnostic results. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a schematic diagram of a thyroid cancer electronic medical record system based on multimodal data fusion in an embodiment of the present application.

[0065] Figure 2 This is a schematic diagram of another thyroid cancer electronic medical record system based on multimodal data fusion in an embodiment of the present application.

[0066] Figure 3 This is a schematic diagram of another thyroid cancer electronic medical record system based on multimodal data fusion in an embodiment of the present application.

[0067] Figure 4This is a schematic diagram of another thyroid cancer electronic medical record system based on multimodal data fusion in an embodiment of the present application.

[0068] Figure 5 This is a schematic diagram of another thyroid cancer electronic medical record system based on multimodal data fusion in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the specific implementation manner of the present invention will be clearly and completely described below.

[0070] The present invention discloses a thyroid cancer electronic medical record system based on multimodal data fusion, which includes a multimodal data acquisition module for collecting multimodal data of thyroid cancer patients, the multimodal data including text data, ultrasound image data, gene data, biochemical indicator data and clinical data, and standardizing and calibrating the data to generate standard data;

[0071] The multimodal feature extraction module is connected to the multimodal data acquisition module. Through natural language processing, computer vision, genetic analysis, biostatistics analysis and time series analysis, it extracts semantic, structural, mutation, change and fluctuation features from each standard data.

[0072] A unimodal prediction model construction module is connected to the multimodal feature extraction module, which constructs a text prediction model based on semantic features, an image prediction model based on structural features, a molecule prediction model based on mutation features, a metabolism prediction model based on change features, and a physical sign prediction model based on fluctuation features, and outputs unimodal prediction results with text, images, molecules, metabolism, and physical signs respectively;

[0073] The multimodal fusion prediction module is connected to the single-modal prediction model construction module, fuses the single-modal prediction model based on the deep learning framework, inputs the single-modal prediction result, and outputs the multimodal fusion prediction result.

[0074] The implementation of the thyroid cancer electronic medical record system of the present invention first uses a multimodal data acquisition module to synchronously collect clinical texts, ultrasound images, genes, biochemical indicators and clinical data of thyroid cancer patients, and standardizes and calibrates various types of data to form standard data in a unified format. Subsequently, the multimodal feature extraction module extracts semantic, structural, mutation and other features from the standard data based on natural language processing, computer vision and other technologies. The single-modal prediction model construction module constructs single-modal prediction models such as text and images according to different feature types and outputs corresponding prediction results. Finally, the multimodal fusion prediction module integrates the outputs of each single-modal prediction model through a deep learning framework to generate a multimodal fusion prediction result.

[0075] The unimodal prediction model construction module uses adaptive algorithms to construct specialized models based on the semantic and structural features output by the multimodal feature extraction module. The text prediction model inputs semantic features extracted by BiLSTM into an MLP, outputting the pathology type probability through softmax. The image prediction model integrates structural features extracted by CNN into an SVM to output the risk probability of benign or malignant nodules. The molecular prediction model encodes mutation features and inputs them into a random forest to output a gene mutation risk score. The metabolic prediction model applies logistic regression to biochemical marker features after PCA dimensionality reduction to output the thyroid function status classification probability. The vital sign prediction model inputs time series features extracted by ARIMA into an LSTM to output a continuous prediction of vital sign trends. Each model achieves precise unimodal prediction through the mapping relationship between features and clinical labels.

[0076] The multimodal data acquisition module interfaces with the hospital's information system, imaging equipment, and gene sequencing platforms. It also connects to the hospital's laboratory information system to obtain test data such as thyroid function indicators and tumor markers, while simultaneously analyzing defect indicators such as test value dispersion and deviation from reference ranges. Clinical data collection integrates with the hospital's electronic medical record system and monitoring equipment to automatically acquire time-series data such as vital signs, palpation results, and medication records. The module also simultaneously evaluates the consistency of palpation results and the completeness of vital sign records. This module automates the collection of multi-source data and standardizes the format of clinical text, grayscale calibration of ultrasound images, quality filtering of genetic data, standardization of biochemical marker data, and time-stamping of clinical data. In the multimodal feature extraction module, natural language processing techniques are used to parse the semantics of lesion descriptions in text, computer vision algorithms are used to extract nodule morphology and structure from images, genetic analysis tools are used to identify mutation sites, biostatistical methods are used to analyze the dynamic changes in biochemical indicators, and time series analysis is used to identify temporal patterns in clinical data. The unimodal prediction model construction module utilizes appropriate algorithms based on various features to construct models for pathology classification and tumor benign and malignant prediction.

[0077] The multimodal fusion prediction module performs weighted fusion of single-modal prediction results through a deep learning network to output a comprehensive diagnostic conclusion. This module performs weighted fusion of the single-modal prediction results by constructing a deep learning fusion network. Specifically, the probability values, risk scores, and other results output by each single-modal prediction model are concatenated into a feature vector and input into the fusion network. Training is then used to learn the weight coefficients of different modalities in the comprehensive diagnosis. Simultaneously, the weights of each modality are dynamically adjusted based on the defect impact factors output by the uncertainty quantification layer, ultimately outputting a comprehensive diagnostic conclusion through a softmax or regression layer.

[0078] The electronic medical record system for thyroid cancer of the present invention has broken through the diagnostic limitations of single modality data through the integration and fusion of multimodal data, and can comprehensively analyze the condition from multiple dimensions such as text semantics, image structure, and gene mutations, thereby improving the accuracy and comprehensiveness of thyroid cancer diagnosis. The standardized data processing process ensures the consistency of data from different sources, providing a reliable basis for subsequent analysis; the multimodal feature extraction and fusion mechanism realizes the deep mining of clinical information, which can assist doctors in obtaining more accurate pathological type judgments, treatment plan recommendations, and prognosis assessments, effectively supporting clinical decision-making, and at the same time promoting the upgrade of the electronic medical record system for thyroid cancer to the direction of intelligence and precision. A schematic diagram of the electronic medical record system for thyroid cancer based on multimodal data fusion of the present invention is shown in FIG. Figures 1 to 5 As shown in .

[0079] As an implementation method, the multimodal data acquisition module includes:

[0080] The defect feature quantification unit is used to collect statistics on defect quantification indicators of each modal data, including:

[0081] The ambiguity of the medical history description and the coverage of symptom keywords in the text;

[0082] Noise intensity and edge clarity of ultrasound images;

[0083] Sequencing error rate and mutation site coverage of genetic data;

[0084] The degree of dispersion of test values ​​of biochemical indicators and the deviation from the reference range;

[0085] The consistency of palpation results and completeness of vital signs records in clinical data;

[0086] Based on the above quantitative indicators, a simulated defect feature library is constructed. According to the defect distribution statistics of real clinical data, multi-dimensional defects are injected into the calibration data at a ratio of 5%-30% to generate enhanced training samples.

[0087] The defect feature quantification unit statistically analyzes quality indicators of multimodal data and combines existing technologies to achieve defect quantification and generate enhanced training samples. First, natural language processing and image processing techniques are used to extract defect quantification indicators for data from different modalities, such as text and ultrasound images. A simulated defect feature library is then constructed based on these indicators. Finally, based on the defect distribution patterns of clinical data, multi-dimensional defects are injected into the calibration data at a ratio of 5%-30% using data augmentation techniques to generate enhanced training samples that reflect various data quality issues.

[0088] Text data is analyzed using natural language processing techniques to analyze medical history texts. A keyword extraction algorithm is used to assess symptom keyword coverage, and a semantic similarity algorithm is used to quantify description ambiguity, such as calculating the degree of difference by comparing with a standard medical history template. Ultrasound images are detected using image processing algorithms to detect noise levels, and edge detection techniques are used to assess lesion edge clarity, such as identifying the proportion of broken or blurred areas. Genetic data are analyzed using existing sequencing quality assessment tools to calculate error rates, and sequence alignment techniques are used to calculate mutation site coverage, such as by matching sequencing results with a reference genome for analysis. Biochemical indicators are analyzed using statistical analysis methods to calculate the degree of dispersion of test values, and deviations are assessed based on medical reference ranges, such as determining the magnitude of abnormal indicators by comparing with normal intervals. Clinical data are analyzed using consistency analysis methods to assess the consistency of palpation results, and data integrity statistical techniques are used to calculate the completeness rate of vital sign records. A simulated defect library is constructed based on a clustering algorithm, and data perturbation techniques are used to generate enhanced training samples based on the proportion of clinical defects.

[0089] The implementation method of the present invention integrates existing mature technologies to efficiently complete data defect quantification and sample enhancement. Existing algorithms in the fields of natural language processing, image processing, etc. are used to convert subjective data quality assessment into objective indicators. For example, semantic analysis is used to accurately quantify the degree of ambiguity in text descriptions, reducing human judgment bias. The multi-dimensional defects injected into the simulated defect library combined with the clinical defect distribution enable the enhanced training samples to adapt to data quality issues in real scenarios during the training phase, thereby improving the ability to handle noisy data. The enhanced samples generated at a ratio of 5%-30% strictly follow the laws of clinical data defects, ensuring that the enhanced training samples can effectively handle various data quality issues in actual applications, providing a reliable input basis for subsequent multimodal analysis, and reducing diagnostic errors caused by data defects.

[0090] As an implementation method, when the multimodal fusion prediction module does not detect defective data, it is built based on the Transformer architecture and includes:

[0091] Cross-modal attention mechanism layer: calculates the interaction weights of each modal feature;

[0092] Multi-scale feature fusion layer: stratifies and fuses single-modal results by time series, spatial location, and biological association;

[0093] Fusion decision layer: Combined with clinical guidelines, it outputs fusion results including diagnosis probability, treatment recommendations and prognosis assessment.

[0094] When the thyroid cancer electronic medical record system detects no data defects, the multimodal fusion prediction module constructs a three-layer processing system based on the Transformer architecture. The cross-modal attention mechanism layer receives single-modal prediction results such as text, images, and genes. Through the attention mechanism, it calculates the interaction weights between features from different modalities and identifies feature associations that are critical to diagnosis. The multi-scale feature fusion layer integrates disease progression information by time series, aligns lesion anatomical structures by spatial location, and connects genes and metabolic indicators by biological associations, achieving multi-dimensional feature fusion. The fusion decision layer inputs the fused features into a preconfigured clinical guideline knowledge base to generate structured outputs containing diagnostic probabilities, treatment plans, and prognostic assessments.

[0095] The cross-modal attention mechanism layer dynamically adjusts the weights of each modal feature through the self-attention mechanism, for example, enhancing the correlation strength between descriptions of irregular boundaries in clinical text and corresponding features in ultrasound images. The multi-scale feature fusion layer uses the positional encoding characteristics of the Transformer to perform time series modeling on examination data at different time points, aligns the lesion location information from images and palpation through the spatial attention mechanism, and establishes an association mapping between genetic data and biochemical indicators based on the medical knowledge graph. The fusion decision layer has a built-in rule library for thyroid cancer diagnosis and treatment guidelines, such as the TI-RADS classification standards and NCCN treatment recommendations. It maps fused features to corresponding rules and automatically generates diagnostic recommendations that meet clinical standards. For example, for specific gene mutations and imaging feature combinations, it outputs surgical type recommendations and recurrence risk assessments to assist doctors in making judgments.

[0096] The implementation method of the present invention achieves deep integration of multimodal data through the Transformer architecture, significantly improving the accuracy and clinical practicality of thyroid cancer diagnosis. The cross-modal attention mechanism can capture the complex dependencies between text, image and other features, avoiding the limitations of single modality information. The multi-scale fusion mechanism comprehensively covers time, space and biological dimensions, providing a more three-dimensional disease assessment. The combination with clinical guidelines ensures that the output results comply with medical standards, directly generates actionable diagnosis and treatment recommendations, provides doctors with efficient decision-making support, and promotes the upgrade of thyroid cancer electronic medical record systems from data recording to intelligent auxiliary diagnosis.

[0097] A common implementation of the multimodal fusion prediction module is:

[0098] The multimodal fusion prediction module is based on a deep learning framework, which can be PyTorch or TensorFlow. It builds a three-layer progressive fusion network to achieve deep integration of single-modal prediction results.

[0099] The single-modal results, such as the pathological type probability output by the text prediction model, the benign and malignant risk score of the image prediction model, and the gene mutation risk value of the molecular prediction model, are spliced ​​into a feature vector of unified dimension.

[0100] A three-layer architecture of cross-modal attention, multi-scale fusion and clinical decision-making is adopted.

[0101] Cross-modal attention layer: This layer uses a self-attention mechanism to calculate the interaction weights between features in each modality. For example, when an ultrasound image shows irregular nodule edges, the weight associated with the description of the unclear boundary in the text and the image features is automatically enhanced, suppressing low-quality modalities.

[0102] Multi-scale feature fusion layer: Features are fused layer by layer based on time series, spatial location, and biological association. For example, a biological association analysis is performed between the TSH value one week after surgery and the TERT mutation status from preoperative genetic testing to generate a comprehensive feature vector.

[0103] The fusion decision layer accesses the NCCN Thyroid Cancer Diagnosis and Treatment Guidelines rule base and maps fusion features into diagnostic conclusions. For example, if the fusion features meet the imaging TI-RADS 5 category, BRAF mutation, and elevated Tg, the output is a 90% probability of malignancy, and surgical treatment is recommended.

[0104] As an embodiment, the multimodal fusion prediction module includes a defect feature alignment unit, which is used to align the defect data through the defect feature alignment unit and fuse the defective data through the hierarchical fusion unit when the defect feature quantization unit detects that the multimodal data has defects;

[0105] The defect feature alignment unit aligns the single-modal prediction results with the simulated defect feature library in feature space, calculates the defect matching degree of multimodal data in semantic, structural, mutation, metabolic and physical dimensions, and generates defect attention weights based on the defect matching degree;

[0106] Hierarchical fusion unit, including bottom-level denoising unit and top-level prediction unit;

[0107] The noise reduction unit is used to filter defects in each modal data and adjust the input weight of each modal based on the defect matching degree;

[0108] The prediction unit uses the cross-modal attention mechanism layer of the Transformer architecture based on the aligned defect features and denoised data, and superimposes the defect attention weights for interactive calculation to generate a fused prediction result.

[0109] When the thyroid cancer electronic medical record system detects defects in multimodal data, the multimodal fusion prediction module processes the defective data through a defect feature alignment unit and a hierarchical fusion unit. The defect feature alignment unit first maps the single-modal prediction results to the simulated defect feature library in feature space, calculates the defect matching degree of each modality in semantic and structural dimensions, and then generates defect attention weights. The denoising unit of the hierarchical fusion unit adjusts the input weights of each modality based on the matching degree, filtering out noisy data. The prediction unit reuses the cross-modal attention mechanism layer of the Transformer architecture, superimposes the defect attention weights, completes the feature interaction calculation, and outputs the fused prediction result.

[0110] The defect feature alignment unit uses a feature mapping algorithm to spatially align the feature vectors of the single-modal results with the standard defect features in the simulated defect library, generates a defect matching matrix by dimension, and then generates defect attention weights through weight normalization. In the hierarchical fusion unit, the denoising unit dynamically adjusts the modal weights according to the defect matching degree. For example, it reduces the input weight of ultrasound images with high noise intensity by 10%-20%, and increases the weight of clinical data with high completeness by 10%-15%. When the prediction unit calls the cross-modal attention mechanism layer of the Transformer, the native interaction weights and the defect attention weights are superimposed in proportion to strengthen the feature interaction of reliable modalities and suppress the interference of defect modalities. The feature mapping algorithm is either a linear mapping algorithm or a metric learning algorithm.

[0111] The implementation of the present invention significantly improves the thyroid cancer electronic medical record system's ability to handle noisy data through defect feature alignment and hierarchical fusion mechanisms. The dynamic adjustment of defect attention weights can reduce the impact of data quality fluctuations on diagnosis. For example, when the error rate of genetic data sequencing is high, the fusion weights of clinical text and image features are automatically enhanced to avoid misjudgments caused by single modality defects. Using the Transformer architecture while superimposing defect compensation logic not only ensures basic fusion capabilities, but also solves data defect problems in a targeted manner, so that the model can still maintain diagnostic accuracy in real clinical scenarios. The hierarchical noise reduction and weight adjustment mechanism can quantify the impact of defects and provide a data quality reference for clinical decision-making. For example, it outputs a probabilistic suggestion that includes a 15% diagnostic confidence level affected by image noise, assisting doctors in determining whether additional examinations are needed.

[0112] As an implementation method, the training method of the hierarchical fusion unit includes:

[0113] Defects are injected into the labeled data based on the simulated defect feature library to generate multimodal sample data containing semantic loss, structural ambiguity, false negative mutations, metabolic index deviations, and incomplete vital sign records;

[0114] Multimodal sample data is used to train the hierarchical fusion unit. By calculating the difference between the defect repair results and the label data, the denoising unit weights and prediction unit parameters of the hierarchical fusion unit are iteratively adjusted until the hierarchical fusion unit's repair accuracy for defect features reaches a preset threshold.

[0115] Labeled data refers to sample data that has been annotated with a gold standard, which is a diagnosis result or clinical indicator confirmed by a doctor;

[0116] The repair accuracy is the consistency ratio between the prediction results after defect data processing and the gold standard.

[0117] The training of the hierarchical fusion unit is based on a simulated defect feature library, which injects defects into gold-standard labeled data to construct a multimodal sample set. First, various defect patterns, such as semantic loss and structural ambiguity, are extracted from the simulated defect feature library and injected into the labeled data at actual clinical proportions to generate training samples containing different defect combinations. These training samples are then fed into the hierarchical fusion unit, which predicts the defect data and compares it with the original gold-standard to calculate the difference in defect repair. Finally, an iterative optimization algorithm, such as stochastic gradient descent, is used to adjust the denoising unit weights and prediction unit parameters until the hierarchical fusion unit's repair accuracy for the defect features reaches a clinically acceptable threshold.

[0118] Defect repair differences are calculated based on cross-entropy loss or mean squared error. Multimodal samples with defects are fed into a hierarchical fusion unit, which outputs predictions that are compared modally with the original gold standard labels. For example, for semantically missing defects in the text modality, the cosine distance between the predicted text feature vector and the gold standard feature vector is calculated. For structural ambiguity defects in the imaging modality, the IoU difference between the segmentation result and the gold standard lesion boundary is calculated. Finally, the weighted sum of the differences across modalities yields the overall repair difference.

[0119] Defect sample generation: Defect patterns are randomly extracted from the simulated defect feature library. For example, the coverage of text keywords is reduced by 20%, and Gaussian noise is added to images. These patterns are injected into the labeled data at a ratio of 5%-30% to generate training samples containing multi-dimensional defects such as semantic ambiguity and structural noise.

[0120] During model training and optimization, defect samples are input into the hierarchical fusion unit. The denoising unit adjusts the weights of each modality input based on the defect matching degree. The prediction unit generates a fusion prediction result through the cross-modal attention mechanism of the Transformer architecture. The prediction result is compared with the original gold standard label, and the difference in defect repair is calculated, including the deviation between the predicted pathology type and the doctor's annotation result. The weight distribution strategy of the denoising unit and the Transformer parameters of the prediction unit are iteratively adjusted through the backpropagation algorithm to gradually improve the model's ability to process defect data. The training termination condition is to continue training until the model's repair accuracy for defect features, that is, the consistency ratio between the prediction result and the gold standard, reaches a preset threshold, such as ≥85%, to ensure that the model can effectively handle various data defects in clinical applications.

[0121] The training method of the present invention significantly improves the practicality of the model by simulating data defects in a real clinical environment. The defect injection strategy strictly follows the defect distribution law of real data. For example, the error rate of gene sequencing is often 10%-20%, so that the model has the ability to handle common data quality problems during the training stage. By calculating the difference between the prediction results and the gold standard, the model can learn the repair rules of defect features in a targeted manner, such as edge enhancement of noisy images and semantic completion of missing text to improve diagnostic accuracy. The repair accuracy rate is used as a training termination indicator, which is directly related to the reliability of clinical diagnosis, ensuring that the model's processing of defective data in actual applications meets medical standards, reducing the risk of misdiagnosis due to data quality, and providing a solid training foundation for multimodal fusion prediction.

[0122] As an implementation method, the defect feature alignment unit generates attention weights based on the defect matching degree, including:

[0123] Defect severity assessment, real-time calculation of each modal defect index;

[0124] When the defect index of a certain modality exceeds the preset threshold, the fusion weight of other modalities will be automatically increased by at least 10-30%.

[0125] The Defect Feature Alignment Unit dynamically adjusts the multimodal fusion weights by assessing the defect severity of each modal data in real time. First, the defect index for each modality is calculated based on the quantitative indicators output by the Defect Feature Quantization Unit. Each modality's defect index is then compared with a preset threshold. If the defect index of a modality exceeds the threshold, the weight of the other modalities in the fusion process is automatically increased to compensate for the negative impact of the defect modality on the overall diagnosis and ensure the reliability of the final prediction result. Defect Index = Noise Intensity × Weight Coefficient.

[0126] Defect severity assessment: For text data, the text defect index is calculated based on keyword coverage and fuzziness; for ultrasound images, the image defect index is generated based on noise intensity and edge clarity; for genetic data, the gene defect index is evaluated based on sequencing error rate and mutation site coverage, and so on to calculate the defect index of biochemical indicators and clinical data.

[0127] Dynamic weight adjustment mechanism to preset clinically acceptable defect thresholds for each modality;

[0128] When the defect index of a certain modality exceeds the threshold, the fusion weight of other modalities is increased according to the preset ratio;

[0129] The degree of improvement is set differently according to the impact of the defective modality on diagnosis. For example, the weight of the imaging modality, which plays a key role in diagnosis, is increased by 30% when the genetic data defects exceed the limit.

[0130] The implementation method of the present invention significantly improves the fault tolerance of the thyroid cancer electronic medical record system to defective data through adaptive weight adjustment.

[0131] Dynamically increase the weight of reliable modalities to effectively offset the interference of defective data. For example, when the quality of genetic sequencing is poor, it enhances the fusion of ultrasound images and clinical text to maintain diagnostic accuracy. Clinical safety is enhanced. Preset thresholds and fixed increase ratios ensure that the weight adjustment range is controllable, avoiding diagnostic bias caused by over-reliance on a single modality, and meeting the robustness requirements of medical decision-making. Quantified quality feedback and real-time calculation of the defect index provide clinicians with a data quality reference. For example, in the output results, the quality of genetic data is marked as general, and it is recommended to combine the imaging results for comprehensive judgment, assisting doctors in interpreting model predictions more accurately, and improving the clinical practicality of the thyroid cancer electronic medical record system.

[0132] As an implementation method, the multimodal feature extraction module is also used to construct a cross-modal defect correlation map;

[0133] Based on the simulated defect feature library, the Apriori algorithm or equivalent algorithm is used to discover strong association rules between defects in each modality, and an association rule library is established to record the association confidence, support, and improvement indicators;

[0134] Based on the Pearl causal graph model or equivalent methods, a causal relationship network between defects is constructed, the average causal effect and natural direct effect are calculated, and a causal effect knowledge base is established to quantify the impact of each defect on the prediction results. Equivalent methods include structural causal models, Do-Calculus causal inference framework, causal Bayesian neural networks, etc.

[0135] The association rule base and the causal effect knowledge base are integrated into a cross-modal defect association graph, where:

[0136] The nodes represent each modal defect type;

[0137] Edges represent the strength of association or causal effect value between defects;

[0138] The parameters of the cross-modal defect association map are updated regularly based on new sample data, and an incremental learning algorithm is used to ensure the timeliness of the cross-modal defect association map; the incremental learning algorithm includes online gradient descent, incremental support vector machine and streaming decision tree.

[0139] The correlation strength value in the graph is used as a correction factor for the cross-modal attention mechanism;

[0140] The defect feature alignment unit adjusts the attention weights according to the correction factors in the association map.

[0141] The multimodal feature extraction module enhances the thyroid cancer electronic medical record system's ability to process defect data by constructing a cross-modal defect association graph. First, based on a simulated defect feature library, the Apriori algorithm is used to discover strong association rules between defects in each modality. Indicators such as association confidence are recorded to form an association rule library. The Pearl causal graph model is then used to analyze the causal relationships between defects, quantifying the impact of each defect on the prediction results and establishing a causal effect knowledge base. Finally, the two knowledge bases are fused into a cross-modal defect association graph, where nodes represent defect types and edges represent association strengths or causal effect values. Incremental learning is used to regularly update the cross-modal defect association graph parameters, which are used to modify the weight distribution of the cross-modal attention mechanism.

[0142] The Apriori algorithm was used to mine frequent itemsets from a simulated defect feature library. Strong association rules were discovered, such as the rule that when text keyword coverage is less than 60%, image edge clarity decreases by 15%. The confidence, support, and lift of the rule were recorded. If the frequency of both a gene sequencing error rate >15% and a biochemical indicator dispersion >20% co-occurring is 70%, and the probability of the latter occurring when the former occurs is 85%, then this rule was stored in the association rule library. A defect causal network was constructed using the Pearl causal graph model. For example, an increase in image noise intensity was determined to directly lead to a decrease in edge clarity, and the average causal effect was calculated, such as a 5% decrease in clarity for every 10 dB increase in noise. The impact of each defect on the final prediction result was quantified, such as a 12% decrease in diagnostic accuracy due to missing text semantics. This formed a causal effect knowledge base. Association rules and causal effects were integrated into a cross-modal defect association graph, with nodes representing defect types and edges representing association strengths or causal effect values. When calculating weights, the cross-modal attention mechanism introduces the correlation strength in the cross-modal defect association map as a correction factor. For example, when high text ambiguity is detected, the attention weights for other reliable modalities are enhanced based on the correlation strength between text ambiguity and image noise in the cross-modal defect association map.

[0143] The cross-modal defect association map of the present invention reveals the correlation and causal relationship between defects, which enables the thyroid cancer electronic medical record system to predict the possible existence of other defects and adjust the processing strategy in advance when a single defect is detected. For example, when the text description is found to be vague, it automatically increases the reliance on imaging and genetic data. The correlation strength serves as a correction factor for the attention weight, allowing the model to more reasonably distribute the fusion weights of each modality, reduce the interference of defect data on diagnosis, and improve diagnostic accuracy in complex scenarios. The causal effect knowledge base in the cross-modal defect association map provides a quantitative explanation for the diagnostic results, assisting doctors in understanding the model's decision-making logic and improving clinical trust.

[0144] As an implementation method, the multimodal fusion prediction module includes:

[0145] Defect perception reasoning engine, which infers and completes defect features based on cross-modal defect association graphs;

[0146] The uncertainty quantification layer calculates the confidence interval of the prediction results through Monte Carlo dropout or Bayesian neural network, and outputs probabilistic diagnostic recommendations containing defect influencing factors that are used to quantify the impact of data defects on diagnostic results, providing credibility assessment and defect attribution reference for clinical decision-making.

[0147] The defect impact factor is generated through Monte Carlo dropout or uncertainty estimation using a Bayesian neural network. Multiple inferences are performed on the defective data to obtain multiple distributions of prediction results. The defect impact factor quantifies the difference in the dispersion of this distribution compared to the original, defect-free data distribution. For example, if a genetic data defect causes the variance of the prediction result to increase by 20%, the defect impact factor is recorded as 20%, indicating that the defect increases the diagnostic uncertainty by 20%. The confidence interval is calculated based on the distribution range of the prediction results. The defect impact factor acts as a correction factor to adjust the width of the confidence interval. When the defect impact factor is 20%, the confidence interval width is correspondingly expanded by 20%, which intuitively reflects the degree to which the data defect weakens the diagnostic confidence. The final probabilistic diagnostic recommendation output will include both the corrected confidence interval and the defect impact factor, providing a data quality reference for clinical decision-making.

[0148] The multimodal fusion prediction module implements intelligent processing of defect data and risk quantification through a defect-aware inference engine and an uncertainty quantification layer. The defect-aware inference engine infers and completes detected defect features based on a cross-modal defect association map. For example, when text descriptions are ambiguous, the association rules between text ambiguity and image noise in the cross-modal defect association map are leveraged to enhance reliance on other reliable modalities. The uncertainty quantification layer calculates confidence intervals for prediction results through Monte Carlo dropout or Bayesian neural networks, generating probabilistic diagnostic recommendations that include defect influencing factors and quantifying the impact of data defects on the final results.

[0149] When the thyroid cancer electronic medical record system detects a defect in a certain modality, such as a high error rate in gene sequencing, the inference engine queries the cross-modal defect association map to obtain the correlation strength and causal effect of the defect with other modalities; according to the cross-modal defect association map information, the attention weight of other modalities is adjusted. For example, when the image noise intensity is high, the fusion weight of genetic data and clinical text is increased, and association rules are used to infer and complete the defect data, such as inferring the characteristics of nodules that may exist but are not clearly imaged through the symptom description in the text.

[0150] The uncertainty quantization layer uses Monte Carlo dropout technology to randomly discard neurons multiple times during the model inference stage to generate multiple prediction results, and calculate the confidence interval based on the result distribution; combined with the output of the defect feature quantization unit, it calculates the impact factor of each defect on the diagnosis result; and integrates the defect impact factor and confidence interval into the diagnostic recommendation. For example, the output probability of thyroid cancer malignancy is 75%, which is mainly affected by image noise.

[0151] The implementation of the present invention significantly improves the clinical practicality and decision-making support capabilities of the thyroid cancer electronic medical record system. The defect-aware reasoning engine uses a cross-modal defect association map to achieve cross-modal knowledge transfer. When the data quality of a certain modality is poor, the diagnostic accuracy is maintained through the complementary information of other modalities. For example, when the ultrasound image is blurred, the reliance on genetic test results is strengthened. The confidence interval and defect impact factor provided by the uncertainty quantification layer enable doctors to intuitively understand the extent of the impact of data quality on the diagnostic results. For example, due to incomplete text descriptions, further inspections are recommended to confirm the conclusions. Probabilistic diagnostic suggestions are combined with defect attribution analysis to provide doctors with a transparent decision-making basis. For example, it is clearly stated that this diagnosis is mainly affected by errors in genetic data sequencing and has a medium degree of credibility, helping doctors to more scientifically weigh the diagnostic results and subsequent treatment plans.

[0152] As an embodiment, the standardized calibration of data by the multimodal data acquisition module includes:

[0153] Perform word segmentation, entity recognition, and medical terminology standardization on text;

[0154] Perform grayscale normalization and lesion labeling on ultrasound images;

[0155] Perform sequencing quality filtering and mutation annotation on gene data;

[0156] Perform outlier detection and normalization on biochemical indicators;

[0157] Clinical data were time aligned and missing values ​​interpolated.

[0158] The multimodal data acquisition module improves data quality and consistency through standardized calibration. Natural language processing techniques are used for word segmentation, entity recognition, and terminology standardization of text data to unify medical descriptions. Ultrasound images are grayscale normalized and lesion annotation is performed to eliminate device variability. Sequencing quality filtering and mutation annotation are performed on genetic data to ensure accurate interpretation of variants. Outlier detection and normalization are performed on biochemical indicators to balance measurement differences between different testing devices. Time alignment and missing value interpolation are performed on clinical data to integrate diagnostic and treatment information from different time points.

[0159] Text processing can use a medical-specific word segmenter to segment clinical text, identify entities such as diseases and symptoms through a named entity recognition model, and then map non-standard terms to standard concepts using the UMLS medical terminology database. Image processing can perform histogram equalization on ultrasound images to achieve grayscale normalization, eliminating brightness differences between different devices; and use deep learning models to automatically annotate thyroid nodule boundaries and features. Gene processing can filter bases with sequencing quality below Q30, perform variant detection using tools such as GATK, and annotate pathogenicity of mutation sites using the ClinVar database. Biochemical marker processing can detect and correct outliers using the IQR method, normalize marker values ​​to the [0, 1] range, and eliminate measurement bias between different testing instruments. Clinical data processing can align clinical records from different sources based on visit timestamps and fill in missing vital signs using multiple interpolation methods.

[0160] Standardized calibration significantly improves data quality. Unifying medical terminology, image grayscale, and biochemical index scales makes multimodal data comparable. For example, thyroid images from different hospitals are processed into a unified grayscale standard to facilitate subsequent feature extraction and fusion. Filtering low-quality data, correcting outliers, and reducing the interference of noise on the prediction model. For example, outlier detection can avoid misdiagnosis due to test errors. Time alignment and missing value interpolation make clinical data more complete. For example, preoperative and postoperative examination records are linked in time to support dynamic analysis of the disease course. Standardized terminology makes the output of the thyroid cancer electronic medical record system more in line with doctors' habits, such as directly displaying standard disease names instead of vague expressions.

[0161] As an implementation method, in the multimodal feature extraction module:

[0162] Natural language processing uses a bidirectional long short-term memory network to extract semantic features of lesion descriptions, pathological types, and treatment plans from texts;

[0163] Computer vision uses convolutional neural networks to extract structural features of nodule boundaries, echo intensity, and blood flow distribution in ultrasound images;

[0164] Genetic analysis uses variant detection algorithms to identify mutational signatures of single nucleotide polymorphisms, indels, and copy number variations in genetic data;

[0165] Biostatistical analysis used principal component analysis and cluster analysis to extract the dynamic change characteristics of thyroid function indicators and tumor markers in biochemical index data;

[0166] Time series analysis uses the autoregressive moving average model to analyze the temporal fluctuation characteristics of vital signs and medication records in clinical data.

[0167] The multimodal feature extraction module extracts key features from various data types using specialized algorithms. Natural language processing uses a bidirectional long-short-term memory network to process text, capturing the temporal dependencies of semantic information such as lesion descriptions and pathological types. Computer vision uses convolutional neural networks to analyze ultrasound images, extracting spatial structural features such as nodule boundaries and echo intensity. Genetic analysis uses variant detection algorithms to identify mutations such as single nucleotide polymorphisms. Biostatistical analysis utilizes principal component analysis and cluster analysis to explore the dynamic changes in biochemical indicators. Time series analysis uses an autoregressive sliding average model to analyze the temporal fluctuations of vital signs and medication records.

[0168] Text feature extraction performs word embedding on the standardized text and inputs it into the BiLSTM network. The bidirectional hidden layer captures the semantic association between the context and the surrounding text, and extracts feature vectors of keywords such as unclear boundaries and papillary carcinoma.

[0169] Image feature extraction: The preprocessed ultrasound image is input into CNN, and features such as nodule morphology, internal echo and blood flow signal distribution are extracted through the convolution layer. After dimensionality reduction through the pooling layer, a structured feature vector is output.

[0170] After quality control of the sequencing data, gene feature extraction was performed using tools such as GATK to detect SNV, InDel and other variations. The mutation sites were annotated using databases such as dbSNP to extract the driver gene mutation features related to thyroid cancer.

[0171] Biochemical index feature extraction: PCA dimensionality reduction was performed on the normalized thyroid function indicators and tumor markers to retain the main variation components, and then K-means clustering was used to identify abnormal change patterns.

[0172] Clinical data feature extraction converts time-aligned vital signs and medication records into time series, uses the ARIMA model to fit data trends, and extracts periodic fluctuation features.

[0173] As an implementation method, the single-modal prediction model construction module may further include a BiLSTM-CRF-based text prediction model for predicting pathological types, building a benign and malignant lesion classifier and a diagnostic term association model;

[0174] A 3D-CNN-based image prediction model is used to predict benign and malignant tumors, and to build a nodule morphology scoring and ultrasound feature quantitative analysis model;

[0175] A molecular prediction model based on random forests is used to predict sensitivity to targeted therapy and build driver gene identification and sensitivity prediction models;

[0176] A metabolic prediction model based on a gradient boosting tree is used to predict the risk of postoperative complications and to build a thyroid function assessment and drug metabolism kinetics model;

[0177] Based on the physical sign prediction model with fluctuation characteristics, a perioperative risk warning and complication prediction model is constructed.

[0178] As an embodiment, the thyroid cancer electronic medical record system of the present invention can also provide data protection. Data protection includes anonymization and desensitization of patient personal information and medical data. Full-link encryption ensures data confidentiality during transmission and storage, preventing leakage or tampering. Security audits and exception monitoring record data operation logs and detect illegal access or data leakage risks in real time. Transmission encryption uses the TLS 1.3 protocol to encrypt multimodal data transmitted between hospital information systems, imaging equipment, and electronic medical record systems, ensuring that data is not intercepted or tampered with by man-in-the-middle attacks. For large file data such as ultrasound images, the AES-256 symmetric encryption algorithm is used for block transmission, combined with RSA asymmetric encryption for key transmission, improving transmission efficiency and security. Storage encryption uses field-level encryption at the database level for sensitive fields such as patient basic information and raw gene sequencing data, encrypted and stored using the national secret SM4 algorithm. At the file system level, full-file encryption is used for binary files such as ultrasound images and pathology slides, and access control lists are used to restrict file read permissions. Key management generates and manages encryption keys through a hardware security module, achieving separate storage of keys and data, and regularly rotating keys.

[0179] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A thyroid cancer electronic medical record system based on multimodal data fusion, characterized in that: include: The multimodal data acquisition module is used to collect multimodal data from thyroid cancer patients. The multimodal data includes text data, ultrasound image data, genetic data, biochemical indicator data, and clinical data, and standardize and calibrate the data to generate standard data. The multimodal feature extraction module is connected to the multimodal data acquisition module. Through natural language processing, computer vision, genetic analysis, biostatistics analysis and time series analysis, it extracts semantic, structural, mutation, change and fluctuation features from each standard data. A unimodal prediction model construction module is connected to the multimodal feature extraction module, which constructs a text prediction model based on semantic features, an image prediction model based on structural features, a molecule prediction model based on mutation features, a metabolism prediction model based on change features, and a physical sign prediction model based on fluctuation features, and outputs unimodal prediction results with text, images, molecules, metabolism, and physical signs respectively; The multimodal fusion prediction module is connected to the single-modal prediction model construction module, fuses the single-modal prediction model based on the deep learning framework, inputs the single-modal prediction result, and outputs the multimodal fusion prediction result.

2. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 1, characterized in that: The multimodal data acquisition module includes: The defect feature quantification unit is used to collect statistics on defect quantification indicators of each modal data, including: The ambiguity of the medical history description and the coverage of symptom keywords in the text; Noise intensity and edge clarity of ultrasound images; Sequencing error rate and mutation site coverage of genetic data; The degree of dispersion of test values ​​of biochemical indicators and the deviation from the reference range; The consistency of palpation results and completeness of vital signs records in clinical data; Based on the above quantitative indicators, a simulated defect feature library is constructed. According to the defect distribution statistics of real clinical data, multi-dimensional defects are injected into the calibration data at a ratio of 5%-30% to generate enhanced training samples.

3. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 2, characterized in that: When the multimodal fusion prediction module does not detect defective data, it is built on the Transformer architecture and includes: Cross-modal attention mechanism layer: calculates the interaction weights of each modal feature; Multi-scale feature fusion layer: stratifies and fuses single-modal results by time series, spatial location, and biological association; Fusion decision layer: Combined with clinical guidelines, it outputs fusion results including diagnosis probability, treatment recommendations and prognosis assessment.

4. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 3, characterized in that: The multimodal fusion prediction module includes a defect feature alignment unit, which is used to align the defect features through the defect feature alignment unit and fuse the defective data through the hierarchical fusion unit when the defect feature quantification unit detects that the multimodal data has defects; The defect feature alignment unit aligns the single-modal prediction results with the simulated defect feature library in feature space, calculates the defect matching degree of multimodal data in semantic, structural, mutation, metabolic and physical dimensions, and generates defect attention weights based on the defect matching degree; Hierarchical fusion unit, including bottom-level denoising unit and top-level prediction unit; The noise reduction unit is used to filter defects in each modal data and adjust the input weight of each modal based on the defect matching degree; The prediction unit uses the cross-modal attention mechanism layer of the Transformer architecture based on the aligned defect features and denoised data, and superimposes the defect attention weights for interactive calculation to generate a fused prediction result.

5. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 4, characterized in that: The training methods for the layered fusion unit include: Defects are injected into the labeled data based on the simulated defect feature library to generate multimodal sample data containing semantic loss, structural ambiguity, false negative mutations, metabolic index deviations, and incomplete vital sign records; Multimodal sample data is used to train the hierarchical fusion unit. By calculating the difference between the defect repair results and the label data, the denoising unit weights and prediction unit parameters of the hierarchical fusion unit are iteratively adjusted until the hierarchical fusion unit's repair accuracy for defect features reaches a preset threshold. Labeled data refers to sample data that has been annotated with a gold standard, which is a diagnosis result or clinical indicator confirmed by a doctor; The repair accuracy is the consistency ratio between the prediction result after defect data processing and the gold standard.

6. According to the thyroid cancer electronic medical record system based on multimodal data fusion as described in claim 4, Its characteristics are: The defect feature alignment unit generates attention weights based on the defect matching degree, including: Defect severity assessment, real-time calculation of each modal defect index; When the defect index of a certain modality exceeds the preset threshold, the fusion weight of other modalities will be automatically increased by at least 10-30%.

7. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 4, characterized in that: The multimodal feature extraction module is also used to construct a cross-modal defect correlation map; Based on the simulated defect feature library, the Apriori algorithm or equivalent algorithm is used to discover strong association rules between defects in each modality, and an association rule library is established to record the association confidence, support, and improvement indicators; Based on the Pearl causal graph model or equivalent methods, construct a causal relationship network between defects, calculate the average causal effect and natural direct effect, and establish a causal effect knowledge base to quantify the impact of each defect on the prediction results; The association rule base and the causal effect knowledge base are integrated into a cross-modal defect association graph, where: The nodes represent each modal defect type; Edges represent the strength of association or causal effect value between defects; Regularly update cross-modal defect association map parameters based on new sample data, and use incremental learning algorithms to ensure the timeliness of cross-modal defect association maps; The correlation strength value in the graph is used as a correction factor for the cross-modal attention mechanism; The defect feature alignment unit adjusts the attention weights according to the correction factors in the association map.

8. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 7, characterized in that: The multimodal fusion prediction module includes: Defect perception reasoning engine, which infers and completes defect features based on cross-modal defect association graphs; The uncertainty quantification layer calculates the confidence interval of the prediction results through Monte Carlo dropout or Bayesian neural network, and outputs probabilistic diagnostic recommendations containing defect influencing factors that are used to quantify the impact of data defects on diagnostic results, providing credibility assessment and defect attribution reference for clinical decision-making.

9. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 1, characterized in that: The standardized calibration of data by the multimodal data acquisition module includes: Perform word segmentation, entity recognition, and medical terminology standardization on text; Perform grayscale normalization and lesion labeling on ultrasound images; Perform sequencing quality filtering and mutation annotation on gene data; Perform outlier detection and normalization on biochemical indicators; Clinical data were time aligned and missing values ​​interpolated.

10. The thyroid cancer electronic medical record system based on multimodal data fusion according to claim 1, characterized in that: In the multimodal feature extraction module: Natural language processing uses a bidirectional long short-term memory network to extract semantic features of lesion descriptions, pathological types, and treatment plans from texts; Computer vision uses convolutional neural networks to extract structural features of nodule boundaries, echo intensity, and blood flow distribution in ultrasound images; Genetic analysis uses variant detection algorithms to identify mutational signatures of single nucleotide polymorphisms, indels, and copy number variations in genetic data; Biostatistical analysis used principal component analysis and cluster analysis to extract the dynamic change characteristics of thyroid function indicators and tumor markers in biochemical index data; Time series analysis uses the autoregressive moving average model to analyze the temporal fluctuation characteristics of vital signs and medication records in clinical data.

Images