A remote medical AI assistance method and system based on multi-modal data fusion

Abstract

The application discloses a kind of remote medical AI auxiliary method and system based on multi-modal data fusion, belong to remote medical AI auxiliary technical field.The method first obtains continuous physiological signal, medical image and other multi-modal medical data, after synchronous alignment, quality control, feature extraction and modal embedding standardization processing, using hierarchical adaptive fusion framework to modal embedding is fused, in combination with task adaptation strategy to generate diagnosis related results, finally through explainability analysis and uncertainty estimation output result to support clinical application.System includes data acquisition, processing, multi-modal fusion, result analysis, man-machine interaction and safe and legal block, with privacy protection and edge-cloud collaborative deployment.This scheme fully exploits the complementary value of multi-modal data, improves the accuracy and stability of diagnosis results, enhances clinical explainability and adaptability, safeguards data security and compliance, and provides reliable technical support for remote medical treatment.

Description

Technical Field

[0001] This invention relates to the field of AI-assisted telemedicine technology, and in particular to a remote medical AI-assisted method and system based on multimodal data fusion. Background Technology

[0002] Telemedicine, as a crucial direction for the digital transformation of the healthcare industry, has seen continuous advancements in technological innovation and clinical application in recent years, becoming a key path to optimize the allocation of medical resources and improve the accessibility of healthcare services. With the widespread adoption of wearable devices, portable diagnostic instruments, and medical imaging systems, as well as the rapid development of big data and artificial intelligence technologies, AI-assisted diagnosis in telemedicine is gradually evolving from single-modal applications to multi-modal data fusion. Currently, the types of multimodal medical data are increasingly diverse, encompassing continuous physiological signals, medical images, laboratory test results, medical record texts, behavioral signals, and more. Data collection has formed a multi-terminal collaborative structure involving patients, remote treatment centers, and hospitals. At the technological application level, models such as deep learning, graph neural networks, and Transformers are widely used for feature extraction and inference of medical data. The combination of cloud deployment and edge computing provides more possibilities for data processing in remote scenarios. Multimodal fusion has been explored in both academic research and clinical practice, driving telemedicine towards greater precision and efficiency. Simultaneously, the healthcare industry's increasing emphasis on cross-institutional collaboration and data security compliance has laid the foundation for the standardized application of AI-assisted technologies in telemedicine.

[0003] Despite significant progress in AI-assisted telemedicine technology, existing technologies still face numerous challenges in practical applications. In multimodal data processing, medical data from different sources and of different types exhibit significant differences in structure, format, and acquisition frequency. Existing technologies often fail to achieve efficient synchronization, alignment, and standardized processing, hindering the full realization of the complementary value of the data. Regarding data fusion, most technologies employ single or simple splicing methods, lacking a fusion framework adaptable to varying data quality and scenario requirements, thus limiting the accuracy and stability of diagnostic results. At the level of result output and clinical application, existing AI models often output diagnostic results in a "black box" manner, lacking interpretability analysis of the basis for result generation and failing to effectively estimate the uncertainty of diagnostic results, reducing clinicians' trust in and willingness to use the models. Furthermore, existing technologies are inadequate in multimodal data quality control and the targeted nature of feature extraction, failing to achieve end-to-end optimization from data acquisition to result application, and thus failing to fully meet the actual needs of accurate diagnosis and clinical adaptation in telemedicine scenarios. These issues all constrain the further promotion and in-depth application of AI-assisted telemedicine technology. Summary of the Invention

[0004] The purpose of this invention is to overcome one or more shortcomings of the prior art and provide a remote medical AI-assisted method and system based on multimodal data fusion.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A remote medical AI-assisted method based on multimodal data fusion is provided, which includes the following steps:

[0007] S1. Acquire multimodal medical data, which includes continuous physiological signals, medical images, laboratory test results, medical record texts, and behavioral signals;

[0008] S2. Perform synchronization alignment, quality control, feature extraction, and modality embedding standardization on the acquired multimodal medical data;

[0009] S3. A hierarchical adaptive fusion framework is used to fuse the standardized modal embeddings, and diagnostic results are generated by combining the task adaptation strategy.

[0010] S4. Perform interpretability analysis and uncertainty estimation on the generated diagnostic results, and output the analysis results to support clinical applications.

[0011] Furthermore, step S2 includes the following sub-steps:

[0012] S2.1. Synchronize and align multimodal medical data with timestamps according to event windows, and adopt nearest neighbor or semantic alignment strategies for multimodal medical data without precise timestamps;

[0013] S2.2. Perform noise detection and quality assessment on the medical data for each modality, and mark low-quality or unusable modal data;

[0014] S2.3. Feature extraction is performed for different types of modal data. For signal modal data, wavelet transform, time-spectrum graph or one-dimensional convolution feature extraction is used. For image modal data, preprocessing, local contrast enhancement combined with convolutional network or visual Transformer feature extraction is used. For text modal data, sentence segmentation, entity extraction combined with encoder feature extraction is used. For structured numerical data, standardization, normalization and statistical feature extraction are used.

[0015] S2.4. Map the features of each modality to an embedding space of a unified dimension and calculate the quality weights corresponding to each modality.

[0016] Furthermore, step S3 includes the following sub-steps:

[0017] S3.1. Perform cross-attention calculation on all available modal embeddings to obtain the intermodal interaction coefficients. The calculation formula is as follows:

[0018] ;

[0019] in, Let be the interaction coefficient between the i-th mode and the j-th mode. and This is the weight matrix. and These are embedding vectors for different modalities. For the embedded vector dimension, For activation functions;

[0020] S3.2. Based on the mutual influence coefficient, the modal embeddings are weighted and summed to obtain the interaction-enhanced embeddings;

[0021] S3.3. Combine quality weights to perform secondary weighting on the interaction enhancement embedding to obtain preliminary fusion results.

[0022] Furthermore, step S3 includes the following sub-steps:

[0023] S3.1. Introduce a gating network and calculate the fusion coefficient of each modality by combining the quality weights, historical reliability, and scene information of each modality. The calculation formula is as follows:

[0024] ;

[0025] in, For modal fusion coefficients, For activation function, For the gated weight matrix, to For the mass weights of each mode, For scene vectors;

[0026] S3.2. Based on the fusion coefficient, perform weighted filtering on the standardized modal embeddings and retain the embedding information of high-contribution modes;

[0027] S3.3. Combine the selected modal embeddings with the task-specific header to generate diagnostic-related results.

[0028] Furthermore, step S3 also includes: splicing and fusing the weighted interaction enhancement embeddings, calculated using the following formula:

[0029] ;

[0030] in, For the final fusion representation, For splicing operations, to The fusion coefficients for each mode are given. to The embedding vectors for each modality are used; the model is optimized based on the final fusion representation and the multi-task learning loss function to obtain diagnostic results.

[0031] Furthermore, step S2 also includes: performing privacy-preserving preprocessing on the multimodal medical data, the privacy-preserving preprocessing including de-identification, sealing and encapsulation, and differential privacy noise injection; the preprocessed multimodal medical data is transmitted through a secure channel, the transmission process adopts a message queue mechanism, supports breakpoint resume, fragmented upload and priority transmission, among which alarm-related data is transmitted first.

[0032] Furthermore, the feature extraction in step S2 adopts an edge-cloud collaborative approach: a lightweight model is used at the edge to perform preliminary feature extraction and anomaly screening on multimodal medical data, while a high-performance model is used in the cloud for deep feature extraction; the data type to be uploaded is selected based on real-time bandwidth and latency, and the data types to be uploaded include raw data, compressed representation, and modal summary.

[0033] Furthermore, the hierarchical adaptive fusion framework in step S3 is trained through federated multimodal learning: different institutions retain local multimodal medical data and only exchange model updates or modal feature summaries during training; a modal feature matcher is introduced to share aligned embedding distribution statistics, and differential privacy or secure multi-party computation is used to encrypt key aggregation steps during the training process.

[0034] Furthermore, after outputting the analysis results in step S4, the process also includes: receiving clinical feedback information and using it as training data to input into the hierarchical adaptive fusion framework; fine-tuning the model in the framework online or offline, while recording the data access process and feedback process through blockchain to form a continuous learning loop.

[0035] A remote medical AI-assisted system based on multimodal data fusion is provided. The system includes a data acquisition module, a data processing module, a multimodal fusion module, a result analysis module, a human-computer interaction module, and a security and compliance module.

[0036] The data acquisition module is used to acquire multimodal medical data composed of continuous physiological signals, medical images, laboratory test results, medical record text, and behavioral signals. The data processing module is used to perform synchronization alignment, quality control, feature extraction, modality embedding standardization, and privacy protection preprocessing on the multimodal medical data. The multimodal fusion module is used to fuse the processed modal data using a hierarchical adaptive fusion framework and complete model training through federated multimodal learning. The result parsing module is used to perform interpretability analysis and uncertainty estimation on the diagnostic results generated by fusion. The human-computer interaction module is used to output analysis results and receive clinical feedback information. The security and compliance module runs through all modules and provides audit logs, access control, differential privacy logs, and compliance reporting interfaces.

[0037] The beneficial effects of this invention are:

[0038] (1) Integrate multimodal medical data and perform synchronous alignment, feature extraction and hierarchical adaptive fusion processing to fully explore the complementary value of data, improve the accuracy and stability of diagnostic results, and provide reliable support for clinical applications;

[0039] (2) Relying on privacy protection preprocessing, edge-cloud collaborative deployment and bandwidth adaptive transmission strategy, data security and compliance are guaranteed, while adapting to the network characteristics of remote scenarios and taking into account the needs of real-time response and in-depth processing.

[0040] (3) By leveraging interpretability analysis, uncertainty estimation and continuous learning driven by clinical feedback, the diagnostic basis becomes clear and verifiable, promoting dynamic optimization of the model and improving the clinical adaptability and reliability of the system. Attached Figure Description

[0041] Figure 1 A flowchart illustrating the specific steps of a remote medical AI-assisted method based on multimodal data fusion;

[0042] Figure 2 A flowchart illustrating the steps of a remote medical AI-assisted method based on multimodal data fusion is provided for this embodiment.

[0043] Figure 3 This is a structural diagram of a remote medical AI-assisted system based on multimodal data fusion, provided as an example. Detailed Implementation

[0044] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Example 1

[0046] See Figure 1 This paper presents a remote medical AI-assisted method based on multimodal data fusion, which includes the following steps:

[0047] S1. Acquire multimodal medical data, which includes continuous physiological signals, medical images, laboratory test results, medical record texts, and behavioral signals;

[0048] S2. Perform synchronization alignment, quality control, feature extraction, and modality embedding standardization on the acquired multimodal medical data;

[0049] S3. A hierarchical adaptive fusion framework is used to fuse the standardized modal embeddings, and diagnostic results are generated by combining the task adaptation strategy.

[0050] S4. Perform interpretability analysis and uncertainty estimation on the generated diagnostic results, and output the analysis results to support clinical applications.

[0051] Step S2 includes the following sub-steps:

[0052] S2.1. Synchronize and align multimodal medical data with timestamps according to event windows, and adopt nearest neighbor or semantic alignment strategies for multimodal medical data without precise timestamps;

[0053] S2.2. Perform noise detection and quality assessment on the medical data for each modality, and mark low-quality or unusable modal data;

[0054] S2.3. Feature extraction is performed for different types of modal data. For signal modal data, wavelet transform, time-spectrum graph or one-dimensional convolution feature extraction is used. For image modal data, preprocessing, local contrast enhancement combined with convolutional network or visual Transformer feature extraction is used. For text modal data, sentence segmentation, entity extraction combined with encoder feature extraction is used. For structured numerical data, standardization, normalization and statistical feature extraction are used.

[0055] S2.4. Map the features of each modality to an embedding space of a unified dimension and calculate the quality weights corresponding to each modality.

[0056] Step S3 includes the following sub-steps:

[0057] S3.1. Perform cross-attention calculation on all available modal embeddings to obtain the intermodal interaction coefficients. The calculation formula is as follows:

[0058] ;

[0059] in, Let be the interaction coefficient between the i-th mode and the j-th mode. and This is the weight matrix. and These are embedding vectors for different modalities. For the embedded vector dimension, For activation functions;

[0060] S3.2. Based on the mutual influence coefficient, the modal embeddings are weighted and summed to obtain the interaction-enhanced embeddings;

[0061] S3.3. Combine quality weights to perform secondary weighting on the interaction enhancement embedding to obtain preliminary fusion results.

[0062] Step S3 includes the following sub-steps:

[0063] S3.1. Introduce a gating network and calculate the fusion coefficient of each modality by combining the quality weights, historical reliability, and scene information of each modality. The calculation formula is as follows:

[0064] ;

[0065] in, For modal fusion coefficients, For activation function, For the gated weight matrix, to For the mass weights of each mode, For scene vectors;

[0066] S3.2. Based on the fusion coefficient, perform weighted filtering on the standardized modal embeddings and retain the embedding information of high-contribution modes;

[0067] S3.3. Combine the selected modal embeddings with the task-specific header to generate diagnostic-related results.

[0068] Step S3 further includes: splicing and blending the weighted interaction enhancement embeddings, calculated using the following formula:

[0069] ;

[0070] in, For the final fusion representation, For splicing operations, to The fusion coefficients for each mode are given. to The embedding vectors for each modality are used; the model is optimized based on the final fusion representation and the multi-task learning loss function to obtain diagnostic results.

[0071] Step S2 further includes: performing privacy-preserving preprocessing on the multimodal medical data, the privacy-preserving preprocessing including de-identification, sealing and encapsulation, and differential privacy noise injection; the preprocessed multimodal medical data is transmitted through a secure channel, the transmission process adopts a message queue mechanism, supports breakpoint resume, fragmented upload and priority transmission, among which alarm-related data is transmitted first.

[0072] The feature extraction in step S2 adopts an edge-cloud collaborative approach: a lightweight model is used at the edge to perform preliminary feature extraction and anomaly screening on multimodal medical data, while a high-performance model is used in the cloud for deep feature extraction; the data type to be uploaded is selected according to the real-time bandwidth and latency status, and the data types to be uploaded include raw data, compressed representation and modal summary.

[0073] The hierarchical adaptive fusion framework in step S3 is trained through federated multimodal learning: different institutions retain local multimodal medical data and only exchange model updates or modal feature summaries during training; a modal feature matcher is introduced to share aligned embedding distribution statistics, and differential privacy or secure multi-party computation is used to encrypt key aggregation steps during the training process.

[0074] After outputting the analysis results in step S4, the process also includes: receiving clinical feedback information and using it as training data to input into the hierarchical adaptive fusion framework; fine-tuning the model in the framework online or offline; and recording the data access and feedback processes through blockchain to form a continuous learning loop.

[0075] In some embodiments, a remote medical AI-assisted system based on multimodal data fusion includes a data acquisition module, a data processing module, a multimodal fusion module, a result analysis module, a human-computer interaction module, and a security and compliance module.

[0076] The data acquisition module is used to acquire multimodal medical data composed of continuous physiological signals, medical images, laboratory test results, medical record text, and behavioral signals. The data processing module is used to perform synchronization alignment, quality control, feature extraction, modality embedding standardization, and privacy protection preprocessing on the multimodal medical data. The multimodal fusion module is used to fuse the processed modal data using a hierarchical adaptive fusion framework and complete model training through federated multimodal learning. The result parsing module is used to perform interpretability analysis and uncertainty estimation on the diagnostic results generated by fusion. The human-computer interaction module is used to output analysis results and receive clinical feedback information. The security and compliance module runs through all modules and provides audit logs, access control, differential privacy logs, and compliance reporting interfaces.

[0077] Example 2

[0078] This embodiment provides a remote medical AI-assisted method based on multimodal data fusion and a system for setting the corresponding method steps, such as... Figure 2 As shown, the specific implementation is as follows:

[0079] S1. Multimodal medical data collection at different points, clear classification, and preliminary integration:

[0080] S1.1. Determine the sources of multimodal medical data:

[0081] The data collection sources cover patients, remote diagnosis and treatment points, and hospitals. Patients include wearable devices, smartphone apps, and home monitoring devices. Remote diagnosis and treatment points are equipped with portable ultrasound equipment, digital stethoscopes, and cameras. Hospitals are connected to PACS imaging systems, LIS laboratory systems, and EHR electronic medical record systems. Each data collection source provides hardware and system support for the acquisition of multimodal medical data in accordance with preset interface specifications and data transmission protocols.

[0082] S1.2. Define the specific types of multimodal medical data:

[0083] Multimodal medical data includes continuous physiological signals, medical images, laboratory test results, medical record texts, and behavioral signals. Continuous physiological signals include ECG, PPG (photoplethysmography) pulse wave, and respiratory flow, etc.; medical images include X-rays, CT (computed tomography) images, ultrasound images, and skin images, etc.; laboratory test results are in structured numerical form, covering various test data such as complete blood count and biochemical indicators; medical record texts are unstructured natural language data, including patient complaints, medical history records, and treatment notes, etc.; behavioral signals include patient videos, facial expressions, and voice, etc., reflecting the patient's condition.

[0084] S1.3. Perform multimodal medical data acquisition operations:

[0085] Wearable devices on the patient's end continuously collect physiological signals, while a smartphone app periodically collects patient symptom text and behavioral signal data. Home monitoring devices record the patient's basic health monitoring data. Remote diagnosis and treatment points use portable ultrasound equipment, digital stethoscopes, and cameras to collect corresponding medical images, audio, and video data during the diagnosis and treatment process. On the hospital side, PACS imaging system, LIS laboratory system, and EHR electronic medical record system are used to extract stored patient medical images, laboratory test results, and medical record text data. During the collection process, stable data transmission between various devices and systems is ensured to avoid data loss or damage.

[0086] S2. Multimodal medical data edge-cloud collaborative preprocessing: alignment, quality control, feature extraction, standardization, and privacy protection:

[0087] S2.1. Perform synchronization and alignment processing on multimodal medical data:

[0088] For multimodal medical data with timestamps, synchronization alignment is performed according to event windows. The event window is set as a time interval around a specific medical event, and all timestamped data within that time interval are matched in terms of time dimension. For multimodal medical data without precise timestamps, a nearest neighbor alignment strategy or a semantic alignment strategy is adopted. The nearest neighbor alignment strategy associates the data without timestamps with the data with the closest time in the timestamp data based on the approximate time range of data collection. The semantic alignment strategy aligns the content-related data without timestamps with the data with timestamps by analyzing the semantic information of the data, ensuring that multimodal medical data from different sources and of different types maintain consistency in terms of time or semantics.

[0089] S2.2. Perform quality control processing on multimodal medical data:

[0090] Noise detection is performed separately for each modality of medical data. Different noise detection methods, such as signal processing algorithms, image quality assessment algorithms, and text denoising algorithms, are used to identify interference noise in continuous physiological signals, noise points in medical images, and redundant information in medical record texts. Each modality of data that has undergone noise detection is scored, and the quality score is set in the range of 0 to 1. A quality score threshold is set, and modality data below the threshold is marked as low-quality or unusable data to avoid adverse effects of low-quality data on subsequent processing.

[0091] S2.3. Perform privacy-preserving preprocessing on multimodal medical data:

[0092] The multimodal medical data that has undergone quality control is de-identified to remove information that can directly identify patients, such as patient names, ID numbers, and contact information. The de-identified data is then sealed and encrypted using encryption algorithms to ensure data security during transmission and storage. Depending on actual privacy protection needs, a local differential privacy noise injection operation can be optionally performed to inject an appropriate amount of differential privacy noise into the data to further prevent reverse inference of patient privacy information. All privacy protection preprocessing operations comply with relevant privacy protection regulations and standards.

[0093] S2.4. Feature extraction of multimodal medical data using an edge-cloud collaborative approach:

[0094] At the edge, a lightweight model is used to perform preliminary feature extraction on multimodal medical data. For continuous physiological signals, wavelet transform, time-spectrum graph, or 1D-CNN feature extraction methods are used to extract the time and frequency domain features of the signals. Medical images are first preprocessed, then local contrast enhancement is performed, and then feature extraction is performed using a lightweight CNN or Vision Transformer (ViT). Medical record text is first processed into sentences, then entity extraction is performed, and finally features are output through a lightweight BERT-like encoder. Structured numerical data are extracted using standardization, normalization, and statistical feature extraction methods. At the same time, the edge also performs preliminary anomaly screening on the data, identifying and marking obvious abnormal data. In the cloud, a high-performance model is used to perform deep feature extraction on the preliminary features or raw data transmitted from the edge. More complex feature extraction algorithms are used for different modal data characteristics to mine deep features in the data and improve the feature representation ability.

[0095] S2.5. Select the data upload type based on real-time bandwidth and latency status:

[0096] The system monitors network bandwidth and data transmission latency in real time. When network bandwidth is sufficient and latency is low, it selects to upload raw medical data or complete feature data. When network bandwidth is limited or latency is high, it selects to upload compressed feature data or only modality summary data. Modality summary data includes data formats that can reflect the core information of the modality, such as embedding vectors, to ensure that data transmission can be carried out efficiently under different network conditions and meet the needs of subsequent processing.

[0097] S2.6. Perform modality embedding standardization on the extracted features:

[0098] Modal features extracted from the edge and cloud are uniformly mapped to a pre-defined embedding space of uniform dimension. The dimension of the embedding space is determined according to the model training requirements and data characteristics. The mapping process uses a standardized embedding algorithm to ensure that each modal feature has a uniform scale and distribution in the embedding space. Based on the quality score results of each modal data, the quality weight corresponding to each modality is calculated through a pre-defined function. The quality weight is used to quantify the importance of each modal feature in the subsequent fusion process. The higher the quality score of the modality, the greater the corresponding quality weight.

[0099] S3. Hierarchical Adaptive Fusion and Task Adaptation: Modal Interaction, Gating Adjustment, Diagnostic Result Generation, and Model Optimization:

[0100] S3.1. Perform cross-attention calculation on the standardized modal embeddings:

[0101] Collect all modal embeddings that have undergone modal embedding normalization and are marked as usable, and construct a modal embedding set; perform cross-attention calculation on the modal embeddings in this set to calculate the mutual influence coefficient between modalities, using the following formula: ;

[0102] in, Let be the interaction coefficient between the i-th mode and the j-th mode. and This is the weight matrix. and These are embedding vectors for different modalities. For the embedded vector dimension, The activation function is used to calculate the degree of interaction between each modality and all other modalities. The embeddings of each modality are weighted and summed based on the interaction coefficient to obtain the interaction-enhanced embedding. The interaction-enhanced embedding fully integrates the correlation information between different modalities and improves the expressive power of features.

[0103] S3.2. Introduce a gated network to calculate the fusion coefficients of each mode:

[0104] The input to the gating network includes the quality weights of each modality, historical reliability, and scenario information. Historical reliability is determined based on the modality's performance in past data processing and fusion processes. Scenario information includes vectors reflecting the current application scenario, such as urgency and bandwidth level. The fusion coefficient for each modality is calculated through the gating network, using the following formula: ;

[0105] in, For modal fusion coefficients, For activation function, For the gated weight matrix, to For the mass weights of each mode, The scene vector is used to represent the contribution of each modality to the current fusion task. Modalities with high quality weights, good historical reliability, and that meet the requirements of the current scene will obtain higher fusion coefficients.

[0106] S3.3. Perform weighted fusion processing on modal embeddings:

[0107] The interaction-enhanced embeddings are weighted twice by combining quality weights, and the quality weights are multiplied by the interaction-enhanced embeddings to highlight the role of high-quality modal embeddings in the fusion result. Based on the fusion coefficients calculated by the gating network, the standardized modal embeddings are weighted and filtered, retaining the embedding information of high-contribution modalities with high fusion coefficients and filtering out modal embeddings with low fusion coefficients and small contributions to the fusion result. The interaction-enhanced embeddings after the second weighting and the filtered high-contribution modal embeddings are then concatenated and fused, calculated as follows:

[0108] ;

[0109] in, For the final fusion representation, For splicing operations, to The fusion coefficients for each mode are given. to These are the embedding vectors for each modality; the splicing and fusion process organically integrates the feature information of different modalities to form a fusion feature representation that can comprehensively reflect the patient's condition.

[0110] S3.4. Generate diagnostic-related results by combining task adaptation strategies:

[0111] The hierarchical adaptive fusion framework sets task-specific headers for different clinical tasks, including diagnostic classification, risk scoring, grading, and segmentation. Corresponding task-specific headers include classification, scoring, grading, and segmentation headers. The final fused representation is input into the task-specific header corresponding to the current clinical task. The task-specific header processes the fused representation and outputs preliminary diagnostic results relevant to the task. During model training, a multi-task learning loss function is used to optimize the model. This function includes task losses for each clinical task, as well as regularization terms such as modality prior compliance and gating sparsity constraints. Through multi-task learning, the model's adaptability and generalization performance to different clinical tasks are improved, ultimately generating accurate and reliable diagnostic results.

[0112] S3.5. Train the hierarchical adaptive fusion framework using federated multimodal learning:

[0113] Multiple institutions participate in model training. Each institution retains its local multimodal medical data and does not transmit raw data to other institutions or the central server to avoid privacy risks caused by raw data leakage. During training, each institution performs local training on the hierarchical adaptive fusion framework based on its local data. After completing local training, it only transmits model update parameters or modal feature summaries to the federated learning coordinator. The federated learning coordinator introduces a modal feature matcher to share the alignment distribution statistics of modal embeddings from each institution in unlabeled cross-domain scenarios. It then aggregates the model update parameters or modal feature summaries transmitted by each institution to generate global model update parameters. Each institution receives the global model update parameters, updates its local model, and repeats the above process of local training, parameter uploading, global aggregation, and model updating until the model performance reaches the preset standard. During training, differential privacy technology is used to add differential privacy noise to the model update parameters, or secure multi-party computation (MPC) is used to encrypt key aggregation steps to ensure the privacy and security of the training process. At the same time, a Trusted Execution Environment (TEE) is set up in the cloud, and the training and inference processes of sensitive models are carried out within the TEE to further enhance the level of privacy protection.

[0114] S3.6. Perform version control and testing on the trained model:

[0115] A model library is established to manage the trained hierarchical adaptive fusion framework models, enabling model version control and supporting A / B testing and rollback operations. Before clinical release, the models undergo Shadow Mode testing, during which the model runs in parallel with the physician's clinical decision-making process without affecting the physician's actual clinical decisions. The Shadow Mode test collects comparative data between the model's output and the physician's clinical decision results to evaluate model performance. After passing the evaluation, an online A / B evaluation method is used to gradually apply the model to real-world clinical scenarios, continuously monitoring model performance. If performance degradation or adverse events occur, the model rollback mechanism restores the model to a previous stable version.

[0116] S4. Interpretability analysis of diagnostic results, quantification of uncertainty: feedback loop, compliance assurance and clinical implementation:

[0117] S4.1. Perform interpretability analysis on diagnostic-related results:

[0118] Attention visualization is employed to analyze diagnostic results, visualizing the attention distribution of each modality embedded in the fusion process and clearly demonstrating the influence of different modalities on the diagnostic results. SHAP (SHapley Additive exPlanations) or Integrated Gradients methods are used to quantify the contribution of each modality's features to the diagnostic results, identifying which features are key factors driving the generation of diagnostic results. For diagnostic results related to medical images, Grad-CAM (Gradient-weighted Class Activation Mapping) is used to locate feature regions in medical images that play a crucial role in the diagnostic results, generating image heatmaps. A multimodal evidence map is constructed, linking the final diagnostic results with specific evidence from each modality, including abnormal bands in continuous physiological signals, feature regions in medical images, key symptom words in medical records, and abnormal indicators in laboratory test results, making the basis for the diagnostic results clearly traceable.

[0119] S4.2. Estimate the uncertainty of diagnostic-related results:

[0120] The uncertainty of diagnostic results is estimated using either Bayesian approximation or MC Dropout. Bayesian approximation quantifies the uncertainty of model predictions by constructing the posterior distribution of model parameters. MC Dropout estimates uncertainty by randomly shutting down some neurons in the model during inference, running the model multiple times to obtain multiple predictions, and estimating the uncertainty based on the dispersion of these predictions. Simultaneously, temperature scaling and confidence calibration methods are used to calibrate the model's output confidence, improving the accuracy of uncertainty estimation. A dynamic threshold is set based on the uncertainty estimation results. When the uncertainty estimate exceeds the dynamic threshold, it indicates that the model has low confidence in the diagnostic results and requires further validation.

[0121] S4.3. Trigger the corresponding processing flow based on the uncertainty estimation results:

[0122] When the uncertainty estimate of the diagnostic-related results is below the dynamic threshold, the diagnostic-related results and interpretability analysis results are directly output to support the clinician's diagnostic decision-making. When the uncertainty estimate exceeds the dynamic threshold, a manual review process is automatically triggered, pushing relevant case information, diagnostic-related results, interpretability analysis results, and uncertainty estimate results to the clinician for review. If the clinician believes that the existing data is insufficient to make an accurate judgment during the review process, the system triggers an additional sampling process. Based on the clinician's instructions or a preset sampling strategy, relevant multimodal medical data is collected, and the data processing, fusion, and result generation processes are re-executed until a diagnostic-related result with the required confidence level is obtained.

[0123] S4.4. Output interpretability analysis results and uncertainty estimation results:

[0124] A dedicated interpretability analysis results display interface for clinicians was designed, providing a case aggregation view, a multimodal evidence map, and interactive interpretability display functions. Doctors can view detailed interpretability analysis content through the interface, including the contribution of each modality, key features, and evidence sources. Doctors can interactively adjust modality weights and view changes in diagnostic results under alternative scenarios, i.e., how diagnostic results change when a certain modality's data is missing, achieving "reverse factual interpretation." Simplified analysis results are output to the patient's mobile app, including real-time alerts and health trend displays, ensuring patients can clearly understand key information related to their health status. The output of results strictly adheres to relevant compliance requirements and does not disclose patient privacy information.

[0125] S4.5. Receive and process clinical feedback information:

[0126] A clinical feedback information receiving channel is established. After reviewing diagnostic results, interpretability analysis results, and uncertainty estimation results, clinicians can submit feedback information through the doctor's workstation. Feedback information includes suggestions for revising diagnostic results, the degree of acceptance of interpretability analysis, suggestions for additional sampling, and evaluation of model performance. Patients can also submit relevant feedback information such as their own feelings and changes in symptoms through the patient APP. The system classifies and organizes the received feedback information, extracts key feedback content, including issues with the accuracy of diagnostic results, the clarity of interpretability analysis, and shortcomings in the model processing.

[0127] S4.6. Continuously optimize the hierarchical adaptive fusion framework based on clinical feedback information:

[0128] Clinical feedback information is input as training data into a hierarchical adaptive fusion framework to construct an active learning loop. Data related to physician-corrected diagnoses are labeled as high-quality annotated data for online or offline model fine-tuning. During fine-tuning, a multi-stage training strategy is employed: first, the encoder is fine-tuned based on newly added feedback data; then, the fusion layer and task-specific heads are jointly optimized. The original modality consistency loss is retained during fine-tuning to ensure semantic aggregation of embeddings of different modalities in the same case, while gating sparsity regularization is preserved to encourage the model to only call key modalities for inference. The model is periodically retrained, incorporating a large amount of accumulated clinical feedback data into the training process to continuously improve the model's diagnostic accuracy, robustness, and clinical adaptability. Simultaneously, blockchain technology is used to record data access and feedback processes, forming an immutable audit log for subsequent compliance audits and traceability.

[0129] S4.7. Build a security and compliance module to ensure compliance throughout the entire process:

[0130] The security and compliance module spans the entire data collection, processing, fusion, analysis, and feedback process. It provides audit log functionality, meticulously recording all relevant behaviors such as data collection time, processing steps, access personnel, access time, operation content, and feedback information. A strict access control mechanism is implemented, with role-based access control policies clearly defining the access permissions for different users (doctors, patients, administrators, etc.), authorizing only authorized users to access corresponding information and functions. Differential privacy logs are generated, injecting differential privacy noise to prevent the inference of patient privacy from log information. A compliance reporting interface is provided, adaptable to privacy protection regulations and medical industry standards in different jurisdictions, capable of generating compliant reports as needed, ensuring the system's entire operation complies with relevant compliance requirements.

[0131] This remote medical AI-assisted method and system based on multimodal data fusion comprehensively acquires and refines multimodal medical data, fully exploring the complementary information of different modalities. Combined with the efficient fusion capabilities of a hierarchical adaptive fusion framework, it significantly improves the accuracy and stability of diagnostic results. Through adaptive modality gating mechanisms and quality weight adjustment strategies, the system automatically adjusts fusion weights and inference processes when some modal data is missing or degraded in quality, ensuring a gradual degradation of system performance and improving the reliability of clinical applications. Employing privacy-preserving technologies such as federated multimodal learning, differential privacy, and secure multi-party computation, it achieves cross-institutional collaborative training, integrating multi-center medical data resources without disclosing patients' original privacy data, and constructing a more generalizable model. Through interpretability analysis methods such as multimodal evidence mapping, attention visualization, and feature attribution, it closely links diagnostic results with original modal evidence, reducing the "black box" nature of the model and improving its overall performance. The system ensures the verifiability of diagnostic results, helping to gain the trust and recognition of clinicians. Utilizing an edge-cloud collaborative deployment architecture and bandwidth-adaptive transmission strategy, the edge handles rapid initial screening and real-time alerts, while the cloud undertakes deep inference and complex fusion tasks, balancing the dual requirements of real-time performance and accuracy in remote scenarios and adapting to fluctuating network bandwidth. Through model compression, knowledge distillation, and multi-stage training technologies, the model can be deployed on various devices such as existing mobile devices and edge gateways, reducing deployment and promotion costs and improving the ease of clinical access. Through a continuous learning loop driven by clinical feedback, the model continuously absorbs new data and experiences from clinical practice, continuously optimizing performance and improving cross-domain adaptability to different hospitals and populations, achieving synergistic development between the model and clinical practice. Simultaneously, a full-process security and compliance module and privacy protection mechanism ensure the compliance of system operation, providing strong support for the standardized application of AI-assisted diagnosis in remote medical care. The improved accuracy of diagnostic results directly stems from the comprehensive acquisition and refined processing of multimodal medical data, as well as the effective integration of a hierarchical adaptive fusion framework—a key technical effect brought about by the core processes of the entire solution.

[0132] Example 3

[0133] In some embodiments, a remote medical AI-assisted system based on multimodal data fusion is provided, such as... Figure 3 As shown, this system specifically includes: a data acquisition module, a data processing module, a multimodal fusion module, a result analysis module, a human-computer interaction module, and a security and compliance module. Each module works collaboratively according to the process to realize a full-link service from multimodal medical data acquisition to clinical application support, providing comprehensive technical support for remote medical diagnosis.

[0134] The data acquisition module acquires multimodal medical data through multi-terminal device linkage. Wearable devices, smartphone apps, and home monitoring devices at the patient end continuously collect data such as physiological signals and symptom records; portable ultrasound, digital stethoscopes, and cameras at remote treatment points capture medical images, audio information, and patient behavioral signals; at the hospital end, the system connects to imaging, testing, and electronic medical record systems to extract stored medical images, laboratory test results, and medical record text data. These continuous physiological signals, medical images, laboratory test results, medical record text, and behavioral signals from different terminals collectively constitute the system's basic data set.

[0135] After data acquisition, the data processing module performs a series of processing operations. First, multimodal medical data with timestamps are synchronized and aligned according to event windows. For data without precise timestamps, nearest neighbor or semantic alignment strategies are used to ensure consistency in time or semantics. Then, noise detection and quality assessment are performed on the data for each modality, marking low-quality or unusable data to prevent invalid data from affecting subsequent processing. Different feature extraction methods are used for different types of modal data: signal modalities are extracted using wavelet transform, time-spectrum mapping, or one-dimensional convolution; image modalities are preprocessed and enhanced with local contrast before feature extraction using convolutional networks or visual Transformers; text modalities are first segmented into sentences and entity extracted, then features are output through an encoder; structured numerical data are extracted using standardization, normalization, and statistical methods. Finally, the features of each modality are mapped to a unified-dimensional embedding space, and the quality weights corresponding to each modality are calculated. Meanwhile, the data processing module also performs privacy-preserving preprocessing on the data, including de-identification, encryption, and differential privacy noise injection. The processed data is transmitted through a secure channel using a message queue mechanism, supporting breakpoint resumption, fragmented upload, and priority transmission, with alarm-related data being transmitted first. The feature extraction process adopts an edge-cloud collaborative approach. The edge terminal uses a lightweight model for preliminary feature extraction and anomaly screening, while the cloud uses a high-performance model for deep feature extraction. The system selects to upload raw data, compressed representation, or modal digest based on real-time bandwidth and latency, adapting to different network environments.

[0136] The multimodal fusion module employs a hierarchical adaptive fusion framework to process the standardized modal embeddings. First, cross-attention is calculated for all available modal embeddings to obtain intermodal interaction coefficients. Then, based on these coefficients, the modal embeddings are weighted and summed to obtain interactive enhancement embeddings. Subsequently, quality weights are used to perform a second weighting of the interactive enhancement embeddings, yielding a preliminary fusion result. Simultaneously, a gating network is introduced to calculate the fusion coefficient for each modality by combining its quality weights, historical reliability, and scene information. Based on these fusion coefficients, the standardized modal embeddings are weighted and filtered, retaining the embedding information of high-contribution modalities. The filtered modal embeddings are then combined with a task-specific header. The weighted interactive enhancement embeddings are concatenated and fused to obtain the final fused representation. Finally, a multi-task learning loss function is used to optimize the model, generating diagnostic-related results. This framework is trained through federated multimodal learning, with different institutions retaining local multimodal medical data. During training, only model updates or modal feature summaries are exchanged. A modal feature matcher is introduced to share aligned embedding distribution statistics. The training process uses differential privacy or secure multi-party computation to encrypt key aggregation steps, thereby improving the model's generalization ability while ensuring data privacy.

[0137] The results parsing module performs interpretability analysis and uncertainty estimation on the generated diagnostic results. It employs methods such as attention visualization and feature attribution to quantify the contribution of each modality's features to the diagnostic results, constructs a multimodal evidence map, and correlates the diagnostic results with specific evidence from each modality, clearly presenting the diagnostic basis. The uncertainty of the diagnostic results is estimated using Bayesian approximation or correlation methods, and the output confidence level is optimized using temperature scaling and confidence level calibration methods. A dynamic threshold is set based on the uncertainty estimation results; when the uncertainty exceeds the threshold, a manual review or additional sampling process is triggered.

[0138] The human-computer interaction module is responsible for outputting analysis results and receiving clinical feedback. The doctor's workstation provides a case aggregation view, a multimodal evidence map, and interactive interpretability display functions, allowing doctors to view the contribution and key features of each modality of data, facilitating consultations and expert scheduling. The patient app pushes real-time alerts and displays health trends, while also providing compliance consent management functions. The system receives feedback from clinicians and patients, using it as training data to input into the multimodal fusion module for online or offline fine-tuning of the model within the framework. Simultaneously, it records the data access and feedback processes via blockchain, forming a continuous learning loop that drives continuous optimization of model performance.

[0139] The security and compliance module runs through all modules of the system, providing audit logs, access control, differential privacy logs, and compliance report interfaces. It records relevant operations throughout the entire data lifecycle, strictly controls data access permissions, generates compliance reports that meet the requirements of different jurisdictions, and ensures the compliance of system operations.

[0140] The entire system leverages the complementarity of different modalities through comprehensive collection and refined processing of multimodal data, improving the accuracy and stability of diagnostic results. Edge-cloud collaborative deployment and bandwidth-adaptive transmission strategies balance real-time performance and processing efficiency, adapting to complex network environments. Privacy protection and compliance mechanisms ensure the secure and compliant use of medical data. Interpretable analysis and a continuous learning loop enhance the system's clinical applicability and reliability, providing strong support for the standardized and efficient implementation of telemedicine.

[0141] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A remote medical AI-assisted method based on multimodal data fusion, characterized in that, Includes the following steps: S1. Acquire multimodal medical data, which includes continuous physiological signals, medical images, laboratory test results, medical record texts, and behavioral signals; S2. Perform synchronization alignment, quality control, feature extraction, and modality embedding standardization on the acquired multimodal medical data; S3. A hierarchical adaptive fusion framework is used to fuse the standardized modal embeddings, and diagnostic results are generated by combining the task adaptation strategy. S4. Perform interpretability analysis and uncertainty estimation on the generated diagnostic results, and output the analysis results to support clinical applications; Step S2 includes the following sub-steps: S2.

1. Synchronize and align multimodal medical data with timestamps according to event windows, and adopt nearest neighbor or semantic alignment strategies for multimodal medical data without precise timestamps; S2.

2. Perform noise detection and quality assessment on the medical data for each modality, and mark low-quality or unusable modal data; S2.

3. Feature extraction is performed for different types of modal data. For signal modal data, wavelet transform, time-spectrum graph or one-dimensional convolution feature extraction is used. For image modal data, preprocessing, local contrast enhancement combined with convolutional network or visual Transformer feature extraction is used. For text modal data, sentence segmentation, entity extraction combined with encoder feature extraction is used. For structured numerical data, standardization, normalization and statistical feature extraction are used. S2.

4. Map the features of each modality to an embedding space of a unified dimension, and calculate the quality weights corresponding to each modality; Step S3 includes the following sub-steps: S3.

1. Perform cross-attention calculation on all available modal embeddings to obtain the intermodal interaction coefficients. The calculation formula is as follows: ； in, Let be the interaction coefficient between the i-th mode and the j-th mode. and This is the weight matrix. and These are embedding vectors for different modalities. For the embedded vector dimension, For activation functions; S3.

2. Based on the mutual influence coefficient, the modal embeddings are weighted and summed to obtain the interaction-enhanced embeddings; S3.

3. Combine quality weights to perform secondary weighting on the interaction enhancement embedding to obtain preliminary fusion results.

2. The method according to claim 1, characterized in that, Step S3 includes the following sub-steps: S3.

1. Introduce a gating network and calculate the fusion coefficient of each modality by combining the quality weights, historical reliability, and scene information of each modality. The calculation formula is as follows: ； in, For modal fusion coefficients, For activation function, For the gated weight matrix, to For the mass weights of each mode, For scene vectors; S3.

2. Based on the fusion coefficient, perform weighted filtering on the standardized modal embeddings and retain the embedding information of high-contribution modes; S3.

3. Combine the selected modal embeddings with the task-specific header to generate diagnostic-related results.

3. The method according to claim 1, characterized in that, Step S3 further includes: splicing and blending the weighted interaction enhancement embeddings, calculated using the following formula: ； in, For the final fusion representation, For splicing operations, to The fusion coefficients for each mode are given. to The embedding vectors for each modality are used; the model is optimized based on the final fusion representation and the multi-task learning loss function to obtain diagnostic results.

4. The method according to claim 1, characterized in that, Step S2 further includes: performing privacy-preserving preprocessing on the multimodal medical data, the privacy-preserving preprocessing including de-identification, sealing and encapsulation, and differential privacy noise injection; the preprocessed multimodal medical data is transmitted through a secure channel, the transmission process adopts a message queue mechanism, supports breakpoint resume, fragmented upload and priority transmission, among which alarm-related data is transmitted first.

5. The method according to claim 1, characterized in that, The feature extraction in step S2 adopts an edge-cloud collaborative approach: a lightweight model is used at the edge to perform preliminary feature extraction and anomaly screening on multimodal medical data, while a high-performance model is used in the cloud for deep feature extraction. The upload data type is selected based on real-time bandwidth and latency status. Upload data types include raw data, compressed representation, and modal digest.

6. The method according to claim 1, characterized in that, The hierarchical adaptive fusion framework in step S3 is trained through federated multimodal learning: different institutions retain local multimodal medical data and only exchange model updates or modal feature summaries during training; a modal feature matcher is introduced to share aligned embedding distribution statistics, and differential privacy or secure multi-party computation is used to encrypt key aggregation steps during the training process.

7. The method according to claim 1, characterized in that, After outputting the analysis results in step S4, the process also includes: receiving clinical feedback information and using it as training data to input into the hierarchical adaptive fusion framework; fine-tuning the model in the framework online or offline; and recording the data access and feedback processes through blockchain to form a continuous learning loop.

8. A remote medical AI-assisted system based on multimodal data fusion, characterized in that, The system includes a data acquisition module, a data processing module, a multimodal fusion module, a result analysis module, a human-computer interaction module, and a security and compliance module. The data acquisition module is used to acquire multimodal medical data composed of continuous physiological signals, medical images, laboratory test results, medical record text, and behavioral signals. The data processing module is used to perform synchronization alignment, quality control, feature extraction, modality embedding standardization, and privacy protection preprocessing on the multimodal medical data. The multimodal fusion module is used to fuse the processed modal data using a hierarchical adaptive fusion framework and complete model training through federated multimodal learning. The result parsing module is used to perform interpretability analysis and uncertainty estimation on the diagnostic results generated by fusion. The human-computer interaction module is used to output analysis results and receive clinical feedback information. The security and compliance module runs through all modules and provides audit logs, access control, differential privacy logs, and compliance reporting interfaces.

Images