Remote intelligent diagnosis and treatment system and method based on multi-modal data fusion and deep learning

Abstract

The invention discloses a remote intelligent diagnosis and treatment system and method based on multi-modal data fusion and deep learning, and relates to the technical field of remote medical treatment. According to the system, a user side collects multi-modal data; the feature extraction and space-time alignment module performs feature extraction and space-time alignment on the data; a cross-modal attention fusion module captures a relationship of internal features of the data, calculates correlation weights among different modal features, and generates a fusion feature vector; the interactive diagnosis module obtains a final diagnosis result according to the fusion feature vector; the diagnosis report generation module generates a structured diagnosis report according to the final diagnosis result; the doctor end provides an interactive interface for doctors, and supports check of diagnostic reports and access of AR remote consultation; and the cloud collaboration platform is used for data storage, model continuous learning and multi-party consultation. According to the invention, deep fusion analysis can be carried out in combination with multi-modal medical data, and interactive diagnosis can be carried out by simulating clinical diagnosis thinking, so that the accuracy and credibility of remote diagnosis are improved.

Description

Technical Field

[0001] This invention relates to the field of telemedicine technology, and in particular to a remote intelligent diagnosis and treatment system and method based on multimodal data fusion and deep learning. Background Technology

[0002] With the development of artificial intelligence technology, telemedicine systems are gradually becoming an important means of compensating for the uneven distribution of medical resources.

[0003] However, existing telemedicine systems mostly focus on communication and data display. Most systems only analyze single-modal data such as electronic medical records (text), medical images (images), and vital signs (time-series signals) independently, or perform simple post-decision splicing. They fail to achieve deep and early fusion of different modal data at the feature level and ignore the inherent correlation and complementarity between cross-modal data. Diagnostic models based on deep learning usually give a final conclusion, lacking an interpretable reasoning path that conforms to the doctor's clinical thinking process. Doctors find it difficult to understand the basis of the model's judgment, resulting in reduced trust and the inability to form effective human-machine collaboration. Existing systems usually output results after inputting all data at once, which cannot simulate the dynamic and progressive process of "further inquiry or examination based on preliminary findings" in real diagnosis and treatment, and the level of intelligence is limited.

[0004] Therefore, proposing a remote intelligent diagnosis and treatment system and method based on multimodal data fusion and deep learning to solve the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a remote intelligent diagnosis and treatment system and method based on multimodal data fusion and deep learning, which can combine multimodal medical data for deep fusion analysis and simulate clinical diagnostic thinking to conduct interactive diagnosis, thereby improving the accuracy and reliability of remote diagnosis.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning includes: On the user end, multimodal data is collected through multimodal physiological sensing devices; The feature extraction and spatiotemporal alignment module connects to the user terminal to perform feature extraction and spatiotemporal alignment on multimodal data. The cross-modal attention fusion module, connected to the feature extraction and spatiotemporal alignment module, captures the relationships between features within multimodal data, calculates the correlation weights between features of different modalities, and generates a fusion feature vector rich in cross-modal information. The interactive diagnostic module, connected to the cross-modal attention fusion module, performs cross-validation and joint inference diagnostics based on the fused feature vectors to obtain the final diagnostic result; The diagnostic report generation module connects to the interactive diagnostic module and automatically generates a structured diagnostic report based on the final diagnostic results. On the physician side, it connects to the diagnostic report generation module, providing doctors with an interactive interface that supports viewing diagnostic reports and accessing AR remote consultations. The cloud-based collaborative platform connects with the user terminal, interactive diagnostic module, and physician terminal for data storage and continuous model learning.

[0007] The above-mentioned system may optionally include user-end multimodal physiological sensing devices, including: electrocardiogram monitors, blood pressure monitors, blood oxygen monitors, body temperature monitors, smart stethoscopes, portable ultrasound, and mobile terminals. Multimodal data includes: text data, time-series data, image data, and audio data; Textual data includes: chief complaint, medical history, and questionnaire; Time-series data include: electrocardiogram, blood pressure, blood oxygen saturation, and body temperature; Image data includes: images of skin lesions, images of the tongue, photographs of the ear canal, and DICOM images; The audio data includes: heart sounds, lung sounds, and cough sounds.

[0008] The above system, optionally, includes a feature extraction and spatiotemporal alignment module comprising: a data normalization unit, a temporal encoder, a spatial encoder, and a data mapping unit; Data standardization unit, which standardizes the collected multimodal data; A time-series encoder extracts the dynamic features of time-series data and timestamps these dynamic features. Spatial encoders extract spatial features from image data; The data mapping unit maps data from different modalities to a unified temporal-spatial framework.

[0009] The above system, optionally, includes a cross-modal attention fusion module that employs a hierarchical cascaded attention mechanism, using intra-modal self-attention to capture the relationships between features within multimodal data; By introducing cross-modal cross-attention, the correlation weights between features of different modalities are dynamically calculated to generate a fusion feature vector rich in cross-modal information.

[0010] The above-mentioned system may optionally include an interactive diagnostic module, comprising: a pre-trained clinical knowledge graph, a reinforcement learning-driven dialogue strategy model, and interactive units; A clinical knowledge graph defines entities and logical relationships of diseases, symptoms, signs, examinations, and complications. Based on the fused feature vectors, it generates a preliminary list of differential diagnoses and confidence levels. A dialogue strategy model that dynamically generates follow-up questions to narrow down the scope of diagnosis; The interactive unit interacts with doctors and patients through a remote interface. Based on follow-up questions and suggestions, it obtains new data or answers, updates the fused feature vector, and conducts a new round of diagnosis until the diagnostic confidence meets the preset requirements or reaches the preset maximum number of interaction rounds, generating the final differential diagnosis list.

[0011] The above system, optionally, features a diagnostic report generation module that automatically generates structured diagnostic reports, highlighting key supporting evidence.

[0012] Optionally, the system described above can be integrated with AR remote consultation on the physician's end. Primary care physicians can request assistance from experts through the physician's end, and the diagnostic report and real-time guidance gestures and annotations from the remote expert can be presented to the primary care physician's field of vision through virtual and real overlay.

[0013] The aforementioned system includes, optionally, a cloud-based collaborative platform that provides multiple web / app interfaces for physicians and application interfaces for patients, supporting multi-party consultations.

[0014] A remote intelligent diagnosis and treatment method based on multimodal data fusion and deep learning, applied to any of the aforementioned remote intelligent diagnosis and treatment systems based on multimodal data fusion and deep learning, comprising: S1. Collect multimodal raw medical data through the user terminal; S2. Standardize, extract features, and spatiotemporally align the collected data; S3. Perform cross-modal attention fusion on the extracted features to generate a fused feature vector rich in cross-modal information; S4. Based on the fused feature vector and combined with the clinical knowledge graph, a preliminary differential diagnosis list and confidence level are generated; if the confidence level is lower than the preset threshold or there are multiple high-probability possibilities, personalized follow-up questions are generated through the dialogue strategy model and interacted with doctors and patients through a remote interface. S5. Based on the new data or answers obtained through interaction, update the fused feature vector, perform iterative analysis, until the diagnostic confidence meets the preset requirements or reaches the preset maximum number of interaction rounds, and obtain the final diagnostic result. S6. Based on the final diagnosis results, generate a final diagnosis report, including: final diagnosis results, confidence level, key evidence and suggestions for subsequent treatment or examination, and push it to the physician for review and confirmation; S7. Doctors review, revise, and confirm the diagnosis results, or request assistance through AR remote consultation. S8. The confirmed diagnostic report is sent back to the cloud collaboration platform as feedback data for the optimization and continuous learning of the interactive diagnostic module model parameters.

[0015] As can be seen from the above technical solution, compared with the prior art, the present invention provides a remote intelligent diagnosis and treatment system and method based on multimodal data fusion and deep learning, which has the following beneficial effects: This invention achieves deep and dynamic fusion of multimodal data at the feature level through spatiotemporal alignment and cross-modal attention mechanisms, fully mining complementary information between data and effectively improving the accuracy of assisted diagnosis. It simulates clinical thinking, providing a progressive and interpretable diagnostic process. The system can proactively ask questions and explain the basis for judgment, transforming the diagnostic process from a "black box" to a "white box," greatly enhancing doctors' trust and the system's acceptability. The system possesses dynamic interactive capabilities, intelligently guiding data supplementation paths based on the uniqueness of different cases, improving its practicality and adaptability. Continuous model optimization is achieved through a closed-loop doctor feedback system, enabling the system to accumulate experience and evolve its diagnostic capabilities with use. AR remote consultation significantly enhances the interactive depth and operational guidance accuracy of telemedicine, lowering the operational threshold for primary care physicians. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 The diagram shows the structure of a remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning provided by this invention. Figure 2 The flowchart illustrates a remote intelligent diagnosis and treatment method based on multimodal data fusion and deep learning, provided by this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Reference Figure 1 As shown, this invention discloses a remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning, comprising: On the user end, multimodal data is collected through multimodal physiological sensing devices; The feature extraction and spatiotemporal alignment module connects to the user terminal to perform feature extraction and spatiotemporal alignment on multimodal data. The cross-modal attention fusion module, connected to the feature extraction and spatiotemporal alignment module, captures the relationships between features within multimodal data, calculates the correlation weights between features of different modalities, and generates a fusion feature vector rich in cross-modal information. The interactive diagnostic module, connected to the cross-modal attention fusion module, performs cross-validation and joint inference diagnostics based on the fused feature vectors to obtain the final diagnostic result; The diagnostic report generation module connects to the interactive diagnostic module and automatically generates a structured diagnostic report based on the final diagnostic results. On the physician side, it connects to the diagnostic report generation module, providing doctors with an interactive interface that supports viewing diagnostic reports and accessing AR remote consultations. The cloud-based collaborative platform connects with the user terminal, interactive diagnostic module, and physician terminal for data storage and continuous model learning.

[0020] Furthermore, user-end multimodal physiological sensing devices include: electrocardiogram monitors, blood pressure monitors, blood oxygen monitors, body temperature monitors, smart stethoscopes, portable ultrasound and mobile terminals, etc. Users can choose to add different physiological sensing devices according to their own needs. Multimodal data includes: text data, time-series data, image data, and audio data; Textual data includes: chief complaint, medical history, and questionnaire; Time-series data include: electrocardiogram, blood pressure, blood oxygen saturation, and body temperature; Image data includes: images of skin lesions, images of the tongue, photographs of the ear canal, and DICOM images; The audio data includes: heart sounds, lung sounds, and cough sounds.

[0021] Furthermore, the feature extraction and spatiotemporal alignment module includes: a data normalization unit, a temporal encoder, a spatial encoder, and a data mapping unit; Data standardization unit, which standardizes the collected multimodal data; A time-series encoder (such as a Transformer or LSTM-based network) extracts the dynamic features of time-series data and timestamps the dynamic features of the time-series data. Spatial encoders (such as CNN and Vision Transformer) extract spatial features from image data; The data mapping unit maps data from different modalities to a unified temporal-spatial framework, establishing an alignment foundation for subsequent fusion.

[0022] Furthermore, the cross-modal attention fusion module adopts a hierarchical cascaded attention mechanism. First, it uses intramodal self-attention to capture the relationships between features within each modality (such as the correlation between different bands in an electrocardiogram). Then, it introduces cross-modal cross-attention, for example, using image features as "queries" to "ask" relevant text description features, dynamically calculating the correlation weights between features of different modalities, and generating a set of fused feature vectors rich in cross-modal information.

[0023] Furthermore, the interactive diagnostic module includes: a pre-trained clinical knowledge graph, a reinforcement learning-driven dialogue strategy model, and interactive units; The clinical knowledge graph defines entities such as diseases, symptoms, signs, examinations, and complications, as well as their logical relationships. Based on the fused feature vectors, it generates a preliminary list of differential diagnoses and confidence levels. The dialogue strategy model dynamically generates follow-up questions that are most helpful in narrowing down the diagnosis based on the uncertainty of the current diagnosis and with reference to the clinical knowledge graph (e.g., "It is recommended to provide the patient's temperature fluctuation curve for the past 3 days", "Please confirm whether the pain is aggravated by pressure?"). The interactive unit interacts with doctors and patients through a remote interface. Based on follow-up questions and suggestions, it obtains new data or answers, updates the fused feature vector, and conducts a new round of diagnosis until the diagnostic confidence meets the preset requirements or reaches the preset maximum number of interaction rounds. Finally, it generates a list of differential diagnoses, realizing a closed loop of "preliminary judgment - intelligent follow-up questioning - updated judgment".

[0024] Furthermore, the structured diagnostic report automatically generated by the diagnostic report generation module highlights key supporting evidence (such as "Diagnosis of pneumonia: high confidence; main evidence: 1. 'cough and thick sputum' in the text description; 2. Audio analysis of lung auscultation shows moist rales; 3. Chest X-ray image detects infiltrative shadow in the right lower lobe").

[0025] Furthermore, the physician end can access AR remote consultation. When a primary care physician requests assistance from an expert through a physician-side terminal (AR glasses or a high-definition tablet), the diagnostic report (such as AI-annotated lesion areas and suggested ultrasound probe scanning paths) and the real-time guidance gestures and annotations from the remote expert are accurately presented in the primary care physician's field of vision in a virtual-real overlay manner, making it seem as if the physician is "in the same room" as the expert and working together.

[0026] Furthermore, the cloud-based collaborative platform provides multiple web / app interfaces for physicians and application interfaces for patients, supporting multi-party consultations.

[0027] Reference Figure 2As shown, a remote intelligent diagnosis and treatment method based on multimodal data fusion and deep learning, applied to any of the aforementioned remote intelligent diagnosis and treatment systems based on multimodal data fusion and deep learning, includes: S1. Collect multimodal raw medical data through the user terminal; S2. Standardize, extract features, and spatiotemporally align the collected data; S3. Perform cross-modal attention fusion on the extracted features to generate a fused feature vector rich in cross-modal information; S4. Based on the fused feature vector and combined with the clinical knowledge graph, a preliminary differential diagnosis list and confidence level are generated; if the confidence level is lower than the preset threshold or there are multiple high-probability possibilities, personalized follow-up questions are generated through the dialogue strategy model and interacted with doctors and patients through a remote interface. S5. Based on the new data or answers obtained through interaction, update the fused feature vector, perform iterative analysis, until the diagnostic confidence meets the preset requirements or reaches the preset maximum number of interaction rounds, and obtain the final diagnostic result. S6. Based on the final diagnosis results, generate a final diagnosis report, including: final diagnosis results, confidence level, key evidence and suggestions for subsequent treatment or examination, and push it to the physician for review and confirmation; S7. Doctors review, revise, and confirm the diagnosis results, or request assistance through AR remote consultation. S8. The confirmed diagnostic report is sent back to the cloud collaboration platform as feedback data for the optimization and continuous learning of the interactive diagnostic module model parameters.

[0028] In one specific embodiment, taking a community-based hypertensive patient as an example, the user-end device configuration includes: an electrocardiogram (ECG) sensor (single-lead / six-lead), a blood pressure monitor (automatic upper arm type), a blood oxygen saturation sensor, a pulse wave velocity (PWV) measurement module, a weight and body fat measurement module, and a mobile medical device; the mobile medical device includes: a high-definition camera (for facial microvascular and tongue imaging), an electronic stethoscope (Bluetooth connection, for collecting heart and lung sounds), a smartphone application (for symptom diary and questionnaire completion), a smartwatch (for continuous heart rate and activity monitoring), and an ambulatory blood pressure monitor (worn 24 hours a day). The physician-side configuration includes: a physician workstation (Web or App), a multimodal data visualization dashboard, and an interactive diagnostic decision support interface; Automatic collection of multimodal data, including: daily timed data, blood pressure, heart rate, blood oxygen, 30-second resting electrocardiogram (twice a day), weight and body fat (once a day). Continuous monitoring data includes: ambulatory blood pressure (automatically measured every 30 minutes), activity level (smartwatch continuously records steps and heart rate variability), and sleep quality (nighttime heart rate and blood oxygen fluctuations). All monitoring data are aligned to UTC time, with the symptom onset time as the key timestamp, and ECG, heart sounds, and pulse waves are aligned according to the cardiac cycle. The system captures the relationships between features within multimodal data, calculates the correlation weights between features of different modalities, and generates a fusion feature vector rich in cross-modal information (e.g., ST segment depression in ECG lead II causes the symptom "chest tightness", weight 0.85; T wave inversion in ECG lead V5 causes the symptom "chest tightness", weight 0.72; decreased heart rate variability causes the symptom "dizziness", weight 0.41; blood pressure fluctuations cause the symptom "chest tightness", weight 0.63). A fragment of the knowledge graph for cardiovascular and cerebrovascular diseases constructed by the system, with the following entity relationships: High blood pressure can lead to left ventricular hypertrophy, coronary atherosclerosis, and kidney damage. Chest tightness and ST segment depression suggest angina pectoris and myocardial ischemia. Left ventricular hypertrophy and reduced diastolic function indicate a high risk and may lead to heart failure; Initial findings: Poorly controlled hypertension, occasional chest tightness; Preliminary diagnosis based on knowledge graph: Step 1: Differentiation between cardiac and non-cardiac causes of chest tightness. Cardiac indicators: ECG abnormalities, myocardial enzymes, BNP; Non-cardiac indicators: pulmonary function, gastroesophageal reflux; The second step is stratification: If it is cardiac in origin, assess the degree of ischemia: Low risk: Asymptomatic ST changes; Intermediate risk: Symptomatic ST segment changes, without persistent changes; High risk: Symptomatic ST depression >0.1mV, lasting more than 1 minute; Preliminary system assessment: Risk assessment: Medium to high risk (overall score: 68 / 100) Differential diagnosis (confidence level): 1. Coronary microvascular dysfunction (65%); 2. Hypertensive heart disease with diastolic dysfunction (58%); 3. Stable angina (42%); 4. Anxiety-related chest tightness (25%); Key evidence: Holter monitoring: 3 instances of asymptomatic ST segment depression within 24 hours (maximum 0.08mV); Blood pressure variability: significant morning peak blood pressure (mean 168 / 102 mmHg); Symptom correlation: The two instances of ST depression were matched with the duration of chest tightness recorded by the patient; Cardiac structure: An ultrasound 3 months ago showed mild left ventricular hypertrophy (IVSd 12mm); To clarify the diagnosis, follow-up questions are generated, the data is updated and reassessed to arrive at the final diagnosis and generate a final diagnosis report; The doctor provides feedback after reviewing the final diagnosis: Consistent diagnosis: Hypertensive heart disease with diastolic dysfunction; Recommendation: Agreed with the medication adjustment plan; Additional advice: It is recommended to perform a carotid ultrasound to assess atherosclerosis. Result confirmed: Marked as "Correct diagnosis, good advice"; The case was uploaded to a cloud-based collaborative platform and added to the training set (after anonymization). Reinforcement learning was used to update the data: when similar features appear, cardiac ultrasound is recommended as the primary method of recommendation.

[0029] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0030] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning, characterized in that, include: On the user end, multimodal data is collected through multimodal physiological sensing devices; The feature extraction and spatiotemporal alignment module connects to the user terminal to perform feature extraction and spatiotemporal alignment on multimodal data. The cross-modal attention fusion module, connected to the feature extraction and spatiotemporal alignment module, captures the relationships between features within multimodal data, calculates the correlation weights between features of different modalities, and generates a fusion feature vector rich in cross-modal information. The interactive diagnostic module, connected to the cross-modal attention fusion module, performs cross-validation and joint inference diagnostics based on the fused feature vectors to obtain the final diagnostic result; The diagnostic report generation module connects to the interactive diagnostic module and automatically generates a structured diagnostic report based on the final diagnostic results. On the physician side, it connects to the diagnostic report generation module, providing doctors with an interactive interface that supports viewing diagnostic reports and accessing AR remote consultations. The cloud-based collaborative platform connects with the user terminal, interactive diagnostic module, and physician terminal for data storage and continuous model learning.

2. The remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 1, characterized in that, Multimodal physiological sensing devices for users include: electrocardiogram monitors, blood pressure monitors, blood oxygen monitors, body temperature monitors, smart stethoscopes, portable ultrasound and mobile terminals; Multimodal data includes: text data, time-series data, image data, and audio data; Textual data includes: chief complaint, medical history, and questionnaire; Time-series data include: electrocardiogram, blood pressure, blood oxygen saturation, and body temperature; Image data includes: images of skin lesions, images of the tongue, photographs of the ear canal, and DICOM images; The audio data includes: heart sounds, lung sounds, and cough sounds.

3. The remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 2, characterized in that, The feature extraction and spatiotemporal alignment module includes: a data normalization unit, a temporal encoder, a spatial encoder, and a data mapping unit; Data standardization unit, which standardizes the collected multimodal data; A time-series encoder extracts the dynamic features of time-series data and timestamps these dynamic features. Spatial encoders extract spatial features from image data; The data mapping unit maps data from different modalities to a unified temporal-spatial framework.

4. The remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 3, characterized in that, The cross-modal attention fusion module adopts a hierarchical cascaded attention mechanism and uses intra-modal self-attention to capture the relationships between features within multimodal data. By introducing cross-modal cross-attention, the correlation weights between features of different modalities are dynamically calculated to generate a fusion feature vector rich in cross-modal information.

5. A remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 4, characterized in that, The interactive diagnostic module includes: a pre-trained clinical knowledge graph, a reinforcement learning-driven dialogue strategy model, and interactive units; A clinical knowledge graph defines entities and logical relationships of diseases, symptoms, signs, examinations, and complications. Based on the fused feature vectors, it generates a preliminary list of differential diagnoses and confidence levels. A dialogue strategy model dynamically generates follow-up questions to narrow down the scope of diagnosis; The interactive unit interacts with doctors and patients through a remote interface. Based on follow-up questions and suggestions, it obtains new data or answers, updates the fused feature vector, and conducts a new round of diagnosis until the diagnostic confidence meets the preset requirements or reaches the preset maximum number of interaction rounds, generating the final differential diagnosis list.

6. A remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 5, characterized in that, The diagnostic report generation module automatically generates structured diagnostic reports, highlighting key supporting evidence.

7. A remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 6, characterized in that, With the integration of AR remote consultations, primary care physicians can request assistance from experts through their devices. The diagnostic reports, along with real-time guidance gestures and annotations from remote experts, are presented to the primary care physicians through a virtual-real overlay method.

8. A remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning according to claim 7, characterized in that, The cloud-based collaborative platform provides multiple web / app interfaces for physicians and application interfaces for patients, supporting multi-party consultations.

9. A remote intelligent diagnosis and treatment method based on multimodal data fusion and deep learning, applied to a remote intelligent diagnosis and treatment system based on multimodal data fusion and deep learning as described in any one of claims 1-8, comprising: S1. Collect multimodal raw medical data through the user terminal; S2. Standardize, extract features, and spatiotemporally align the collected data; S3. Perform cross-modal attention fusion on the extracted features to generate a fused feature vector rich in cross-modal information; S4. Based on the fused feature vector and combined with the clinical knowledge graph, a preliminary differential diagnosis list and confidence level are generated; if the confidence level is lower than the preset threshold or there are multiple high-probability possibilities, personalized follow-up questions are generated through the dialogue strategy model and interacted with doctors and patients through a remote interface. S5. Based on the new data or answers obtained through interaction, update the fused feature vector, perform iterative analysis, until the diagnostic confidence meets the preset requirements or reaches the preset maximum number of interaction rounds, and obtain the final diagnostic result. S6. Based on the final diagnosis results, generate a final diagnosis report, including: final diagnosis results, confidence level, key evidence and suggestions for subsequent treatment or examination, and push it to the physician for review and confirmation; S7. Doctors review, revise, and confirm the diagnosis results, or request assistance through AR remote consultation. S8. The confirmed diagnostic report is sent back to the cloud collaboration platform as feedback data for the optimization and continuous learning of the interactive diagnostic module model parameters.

Images