A multimodal data fusion intelligent auxiliary diagnosis platform implementation method and device

Abstract

This application provides a method and device for implementing an intelligent assisted consultation platform based on multimodal data fusion, relating to the field of intelligent assisted consultation platforms. The method includes steps S1: acquiring multimodal interaction data from users; step S2: constructing an information state tracking model based on clinical cognition; step S3: performing multimodal information fusion on the multimodal data acquired in each round of interaction; and step S4: designing an uncertainty measurement module to calculate the overall uncertainty index of the current information state based on the dynamic information state graph. The overall uncertainty index includes the information entropy, information consistency score, and information sufficiency score of each information node. This application can determine the sufficiency and consistency of the acquired information in real time during the consultation process and adaptively adjust the follow-up questioning strategy according to the user's identity, thereby significantly improving consultation efficiency and the completeness of information collection, reducing invalid or redundant questioning rounds, and lowering the risk of misdiagnosis due to contradictory information.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted medical consultation platforms, and more specifically, to a method and device for implementing an intelligent assisted medical consultation platform that integrates multimodal data fusion. Background Technology

[0002] Existing intelligent assisted consultation platforms primarily utilize medical big data models, medical knowledge graphs, and natural language processing as their core technologies. They are divided into C-end services for patients and B-end auxiliary systems for hospital doctors. The C-end platform provides users with 24 / 7 symptom consultation, intelligent triage, report interpretation, medication guidance, and chronic disease management services through text, voice, and images. It helps users understand their condition through multiple rounds of interaction and guides them to seek appropriate medical treatment. The B-end system serves as a clinical auxiliary tool, assisting doctors in completing pre-consultation, automatically generating structured medical records, providing clinical decision-making references, and improving the efficiency and standardization of diagnosis and treatment.

[0003] Most existing intelligent assisted consultation platforms adopt fixed-path questionnaire-style or slot-filling dialogue management, lacking information status tracking and uncertainty measurement mechanisms based on clinical cognition. They are unable to dynamically judge the sufficiency, consistency, and key missing items of the information obtained during the consultation process, resulting in low consultation efficiency or incomplete collection of key information.

[0004] Therefore, we have made improvements to this and proposed a method and device for implementing an intelligent assisted diagnosis platform based on multimodal data fusion. Summary of the Invention

[0005] The purpose of this invention is to address the problem that most existing intelligent assisted consultation platforms adopt fixed-path questionnaire-style or slot-filling dialogue management, lack information status tracking and uncertainty measurement mechanisms based on clinical cognition, and are unable to dynamically judge the sufficiency, consistency and key missing items of the information obtained during the consultation process, resulting in low consultation efficiency or incomplete collection of key information.

[0006] To achieve the above-mentioned objectives, this invention provides a method and device for implementing an intelligent assisted diagnosis platform based on multimodal data fusion, in order to solve the aforementioned problems.

[0007] The application is as follows: A method for implementing an intelligent assisted diagnosis platform based on multimodal data fusion includes the following steps: Step S1: Obtain the user's multimodal interaction data, wherein the multimodal interaction data includes at least one or more of the following: text description, voice signal, medical image and physiological signal data; Step S2: Construct an information state tracking model based on clinical cognition. The information state tracking model maintains a dynamic information state graph, which includes collected information nodes, information confidence, information consistency edges, and information missing degree. The logical dependencies between information nodes are modeled through a clinical cognition rule base. Step S3: For the multimodal data obtained in each round of interaction, perform multimodal information fusion, inject the fused information into the dynamic information state graph, and update the confidence and information consistency edges of each information node; Step S4: Design an uncertainty measurement module to calculate the overall uncertainty index of the current information state based on the dynamic information state diagram. The overall uncertainty index includes the information entropy, information consistency score, and information sufficiency score of each information node. Step S5: Based on the overall uncertainty index and the information missing degree, combined with the user's identity characteristics, the key missing information items in the current consultation process are dynamically identified through the decision network, and the next round of consultation questions or consultation paths are generated in accordance with the user's identity until the preset termination conditions are met.

[0008] As a preferred technical solution of this application, the step S2 of constructing an information state tracking model based on clinical cognition specifically includes: Construct a clinical cognitive ontology knowledge graph, which includes entities such as symptoms, signs, examinations, diagnoses, medications, time attributes, and triggering factors, as well as their semantic relationships. Based on the ontology knowledge graph, clinical logical dependency rules between information nodes are defined, including necessary condition rules, sufficient condition rules, mutual exclusion rules, and time sequence rules. Establish a data structure for the information state graph, where each information node corresponds to a clinical information item, and each information node includes the value status (known / unknown / contradictory), confidence distribution, and timestamp.

[0009] As a preferred technical solution of this application, the multimodal information fusion in step S3 includes: Speech recognition and sentiment analysis are performed on speech signals to extract text content and tone features; Feature extraction is performed on medical images, and lesion features, anatomical structures and abnormal indicators in the images are identified through a pre-trained deep learning model; Time-domain and frequency-domain analysis of physiological signal data is performed to extract key physiological parameters; The information extracted from the different modalities is aligned and fused through a cross-modal attention mechanism to output a unified information tuple (entity, attribute, value, confidence).

[0010] As a preferred technical solution of this application, the uncertainty measurement module in step S4 is specifically executed as follows: Calculate the information entropy of each information node. The probability distribution of different values ​​for this information node; Calculate the information consistency score: Based on the consistency edge in the information state graph, detect whether there is a logical contradiction between the currently collected information nodes. If there is a contradiction, mark the contradictory node and reduce the consistency score. Calculate the information sufficiency score: Based on the minimum information set required for consultation as defined in the clinical cognitive rule base, calculate the coverage of the currently collected information set and the minimum information set, and perform a weighted summation by combining the clinical weights of the information nodes.

[0011] As a preferred technical solution of this application, step S5, which combines user identity features, specifically includes: During the initial stage of consultation or interaction, user identity feature vectors are extracted. The user identity features include at least one or more of the following: age, gender, medical history, health literacy level, language habits, and cognitive ability. The state space of the decision network is dynamically adjusted based on the user identity features, with the user identity features serving as an additional input dimension. When generating the next round of consultation questions, the decision network adaptively adjusts the complexity of the question wording, the degree of use of medical terminology, the granularity of follow-up questions, and the priority of information collection based on the user's identity characteristics. For users with low health literacy, medical questions are automatically translated into simpler terms and example options are provided. For users with a clear history of chronic diseases, follow-up questions are given on complications and medication adherence related to the chronic disease.

[0012] As a preferred technical solution of this application, step S5, which involves dynamically identifying missing key information items through a decision network, specifically includes: A dynamic consultation decision network based on reinforcement learning is constructed. The state space of the decision network is the concatenation of the state representation vector of the current dynamic information state graph and the user identity feature vector, and the action space is the set of candidate consultation questions or the set of information collection actions. The reward function of the decision network is designed by comprehensively considering the degree of uncertainty reduction, the efficiency of consultation rounds, the completeness of information collection, and the user-friendliness of interaction. The decision network selects the action that minimizes the overall uncertainty index and has the highest information gain based on the current state, whereby the information gain is calculated based on the expected change in confidence.

[0013] As a preferred technical solution of this application, the preset termination conditions include: The overall uncertainty index is below the preset threshold; The information sufficiency score exceeds the preset sufficiency threshold; No new high-confidence information was obtained in two consecutive rounds of consultations; The number of consultation rounds has reached the maximum limit.

[0014] As a preferred technical solution of this application, step S6 is also included: During or after a consultation, a structured electronic medical record is automatically generated. The structured electronic medical record includes confirmed information nodes, uncertain information annotations, and suggested supplementary examination items.

[0015] A multimodal data fusion-based intelligent assisted diagnosis device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes a method for implementing a multimodal data fusion-based intelligent assisted diagnosis platform.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: In the scheme of this application: By introducing dynamic information status tracking, uncertainty measurement, and user identity-based decision networks, the sufficiency and consistency of the information obtained can be judged in real time during the consultation process, key missing information items can be automatically identified, and follow-up questioning strategies can be adaptively adjusted according to user identity. This significantly improves consultation efficiency and the completeness of information collection, reduces invalid or redundant questioning rounds, and reduces the risk of misdiagnosis due to contradictory information. Attached Figure Description

[0017] Figure 1 A schematic diagram illustrating the implementation method and steps of the intelligent assisted diagnosis platform based on multimodal data fusion provided in this application; Figure 2 A schematic diagram of the structure of the intelligent assisted diagnosis device with multimodal data fusion provided in this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] The present invention will be further described below with reference to embodiments.

[0020] Example: Figure 1 As shown, a method for implementing an intelligent assisted diagnosis platform based on multimodal data fusion includes the following steps: Step S1: Obtain the user's multimodal interaction data, wherein the multimodal interaction data includes at least one or more of the following: text description, voice signal, medical image and physiological signal data; Preferably, the text description receives the user's freely input description of the patient's condition via a natural language interface; the speech signal is acquired by a microphone array, first segmented by VAD (voice activity detection), and then fed into an end-to-end speech recognition model (such as Whisper or Conformer) to convert it into text, while simultaneously extracting prosodic features such as fundamental frequency and energy for emotion recognition; medical images, including but not limited to X-rays, CT scans, and MRIs, are read via the DICOM protocol, preprocessed (resampled, normalized), and input into a pre-trained ResNet-3D or Vision Transformer model to extract lesion features; physiological signal data such as ECG, blood oxygen, and blood pressure can be acquired in real time via Bluetooth wearable devices, and after filtering and denoising, time-domain features (mean, variance) and frequency-domain features (power spectral density) are calculated. Step S2: Construct an information state tracking model based on clinical cognition. The information state tracking model maintains a dynamic information state graph, which includes collected information nodes, information confidence, information consistency edges, and information missing degree. The logical dependencies between information nodes are modeled through a clinical cognition rule base. Preferably, the dynamic information state graph is stored in a graph database (such as Neo4j) or an in-memory graph structure. In addition to the value state, each information node also records the parameter form of the confidence distribution (such as the α and β values ​​of the Beta distribution) and a timestamp for subsequent time-series inference.

[0021] Step S3: For the multimodal data obtained in each round of interaction, perform multimodal information fusion, inject the fused information into the dynamic information state graph, and update the confidence and information consistency edges of each information node; Preferably, the multimodal fusion adopts a cross-modal gating attention mechanism, which performs a weighted summation of the feature vectors extracted from different modalities, with the weights dynamically adjusted by the current confidence of each modality.

[0022] Step S4: Design an uncertainty measurement module to calculate the overall uncertainty index of the current information state based on the dynamic information state diagram. The overall uncertainty index includes the information entropy, information consistency score, and information sufficiency score of each information node. Preferably, the information entropy is calculated using the differential entropy of continuous variables or the Shannon entropy of discrete variables. The consistency score is based on predefined logical constraints in the clinical cognitive rule base (such as "fever" and "normal body temperature" cannot be true at the same time) for conflict detection. The sufficiency score uses an adaptive threshold based on the information gain ratio.

[0023] Step S5: Based on the overall uncertainty index and the information missing degree, combined with the user's identity characteristics, the key missing information items in the current consultation process are dynamically identified through the decision network, and the next round of consultation questions or consultation paths are generated in accordance with the user's identity until the preset termination conditions are met.

[0024] Preferably, the decision network adopts a deep Q-network (DQN) or a policy gradient method, the state space is a concatenation of graph embedding vectors and user identity feature vectors, and the action space includes 400 common consultation question templates and information collection actions (such as requesting to upload images).

[0025] Compared to existing questionnaire-based consultation platforms with fixed paths, this new platform, by introducing dynamic information status tracking, uncertainty measurement, and a user-identity-based decision network, can judge the sufficiency and consistency of the information obtained in real time during the consultation process, automatically identify key missing information items, and adaptively adjust follow-up questioning strategies according to user identity. This significantly improves consultation efficiency and the completeness of information collection, reduces invalid or redundant questioning rounds, and lowers the risk of misdiagnosis due to contradictory information.

[0026] Furthermore, constructing an information status tracking model based on clinical cognition specifically includes: A clinical cognitive ontology knowledge graph is constructed, which includes entities such as symptoms, signs, examinations, diagnoses, medications, time attributes, and inducing factors, as well as their semantic relationships. Preferably, the clinical cognitive ontology knowledge graph is formally described using OWL (Web Ontology Language). Based on the ontology knowledge graph, clinical logical dependency rules between information nodes are defined, including necessary condition rules, sufficient condition rules, mutual exclusion rules, and time sequence rules. Necessary condition rules include "if pneumonia is diagnosed, there must be fever or cough"; sufficient condition rules include "if fever, dry cough, and ground-glass opacities in the lungs occur at the same time, it is highly suggestive of pneumonia caused by COVID-19 infection"; mutually exclusive rules include "positive pregnancy test" and "post-hysterectomy" are mutually exclusive; and temporal rules include "rash" must occur after "fever". Establish a data structure for the information state graph, where each information node corresponds to a clinical information item, and each information node includes the value status (known / unknown / contradictory), confidence distribution, and timestamp.

[0027] The confidence distribution of each information node in the information state graph is represented by Bayesian probability or fuzzy membership degree. "Contradictions" in the value state are marked when triggered by mutual exclusion rules, and the source edge of the contradiction is recorded. The timestamp uses UNIX millisecond-level time to determine the timeliness weight of the information. In addition, the model also maintains a dynamic "information demand set", that is, a list of information nodes required for the current diagnosis but not yet collected. This list is updated in real time by the rule engine as the collected information changes.

[0028] By constructing a cognitive ontology knowledge graph and information state graph with rich semantic relationships and clinical logical rules, the platform can understand the inherent connections between symptoms, signs, and examination results, just like a clinician, automatically detect information contradictions, and identify key information that is currently missing. This structured knowledge representation not only improves the logical rigor of the consultation but also provides a computable foundation for subsequent uncertainty measurement and decision networks.

[0029] Furthermore, the multimodal information fusion in step S3 includes: Speech recognition and sentiment analysis are performed on speech signals to extract text content and tone features; Feature extraction is performed on medical images, and lesion features, anatomical structures and abnormal indicators in the images are identified through a pre-trained deep learning model; Time-domain and frequency-domain analysis of physiological signal data is performed to extract key physiological parameters; The information extracted from the different modalities is aligned and fused through a cross-modal attention mechanism to output a unified information tuple (entity, attribute, value, confidence).

[0030] Preferably, the text obtained after speech recognition and the directly input text are fed together into a pre-trained medical language model (such as BioBERT or MedBERT) for entity recognition and relation extraction; tone features, including speech rate, pause frequency, and pitch variation, are classified into states such as "anxiety", "hesitation", and "calm" through SVM or lightweight neural networks, and used as a moderating factor for information confidence (such as the reduced confidence of symptom description when hesitant). In medical image processing, DenseNet-121 is used to extract lung field, heart contour and nodule features for chest X-rays; 3D U-Net is used to segment lesion regions for CT sequences and quantitative indicators such as lesion volume, density and edge smoothness are calculated; all image features are mapped to 128-dimensional vectors through fully connected layers. Preferably, physiological signal processing includes: detecting R waves from electrocardiogram signals using the Pan-Tompkins algorithm, calculating heart rate and heart rate variability; extracting blood oxygen saturation and perfusion index from blood oxygen signals; and generating discrete events (such as "blood oxygen is below 94%) after comparing these parameters with clinical thresholds.

[0031] The cross-modal attention mechanism employs a multi-head attention structure, using text features as queries and image and physiological features as keys and values. It calculates a weighted fused feature vector, which is then mapped to information tuples through an output layer. Each information tuple contains a clinical entity (e.g., "lung nodule"), attribute (e.g., "size"), value (e.g., "8mm"), and confidence (calculated jointly by attention weights and original modality confidence). The fused information tuples are injected into the dynamic information state graph in chronological order. If there is a conflict with existing nodes, a consistency detection process is triggered.

[0032] By comprehensively utilizing multiple data sources such as text, voice, images, and physiological signals, especially for children, the elderly, or critically ill patients who cannot clearly express their symptoms, images and physiological signals can provide objective supplements; the cross-modal attention mechanism effectively solves the semantic alignment problem between different modal information, outputs unified information tuples with confidence, provides high-quality input for the information state graph, thereby improving the robustness and accuracy of the consultation platform in complex clinical scenarios.

[0033] Furthermore, the uncertainty measurement module in step S4 specifically performs the following: Calculate the information entropy of each information node. The probability distribution of different values ​​for this information node; Calculate the information consistency score: Based on the consistency edge in the information state graph, detect whether there is a logical contradiction between the currently collected information nodes. If there is a contradiction, mark the contradictory node and reduce the consistency score. Calculate the information sufficiency score: Based on the minimum information set required for consultation as defined in the clinical cognitive rule base, calculate the coverage of the currently collected information set and the minimum information set, and perform a weighted summation by combining the clinical weights of the information nodes.

[0034] In practice, for continuous information nodes (such as body temperature), the probability density function is obtained by kernel density estimation, and then the differential entropy is calculated; for discrete nodes (such as whether symptoms are present or absent), the Shannon entropy is calculated directly; the initial value of the confidence distribution p(v) comes from the multimodal fusion module, and is iteratively corrected after each round of consultation by combining the Bayesian update rule with the user's answer. The initial information consistency score is 1.0. For each pair of contradictory nodes detected (e.g., "body temperature 38.5℃" and "self-reported no fever"), 0.1 to 0.3 points are deducted according to the severity of the contradiction, and the contradictory node is marked as "needs clarification".

[0035] In the information sufficiency score, the minimum information set is dynamically generated based on the chief complaint. For example, the minimum information set for the chief complaint "chest pain" includes the location, nature, duration, radiation location, triggering and relieving factors, and accompanying symptoms (dyspnea, sweating), etc. Each information node is assigned a clinical weight (e.g., "chest pain radiating to the left arm" has a weight of 0.8, and "pain is squeezing" has a weight of 0.9), and the coverage is weighted, summed, and normalized to 0-1.

[0036] Preferably, the overall uncertainty index is the weighted sum of the three scores, calculated using the following formula: in For normalized entropy, For consistency scoring, For sufficiency scoring, the weights are adjustable (typical value). =0.3, =0.3, =0.4). When When the uncertainty level is below the threshold of 0.2, it is considered low uncertainty and the consultation can be terminated.

[0037] By providing a quantitative uncertainty measurement mechanism, the platform can accurately determine the quality of the current information state. Information entropy reflects the degree of ambiguity of information, consistency score reveals logical contradictions in the patient's description, and sufficiency score directly guides whether further questioning is needed. The weighted fusion of these three factors outputs an overall uncertainty index, providing an objective and calculable basis for the termination conditions and action selection of the decision network. This avoids the arbitrariness of judgments based on experience or fixed rules in traditional methods, thereby achieving adaptive optimization of the consultation process.

[0038] Furthermore, step S5, which combines user identity features, specifically includes: During the initial stage of consultation or interaction, user identity feature vectors are extracted. The user identity features include at least one or more of the following: age, gender, medical history, health literacy level, language habits, and cognitive ability. The state space of the decision network is dynamically adjusted based on the user identity features, with the user identity features serving as an additional input dimension. When generating the next round of consultation questions, the decision network adaptively adjusts the complexity of the question wording, the degree of use of medical terminology, the granularity of follow-up questions, and the priority of information collection based on the user's identity characteristics. For users with low health literacy, medical questions are automatically translated into simpler terms and example options are provided; for users with a clear history of chronic diseases, information on complications and medication adherence related to the chronic disease is prioritized.

[0039] In practice, user identity features are extracted in the following ways: age and gender are obtained from user registration information or the first round of conversation; past medical history is extracted from historical conversations through keyword matching or NER; health literacy level is inferred by a short assessment scale (such as a single-item self-report "How well do you understand medical terms?" 1-5 points) or by whether medical terms are used in the conversation (such as "hypertension" vs "high blood pressure"); language habits are analyzed by sentence length and word complexity in the conversation history; cognitive ability is judged by reaction time and whether repeated questions are needed; the identity feature vector is normalized and then concatenated into the state vector.

[0040] Preferably, the output layer of the decision network's strategy network is divided into two branches: one branch selects the question template ID, and the other branch outputs the question rewriting parameters (such as a popularization coefficient of 0~1 and a terminology masking list). For popularization, a mapping dictionary from medical terms to common expressions is maintained; for example, "vomiting" is mapped to "nausea, vomiting," and "frequent urination" is mapped to "urinating many times a day." For users with a history of chronic diseases (such as diabetes), the system automatically increases the weight of nodes such as "frequent blood glucose monitoring," "abnormal foot sensation," and "vision changes" in the information demand set, and prioritizes asking follow-up questions about these contents in the early stages of the consultation. At the same time, the granularity of follow-up questions is adjusted according to the user's age: for elderly users, complex questions are broken down into multiple simple true / false questions; for children, instructions for guardians are added when guardians answer.

[0041] By incorporating user identity features, the online consultation platform can provide personalized interactions for different groups, significantly improving user experience and the quality of information collection. For users with low health literacy or the elderly, simplified language and granular follow-up questions lower the barrier to understanding and cognitive burden, improving the accuracy of answers. For patients with chronic diseases, prioritizing follow-up questions about related complications and medications helps in the early detection of changes in their condition. This adaptive mechanism allows the platform to serve both the general public and the clinical needs of specific populations, expanding its applicability and clinical value.

[0042] Furthermore, step S5, which involves dynamically identifying missing key information items through a decision network, specifically includes: A dynamic consultation decision network based on reinforcement learning is constructed. The state space of the decision network is the concatenation of the state representation vector of the current dynamic information state graph and the user identity feature vector, and the action space is the set of candidate consultation questions or the set of information collection actions. The reward function of the decision network is designed by comprehensively considering the degree of uncertainty reduction, the efficiency of consultation rounds, the completeness of information collection, and the user-friendliness of interaction. The decision network selects the action that minimizes the overall uncertainty index and has the highest information gain based on the current state, whereby the information gain is calculated based on the expected change in confidence.

[0043] In practice, the state representation vector encodes the dynamic information state graph using a graph neural network (GNN): the features of each information node (value state, confidence level, timestamp) are used as the initial node embedding, which is then aggregated with the information of neighboring nodes through a two-layer graph convolutional network (GCN), and then global pooling is used to obtain the graph-level embedding. Finally, it is concatenated with the user identity feature vector to form the state vector (256 dimensions). The action space includes 400 candidate question templates (such as "Where is your pain located?", "When did the pain start?") and 10 information collection actions (such as "Please upload a chest X-ray", "Please measure your blood pressure").

[0044] Preferably, the reward function is designed as follows: in The reduction in overall uncertainty (the difference in U-value between the current state and the next state). For the current consultation round (short paths are encouraged). This refers to the improvement in information integrity (the number of newly added high-confidence information nodes). User interaction friendliness (calculated based on question repetition rate, question length, and whether plain language is used, with a value of 0-1). , , , Hyperparameters (typical values) =0.5, =0.1, =0.3, =0.1).

[0045] Information gain is calculated using a Bayesian experimental design: for each candidate action 'a', the probability distribution of possible user responses is simulated (based on historical data or a pre-trained model), and the expected entropy reduction of the information state graph after the action is executed is calculated. The decision network is trained using a dual deep Q-network (DDQN), and the experience replay pool stores a quadruple (state, action, reward, next state). The target network is updated every 100 steps. In actual deployment, the inference time of each round of the decision network is controlled within 50ms to meet the requirements of real-time interaction.

[0046] By employing a reinforcement learning-driven dynamic triage decision network, the limitations of traditional fixed-path or rule-based methods are overcome. Through maximizing information gain and comprehensively considering multi-objective optimization of the reward function, the decision network can automatically select the most helpful follow-up questions or data collection actions in each round to reduce uncertainty and improve information completeness, while simultaneously balancing triage efficiency and user-friendliness. This enables the platform to acquire the most diagnostically valuable information within a limited number of rounds, making it particularly suitable for the rapid triage of complex or atypical cases.

[0047] Furthermore, the preset termination conditions include: The overall uncertainty index is below the preset threshold; The information sufficiency score exceeds the preset sufficiency threshold; No new high-confidence information was obtained in two consecutive rounds of consultations; The number of consultation rounds has reached the maximum limit.

[0048] Preferably, the preset threshold can be dynamically configured according to the clinical scenario: for emergency scenarios, the overall uncertainty threshold is set to 0.4 (allowing for higher uncertainty to facilitate rapid triage); for chronic disease follow-up, the threshold is set to 0.2. The sufficiency threshold is usually set to 0.85, that is, more than 85% of the minimum information set has been collected.

[0049] Two consecutive rounds of no new high-confidence information means that in two consecutive rounds, the confidence level of the information nodes in the newly injected information state graph is lower than 0.6 (out of 1), or no new information nodes are added. The maximum number of rounds is set to 15 by default. For simple cases, the process can be terminated within 5 rounds; for complex cases, the user can manually adjust it to 25 rounds. When any termination condition is met, the consultation process automatically ends and triggers step S6 to generate a structured medical record.

[0050] In addition, the system also supports the "early termination" function: if the user actively expresses the intention of "I am sure it is just a cold" or "no need to ask any more questions", the remaining follow-up questions will be skipped and the process will end directly; all termination conditions are evaluated before each action of the decision network, and the evaluation priority is: user-initiated termination > no new information for two consecutive rounds > sufficiency meets the standard > uncertainty meets the standard > round limit.

[0051] By setting multiple termination conditions, the system balances information sufficiency, efficiency, user experience, and security. The combination of an overall uncertainty threshold and a sufficiency threshold ensures sufficient clinical credibility of the consultation results; two consecutive rounds of no new information detection prevent falling into an ineffective loop. The upper limit on the number of rounds serves as a safety net to prevent endless follow-up questions in extreme cases; this multi-level termination mechanism allows the platform to ensure the quality of consultations while flexibly adapting to the timeliness requirements of different scenarios.

[0052] The method also includes step S6: During or after a consultation, a structured electronic medical record is automatically generated. The structured electronic medical record includes confirmed information nodes, uncertain information annotations, and suggested supplementary examination items.

[0053] The structured electronic medical records follow the HL7 FHIR standard or the domestic electronic medical record dataset standard (such as the "Basic Dataset of Electronic Medical Records") and are output in JSON or XML format. Confirmed information nodes refer to information nodes with a confidence level higher than 0.8, and are organized into chapters such as chief complaint, present illness, past medical history, physical examination, and auxiliary examinations.

[0054] Uncertainty information labeling includes: marking nodes with confidence levels between 0.5 and 0.8 as "to be confirmed" and attaching the confidence level value; marking contradictory nodes as "requires verification" and listing the specific descriptions and possible causes of the contradictory parties (such as "the patient reported no fever, but the body temperature was measured at 38.2℃").

[0055] The suggested supplementary examination items are generated based on the current information gaps and the clinical cognitive rule base, such as "It is recommended to perform an electrocardiogram to rule out myocardial infarction" or "It is recommended to test C-reactive protein and complete blood count." During the generation process, the system calls the template engine to fill the node values ​​in the information status diagram into the preset paragraph structure.

[0056] For uncertain or contradictory information, the medical record will be highlighted with a special color or annotation to remind doctors to pay special attention. After the medical record is generated, it can be exported as a PDF with one click or directly uploaded to the hospital's HIS system. In addition, "intermediate medical records" can be generated in real time during the consultation process, showing the currently confirmed information and the uncertainties that still exist, so that users (patients or doctors) can view and correct them at any time.

[0057] By automatically converting consultation results into structured electronic medical records, the burden on doctors to write medical records is greatly reduced, while ensuring the standardization and completeness of the records. The annotation of uncertainty information and suggestions for supplementary examinations provide doctors with clear clinical decision support, helping them quickly identify areas requiring further verification or investigation. This end-to-end automated process significantly improves diagnostic and treatment efficiency and reduces medical risks caused by omissions in medical record writing.

[0058] like Figure 2 The diagram also includes a multimodal data fusion intelligent auxiliary diagnosis device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes a method for implementing a multimodal data fusion intelligent auxiliary diagnosis platform.

[0059] The electronic device can be a general-purpose computer (such as a PC or server), a dedicated medical terminal (such as an intelligent consultation all-in-one machine), a tablet computer, a smartphone, or an embedded medical device; the processor includes at least a CPU, and for deep learning inference tasks, it may include a GPU, NPU, or TPU; the memory includes main memory (DDR) and secondary storage (SSD). The device is also equipped with necessary input / output interfaces: microphone, camera, touch screen, network interface, and Bluetooth / Wi-Fi module, for collecting multimodal data and displaying the consultation interaction interface.

[0060] For mobile devices, the program runs as an app and can utilize the device's built-in sensors (such as an accelerometer for fall detection) to expand data acquisition capabilities. The device and cloud collaborate: a lightweight model processes speech and simple text in real-time on the device, while complex image analysis and reinforcement learning decisions are uploaded to the cloud server for completion, with the results returned to the device for display. This edge-cloud collaborative architecture balances response speed and computing power.

[0061] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0062] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A method for implementing an intelligent assisted diagnosis platform based on multimodal data fusion, characterized in that, Includes the following steps: Step S1: Obtain the user's multimodal interaction data, wherein the multimodal interaction data includes at least one or more of the following: text description, voice signal, medical image and physiological signal data; Step S2: Construct an information state tracking model based on clinical cognition. The information state tracking model maintains a dynamic information state graph, which includes collected information nodes, information confidence, information consistency edges, and information missing degree. The logical dependencies between information nodes are modeled through a clinical cognition rule base. Step S3: For the multimodal data obtained in each round of interaction, perform multimodal information fusion, inject the fused information into the dynamic information state graph, and update the confidence and information consistency edges of each information node; Step S4: Design an uncertainty measurement module to calculate the overall uncertainty index of the current information state based on the dynamic information state diagram. The overall uncertainty index includes the information entropy, information consistency score, and information sufficiency score of each information node. Step S5: Based on the overall uncertainty index and the information missing degree, combined with the user's identity characteristics, the key missing information items in the current consultation process are dynamically identified through the decision network, and the next round of consultation questions or consultation paths are generated in accordance with the user's identity until the preset termination conditions are met.

2. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 1, characterized in that, The construction of the information state tracking model based on clinical cognition in step S2 specifically includes: Construct a clinical cognitive ontology knowledge graph, which includes entities such as symptoms, signs, examinations, diagnoses, medications, time attributes, and triggering factors, as well as their semantic relationships. Based on the ontology knowledge graph, clinical logical dependency rules between information nodes are defined, including necessary condition rules, sufficient condition rules, mutual exclusion rules, and time sequence rules. Establish a data structure for the information state graph, where each information node corresponds to a clinical information item, and each information node includes the value status (known / unknown / contradictory), confidence distribution, and timestamp.

3. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 2, characterized in that, The multimodal information fusion in step S3 includes: Speech recognition and sentiment analysis are performed on speech signals to extract text content and tone features; Feature extraction is performed on medical images, and lesion features, anatomical structures and abnormal indicators in the images are identified through a pre-trained deep learning model; Time-domain and frequency-domain analysis of physiological signal data is performed to extract key physiological parameters; The information extracted from the different modalities is aligned and fused through a cross-modal attention mechanism to output a unified information tuple (entity, attribute, value, confidence).

4. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 3, characterized in that, The uncertainty measurement module in step S4 is specifically executed as follows: Calculate the information entropy of each information node. The probability distribution of different values ​​for this information node; Calculate the information consistency score: Based on the consistency edge in the information state graph, detect whether there is a logical contradiction between the currently collected information nodes. If there is a contradiction, mark the contradictory node and reduce the consistency score. Calculate the information sufficiency score: Based on the minimum information set required for consultation as defined in the clinical cognitive rule base, calculate the coverage of the currently collected information set and the minimum information set, and perform a weighted summation by combining the clinical weights of the information nodes.

5. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 4, characterized in that, Step S5, which combines user identity features, specifically includes: During the initial stage of consultation or interaction, user identity feature vectors are extracted. The user identity features include at least one or more of the following: age, gender, medical history, health literacy level, language habits, and cognitive ability. The state space of the decision network is dynamically adjusted based on the user identity features, with the user identity features serving as an additional input dimension. When generating the next round of consultation questions, the decision network adaptively adjusts the complexity of the question wording, the degree of use of medical terminology, the granularity of follow-up questions, and the priority of information collection based on the user's identity characteristics. For users with low health literacy, medical questions are automatically translated into simpler terms and example options are provided. For users with a clear history of chronic diseases, follow-up questions are given on complications and medication adherence related to the chronic disease.

6. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 5, characterized in that, The step S5, which involves dynamically identifying missing key information items through a decision network, specifically includes: A dynamic consultation decision network based on reinforcement learning is constructed. The state space of the decision network is the concatenation of the state representation vector of the current dynamic information state graph and the user identity feature vector, and the action space is the set of candidate consultation questions or the set of information collection actions. The reward function of the decision network is designed by comprehensively considering the degree of uncertainty reduction, the efficiency of consultation rounds, the completeness of information collection, and the user-friendliness of interaction. The decision network selects the action that minimizes the overall uncertainty index and has the highest information gain based on the current state, whereby the information gain is calculated based on the expected change in confidence.

7. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 6, characterized in that, The preset termination conditions include: The overall uncertainty index is below the preset threshold; The information sufficiency score exceeds the preset sufficiency threshold; No new high-confidence information was obtained in two consecutive rounds of consultations; The number of consultation rounds has reached the maximum limit.

8. The method for implementing a multimodal data fusion intelligent assisted diagnosis platform according to claim 7, characterized in that, It also includes step S6: During or after a consultation, a structured electronic medical record is automatically generated. The structured electronic medical record includes confirmed information nodes, uncertain information annotations, and suggested supplementary examination items.

9. A multimodal data fusion intelligent auxiliary diagnosis device, implemented using the multimodal data fusion intelligent auxiliary diagnosis platform method as described in claim 8, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1 to 8.

Images