Medical data aided decision-making method and device fusing multi-modal data

Abstract

The invention discloses a medical data aided decision-making method and device fusing multi-modal data, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining the original medical data of a to-be-predicted patient; aligning semantics among the data of various modalities, and generating personalized feature vectors of the patient to be predicted; determining an activation weight according to the feature information of the personalized feature vector; according to knowledge data in the original medical data, a clinical rule is converted into a fine-adjustable constraint condition, and according to the constraint condition and the activated identification expert layer, feature analysis is performed on the personalized feature vector to obtain an illness state analysis state vector; assembling cue words, and reasoning the cue words; and generating a plurality of candidate reasoning paths, determining an optimal reasoning path in the candidate reasoning paths, and generating an optimal reasoning result by using the optimal reasoning path. The invention provides an LLM for performing dynamic verification and adjustment by fusing multi-dimensional contrast signals in real time in the process of medical data analysis and auxiliary decision generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a medical data-assisted decision-making method and apparatus that integrates multimodal data. Background Technology

[0002] Currently, artificial intelligence technology, especially large language models (LLM), is being applied more and more deeply in the medical field, covering multiple scenarios such as assisted diagnosis, information integration, medical record analysis, and patient follow-up.

[0003] Current LLM models, when applied to medical decision support, can only process structured data. They have weak capabilities in fusing unstructured text, time-series physiological signals, images, and other multimodal data, failing to comprehensively mine patient information. The content generated by the models lacks rigorous real-time factual and logical verification, and the "illusion" phenomenon may lead to serious misleading in medical scenarios. Moreover, general-purpose LLM models lack medical expertise, while specialized models are often limited to a single disease or task, resulting in insufficient generalization and dynamic update capabilities. The output of LLM models is a "black box," lacking interpretable signal support, making it difficult for clinicians to understand and trust their recommendations. In addition, existing validation methods are mostly post-processing, unable to dynamically intervene during the decision generation process, leading to low efficiency and risk exposure windows.

[0004] It can be seen that LLM has made significant progress in understanding and generating single modalities. However, when faced with multimodal task scenarios such as the medical field, existing LLMs struggle to achieve a comprehensive understanding.

[0005] In summary, there is an urgent need to provide an LLM that can be applied in the medical field and process multimodal medical data, thereby improving the accuracy and interpretability of intelligent understanding and matching of multimodal medical data, and integrating LLM into the medical decision support process. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a medical data-assisted decision-making method and apparatus that integrates multimodal data, thereby addressing the significant progress made by LLM in understanding and generating single modalities. However, when faced with multimodal task scenarios in the medical field, existing LLM methods struggle to achieve comprehensive understanding.

[0007] According to a first aspect, embodiments of the present invention provide a medical data-assisted decision-making method that integrates multimodal data, the method comprising: Obtain the raw medical data of the patient to be predicted; The decision-making agent encodes raw medical data containing various modalities into a pre-defined semantic vector space, aligns the semantics between data of various modalities, and generates personalized feature vectors for patients to be predicted. Based on the feature information of the personalized feature vector, determine the preset activation weight of each expert layer and establish the routing between the personalized feature vector and the expert layer. Based on the knowledge data in the original medical data, clinical rules are transformed into fine-tunable constraints. Based on the constraints and the activated expert layer, personalized feature vectors are analyzed to obtain the disease analysis state vector. The constraint parameters of the constraints are set according to the data distribution of historical medical data. Obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the large language model embedded in the decision agent to reason about the prompt words; At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated, and the candidate reasoning paths are compared in terms of fact, logic, and language dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result.

[0008] In conjunction with the first aspect, in the first embodiment of the first aspect, the decision intelligence uses a trained cross-modal semantic coding model to generate a personalized feature vector of the patient to be predicted. The cross-modal semantic coding model includes a modality data encoder and a feature fusion layer corresponding to the modalities contained in the original medical data; The input of the feature fusion layer is connected to the output of all modal data encoders, and the input of the modal data encoders is connected to the raw medical data.

[0009] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, the step of using a decision-making agent to uniformly encode raw medical data containing various modalities into a preset semantic vector space, aligning the semantics between data of various modalities, and generating a personalized feature vector for the patient to be predicted specifically includes: The raw medical data is classified according to the disease condition dimension using a decision-making intelligent agent; the disease condition dimension includes disease type, disease severity, and treatment stage. The classified raw medical data is labeled to generate tag information for the raw medical data. The raw medical data carrying the tag information is input into each modal data encoder to obtain the feature vector output by the modal data encoder; Project the feature vectors of various modalities into a preset encoding into a preset semantic vector space; The projected feature vectors are input into the feature fusion layer, and the feature fusion layer is used to align the feature vectors of various modalities, establish semantic associations between the feature vectors of various modalities, and generate personalized feature vectors for the patients to be predicted.

[0010] In conjunction with the first aspect, in the third implementation of the first aspect, at each step of the reasoning decoding, multiple candidate reasoning paths are generated, and the candidate reasoning paths are compared in terms of factual, logical, and linguistic dimensions to determine the optimal reasoning path among the candidate reasoning paths. The optimal reasoning path is then used to generate the optimal reasoning result, specifically including: In each step of the reasoning and decoding process, based on the disease state vector and the current task context, the probability distribution of the next word is predicted, resulting in several candidate words and forming several candidate reasoning paths. An initial probability is assigned to each candidate reasoning path based on the candidate words. Medical entities are extracted from each candidate reasoning path, and the extracted entities are matched and verified with a preset database to obtain the fact consistency score of each candidate reasoning path. Perform internal logical judgment on each candidate reasoning path to obtain the logical consistency score of each candidate reasoning path; Each candidate reasoning path is evaluated for its linguistic representation, and a linguistic standardization score is obtained for each candidate reasoning path. The comprehensive reasoning score for each candidate reasoning path is determined by weighted summation based on preset weights, factual consistency scores, logical consistency scores, language standardization scores, and initial probabilities. At each step of the decoding process, candidate reasoning paths are sorted according to the comprehensive reasoning score. The candidate reasoning path with the highest comprehensive reasoning score is taken as the optimal reasoning path for the corresponding step, and the candidate reasoning result of the optimal reasoning path is taken as the optimal reasoning result for the corresponding step.

[0011] In conjunction with the third implementation of the first aspect, and in the fourth implementation of the first aspect, the step of extracting medical entities from each candidate reasoning path and matching and verifying the extracted entities with a preset database to obtain a fact consistency score for each candidate reasoning path specifically includes: The prompt words are reasoned using candidate reasoning paths to obtain the candidate reasoning results for each candidate reasoning path, and medical entities are extracted from each candidate reasoning result. Check whether the extracted medical entities are within the physiological range, and obtain the first fact diagnosis result for each candidate reasoning path; Check whether the relationship between the extracted medical entities and the clinical guideline entries is correct, and obtain the second fact diagnosis result for each candidate reasoning path; Based on the first and second fact diagnostic results, a fact consistency score is obtained for each candidate reasoning path.

[0012] In conjunction with the third implementation of the first aspect, and in the fifth implementation of the first aspect, the step of performing internal logical judgment on each candidate reasoning path to obtain a logical consistency score for each candidate reasoning path specifically includes: The prompt words are reasoned using candidate reasoning paths to obtain the candidate reasoning results for each candidate reasoning path. The causal and temporal logic of each candidate reasoning result is then checked to obtain the first logical diagnosis result for each candidate reasoning path. Check whether each sentence in each candidate reasoning result contradicts the already generated context, and obtain the second logical diagnosis result for each candidate reasoning path; Check whether each candidate reasoning result matches the disease analysis state vector to obtain the third logical diagnosis result for each candidate reasoning path; Based on the first, second, and third logical diagnostic results, a logical consistency score is obtained for each candidate reasoning path.

[0013] In conjunction with the third implementation of the first aspect, and in the sixth implementation of the first aspect, the step of performing linguistic expression judgment on each candidate reasoning path to obtain a linguistic normativity score for each candidate reasoning path specifically includes: The candidate inference paths are used to infer the prompt words, and the candidate inference results of each candidate inference path are obtained. The domain language style of each candidate inference result is checked, and the first language specification result of each candidate inference path is obtained. Check the clarity of expression for each candidate reasoning result to obtain the second language standardization result for each candidate reasoning path; By comparing the semantic similarity between each candidate reasoning result, the redundancy and missing information in the candidate reasoning results are determined, and the third language standardization result of each candidate reasoning path is obtained. Based on the first, second, and third language norm scores, a language norm score is obtained for each candidate reasoning path.

[0014] According to a second aspect, embodiments of the present invention also provide a medical data-assisted decision-making device that integrates multimodal data, the device comprising: The data acquisition module is used to acquire the raw medical data of the patient to be predicted; Data encoding module. This module utilizes a decision-making agent to uniformly encode raw medical data containing various modalities into a pre-defined semantic vector space, aligning the semantics between data from different modalities to generate personalized feature vectors for patients to be predicted. The data routing module is used to determine the activation weight of each preset expert layer based on the feature information of the personalized feature vector, and to establish the routing between the personalized feature vector and the expert layer. The constraint transformation module is used to transform clinical rules into fine-tunable constraints based on knowledge data in the original medical data. Based on the constraints and the activated expert layer, it performs feature analysis on the personalized feature vector to obtain the disease analysis state vector. The constraint parameters of the constraints are set according to the data distribution of historical medical data. The data assembly module is used to obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the large language model embedded in the decision agent to reason about the prompt words; The optimal reasoning module is used to generate multiple candidate reasoning paths at each step of the reasoning and decoding process, compare the candidate reasoning paths in terms of facts, logic, and language, determine the optimal reasoning path among the candidate reasoning paths, and use the optimal reasoning path to generate the optimal reasoning result.

[0015] According to a third aspect, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the medical data-assisted decision-making method for fusing multimodal data as described above.

[0016] According to a fourth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the medical data-assisted decision-making method for fusing multimodal data as described above.

[0017] The medical data-assisted decision-making method and apparatus of the present invention, which integrates multimodal data, generates multiple candidate reasoning paths at each step of the reasoning and decoding process. These candidate paths are then compared across factual, logical, and linguistic dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result. By deeply embedding multidimensional signal verification into the generation process, real-time suppression of AI illusions is achieved, ensuring high quality in terms of facts, logic, and expression in the output content, significantly reducing the risk of misleading information that AI-assisted medical care may bring. Furthermore, by determining the activation weight of each pre-set expert layer based on the feature information of personalized feature vectors and establishing a route between personalized feature vectors and expert layers, and by transforming clinical rules into finely adjustable constraints based on knowledge data in the original medical data, feature analysis is performed on the personalized feature vectors based on the constraints and the activated expert layers to obtain the disease analysis state vector. The innovative domain-specific expert cluster design enables the decision-making agent to efficiently and accurately process and analyze complex data such as long-term physiological signals, deeply integrating with text and structured data to uncover more comprehensive disease information. LLM can rapidly process massive amounts of data, providing multi-validated key information summaries and risk alerts, improving doctors' work efficiency and decision-making quality, freeing them from the heavy workload of data screening and preliminary analysis, and allowing them to focus more on higher-order clinical thinking and doctor-patient communication.

[0018] This invention provides an LLM (Limited Ledger Model) that integrates multidimensional comparative signals for dynamic verification and adjustment in real time during medical data analysis and decision support generation. The LLM does not directly generate the final treatment plan, but rather constructs an intelligent, reliable, and interpretable medical data analysis and reasoning framework to provide clinicians with multi-verified, structured disease assessment conclusions, key indicator analysis, and decision support references, thus entrusting the final decision-making power to the doctor and improving both the efficiency and quality of medical work. Attached Figure Description

[0019] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings: Figure 1 This invention illustrates one of the flowcharts of a medical data-assisted decision-making method that integrates multimodal data. Figure 2 The second flowchart of the medical data-assisted decision-making method that integrates multimodal data provided by the present invention is shown. Figure 3 A schematic diagram of the structure of the medical data-assisted decision-making device that integrates multimodal data provided by the present invention is shown. Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Currently, artificial intelligence technology, especially LLM (Limited Learning), is being applied more and more deeply in the medical field, covering multiple scenarios such as assisted diagnosis, information integration, medical record analysis, and patient follow-up. LLM in the medical field mainly includes: Rule-based medical assistance systems use pre-defined clinical guidelines, diagnostic criteria, and treatment pathways, combined with structured patient data such as vital signs and laboratory indicators, to perform logical reasoning and output diagnostic suggestions. However, these systems are often characterized by rigid rules, difficulty in handling complex and changing clinical situations, inability to process unstructured data such as patient complaints and medical records, and high update and maintenance costs.

[0022] Traditional machine learning-based medical assistance models utilize statistical models or classic machine learning algorithms, such as logistic regression, random forests, and support vector machines, to classify or regress medical data, thereby assisting doctors in disease risk prediction and treatment efficacy evaluation. However, these models are highly feature-dependent, have poor interpretability, and struggle to integrate multimodal and heterogeneous medical data. With the acceleration of digitalization, massive amounts of heterogeneous data have been generated across various industries. In the field of medical diagnostic assistance, key information often exists not in a single modality but rather in a complex mix of text, tables, images, charts, and structured database records.

[0023] Medical text analysis based on deep learning and natural language processing: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformer architectures are used to extract information, identify entities, and mine relationships from medical texts, including medical records, image reports, and documents. However, the content generated by the model may suffer from "hallucination," meaning the output may contain information that is inconsistent with the facts or logically contradictory, posing a significant risk in high-risk scenarios such as medicine.

[0024] Medical question-answering and auxiliary generation systems based on large language models: These systems directly use or fine-tune general-purpose large language models for medical knowledge question answering, report generation, or suggestion output. However, the models may generate content lacking medical basis or inconsistent with the latest clinical guidelines. Furthermore, these auxiliary generation systems may exhibit inconsistencies in multi-turn dialogues or complex disease reasoning, leading to potentially ambiguous and unclear output that does not conform to clinical documentation standards.

[0025] Current LLM models, when applied to medical decision support, can only process structured data. They have weak capabilities in fusing unstructured text, time-series physiological signals, images, and other multimodal data, failing to comprehensively mine patient information. The content generated by the models lacks rigorous real-time factual and logical verification, and the "illusion" phenomenon may lead to serious misleading in medical scenarios. Moreover, general-purpose LLM models lack medical expertise, while specialized models are often limited to a single disease or task, resulting in insufficient generalization and dynamic update capabilities. The output of LLM models is a "black box," lacking interpretable signal support, making it difficult for clinicians to understand and trust their recommendations. In addition, existing validation methods are mostly post-processing, unable to dynamically intervene during the decision generation process, leading to low efficiency and risk exposure windows.

[0026] It can be seen that LLM has made significant progress in understanding and generating single modalities. However, when faced with multimodal task scenarios such as the medical field, existing LLMs struggle to achieve a comprehensive understanding.

[0027] In summary, there is an urgent need to provide an LLM that can be applied in the medical field and process multimodal medical data, thereby improving the accuracy and interpretability of intelligent understanding and matching of multimodal medical data, and integrating LLM into the medical decision support process.

[0028] Due to the aforementioned technical problems, this invention provides a medical data-assisted decision-making method that integrates multimodal data. The aim is to provide a multi-dimensional comparative signal dynamic verification and adjustment mechanism (LLM) that integrates multi-dimensional comparative signals in real-time during medical data analysis and decision-making generation. The LLM does not directly generate the final treatment plan; instead, it constructs an intelligent, reliable, and interpretable medical data analysis and reasoning framework to provide clinicians with multi-verified, structured disease assessment conclusions, key indicator analysis, and decision support references, ultimately empowering doctors with the final decision-making authority. This improves both the efficiency and quality of medical work. This medical data-assisted decision-making method integrating multimodal data can be used in electronic devices, including but not limited to computers and mobile terminals. Figure 1 This is a flowchart illustrating a medical data-assisted decision-making method that integrates multimodal data according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method may include the following steps: S101. Obtain the original medical data of the patient to be predicted.

[0029] In this embodiment of the invention, the raw medical data includes multiple data modalities, including at least: text data, tabular data, image data, structured medical data, time-series physiological signals, and external knowledge data.

[0030] Structured medical data can include basic patient information from digital management platforms, such as age, gender, and past medical history, as well as patient treatment data. Unstructured data, such as text data, tabular data, and image data, can include medical records, pathology records, follow-up records, laboratory test results, and imaging test results. Temporal physiological signals include data from polysomnography, Holter monitoring, continuous glucose monitoring, and ventilator therapy. For example, in the field of sleep-disordered breathing, this includes signals such as respiratory airflow, blood oxygen saturation, chest and abdominal respiratory movements, snoring, and electroencephalograms (EEGs). External knowledge data includes the latest clinical guidelines, medical textbooks, authoritative journal articles, drug instructions, and disease knowledge graphs.

[0031] Raw medical data can be stored in electronic devices beforehand, or it can be acquired by electronic devices from external sources. For example, electronic devices may acquire medical data from external data collection devices, or they may acquire data from interconnected hospitals, medical institutions, etc.

[0032] There are no restrictions on the specific form of obtaining the original medical data, as long as the electronic device can obtain the original medical data.

[0033] The collection of raw medical data is achieved through interface integration and manual entry, while strictly adhering to the requirements for medical data privacy protection. The data is anonymized to ensure the security of the data and the transmission process.

[0034] S102. Using a decision-making intelligent agent, the raw medical data containing various modalities are uniformly encoded into a preset semantic vector space, the semantics between the data of various modalities are aligned, and personalized feature vectors of patients to be predicted are generated.

[0035] In this embodiment of the invention, the decision agent is not the LLM itself, but a complete application system oriented towards achieving a specific goal. The decision agent consists of three core components: the LLM as the reasoning center, the external tools that can be invoked, and the orchestration layer responsible for task planning and execution loops. The prompt words are the key carriers of the orchestration layer. The prompt words specify the format in which the LLM embedded in the decision agent should think, and define the decomposition logic, tool invocation strategy and termination conditions through structured instructions, tool descriptions and examples.

[0036] In this embodiment of the invention, the decision agent uses a trained cross-modal semantic coding model to encode the feature information of different modalities into a shared, high-dimensional semantic vector space. In the semantic vector space, the data of various modalities contained in the original medical data are semantically aligned to establish the semantic correlation between these data, thereby achieving deep fusion between multiple modalities of data.

[0037] The cross-modal semantic coding model specifically includes multiple parallel modal data encoders and a feature fusion layer that is connected to all the modal data encoders.

[0038] The input of the feature fusion layer is connected to the output of all modal data encoders and serves as the deep fusion layer for these modal data encoders. It can be understood that the modal data encoders will have modal data encoders corresponding to the total number of modalities in the original medical data, ensuring that the corresponding data features can be obtained for each modality.

[0039] Specifically, the modal data encoder includes: Text data encoder: Based on pre-trained medical language models such as BioBERT and ClinicalBERT, it is fine-tuned to encode text information such as medical history and reports.

[0040] Table data encoder: Extracts the content of each cell in a table data and obtains the feature vector of the table data.

[0041] Image data encoder: Based on a pre-trained image extraction model, it extracts image features of regions of interest from image data from both local and global perspectives.

[0042] Timing signal encoder: Integrates a timing group attention module. For long timing signals, traditional Transformer attention suffers from excessive complexity. The timing signal encoder adopts a group attention mechanism, more specifically: The temporal signal encoder uses a preset time window to divide the entire temporal physiological signal into K groups of fixed duration. Then, it calculates self-attention within each group to capture strong correlations of events within a local range, such as the synchronicity of airflow cessation and blood oxygen drop within a single apnea event. After that, it calculates cross-group global attention between groups to capture long-term dependence and overall trends, such as identifying periods when respiratory events are more concentrated at night.

[0043] Assuming each group has a length of L, the number of groups is K, and the total length is n=KL, by reasonably setting L through a preset time window, the overall complexity can be reduced, significantly reducing computation and memory overhead, making it possible to process overnight PSG data on a regular GPU.

[0044] Structured data encoder: Uses a fully connected network to encode numerical and categorical features into vectors.

[0045] Feature fusion layer: Utilizing a cross-modal attention mechanism, text, tables, images, time series, and structured feature vectors mutually query attention, i.e., query keys, values, etc., to achieve deep information fusion. For example, the model can learn the temporal correspondence between nighttime wakefulness due to shortness of breath in text features and sudden drops in blood oxygen detected by PSG.

[0046] S103. Based on the feature information of the personalized feature vector, determine the preset activation weight of each expert layer and establish the routing between the personalized feature vector and the expert layer.

[0047] In this embodiment of the invention, at the deep layer of the LLM model embedded in the decision agent, the general language expert in the general hybrid expert (MoE) model is abandoned, and a set of irreplaceable domain-specific experts that are strongly related to specific medical analysis sub-tasks are provided, namely the preset identification expert layer.

[0048] Taking the field of sleep apnea as an example, the expert pool, composed of a pre-defined layer of expert evaluators, includes: Physiological event identification expert: specializing in differentiating between obstructive, central, and mixed sleep apnea events; Severity grading experts: Based on indicators such as AHI and blood oxygen, they assess the severity of the disease, such as mild, moderate, and severe.

[0049] Comorbidity risk assessment expert: Analyzes the association risk between the current condition and comorbidities such as hypertension, arrhythmia, and diabetes; Treatment response prediction expert: Based on historical data, predict patients' possible responses to different interventions.

[0050] Based on the feature information of the current personalized feature vector, such as the significant features of "airflow interruption" and "contradictory chest and abdomen movement" in the encoded personalized feature vector, the activation weight of each expert layer is calculated, and the corresponding routing network is established.

[0051] The routing logic is strongly bound to the physiological signal features reflected in the personalized feature vectors, rather than general semantics. For example, when a prolonged airflow cessation accompanied by the disappearance of chest and abdominal movements is detected, the central event identification expert layer is activated with a high probability; while when airflow cessation is accompanied by continuous chest and abdominal exertion, the obstructive event identification expert layer is activated.

[0052] In this way, during subsequent reasoning processes, only the top-2 most relevant experts are activated for each inference, rather than all expert layers, significantly saving computational resources. Each activated expert layer outputs preliminary analysis results within its professional field, such as "Obstructive event index: 25 times / hour", "Lowest blood oxygen level: 82%", and "According to the AHI classification standard, it is recommended to classify it as moderate".

[0053] S104. Based on the knowledge data in the original medical data, the clinical rules are transformed into fine-tunable constraints. Based on these constraints and the activated expert layer, feature analysis is performed on the personalized feature vector to obtain the disease analysis state vector. The constraint parameters are set according to the data distribution of historical medical data.

[0054] In this embodiment of the invention, to resolve the contradiction between the black box nature of the LLM model and the rigid requirements of clinical rules, key clinical diagnostic criteria and guideline recommendations are encoded as learnable parameter layers, achieving a fusion of hard constraints and soft learning. These are transformed into differentiable logical functions or constraints and embedded into the network's forward propagation process. For example, the transformation yields a constraint for rule compliance loss; when the intermediate output of the LLM model violates the hard constraints, a significant loss occurs.

[0055] The key thresholds or weights of the constraints themselves, i.e., the constraint parameters, are not completely fixed. Instead, they are learnable parameters that are trained on a large amount of real historical medical data. This allows the LLM model to learn to fine-tune based on the specific data distribution while adhering to the general rules. For example, the LLM model might learn that for elderly patients, the weight of blood oxygen saturation in the classification should be slightly higher than that for younger patients.

[0056] By transforming the constraints and adjusting their parameters, we can ensure that all model outputs, especially in critical diagnostic classification and severity assessment, comply with current clinical guidelines, thus eliminating LLM-related decision-making errors in auxiliary diagnosis.

[0057] In this way, the decision-making agent ultimately outputs a disease analysis state vector that integrates multimodal information, has undergone preliminary analysis by domain experts, and conforms to clinical rule constraints. The disease analysis state vector is the basis for generating all subsequent text descriptions and conducting multidimensional verification.

[0058] In this embodiment of the invention, historical medical data is derived from historical patients. This allows for the optimization of the weighting of the expert layer through historical medical data, and enables the marking of high-risk warnings for results that exceed the common clinical range, such as a minimum blood oxygen value ≤80%.

[0059] S105. Obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the LLM embedded in the decision agent to reason about the prompt words.

[0060] In this embodiment of the invention, the current task context is the actual medical type task that needs to be solved. The decision agent will fuse and assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and call the LLM embedded in the decision agent to perform reasoning to obtain the reasoning result.

[0061] In this embodiment of the invention, the reasoning result may include information in dimensions such as diagnostic conclusion, medical indicator report, personalized medical advice, and follow-up advice. The diagnostic conclusion may be, for example, "obstructive sleep apnea syndrome (moderate)"; the medical indicator report may include key indicators such as AHI value, lowest blood oxygen value, and respiratory event distribution time period.

[0062] S106. At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated, and the candidate reasoning paths are compared in terms of fact, logic, and language dimensions to determine the optimal reasoning path among the candidate reasoning paths. The optimal reasoning path is then used to generate the optimal reasoning result.

[0063] In this embodiment of the invention, during the decoding process, multiple candidate output sequences are generated using methods such as beam search and sampling, and initial probabilities are assigned. Then, multi-dimensional signal fusion is performed using three dimensions: fact signal comparison, logic signal comparison, and language signal comparison, in order to determine the optimal candidate path until the generation is complete.

[0064] The high-quality, reliable text sequence generated after multidimensional signal dynamic verification yields the optimal inference results, which are then organized into a clearly structured, easy-to-read auxiliary analysis report for doctors. It is emphasized again that this report does not contain direct, conclusive treatment plans, but rather provides decision support information.

[0065] For example, the optimal reasoning result may include: Patient's condition summary: Use concise and accurate language to summarize the core analysis results. For example: "Patient XXX, nocturnal polysomnography showed: sleep apnea mainly characterized by obstructive events, AHI 28 times / hour, which is moderate; mean blood oxygen saturation 94%, lowest blood oxygen saturation 82%." Each key indicator is explained, including its numerical value, clinical significance, and the clinical guidelines from which it is based.

[0066] List of issues found: The main abnormalities found in the analysis are listed in bullet points, such as "1. Moderate obstructive sleep apnea; 2. Nocturnal intermittent hypoxemia; 3. Fragmentation of sleep structure (low proportion of N3 stage sleep)".

[0067] Associated Risk Warning: Based on the output of comorbidity risk assessment experts, this warning indicates risks that may be related to the current condition.

[0068] Supporting medical evidence: Next to the relevant conclusions in the report, indicate the sources of factual signals that support the conclusions in the form of footnotes or sidebars.

[0069] Intervention directions for reference: Based on the analysis of knowledge base and treatment response prediction experts, we have listed the major categories of interventions that are common and applicable to this type of condition within the framework of clinical guidelines, as well as their general principles.

[0070] Important Notes and Contraindications: Based on contraindication information in the knowledge base, important reminders are provided. For example: "If considering the use of a specific orthodontic appliance, a detailed dental evaluation is required to rule out contraindications such as severe periodontal disease." Follow-up monitoring recommendations: Provide key indicators for subsequent observation and suggested follow-up examination times. For example: "It is recommended to repeat PSG at the start of treatment and 3 months later to assess efficacy and optimize parameters." Declaration of Credibility and Limitations of the Report: The report automatically generates a disclaimer at the end: "This report was generated by an AI system. All conclusions have been verified in real time against authoritative medical knowledge bases and follow [mentioned clinical guideline name]. The report content is for clinicians' decision-making reference only and cannot replace the professional judgment of licensed physicians. The final diagnosis and treatment decision must be made by the physician based on the patient's comprehensive condition." Explain the limitations of the analysis, such as "This analysis is based on the monitoring data provided and does not include information such as the Daytime Sleepiness Scale (ESS) score, which may affect the comprehensive assessment of the subjective impact on the condition." The optimal reasoning results and output module assembles the generated text components with visualizations, evidence citations, and other elements into a final, credible auxiliary analysis report, which is then presented to the physician. The optimal reasoning results can also be presented in HTML, PDF, or a structured view integrated into the physician's workstation interface, ensuring clear information hierarchy and highlighting key points.

[0071] This invention presents a medical data-assisted decision-making method that integrates multimodal data. At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated. These paths are then compared across factual, logical, and linguistic dimensions to determine the optimal reasoning path. The optimal reasoning path is used to generate the optimal reasoning result. By deeply embedding multidimensional signal verification into the generation process, real-time suppression of AI illusions is achieved, ensuring high quality in terms of facts, logic, and expression in the output content, significantly reducing the risk of misleading information that AI-assisted medical care may bring. Furthermore, by determining the activation weights of each pre-defined expert layer based on the feature information of personalized feature vectors and establishing a route between personalized feature vectors and expert layers, and by transforming clinical rules into finely adjustable constraints based on knowledge data in the original medical data, feature analysis is performed on the personalized feature vectors based on the constraints and the activated expert layers to obtain a disease analysis state vector. The innovative domain-specific expert cluster design enables the decision-making agent to efficiently and accurately process and analyze complex data such as long-term physiological signals, deeply integrating with text and structured data to uncover more comprehensive disease information. LLM can rapidly process massive amounts of data, providing multi-validated key information summaries and risk alerts, improving doctors' work efficiency and decision-making quality, freeing them from the heavy workload of data screening and preliminary analysis, and allowing them to focus more on higher-order clinical thinking and doctor-patient communication.

[0072] This invention provides an LLM (Limited Ledger Model) that integrates multidimensional comparative signals for dynamic verification and adjustment in real time during medical data analysis and decision support generation. The LLM does not directly generate the final treatment plan, but rather constructs an intelligent, reliable, and interpretable medical data analysis and reasoning framework to provide clinicians with multi-verified, structured disease assessment conclusions, key indicator analysis, and decision support references, thus entrusting the final decision-making power to the doctor and improving both the efficiency and quality of medical work.

[0073] In this embodiment of the invention, step S102 specifically includes: S1021. Use a decision-making intelligent agent to classify the raw medical data according to the disease condition dimension, which includes the disease type, disease severity, and treatment stage.

[0074] S1022. Label the classified raw medical data to generate label information for the raw medical data.

[0075] S1023. Input the raw medical data carrying the tag information into each modal data encoder to obtain the feature vector output by the modal data encoder.

[0076] The decision-making agent uses the modal data encoders of the cross-modal semantic coding model to convert the raw medical data of the corresponding modality into the corresponding feature vectors.

[0077] S1024. Project the feature vectors of various modalities into a preset encoding into a preset semantic vector space, thus projecting the feature vectors of all modalities into the same dimension.

[0078] S1025. The projected feature vectors are input into the feature fusion layer, and the feature fusion layer is used to align the feature vectors of various modalities, establish semantic associations between the feature vectors of various modalities, and generate personalized feature vectors for the patient to be predicted.

[0079] In this embodiment of the invention, step S104 specifically includes: S1041. Determine the expert pool consisting of a preset layer of identification experts.

[0080] S1042. The decision-making agent determines the identification expert layer based on the activation weight.

[0081] In this embodiment of the invention, an activation threshold is also set. When the activation weight of the authentication expert layer route exceeds the activation threshold, the corresponding authentication expert layer will be activated.

[0082] S1043. Based on the knowledge data in the original medical data, the clinical rules are transformed into several constraints, and the constraint parameters of each constraint are determined according to the data distribution of the historical medical data of historical patients.

[0083] S1044. During the forward propagation of the LLM network embedded with the decision agent, the constraints are embedded into the network. Based on the constraints, the personalized feature vector is analyzed to obtain the disease analysis state vector.

[0084] In this embodiment of the invention, step S106 specifically includes: S1061. At each step of the reasoning and decoding, based on the disease analysis state vector and the current task context, predict the probability distribution of the next word, obtain several candidate words and form several candidate reasoning paths, and assign an initial probability to each candidate reasoning path based on the candidate words.

[0085] At each step of the inference and decoding process, methods such as beam search and Top-k / Top-p sampling are employed to generate higher-quality inference results. Beam search retains the k candidate sequences with the highest probabilities at each generation step.

[0086] S1062. Extract medical entities from each candidate reasoning path and match and verify the extracted entities with a preset database to obtain the fact consistency score for each candidate reasoning path. Consistency score The higher the score, the more consistent it is with authoritative medical knowledge.

[0087] In this embodiment of the invention, the preset database can be constructed using historical medical data.

[0088] More specifically, step S1062 includes: S10621. Use candidate reasoning paths to reason about the prompt words, obtain the candidate reasoning results for each candidate reasoning path, and extract medical entities from each candidate reasoning result.

[0089] S10622. Check whether the extracted medical entities are within the physiological range, and obtain the first fact diagnosis result for each candidate reasoning path, such as whether the adult's blood oxygen saturation is between 80-100%. In this way, determine whether it is a numerical fact.

[0090] S10623. Check whether the relationship between the extracted medical entities and clinical guideline entries is correct, and obtain the second fact diagnosis result for each candidate reasoning path. In this way, determine whether it is a relational fact. If the preset database is successfully matched, the matched clinical guideline entry or document ID can be recorded as the source of subsequent evidence.

[0091] S10624. Based on the first and second fact diagnostic results, obtain the fact consistency score for each candidate reasoning path. .

[0092] S1063. Perform internal logical judgment on each candidate reasoning path to obtain the logical consistency score of each candidate reasoning path. Logical consistency score A higher score indicates a more reasonable causal relationship, temporal relationship, etc.

[0093] More specifically, step S1063 includes: S10631. Use candidate reasoning paths to reason about the prompt words, obtain the candidate reasoning results for each candidate reasoning path, and check the causal and temporal logic of each candidate reasoning result to obtain the first logical diagnosis result for each candidate reasoning path.

[0094] In this embodiment of the invention, the causal and temporal relationships of the candidate inference results are checked to see if they are reasonable. For example, if the preceding text states "After the patient used CPAP, the AHI dropped to 5 times / hour", and the following text expands to "Therefore, it is recommended to perform uvulopalatopharyngoplasty (UPPP) immediately", there is a contradiction in the treatment logic between these two statements. This can be captured by checking the causal and temporal logic.

[0095] S10632. Check whether there is a contradiction between each sentence in each candidate reasoning result and the already generated context, and obtain the second logical diagnosis result for each candidate reasoning path.

[0096] Natural language reasoning techniques are used to check for contradictions between newly generated sentences and existing context. For example, if the existing sentence is "no obvious hypoxemia was observed during overnight monitoring," but a new candidate sentence is to generate "the lowest blood oxygen level was 72%", this will be identified as a contradiction.

[0097] S10633. Check whether each candidate reasoning result matches the disease analysis state vector, that is, check the consistency with the original disease analysis state vector, and obtain the third logical diagnosis result of each candidate reasoning path.

[0098] Inconsistencies arise when the description of the candidate inference results matches the state vector of the disease analysis. For example, if the state vector shows "predominantly obstructive events," but the candidate text emphasizes "predominantly central events," then inconsistencies are generated.

[0099] S10634. Based on the first, second, and third logical diagnostic results, obtain the logical consistency score for each candidate reasoning path. .

[0100] S1064. Perform a linguistic expression judgment on each candidate reasoning path to obtain a linguistic normativity score for each candidate reasoning path. Language standardization score A higher score indicates that the content generated by the reasoning and decoding conforms more to the medical document standards and is clear and unambiguous.

[0101] More specifically, step S1064 includes: S10641. Use candidate reasoning paths to reason about the prompt words, obtain the candidate reasoning results for each candidate reasoning path, check the domain language style of each candidate reasoning result, and obtain the first language specification result for each candidate reasoning path.

[0102] By examining whether the language used is professional and formal, avoiding colloquial and vague vocabulary, the domain language style of each candidate inference result can be determined.

[0103] S10642. Check the clarity of expression of each candidate reasoning result to obtain the second language standardization result of each candidate reasoning path.

[0104] By calculating syntactic complexity and entity reference clarity, we avoid generating overly long, complexly nested, or ambiguous sentences, and then check the clarity of expression for each candidate inference result.

[0105] S10643. Compare the semantic similarity between each candidate reasoning result, determine the information redundancy and missing information in the candidate reasoning results, and obtain the third language standardization result for each candidate reasoning path.

[0106] By comparing multiple candidate paths, paths that repeatedly state the same information or omit key conclusions are penalized. For example, paths that only describe a phenomenon without providing a severity rating are penalized.

[0107] S10644. Based on the first, second, and third language standardization scores, obtain the language standardization score for each candidate inference path. .

[0108] S1065. Based on preset weights and factual consistency scores. Logical consistency score Language standardization score And the initial probability, to determine the comprehensive reasoning score for each candidate reasoning path, specifically: in, Indicates the first The combined reasoning score of the candidate reasoning paths; Indicates the first The initial probability of each candidate reasoning path; Indicates the first Fact consistency score of each candidate reasoning path; Indicates the first Logical consistency score of each candidate reasoning path; Indicates the first The linguistic normativity score of each candidate reasoning path; Indicates the first weight; Indicates the second weight; Indicates the third weight; This indicates the fourth weight.

[0109] By using multi-dimensional signal fusion and path reordering, the comprehensive reasoning score of each candidate reasoning path can be determined during the decoding process. Then, based on the comprehensive reasoning score, the optimal candidate reasoning path for each decoding step is selected, and decoding is performed step by step until the reasoning is completed.

[0110] It should be noted that the first to fourth weights are all adjustable weight parameters, which are determined during the training phase through reinforcement learning or by tuning based on the validation set. Weights are usually higher to maintain fluency of language; and In medical settings, weight should be given significant weight to ensure that facts and logic take precedence.

[0111] S1066. At each step of decoding, the candidate reasoning paths are sorted according to the comprehensive reasoning score. The candidate reasoning path with the highest comprehensive reasoning score is taken as the optimal reasoning path for the corresponding step, and the candidate reasoning result of the optimal reasoning path is taken as the optimal reasoning result for the corresponding step.

[0112] The entire generation-verification loop is performed as each word / segment is output, enabling full-process, real-time, and fine-grained control over the generated content, thus preserving and further expanding the accurate path.

[0113] Please see Figure 2 In this embodiment of the invention, the method may further include the following steps: S201. Obtain the original medical data of the patient to be predicted. The specific details are as shown in step S101, and will not be repeated here.

[0114] S202. Preprocess the raw medical data. Preprocessing includes data cleaning, data anonymization, medical terminology standardization, entity recognition, relation extraction, text vectorization, and feature extraction.

[0115] Among these processes, the acquired raw medical data undergoes data cleaning and data anonymization to remove duplicate and invalid records, correct obvious errors, and anonymize patient personal information.

[0116] For unstructured data in raw medical data, medical ontology such as UMLS and SNOMED CT can be used to map colloquial, synonymous or abbreviated terms to standard medical terms, such as: "snoring" → "snoring", "palpitation" → "heart palpitation", etc.

[0117] Using a pre-trained medical named entity recognition model, we can identify entities such as diseases, symptoms, drugs, examinations, surgeries, and body parts in unstructured data, and extract the temporal, causal, and attribute relationships between them.

[0118] Specifically, for raw medical data in text format, the processed text is converted into a dense vector representation for use by downstream models.

[0119] For time-series physiological signal data in the raw medical data, filters are used to remove noise such as power frequency interference and motion artifacts. For short-term data gaps caused by equipment factors, interpolation is used to complete the data. Clinically significant features are extracted for different signals, such as identifying and extracting features like apnea and hypoventilation. Long-series signals are divided into fixed-length windows, and the features within each window are normalized.

[0120] For external knowledge data, a knowledge base can be built, transforming it into a structured knowledge graph or vector database to support efficient retrieval and matching. Nodes in the knowledge graph represent diseases, symptoms, drugs, etc., while edges represent indications, contraindications, causal relationships, etc.

[0121] Correspondingly, the subsequent steps involve classifying and labeling the preprocessed raw medical data. This means classifying and labeling the preprocessed raw medical data according to disease type, severity of illness, treatment stage, etc., thus giving the raw medical data a label.

[0122] S203. Utilize a decision-making intelligent agent to uniformly encode the raw medical data containing various modalities into a preset semantic vector space, align the semantics between the data of various modalities, and generate personalized feature vectors for the patients to be predicted. The specific content is as shown in step S102, and will not be repeated here.

[0123] S204. Based on the feature information of the personalized feature vector, determine the preset activation weights of each expert layer and establish the routing between the personalized feature vector and the expert layer. The specific details are as shown in step S102 and will not be repeated here.

[0124] S205. Based on the knowledge data in the original medical data, the clinical rules are transformed into fine-tunable constraints. Based on these constraints and the activated expert layer, feature analysis is performed on the personalized feature vector to obtain the disease analysis state vector. The constraint parameters are set according to the data distribution of historical medical data. The specific details are shown in step S102 and will not be repeated here.

[0125] S206. Obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the large language model embedded in the decision agent to reason about the prompt words. The specific content is as shown in step S102, and will not be repeated here.

[0126] S207. At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated, and these paths are compared across factual, logical, and linguistic dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result. The specific details are as shown in step S102 and will not be repeated here.

[0127] The following describes the medical data-assisted decision-making device that integrates multimodal data provided in the embodiments of the present invention. The medical data-assisted decision-making device that integrates multimodal data described below can be referred to in correspondence with the medical data-assisted decision-making method that integrates multimodal data described above.

[0128] Due to the aforementioned technical problems, this invention also provides a medical data-assisted decision-making device that integrates multimodal data. This device aims to provide an LLM (Limited Least Meaning) that dynamically verifies and adjusts multidimensional comparative signals in real time during medical data analysis and decision-making generation. The LLM does not directly generate the final treatment plan; instead, it constructs an intelligent, reliable, and interpretable medical data analysis and reasoning framework to provide clinicians with multi-verified, structured disease assessment conclusions, key indicator analyses, and decision support references, ultimately empowering doctors with the final decision-making authority. This improves both the efficiency and quality of medical work. Figure 3 is a schematic diagram of the structure of the medical data-assisted decision-making method integrating multimodal data according to an embodiment of this invention. Figure 3 As shown, the device may include: The data acquisition module 10 is used to acquire the original medical data of the patient to be predicted.

[0129] In this embodiment of the invention, the raw medical data includes multiple data modalities, including at least: text data, tabular data, image data, structured medical data, time-series physiological signals, and external knowledge data.

[0130] Structured medical data can include basic patient information from digital management platforms, such as age, gender, and past medical history, as well as patient treatment data. Unstructured data, such as text data, tabular data, and image data, can include medical records, pathology records, follow-up records, laboratory test results, and imaging test results. Temporal physiological signals include data from polysomnography, Holter monitoring, continuous glucose monitoring, and ventilator therapy. For example, in the field of sleep-disordered breathing, this includes signals such as respiratory airflow, blood oxygen saturation, chest and abdominal respiratory movements, snoring, and electroencephalograms (EEGs). External knowledge data includes the latest clinical guidelines, medical textbooks, authoritative journal articles, drug instructions, and disease knowledge graphs.

[0131] Raw medical data can be stored in electronic devices beforehand, or it can be acquired by electronic devices from external sources. For example, electronic devices may acquire medical data from external data collection devices, or they may acquire data from interconnected hospitals, medical institutions, etc.

[0132] There are no restrictions on the specific form of obtaining the original medical data, as long as the electronic device can obtain the original medical data.

[0133] The collection of raw medical data is achieved through interface integration and manual entry, while strictly adhering to the requirements for medical data privacy protection. The data is anonymized to ensure the security of the data and the transmission process.

[0134] The data encoding module 20 is used to encode raw medical data containing various modalities into a preset semantic vector space using a decision-making intelligent agent, align the semantics between data of various modalities, and generate personalized feature vectors for patients to be predicted.

[0135] In this embodiment of the invention, the decision agent is not the LLM itself, but a complete application system oriented towards achieving a specific goal. The decision agent consists of three core components: the LLM as the reasoning center, the external tools that can be invoked, and the orchestration layer responsible for task planning and execution loops. The prompt words are the key carriers of the orchestration layer. The prompt words specify the format in which the LLM embedded in the decision agent should think, and define the decomposition logic, tool invocation strategy and termination conditions through structured instructions, tool descriptions and examples.

[0136] In this embodiment of the invention, the decision agent uses a trained cross-modal semantic coding model to encode the feature information of different modalities into a shared, high-dimensional semantic vector space. In the semantic vector space, the data of various modalities contained in the original medical data are semantically aligned to establish the semantic correlation between these data, thereby achieving deep fusion between multiple modalities of data.

[0137] The cross-modal semantic coding model specifically includes multiple parallel modal data encoders and a feature fusion layer that is connected to all the modal data encoders.

[0138] The input of the feature fusion layer is connected to the output of all modal data encoders and serves as the deep fusion layer for these modal data encoders. It can be understood that the modal data encoders will have modal data encoders corresponding to the total number of modalities in the original medical data, ensuring that the corresponding data features can be obtained for each modality.

[0139] The data routing module 30 is used to determine the preset activation weight of each expert layer based on the feature information of the personalized feature vector, and to establish the routing between the personalized feature vector and the expert layer.

[0140] In this embodiment of the invention, at the deep layer of the LLM model embedded in the decision agent, the general language expert in the general hybrid expert (MoE) model is abandoned, and a set of irreplaceable domain-specific experts that are strongly related to specific medical analysis sub-tasks are provided, namely the preset identification expert layer.

[0141] Taking the field of sleep apnea as an example, the expert pool, composed of a pre-defined layer of expert evaluators, includes: Physiological event identification expert: specializing in differentiating between obstructive, central, and mixed sleep apnea events; Severity grading experts: Based on indicators such as AHI and blood oxygen, they assess the severity of the disease, such as mild, moderate, and severe.

[0142] Comorbidity risk assessment expert: Analyzes the association risk between the current condition and comorbidities such as hypertension, arrhythmia, and diabetes; Treatment response prediction expert: Based on historical data, predict patients' possible responses to different interventions.

[0143] Based on the feature information of the current personalized feature vector, such as the significant features of "airflow interruption" and "contradictory chest and abdomen movement" in the encoded personalized feature vector, the activation weight of each expert layer is calculated, and the corresponding routing network is established.

[0144] The routing logic is strongly bound to the physiological signal features reflected in the personalized feature vectors, rather than general semantics. For example, when a prolonged airflow cessation accompanied by the disappearance of chest and abdominal movements is detected, the central event identification expert layer is activated with a high probability; while when airflow cessation is accompanied by continuous chest and abdominal exertion, the obstructive event identification expert layer is activated.

[0145] In this way, during subsequent reasoning processes, only the top-2 most relevant experts are activated for each inference, rather than all expert layers, significantly saving computational resources. Each activated expert layer outputs preliminary analysis results within its professional field, such as "Obstructive event index: 25 times / hour", "Lowest blood oxygen level: 82%", and "According to the AHI classification standard, it is recommended to classify it as moderate".

[0146] The constraint transformation module 40 is used to transform clinical rules into fine-tunable constraints based on knowledge data in the original medical data. Based on these constraints and the activated expert layer, it performs feature analysis on the personalized feature vector to obtain the disease analysis state vector. The constraint parameters are set according to the data distribution of historical medical data.

[0147] In this embodiment of the invention, to resolve the contradiction between the black box nature of the LLM model and the rigid requirements of clinical rules, key clinical diagnostic criteria and guideline recommendations are encoded as learnable parameter layers, achieving a fusion of hard constraints and soft learning. These are transformed into differentiable logical functions or constraints and embedded into the network's forward propagation process. For example, the transformation yields a constraint for rule compliance loss; when the intermediate output of the LLM model violates the hard constraints, a significant loss occurs.

[0148] The key thresholds or weights of the constraints themselves, i.e., the constraint parameters, are not completely fixed. Instead, they are learnable parameters that are trained on a large amount of real historical medical data. This allows the LLM model to learn to fine-tune based on the specific data distribution while adhering to the general rules. For example, the LLM model might learn that for elderly patients, the weight of blood oxygen saturation in the classification should be slightly higher than that for younger patients.

[0149] By transforming the constraints and adjusting their parameters, we can ensure that all model outputs, especially in critical diagnostic classification and severity assessment, comply with current clinical guidelines, thus eliminating LLM-related decision-making errors in auxiliary diagnosis.

[0150] In this way, the decision-making agent ultimately outputs a disease analysis state vector that integrates multimodal information, has undergone preliminary analysis by domain experts, and conforms to clinical rule constraints. The disease analysis state vector is the basis for generating all subsequent text descriptions and conducting multidimensional verification.

[0151] In this embodiment of the invention, historical medical data is derived from historical patients. This allows for the optimization of the weighting of the expert layer through historical medical data, and enables the marking of high-risk warnings for results that exceed the common clinical range, such as a minimum blood oxygen value ≤80%.

[0152] The data assembly module 50 is used to obtain the current task context, assemble the current task context, personalized feature vector and disease analysis state vector into prompt words, and use the LLM embedded in the decision agent to reason about the prompt words.

[0153] In this embodiment of the invention, the current task context is the actual medical type task that needs to be solved. The decision agent will fuse and assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and call the LLM embedded in the decision agent to perform reasoning to obtain the reasoning result.

[0154] In this embodiment of the invention, the reasoning result may include information in dimensions such as diagnostic conclusion, medical indicator report, personalized medical advice, and follow-up advice. The diagnostic conclusion may be, for example, "obstructive sleep apnea syndrome (moderate)"; the medical indicator report may include key indicators such as AHI value, lowest blood oxygen value, and respiratory event distribution time period.

[0155] The reasoning decoding module 60 is used to generate multiple candidate reasoning paths at each step of the reasoning decoding process, compare the candidate reasoning paths in terms of facts, logic and language dimensions, determine the optimal reasoning path among the candidate reasoning paths, and use the optimal reasoning path to generate the optimal reasoning result.

[0156] In this embodiment of the invention, during the decoding process, multiple candidate output sequences are generated using methods such as beam search and sampling, and initial probabilities are assigned. Then, multi-dimensional signal fusion is performed using three dimensions: fact signal comparison, logic signal comparison, and language signal comparison, in order to determine the optimal candidate path until the generation is complete.

[0157] The high-quality, reliable text sequence generated after multidimensional signal dynamic verification yields the optimal inference results, which are then organized into a clearly structured, easy-to-read auxiliary analysis report for doctors. It is emphasized again that this report does not contain direct, conclusive treatment plans, but rather provides decision support information.

[0158] This invention relates to a medical data-assisted decision-making device that integrates multimodal data. At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated. These paths are compared across factual, logical, and linguistic dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result. By deeply embedding multidimensional signal verification into the generation process, real-time suppression of AI illusions is achieved, ensuring high quality in terms of facts, logic, and expression in the output content, significantly reducing the risk of misleading information that AI-assisted medical care may bring. Furthermore, by determining the activation weights of each pre-set expert layer based on the feature information of personalized feature vectors and establishing a route between personalized feature vectors and expert layers, and by transforming clinical rules into finely adjustable constraints based on knowledge data in the original medical data, feature analysis is performed on the personalized feature vectors based on the constraints and the activated expert layers to obtain a disease analysis state vector. The innovative domain-specific expert cluster design enables the decision-making agent to efficiently and accurately process and analyze complex data such as long-term physiological signals, deeply integrating with text and structured data to uncover more comprehensive disease information. LLM can rapidly process massive amounts of data, providing multi-validated key information summaries and risk alerts, improving doctors' work efficiency and decision-making quality, freeing them from the heavy workload of data screening and preliminary analysis, and allowing them to focus more on higher-order clinical thinking and doctor-patient communication.

[0159] This invention provides an LLM (Limited Ledger Model) that integrates multidimensional comparative signals for dynamic verification and adjustment in real time during medical data analysis and decision support generation. The LLM does not directly generate the final treatment plan, but rather constructs an intelligent, reliable, and interpretable medical data analysis and reasoning framework to provide clinicians with multi-verified, structured disease assessment conclusions, key indicator analysis, and decision support references, thus entrusting the final decision-making power to the doctor and improving both the efficiency and quality of medical work.

[0160] This invention provides a complete and systematic context-adaptive reasoning solution for complex scenarios, from data preprocessing to final output and optimization. It forms a closed loop, is easy to integrate into actual business systems, and has high practical value and broad prospects for industrial application.

[0161] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical commands in the memory 430 to execute a medical data-assisted decision-making method that fuses multimodal data. This method includes: Obtain the raw medical data of the patient to be predicted; The decision-making agent encodes raw medical data containing various modalities into a pre-defined semantic vector space, aligns the semantics between data of various modalities, and generates personalized feature vectors for patients to be predicted. Based on the feature information of the personalized feature vector, determine the preset activation weight of each expert layer and establish the routing between the personalized feature vector and the expert layer. Based on the knowledge data in the original medical data, clinical rules are transformed into fine-tunable constraints. Based on the constraints and the activated expert layer, personalized feature vectors are analyzed to obtain the disease analysis state vector. The constraint parameters of the constraints are set according to the data distribution of historical medical data. Obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the large language model embedded in the decision agent to reason about the prompt words; At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated, and the candidate reasoning paths are compared in terms of fact, logic, and language dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result.

[0162] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the medical data-assisted decision-making method for fusing multimodal data provided by the above methods, the method comprising: Obtain the raw medical data of the patient to be predicted; The decision-making agent encodes raw medical data containing various modalities into a pre-defined semantic vector space, aligns the semantics between data of various modalities, and generates personalized feature vectors for patients to be predicted. Based on the feature information of the personalized feature vector, determine the preset activation weight of each expert layer and establish the routing between the personalized feature vector and the expert layer. Based on the knowledge data in the original medical data, clinical rules are transformed into fine-tunable constraints. Based on the constraints and the activated expert layer, personalized feature vectors are analyzed to obtain the disease analysis state vector. The constraint parameters of the constraints are set according to the data distribution of historical medical data. Obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the large language model embedded in the decision agent to reason about the prompt words; At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated, and the candidate reasoning paths are compared in terms of fact, logic, and language dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result.

[0164] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned medical data-assisted decision-making methods for fusing multimodal data, the method comprising: Obtain the raw medical data of the patient to be predicted; The decision-making agent encodes raw medical data containing various modalities into a pre-defined semantic vector space, aligns the semantics between data of various modalities, and generates personalized feature vectors for patients to be predicted. Based on the feature information of the personalized feature vector, determine the preset activation weight of each expert layer and establish the routing between the personalized feature vector and the expert layer. Based on the knowledge data in the original medical data, clinical rules are transformed into fine-tunable constraints. Based on the constraints and the activated expert layer, personalized feature vectors are analyzed to obtain the disease analysis state vector. The constraint parameters of the constraints are set according to the data distribution of historical medical data. Obtain the current task context, assemble the current task context, personalized feature vector, and disease analysis state vector into prompt words, and use the large language model embedded in the decision agent to reason about the prompt words; At each step of the reasoning and decoding process, multiple candidate reasoning paths are generated, and the candidate reasoning paths are compared in terms of fact, logic, and language dimensions to determine the optimal reasoning path. The optimal reasoning path is then used to generate the optimal reasoning result.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for medical data assisted decision fusion of multi-modal data, characterized in that, The method comprises: acquiring original medical data of a patient to be predicted; uniformly encoding the original medical data containing various modalities into a preset semantic vector space by using a decision agent, aligning the semantics between the data of various modalities, and generating a personalized feature vector of the patient to be predicted; determining the activation weight of each discriminant expert layer according to the feature information of the personalized feature vector, and establishing the routing of the personalized feature vector and the discriminant expert layer; translating the clinical rules into adjustable constraint conditions according to the knowledge data in the original medical data, performing feature analysis on the personalized feature vector according to the constraint conditions and the activated discriminant expert layer, and obtaining a disease analysis state vector; the constraint parameters of the constraint conditions are set according to the data distribution of historical medical data; acquiring a current task context, assembling the current task context, the personalized feature vector and the disease analysis state vector into a prompt word, and using a large language model embedded in the decision agent to reason the prompt word; at each step of the reasoning decoding, a plurality of candidate reasoning paths are generated, and the candidate reasoning paths are compared in the dimensions of fact, logic and language to determine an optimal reasoning path in the candidate reasoning paths, and an optimal reasoning result is generated by using the optimal reasoning path.

2. The medical data assisted decision making method of fusing multi-modal data according to claim 1, characterized in that, The decision intelligence uses a trained cross-modal semantic encoding model to generate a personalized feature vector of a patient to be predicted. The cross-modal semantic encoding model comprises a modal data encoder corresponding to the modalities contained in the original medical data and a feature fusion layer. The input end of the feature fusion layer is connected with the output ends of all the modal data encoders, and the input end of the modal data encoder is connected with the original medical data.

3. The medical data assisted decision making method of fusing multi-modal data according to claim 2, characterized in that, The decision agent classifies the original medical data according to the disease dimension; the disease dimension includes disease type, disease severity and treatment stage. The classified original medical data is labeled to generate label information of the original medical data. The original medical data carrying the label information is input into each modal data encoder to obtain the feature vectors output by the modal data encoders. The feature vectors of various modalities are projected into a preset semantic vector space. The projected feature vectors are input into the feature fusion layer, and the feature fusion layer aligns the feature vectors of various modalities to establish the semantic association between the feature vectors of various modalities, and generates a personalized feature vector of the patient to be predicted. At each step of the reasoning decoding, a plurality of candidate reasoning paths are generated, and the candidate reasoning paths are compared in the dimensions of fact, logic and language to determine an optimal reasoning path in the candidate reasoning paths, and an optimal reasoning result is generated by using the optimal reasoning path.

4. The medical data assisted decision making method of fusing multi-modal data according to claim 1, wherein, ​ At each step of the push decoding, a probability distribution of the next word is predicted according to the illness analysis state vector and the current task context, a plurality of candidate words are obtained, and a plurality of candidate reasoning paths are formed, and an initial probability is assigned to each candidate reasoning path according to the candidate words; Medical entities are extracted from each candidate reasoning path, and the extracted entities are matched and verified with a preset database to obtain a fact consistency score of each candidate reasoning path; Internal logic of each candidate reasoning path is judged to obtain a logic consistency score of each candidate reasoning path; Language expression of each candidate reasoning path is judged to obtain a language standardization score of each candidate reasoning path; The preset weight, the fact consistency score, the logic consistency score, the language standardization score, and the initial probability are weighted and summed to determine a comprehensive reasoning score of each candidate reasoning path; At each step of the decoding, the candidate reasoning paths are sorted according to the comprehensive reasoning scores, the candidate reasoning path with the highest comprehensive reasoning score is taken as the optimal reasoning path of the corresponding step, and the candidate reasoning result of the optimal reasoning path is taken as the optimal reasoning result of the corresponding step.

5. The medical data assisted decision making method of fusing multi-modal data according to claim 4, characterized in that, The medical entities are extracted from each candidate reasoning path, and the extracted entities are matched and verified with a preset database to obtain a fact consistency score of each candidate reasoning path, and specifically includes: The candidate reasoning result of each candidate reasoning path is obtained by reasoning the prompt word using the candidate reasoning path, and medical entities are extracted from each candidate reasoning result; It is checked whether the extracted medical entities are within the physiological range to obtain a first fact diagnosis result of each candidate reasoning path; It is checked whether the relationship between the extracted medical entities and the clinical guideline items is correct to obtain a second fact diagnosis result of each candidate reasoning path; According to the first and second fact diagnosis results, a fact consistency score of each candidate reasoning path is obtained.

6. The medical data assisted decision making method of fusing multi-modal data according to claim 4, wherein, The internal logic of each candidate reasoning path is judged to obtain a logic consistency score of each candidate reasoning path, and specifically includes: The candidate reasoning result of each candidate reasoning path is obtained by reasoning the prompt word using the candidate reasoning path, and the cause-effect and time sequence logic of each candidate reasoning result is checked to obtain a first logic diagnosis result of each candidate reasoning path; It is checked whether each sentence in each candidate reasoning result is contradictory to the generated context to obtain a second logic diagnosis result of each candidate reasoning path; It is checked whether each candidate reasoning result is consistent with the illness analysis state vector to obtain a third logic diagnosis result of each candidate reasoning path; According to the first, second, and third logic diagnosis results, a logic consistency score of each candidate reasoning path is obtained.

7. The medical data assisted decision making method of fusing multi-modal data according to claim 4, wherein, The language expression of each candidate reasoning path is judged to obtain a language standardization score of each candidate reasoning path, and specifically includes: The candidate reasoning result of each candidate reasoning path is obtained by reasoning the prompt word using the candidate reasoning path, and the domain language style of each candidate reasoning result is checked to obtain a first language standardization result of each candidate reasoning path; The expression clarity of each candidate reasoning result is checked to obtain a second language standardization result of each candidate reasoning path; The semantic similarity between each candidate reasoning result is compared to determine information redundancy and missing in the candidate reasoning results, and a third language specification result of each candidate reasoning path is obtained; According to the first, second and third language specification scores, a language specification score of each candidate reasoning path is obtained.

8. A medical data assisted decision fusion multi-modal data apparatus, characterized in that, The device comprises: a data acquisition module configured to acquire original medical data of a patient to be predicted; a data encoding module configured to uniformly encode the original medical data containing various modalities into a preset semantic vector space by using a decision-making agent, align the semantics between the data of various modalities, and generate a personalized feature vector of the patient to be predicted; a data routing module configured to determine an activation weight of each discriminative expert layer according to feature information of the personalized feature vector, and establish a routing between the personalized feature vector and the discriminative expert layer; a constraint conversion module configured to convert clinical rules into adjustable constraint conditions according to knowledge data in the original medical data, perform feature analysis on the personalized feature vector according to the constraint conditions and the activated discriminative expert layer, and obtain a disease analysis state vector; constraint parameters of the constraint conditions are set according to data distribution of historical medical data; a data assembly module configured to obtain a current task context, assemble the current task context, the personalized feature vector and the disease analysis state vector into a prompt word, and use a large language model embedded in the decision-making agent to reason the prompt word; an optimal reasoning module configured to generate multiple candidate reasoning paths at each step of the reasoning decoding, compare the candidate reasoning paths in the fact, logic and language dimensions, determine an optimal reasoning path in the candidate reasoning paths, and generate an optimal reasoning result by using the optimal reasoning path.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the medical data aided decision-making method for fusing multi-modal data according to the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the medical data aided decision-making method for fusing multi-modal data according to any one of claims 1 to 7.

Images