Multi-modal reasoning model construction method and system for senile comorbidity

Abstract

The present disclosure is suitable for the field of artificial intelligence and intelligent medical technology, and provides a multi-modal reasoning model construction method for senile comorbidity, collects and pre-processes multi-modal clinical data containing senile comorbidity, and constructs a knowledge-enhanced training data set; the specialist knowledge of senile comorbidity is injected into the basic model by using the supervised fine-tuning technology, and the model after supervised fine-tuning is obtained; a progressive curriculum reinforcement learning framework is constructed, and finally a multi-modal reasoning model is obtained. The present disclosure solves the problems of lack of high-quality multi-modal medical data and lack of deep reasoning logic annotation in the prior art, through the collection and standardized processing of multi-source clinical data, combined with the knowledge distillation technology based on the medical literature knowledge base, the implicit knowledge in the authoritative medical literature can be made explicit and structured, and the data quality is ensured through automatic verification and manual review, breaking through the limitation of traditional limited labeled data, and providing a solid and rich medical logic basis for model training.

Description

Technical Field

[0002] This application belongs to the field of artificial intelligence and smart healthcare technology, and specifically relates to a method for constructing a multimodal reasoning model for comorbidities in the elderly. Background Technology

[0003] Deep population aging is a major global challenge. Coronary heart disease (CHD) is a serious threat to the health of the elderly, with a prevalence rate as high as 27.8% in people over 60 years of age. Elderly CHD patients often suffer from multiple geriatric syndromes, such as frailty, sarcopenia, cognitive impairment, and malnutrition. These comorbidities are intertwined, making disease assessment and management extremely complex and constituting a core challenge in clinical practice. Comprehensive geriatric assessment, as a key means of screening and managing these syndromes and achieving individualized treatment, requires multidisciplinary collaboration to comprehensively assess the elderly's physical function, psychological state, cognitive ability, social support, and environment. In recent years, artificial intelligence technology has been widely applied in the medical field. Multimodal large language models, due to their powerful cross-modal information fusion and complex semantic reasoning capabilities, provide a new technical approach to solving the above-mentioned challenges. Multimodal large language models can collaboratively process heterogeneous data such as electronic medical record text, medical images, physiological signals, assessment scales, and even behavioral video and audio, achieving a panoramic and in-depth understanding and reasoning of patients' health status, and are expected to surpass traditional single-modal or structured rule-based analysis methods.

[0004] However, applying multimodal large language models to the highly specialized and complex clinical scenario of elderly patients with coronary heart disease and geriatric syndromes still faces multiple severe challenges. First, high-quality multimodal labeled data, especially those containing deep reasoning logic, is extremely scarce in the medical field. Existing data is mostly heterogeneous and scattered, and lacks annotations that describe the coherent logic of "symptom-mechanism-decision". Second, general-purpose multimodal large language models lack specialized medical knowledge, which can easily lead to factual errors or "illusions" when applied to complex clinical reasoning, resulting in outputs that do not conform to diagnostic and treatment guidelines or are even harmful, posing risks to safety and reliability. In addition, how to effectively inject domain knowledge and stimulate deep reasoning capabilities during the model training process, while ensuring that its output is strictly aligned with clinical logic, is a core technical challenge.

[0005] In summary, there is still a lack of a systematic methodology that can build a highly reliable and interpretable multimodal reasoning model for the diagnosis and treatment of comorbidities in the elderly, covering the entire process from data construction and model training to optimization. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a method for constructing a multimodal reasoning model for comorbidities in the elderly, including: We collected and preprocessed multimodal clinical data containing comorbidities in the elderly, determined the rules and framework for data construction based on the clinical diagnosis and treatment pathways of comorbidities in the elderly, and used knowledge distillation technology to extract structured knowledge from medical literature. Based on the rule framework and structured knowledge, we constructed a knowledge-enhanced training dataset. Using a pre-trained multimodal large language model as the base model, and based on a knowledge-enhanced training dataset, supervised fine-tuning techniques are employed to inject specialized domain knowledge of geriatric comorbidities into the base model, resulting in a supervised fine-tuned model. A progressive curriculum reinforcement learning framework is constructed, and based on this framework, a composite reward function is used to optimize the supervised fine-tuned model through reinforcement learning. This aims to stimulate the model's deep reasoning ability and suppress output illusions, ultimately obtaining a multimodal reasoning model.

[0007] Furthermore, constructing the knowledge-enhanced training dataset includes: Collect and standardize multi-source clinical data, which includes de-identified electronic medical record text, medical images, electrophysiological signals, and behavioral video and audio data. We collected and constructed a medical literature knowledge base in the field of geriatric comorbidities. Using a large language model as a teacher model, we extracted structured diagnosis and treatment rules, drug interactions, prognostic factors, and key reasoning paths from the medical literature knowledge base to generate knowledge-enhanced data. Based on clinical treatment pathways, the system automatically or / and semi-automatically generates instruction-response pairs of data that include risk assessment, multimodal diagnosis, and individualized treatment decision-making dimensions, and forcibly generates thought chains during the generation process. The generated command responses undergo automated quality checks, and data that fails the checks is manually reviewed and corrected, resulting in a knowledge-enhanced training dataset that includes training, validation, and test datasets. Furthermore, the medical images include at least one of coronary CT angiography or cardiac ultrasound images; Electrophysiological signals include at least one of resting electrocardiogram or Holter monitoring; Behavioral data includes at least one of video for gait analysis or audio clips for cognitive or emotional assessment.

[0008] Furthermore, the supervised fine-tuning technique employs low-rank adaptive technology to fine-tune the parameters of the base model. The fine-tuning tasks include at least one of multimodal data feature extraction and association analysis, text structured element extraction and scoring, or time-series data pattern recognition.

[0009] Furthermore, the progressive curriculum reinforcement learning framework includes: The course is designed with progressive stages from easy to difficult, including lesion area identification, multimodal data association selection, open clinical data analysis, and comprehensive management plan generation. A hybrid reward function is designed for each course stage, comprising a basic reward based on verifiable clinical rules and a generalized reward based on referee model evaluation. A reinforcement learning algorithm is used to train the supervised fine-tuned model based on the progressive course stages and the hybrid reward function. During training, samples that the model repeatedly fails to reason are automatically identified as difficult samples. These difficult samples are placed in a priority experience replay buffer, and the model is trained with emphasis through a weighted sampling mechanism.

[0010] Furthermore, verifiable clinical rules include at least one of diagnostic criteria compliance, medication contraindication conflict detection, or consistency of scoring logic in assessment scales; the referee model is a professional domain large language model with a small parameter scale optimized by professional domain data.

[0011] Furthermore, after obtaining the multimodal inference model, it also includes: The multimodal reasoning model is deployed in a clinical validation environment to process real multimodal clinical data, output structured reasoning results including comprehensive geriatric assessment reports, risk warnings, and individualized treatment recommendations, and iteratively optimizes the model based on clinical feedback.

[0012] A system for building multimodal inference models for comorbidities in the elderly, comprising: The dataset construction module is used to collect and preprocess multimodal clinical data containing elderly patients with coronary heart disease and their comorbid geriatric syndromes, and to construct a knowledge-enhanced training dataset based on clinical diagnosis and treatment pathways and knowledge distillation technology. The supervised fine-tuning module is used to inject specialized domain knowledge of geriatric coronary heart disease and geriatric syndromes into the base model based on a pre-trained multimodal large language model and a knowledge-enhanced training dataset, thereby obtaining a supervised fine-tuned model. The reinforcement learning optimization module is used to construct a progressive curriculum reinforcement learning framework. Based on the progressive curriculum reinforcement learning framework, it uses a hybrid reward function to perform reinforcement learning optimization on the supervised fine-tuned model to stimulate the model's deep reasoning ability and suppress output illusions, ultimately obtaining a multimodal reasoning model.

[0013] An electronic device includes at least one processor and at least one memory, the memory being data-connected to the processor, wherein the memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform any of the methods described above.

[0014] A computer storage medium stores computer instructions, which, when executed by a processor, specifically perform the steps described in any of the methods above.

[0015] A computer program product includes computer instructions, which, when executed by a processor, specifically perform the steps described in any of the methods above.

[0016] Compared with the prior art, this application has the following advantages: This invention addresses the challenges of scarce high-quality multimodal medical data and a lack of in-depth reasoning logic annotation in existing technologies. By collecting and standardizing multi-source clinical data, and combining it with knowledge distillation technology based on medical literature knowledge bases, it can make implicit knowledge in authoritative medical literature explicit and structured, automatically generating instruction-response pairs of data that include a coherent logic of "symptom-mechanism-decision". Through automated verification and manual review, the data quality is ensured, breaking through the limitations of traditional methods that rely solely on limited labeled data. This provides a solid and rich medical logic foundation for model training and solves the core pain points of data scarcity and insufficient annotation.

[0017] To address the problem that general multimodal large language models lack specialized medical knowledge and are prone to factual errors or hallucinations in complex clinical reasoning, this invention adopts a two-stage training architecture of supervised fine-tuning and reinforcement learning. Supervised fine-tuning technology injects specialized knowledge, treatment guidelines, and assessment logic related to geriatric coronary heart disease and geriatric syndromes into the pre-trained basic model, enabling rapid adaptation and cold start of domain knowledge. Subsequently, a progressive curriculum reinforcement learning framework is constructed, and the model is optimized using a hybrid reward function that integrates verifiable clinical rules and judged by a referee model. This not only enables the model to master professional knowledge but also strictly constrains its output to conform to treatment guidelines through rule-based rewards, significantly reducing logical contradictions and safety violations, effectively suppressing hallucinations, and improving the reliability and credibility of the model in key medical applications.

[0018] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a method according to an embodiment of this application is shown; Figure 2 A system framework diagram according to an embodiment of this application is shown; Figure 3 A system implementation phase diagram according to an embodiment of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This invention relates to a method for constructing multimodal reasoning models for comorbidities in the elderly, belonging to the fields of artificial intelligence and smart healthcare technology.

[0023] With the increasing aging of the population, comorbidities in the elderly have become a common phenomenon in clinical practice. Coronary heart disease, as one of the most common chronic diseases in the elderly, often coexists with various geriatric syndromes such as frailty, sarcopenia, cognitive impairment, and malnutrition, forming a complex comorbid state. Comprehensive geriatric assessment is the core means of managing such comorbidities, but traditional assessment processes rely on multidisciplinary collaboration, are time-consuming and highly subjective, and the resulting multimodal data have not been effectively integrated and utilized.

[0024] While multimodal large language models have the ability to process heterogeneous data, they still face three major technical bottlenecks in their application in the medical field: First, high-quality, logically rich, multimodal labeled data is scarce; Second, the general model lacks specialized medical knowledge and is prone to producing "illusionary" outputs that contradict the diagnostic and treatment guidelines; Third, there is a lack of systematic methods for model training to achieve domain knowledge injection and stimulate deep reasoning capabilities.

[0025] To address the aforementioned technical problems, this invention provides a complete technical solution. First, it solves the problem of scarce high-quality training data by employing a data construction method based on clinical treatment pathways and knowledge distillation techniques, providing the model with a foundation of medical knowledge rich in logical reasoning. Second, it adopts a two-stage training architecture of "supervised fine-tuning + reinforcement learning." Supervised fine-tuning first enables rapid injection of domain knowledge, and then progressive course reinforcement learning stimulates the model's deep reasoning ability. Simultaneously, a hybrid reward function is used to constrain the output to conform to clinical standards. Finally, a dynamic backtracking mechanism for difficult samples enhances the model's ability to handle complex cases.

[0026] This invention addresses the aforementioned problems by systematically solving the core challenges of scarce multimodal data, insufficient model expertise, and inadequate reasoning ability in the diagnosis and treatment of comorbidities in the elderly. It constructs a highly reliable, low-illusion, and interpretable professional multimodal reasoning model, providing precise auxiliary decision support for clinical practice.

[0027] The following is in conjunction with the appendix Figure 1-3 The technical solution of the present invention will be further described in detail below.

[0028] S1. Construction of Multimodal Knowledge Augmentation Dataset This step aims to construct a high-quality, logically rich, multimodal training dataset by integrating multi-source clinical data and medical literature knowledge.

[0029] S1-1, Perform data collection and standardization processing.

[0030] De-identified clinical data is collected from partner medical institutions, including: text data such as admission records, medical records, examination and test reports, discharge summaries, etc.; imaging data such as coronary CT angiography images, cardiac ultrasound images, etc., stored in DICOM format; physiological signals such as resting electrocardiogram, Holter electrocardiogram, etc.; and behavioral data such as gait analysis videos, cognitive assessment task voice, etc.

[0031] Subsequently, the above data underwent quality control, cleaning, format standardization, and preprocessing: text data was segmented and entity recognized using natural language processing tools; image data underwent format unification and resolution standardization, and regions of interest were initially segmented using a pre-trained model; physiological signals underwent denoising, filtering, and baseline drift correction; behavioral videos underwent keypoint detection to extract gait parameters, and speech data underwent feature extraction. Using the patient's unique identifier and timestamp, the above multimodal data for the same patient were aligned and correlated on a timeline, forming a complete "patient-time point-multimodal data" record.

[0032] S1-2, Perform medical knowledge distillation and enhancement.

[0033] The system collects clinical guidelines, expert consensus, authoritative textbooks, and research literature related to coronary artery disease and geriatric syndromes in the elderly to construct a local medical literature knowledge base. Utilizing a large language model with strong reasoning capabilities as the teacher model, prompts are designed to guide the model in extracting structured knowledge from the literature, including: treatment rules, such as "For stable coronary artery disease patients ≥75 years old with moderate to severe frailty, revascularization decisions require a comprehensive assessment of life expectancy and surgical risks"; reasoning pathways, such as "A patient's chest CT scan shows a coronary artery calcium score >400, combined with typical angina symptoms and ST segment depression on Holter monitoring, suggesting a high probability of coronary artery disease; further evaluation with coronary CTA is recommended"; drug knowledge, such as "The combined use of aspirin and warfarin increases the risk of bleeding; close monitoring of INR is necessary"; and assessment logic, such as "Frailty assessment includes five dimensions: fatigue, resistance, walking ability, disease, and weight loss; three or more positive dimensions indicate frailty."

[0034] The extracted knowledge is stored in the form of "head entity-relationship-tail entity" triples or "question-thought chain-answer" to generate knowledge-enhanced data.

[0035] S1-3, Perform inference data synthesis and quality assurance.

[0036] S1-3-1, Synthesis of inference data.

[0037] Using a multimodal large language model as the engine, combined with the aforementioned knowledge-enhanced data and standardized real data, instruction-response pairs are automatically generated based on clinical diagnosis and treatment pathways.

[0038] The instruction template is based on clinical pathway design and covers multiple reasoning steps, such as "medical history summarization and feature extraction", "multimodal sign recognition and association", "pathophysiological mechanism and risk assessment analysis", and "personalized comprehensive management plan formulation". During the generation process, the model is forced to output a coherent thought chain. For example, for the frailty assessment task, the model is required to first generate the thought chain: "1. Extract age and comorbidities from the medical history summary; 2. Read the LVEF value from the echocardiogram report, indicating possible cardiogenic weakness; 3. Gait speed is lower than the frailty threshold; 4. Based on the above information, it meets the diagnostic criteria for frailty", and then generate the final answer.

[0039] S1-3-2. Introduce a rule-based automatic verification mechanism to check the logical consistency of the generated content; at the same time, establish an “LVLM-as-a-Judge” mechanism to use a lightweight judge model to score the quality of all generated content and filter out low-confidence samples.

[0040] Domain experts manually reviewed and corrected difficult samples and complex, long-tail comorbid cases, providing feedback to the judging model to improve its judgment capabilities. Based on the complexity of the generated samples and the perplexity of the model, the final dataset was divided into a "basic knowledge layer," a "core capability layer," and a "higher-order reasoning layer," and further divided into training, validation, and test sets in a 7:2:1 ratio to form a knowledge-enhanced training dataset.

[0041] S2, Supervised Fine-tuning of Domain Knowledge Injection This step aims to inject domain-specific knowledge into a pre-trained multimodal large language model through supervised fine-tuning techniques to achieve domain adaptation.

[0042] We selected an open-source multimodal large language model that performs well on general multimodal understanding tasks as the base model, and used a low-rank adaptive parameter efficient fine-tuning technique. We then used the knowledge-enhanced training dataset constructed by S1 to perform full-supervised fine-tuning of the base model.

[0043] The fully supervised fine-tuning process employs a multi-task learning strategy, enabling the model to simultaneously learn the following clinical tasks: The image and text comprehension task requires the model to describe lesion features based on images or select corresponding images based on descriptions; Text-based reasoning tasks require models to answer disease diagnoses and assess the rationality of medication based on medical record texts; Multimodal question answering tasks require models to integrate text, image, and signal data to answer questions about diagnosis, assessment, and prognosis; The report generation task requires the model to automatically generate a structured summary of comprehensive geriatric assessments or a draft of treatment recommendations based on the input information.

[0044] The learning rate adopts a warm-up and decay strategy, and the batch size is set according to the computing resources. After this stage, a model with preliminary domain knowledge is obtained, denoted as GerVLM-R1-SFT. Thus, domain knowledge transfer can be quickly achieved through supervised fine-tuning, enabling the model to master the basic concepts and task framework of diagnosis and treatment of comorbidities in the elderly.

[0045] S3, progressive course reinforcement learning optimization.

[0046] This step aims to further enhance the model's deep reasoning capabilities, complex decision-making abilities, and output security through a progressive curriculum reinforcement learning framework.

[0047] S3-1, Progressive Course Stage Design.

[0048] The complex clinical reasoning task is broken down into four progressively more difficult stages: Phase 1 is lesion localization. Input a coronary CTA image and ask the model to use a bounding box to mark the location of the most severe stenosis and describe the degree of stenosis. Phase 2 is a multi-image selection task. Given a description of clinical symptoms and multiple echocardiogram images, the model is required to select the image that best supports the diagnosis. Phase three involves open-ended text and image questions, providing patients with complete multimodal data and posing open-ended questions such as "What is the risk of this patient developing acute heart failure? Please list the main risk factors." Phase four involves developing a comprehensive management plan. Based on all information of elderly coronary heart disease patients with comorbid frailty and sarcopenia, the model is required to generate an individualized comprehensive management plan that includes medication adjustment recommendations, exercise prescription principles, nutritional support goals, key points of psychological intervention, and follow-up plans.

[0049] S3-2, Construct a hybrid reward function.

[0050] A dedicated hybrid reward function is designed for each course stage, which can be expressed as follows:

[0051] Where R represents the total reward value; This indicates a rule-based reward, calculated based on explicit, programmable clinical rules, such as compliance with diagnostic criteria (must include specific keywords), detection of medication contraindications, and logical consistency of assessment scale scoring (e.g., FRAIL scale scoring rules). For example, when developing a medication regimen, if the model recommends a contraindicated drug for the patient (e.g., using full-dose metformin in patients with severe renal insufficiency), a large negative reward is given. If the regimen includes essential drugs (e.g., dual antiplatelet therapy for secondary prevention of coronary artery disease), a positive reward is given.

[0052] The referee reward is represented by a professional domain large language model finely tuned on high-quality medical dialogue data. The current model's output and input state are submitted to the referee model, which generates a comprehensive quality score between 0 and 1.

[0053] λ rule With λ judge for and The weighting coefficients.

[0054] S3-2. Model optimization is performed using reinforcement learning algorithms.

[0055] The decoupled truncation strategy optimization algorithm is adopted, and the GerVLM-R1-SFT model is trained using the above-mentioned course stages and mixed reward function. This algorithm can effectively alleviate the problems of gradient instability and low exploration efficiency in reinforcement learning training through decoupled strategy update and value function estimation.

[0056] During training, the agent (GerVLM-R1-SFT) explores the course environment, receives rewards based on generated text actions, and gradually updates policy parameters.

[0057] Finally, a dynamic backtracking mechanism for difficult samples is introduced. During training, the reward value of each round is continuously monitored, and samples that repeatedly receive low rewards are automatically marked as "difficult samples" and stored in a priority experience replay buffer.

[0058] In subsequent sampling of training data, these difficult samples are drawn from the buffer with higher probability for weighted sampling training, forcing the model to focus its learning until its performance on this type of sample reaches a preset threshold. After this stage of training is completed, the final reinforcement learning optimized model is obtained, denoted as GerVLM-R1-RL.

[0059] S4. Clinical validation and iterative closed loop.

[0060] This step aims to deploy the optimized model in a clinical environment and continuously iterate and optimize it based on actual feedback.

[0061] The GerVLM-R1-RL model was encapsulated as an inference API and integrated into a clinical research prototype platform.

[0062] In the specific implementation process, eligible patients are recruited from the geriatrics or cardiology departments of partner hospitals. After obtaining informed consent, their anonymized multimodal data is input into the platform. The model then automatically processes the data and outputs the following: A structured comprehensive assessment report for the elderly, listing scores and interpretations for each dimension; The risk of major adverse cardiovascular events or falls in the next six months can be divided into high-risk, medium-risk, and low-risk levels. The draft of five personalized prescription recommendations can cover aspects such as medication, exercise, nutrition, psychology, and follow-up.

[0063] When clinicians use the platform to assist in decision-making, they rate the accuracy, usability, and safety of the model's output and provide suggestions for improvement.

[0064] The system records the discrepancies between the model's output and the medical team's final decision. Regularly collected feedback and discrepancy data are considered new "gold standards" or "hard samples" to construct new fine-tuning or reinforcement learning data, initiating a new round of model iteration and forming a continuous improvement loop of "R&D-application-feedback-optimization".

[0065] Through the above specific implementation methods, the present invention has successfully constructed a high-performance, high-reliability multimodal reasoning model for elderly patients with coronary heart disease and geriatric syndromes, providing clinicians with a powerful intelligent auxiliary tool and significantly improving the standardization, accuracy and efficiency of diagnosis and treatment of geriatric comorbidities.

[0066] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing a multimodal reasoning model for comorbidities in the elderly, characterized in that, include: Multimodal clinical data containing comorbidities in the elderly are collected and preprocessed. Based on the clinical diagnosis and treatment pathways of comorbidities in the elderly, the rules and framework for data construction are determined. Knowledge distillation technology is used to extract structured knowledge from medical literature. Based on the rule framework and the structured knowledge, a knowledge-enhanced training dataset is constructed. Using a pre-trained multimodal large language model as the base model, and based on the knowledge-enhanced training dataset, supervised fine-tuning techniques are employed to inject specialized domain knowledge of geriatric comorbidities into the base model, resulting in a supervised fine-tuned model. A progressive curriculum reinforcement learning framework is constructed, and based on the progressive curriculum reinforcement learning framework, a composite reward function is used to optimize the supervised fine-tuned model through reinforcement learning to stimulate the deep reasoning ability of the model and suppress output illusion, ultimately obtaining a multimodal reasoning model.

2. The method according to claim 1, characterized in that, Constructing a knowledge-enhanced training dataset includes: Collect and standardize multi-source clinical data, including de-identified electronic medical record text, medical images, electrophysiological signals, and behavioral video and audio data. A medical literature knowledge base in the field of geriatric comorbidities was collected and constructed. A large language model was used as a teacher model to extract structured diagnosis and treatment rules, drug interactions, prognostic factors and key reasoning paths from the medical literature knowledge base to generate knowledge-enhanced data. Based on clinical treatment pathways, the system automatically or / and semi-automatically generates instruction-response pairs of data that include risk assessment, multimodal diagnosis, and individualized treatment decision-making dimensions, and forcibly generates thought chains during the generation process. The generated instruction response performs automated quality checks on the data, and manually reviews and corrects the data that fails the checks, forming a knowledge-enhanced training dataset that includes a training dataset, a validation dataset, and a test dataset.

3. The method according to claim 2, characterized in that, The medical images include at least one of coronary CT angiography images and cardiac ultrasound images; The electrocardiographic signals include at least one of resting electrocardiogram and Holter monitoring; The behavioral data includes at least one of video for gait analysis and audio clips for cognitive or emotional assessment.

4. The method according to claim 1, characterized in that, The supervised fine-tuning technique employs low-rank adaptive technology to fine-tune the parameters of the base model. The fine-tuning tasks include at least one of multimodal data feature extraction and association analysis, text structured element extraction and scoring, and time-series data pattern recognition.

5. The method according to claim 1, characterized in that, The progressive curriculum reinforcement learning framework includes: The course is designed with progressive stages from easy to difficult, including lesion area identification, multimodal data association selection, open clinical data analysis, and comprehensive management plan generation. A hybrid reward function is designed for each course stage, comprising a basic reward based on verifiable clinical rules and a generalized reward based on referee model evaluation. A reinforcement learning algorithm is used to train the supervised fine-tuned model based on the progressive course stages and the hybrid reward function. During training, samples that the model repeatedly fails to reason are automatically identified as difficult samples. These difficult samples are placed in a priority experience replay buffer, and the model is trained with emphasis through a weighted sampling mechanism.

6. The method according to claim 5, characterized in that, The verifiable clinical rules include at least one of diagnostic criteria compliance, medication contraindication conflict detection, and / or assessment scale scoring logic consistency; the adjudication model is a professional domain large language model with a small parameter scale optimized by professional domain data.

7. The method according to any one of claims 1 to 6, characterized in that, After obtaining the multimodal inference model, the following steps are also included: The multimodal reasoning model is deployed in a clinical validation environment to process real multimodal clinical data, output structured reasoning results including comprehensive geriatric assessment reports, risk warning prompts, and individualized treatment recommendations, and iteratively optimizes the model based on clinical feedback.

8. A multimodal reasoning model construction system for comorbidities in the elderly, characterized in that, The system includes: The dataset construction module is used to collect and preprocess multimodal clinical data containing elderly patients with coronary heart disease and their comorbid geriatric syndromes, and to construct a knowledge-enhanced training dataset based on clinical diagnosis and treatment pathways and knowledge distillation technology. The supervised fine-tuning module is used to inject specialized domain knowledge of geriatric coronary heart disease and geriatric syndromes into the pre-trained multimodal large language model as the base model, based on the knowledge-enhanced training dataset, to obtain the supervised fine-tuned model. The reinforcement learning optimization module is used to construct a progressive curriculum reinforcement learning framework and, based on the progressive curriculum reinforcement learning framework, to perform reinforcement learning optimization on the supervised fine-tuned model using a hybrid reward function, so as to stimulate the deep reasoning ability of the model and suppress output illusion, and finally obtain a multimodal reasoning model.

Images