Multi-modal drug recommendation method and system based on large language model driving

Abstract

The invention discloses a multi-modal drug recommendation method and system based on large language model driving, and relates to the technical field of medical informatics and artificial intelligence. The invention aims to solve the limitation of an existing drug recommendation model in the aspects of patient characterization construction, drug multi-modal modeling and large language model application. Comprising the following steps: constructing patient characterization, and aligning a patient state and a potential medication space in a characterization learning stage through a cooperative prompt project driven by a large language model; multi-modal drug characterization is constructed, and clinical logic and chemical characteristics of drugs are deeply depicted through semantic expert and molecular structure expert dual-channel design; a'static anchoring dynamic 'time sequence reasoning mechanism is provided, context modulation is performed on a dynamic evolutionary clinical event sequence by utilizing a static treatment baseline formed by global medication history, and finally, an accurate and safe medication combination is generated through a label sensing prediction module. According to the method, the accuracy, safety and clinical logic self-consistency of drug recommendation are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of medical informatics and artificial intelligence, specifically to a method for clinical decision support using electronic health records (EHRs), and more particularly to an intelligent and safe drug recommendation method and system based on large language model (LLM) driven and multimodal data representation. Background Technology

[0002] Utilizing electronic health record (EHR) data to assist in clinical drug recommendations is an important research direction in the fields of medical informatics and artificial intelligence. Its core task is to predict the safest and most effective drug combination for the next visit based on a patient's longitudinal medical history. An ideal recommendation system not only needs high accuracy but must also be able to avoid adverse drug interactions (DDIs) to ensure patient medication safety.

[0003] Existing drug recommendation methods mainly follow two technical approaches: The first is instance-based recommendation, such as logistic regression (LR) or classifier chains (ECC). These methods primarily predict based on the patient's current health status but generally ignore the temporal evolution of the disease, resulting in limited representational capabilities. The second is recommendation based on longitudinal models, such as the RETAIN model using recurrent neural networks (RNNs) and the GAMENet model combining graph neural networks (GNNs) and memory networks. These methods capture dynamic changes in health status by modeling the patient's complete medical history, achieving significant progress.

[0004] However, despite the continuous evolution of existing technologies, three key technical bottlenecks remain in building comprehensive, accurate, and clinically logical recommendation models: The construction of patient representations lacks foresight: Existing methods often retrospectively aggregate historical clinical states when constructing patient representations, failing to proactively consider the final medication decision needs. This leads to a semantic gap between patient representations and the drug space; for example, the representation of a patient with "hypertension" fails to proactively associate with the semantics of "antihypertensive drugs," weakening the accuracy of recommendations.

[0005] Insufficient depth in drug multimodal modeling: Early models treated drugs as abstract discrete codes, ignoring their rich chemical structures and multi-level clinical semantic information. Although subsequent studies attempted to introduce molecular graph neural networks or external knowledge graphs, they were mostly limited to single modalities or simple feature splicing, failing to achieve a systematic and deep integration of chemical features and clinical knowledge, thus limiting the cognitive depth and generalization ability of the models.

[0006] Large Language Models (LLMs) have inherent limitations: while LLMs possess powerful capabilities for encoding medical knowledge and generating semantics, their direct use in drug recommendation engines presents the core challenge of "output not being in the database." Natural language drug names generated by LLMs are difficult to reliably map back to the standardized, limited drug databases of medical institutions, leading to unpredictable recommendation results and severely impacting the model's practicality and reliability.

[0007] Therefore, existing technologies have not yet been able to effectively address the aforementioned challenges in a coordinated manner. There is an urgent need in this field for an innovative, unified framework that can safely and controllably integrate the semantic understanding capabilities of LLM into recommendation systems and achieve deep semantic alignment between patients and drugs at the multimodal level, ultimately leading to accurate, safe, and clinically interpretable intelligent drug recommendations. Summary of the Invention

[0008] The purpose of this invention is to address the problems in current technologies, such as the lack of prospective guidance in patient representation, insufficient fusion of multimodal drug information, and inherent defects in the application of large language models, by providing a novel drug recommendation method and system based on a large language model as the semantic center.

[0009] The technical solution of the present invention is as follows: A multimodal drug recommendation method driven by a large language model includes the following steps: Acquire and preprocess patients' electronic health record data, including diagnosis codes, surgical codes, and historical medication codes from each patient's visit, and clean, standardize, and map the data to construct an initial learnable embedding vector for each medical entity. Constructing prospective patient representations: A large language model is used to generate enhanced semantics for drug therapy for diagnostic and surgical codes, and the enhanced semantics are deeply aligned with the structured representations extracted from electronic health records through a bidirectional fusion mechanism to generate patient representations; Constructing a dual-expert multimodal drug characterization: setting up molecular structure expert pathways and semantic expert pathways in parallel, enhancing the patient's historical medication records with knowledge, and integrating the chemical structure information and clinical semantic information of the drug into a global medication history characterization through a hierarchical deep fusion strategy; Context-aware temporal reasoning: The global medication representation is used as a static context. Each time step of the dynamic clinical event sequence composed of the patient representations of historical visits is modulated. The modulated sequence is then input into the temporal model to generate a context-aware patient health trajectory representation. Generate drug recommendation results: Based on health trajectory representation, the recommendation probability of each candidate drug is dynamically calculated using a label-aware prediction module, and combined with a conditional drug interaction risk control strategy, the final drug combination is output.

[0010] Furthermore, the construction of prospective patient characterization specifically includes: Generate initial structured representations: For the set of diagnostic codes in each visit, the discrete codes are converted into continuous vectors through the embedding layer, and then aggregated using the mask mean pooling method to generate initial diagnostic and surgical representations. A collaborative prompting strategy is implemented for semantic enhancement: instructions for medication decisions are set up to guide a large language model to generate enhanced semantics for diagnoses and surgeries; specifically, for diagnostic codes, the large language model is guided to generate semantic descriptions of their definitions, core treatment goals, typical drug categories, contraindications, and related surgeries; for surgical codes, the large language model is guided to generate semantic descriptions of their indications, preoperative medication adjustments, and commonly used postoperative medications; and all medical entities are bound to unique codes in a standard database. Deep fusion is achieved by employing bidirectional cross-attention: structured representations and semantic representations serve as queries, keys, and values ​​for each other, enabling bidirectional information retrieval and extraction. The intensity of information injection is dynamically controlled through a gated residual network to generate the fused patient representation.

[0011] Furthermore, the deep fusion of bidirectional cross-attention specifically includes: Project the initial structured representation and the enhanced semantic representation into a unified fusion space; Perform two multi-head cross-attention operations to make the two representations mutually serve as queries, keys, and values, and perform bidirectional information extraction; By dynamically controlling the information injection intensity through a gated residual network, information from one side is prevented from being overwhelmed by information from the other. Finally, the refined representations are added together and layer normalized to generate a fused patient representation.

[0012] Furthermore, the construction of the dual-expert multimodal drug characterization specifically includes: Setting up a molecular structure expert pathway: using a pre-trained molecular model to convert the drug's SMILES string into a molecular structure embedding, and then fusing it with the initial embedding of historical drug use through a cross-attention mechanism to generate enhanced molecular features; Establish a semantic expert pathway: Based on the hierarchical prompting strategy of the anatomy-therapeutics-chemistry classification system, guide a large language model to generate structured semantic knowledge of drugs at the anatomical, therapeutic and pharmacological levels, and integrate it with the initial drug use embedding to generate enhanced semantic features; A two-stage hierarchical fusion strategy is adopted: enhanced molecular features and enhanced semantic features are input into the MLP-Mixer architecture for intermodal interaction, and then adaptive weights are calculated through the gated attention fusion module to perform weighted summation to generate a global drug use history representation.

[0013] Furthermore, the setting of the molecular structure expert pathway specifically includes: Using molecular models pre-trained on massive molecular graph data, drug SMILES strings are transformed into high-quality molecular structure embeddings. The molecular structure embedding is fused with the initial embedding of the patient's historical medication record through a bidirectional cross-attention mechanism to generate enhanced molecular features that deeply integrate the patient's medication context.

[0014] Furthermore, the setting of the semantic expert pathway specifically includes: A hierarchical prompting strategy based on the anatomy-therapeutics-chemistry classification system is adopted to guide a large language model to generate detailed descriptions of drug action organs, therapeutic subgroups, clinical indications, mechanisms of action, and drug interactions in three dimensions: anatomy, therapeutics, and pharmacology / chemistry. The generated hierarchical semantic embeddings are fused with the initial medication embeddings through cross-attention to generate enhanced semantic features.

[0015] Furthermore, the two-stage hierarchical fusion strategy specifically includes: By inputting enhanced molecular features and enhanced semantic features into the MLP-Mixer architecture, the deep nonlinear mapping relationship between the two modalities is explicitly modeled through token mixing and channel mixing MLP blocks; The interactive modal view is input into the gated attention fusion module, where an adaptive weight is calculated for each modality through a small attention network, and a weighted sum is performed to generate a unified global medication history representation.

[0016] Furthermore, the context-aware temporal reasoning specifically includes: Static context broadcasting: The global medication history representation is copied and expanded along the time dimension to form a static context matrix with the same length as the medical visit sequence; Forming a comprehensive input vector: At each time step, the dynamic patient representation of that time step is concatenated with the corresponding vector of the static context matrix to form a comprehensive input vector; Temporal network coding: The integrated input vector sequence is input into a bidirectional gated recurrent unit network to encode and generate a context-aware patient health trajectory representation.

[0017] Furthermore, the generated drug recommendation results specifically include: A label-level attention mechanism is used as the prediction head: the embedding matrix of the drug vocabulary is used as the query matrix, the health trajectory representation sequence is used as the key and value matrix, attention score is calculated for each candidate drug, a unique context vector is generated, and the recommendation probability is calculated based on this. The model is trained using a composite optimization objective: the task loss is DiceBCELoss, which is a weighted combination of binary cross-entropy loss and Dice loss; the drug interaction safety loss is a conditional regularization term that is activated only when the predicted drug interaction rate exceeds the clinical threshold, and is obtained by calculating the co-occurrence probability of the predicted drug pair and multiplying it by the prior risk value.

[0018] This application also includes a multimodal drug recommendation system driven by a large language model, which applies a multimodal drug recommendation method driven by a large language model, including: Preprocessing module: Acquires and preprocesses the patient's electronic health record data, including the diagnosis codes, surgical codes, and historical medication codes of the patient's previous visits, and cleans, standardizes, and maps the data to codes, and constructs an initial learnable embedding vector for each medical entity; Prospective Patient Representation Module: Utilizes a large language model to generate enhanced semantics for diagnostic and surgical codes, and deeply aligns the enhanced semantics with structured representations extracted from electronic health records through a bidirectional fusion mechanism to generate patient representations; Dual-expert multimodal drug characterization module: The molecular structure expert pathway and the semantic expert pathway are set up in parallel to enhance the patient's historical medication records. Through a hierarchical deep fusion strategy, the chemical structure information and clinical semantic information of the drug are integrated into a global medication history characterization. Context-aware temporal reasoning module: It uses the global medication representation as a static context, modulates each time step of the dynamic clinical event sequence composed of patient representations from historical visits, and then inputs the modulated sequence into the temporal model to generate a context-aware patient health trajectory representation. Drug recommendation result generation module: Based on health trajectory representation, the module dynamically calculates the recommendation probability for each candidate drug using a label-aware prediction module, and outputs the final drug combination by combining a conditional drug interaction risk control strategy.

[0019] Compared with existing technologies, the advantages of this invention are: 1. Achieved forward-looking and collaborative representation: Through LLM-driven collaborative prompting engineering, this application proactively constructs a bidirectional semantic association between patient status and drug information in the initial stage of representation learning. The generated patient representation has a clear downstream decision-making orientation, which significantly improves the accuracy of recommendations. 2. A deeply integrated multimodal drug characterization was constructed: Through a dual-expert parallel enhancement and hierarchical fusion architecture, this application comprehensively characterizes the chemical properties and multi-level clinical semantics of drugs, providing the model with richer knowledge, enabling it to perform inductive reasoning and improving its generalization ability. 3. Innovative application paradigm of LLM in recommender system: Through the "encoding anchoring" principle and the "static anchoring dynamic" reasoning mechanism, this application ingeniously integrates the powerful semantic understanding capability of LLM into the structured recommender framework, which not only overcomes the limitations of direct generation of LLM, but also makes full use of its knowledge integration advantages, providing a new solution for complex time-series decision-making tasks. 4. Improved safety and reliability of recommendations: By introducing a conditional DDI loss function and a label-aware prediction mechanism, this method can improve accuracy while effectively controlling the recommendation of high-risk drug combinations, making the results more in line with clinical safety requirements. Attached Figure Description

[0020] Figure 1 This is the overall flowchart of the method in this application.

[0021] Figure 2 This is a model structure diagram of the method in this application.

[0022] Figure 3 This is a schematic diagram of the bidirectional cross-attention mechanism.

[0023] Figure 4 This is a schematic diagram of the intermodal interaction in the MLP-Mixer.

[0024] Figure 5 This is a schematic diagram of the "static anchoring dynamic" temporal reasoning mechanism. Detailed Implementation

[0025] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0026] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0027] Please see Figure 1-5 A multimodal drug recommendation method driven by a large language model, such as Figure 1 As shown, it includes the following steps: First, acquire and preprocess the patient's electronic health record (EHR) data, including their diagnosis, surgery, and medication records from previous visits.

[0028] Secondly, a prospective patient representation is constructed. The core of this step is the introduction of collaborative prompting engineering driven by a large language model (LLM). By designing prompts for medication decisions, the LLM is guided to generate enhanced semantics for diagnosis and surgery coding, including typical treatment options, potential medication needs, and key contraindications. Simultaneously, following the "code anchoring" principle, it is ensured that all generated semantics are precisely aligned with standard medical codes, fundamentally solving the problem of "output not in the database." Finally, a bidirectional cross-attention mechanism is used to deeply integrate the external semantic representation generated by the LLM with the internal structured representation extracted from EHR data, generating a patient representation that contains both statistical regularities and prospective clinical guidance.

[0029] Next, a dual-expert multimodal drug characterization was constructed. This step aimed to comprehensively characterize the drug from two orthogonal dimensions: chemical structure and clinical semantics. Two parallel pathways, one for "molecular structure experts" and the other for "semantic experts," were used to augment the patient's historical medication records. The molecular structure experts extracted the drug's structural and physicochemical features using a pre-trained molecular graph model; the semantic experts, using the Anatomy-Therapeutics-Chemistry (ATC) classification system as a framework, guided LLM to generate deep semantic representations of the drug at the anatomical, therapeutic, and pharmacological levels through a hierarchical prompting strategy. Finally, an innovative two-stage hierarchical fusion strategy (first using an MLP-Mixer for intermodal interaction, then adaptively unifying through gated attention) integrated the outputs of the two experts into a single, highly condensed global medication history characterization.

[0030] Then, context-aware temporal reasoning is performed. This invention proposes an innovative concept of "Static Anchoring of Dynamics." The global medication history representation generated in the previous step is regarded as a "static" treatment baseline and attached to each time step of the "dynamic" clinical event sequence composed of patient representations from previous visits. In this way, temporal models such as Bi-Gated Recurrent Units (Bi-GRUs) can be aware of the overall medication context at all times when encoding the patient's health trajectory, thereby learning a more clinically logical pattern of disease evolution.

[0031] Finally, drug recommendations are generated. A label-aware prediction module dynamically aggregates the most relevant historical evidence from the coded patient health trajectory for each candidate drug in the vocabulary, thereby calculating the recommendation probability. Simultaneously, a conditional drug interaction (DDI) penalty term is introduced, activated only when the risk of the predicted combination exceeds a preset threshold, to achieve a balance between safety and accuracy, outputting the final medication recommendation.

[0032] This application also includes a multimodal drug recommendation system driven by a large language model, which applies a multimodal drug recommendation method driven by a large language model, including: Preprocessing module: Acquires and preprocesses the patient's electronic health record data, including the diagnosis codes, surgical codes, and historical medication codes of the patient's previous visits, and cleans, standardizes, and maps the data to codes, and constructs an initial learnable embedding vector for each medical entity; Prospective Patient Representation Module: Utilizes a large language model to generate enhanced semantics for diagnostic and surgical codes, and deeply aligns the enhanced semantics with structured representations extracted from electronic health records through a bidirectional fusion mechanism to generate patient representations; Dual-expert multimodal drug characterization module: The molecular structure expert pathway and the semantic expert pathway are set up in parallel to enhance the patient's historical medication records. Through a hierarchical deep fusion strategy, the chemical structure information and clinical semantic information of the drug are integrated into a global medication history characterization. Context-aware temporal reasoning module: It uses the global medication representation as a static context, modulates each time step of the dynamic clinical event sequence composed of patient representations from historical visits, and then inputs the modulated sequence into the temporal model to generate a context-aware patient health trajectory representation. Drug recommendation result generation module: Based on health trajectory representation, the module dynamically calculates the recommendation probability for each candidate drug using a label-aware prediction module, and outputs the final drug combination by combining a conditional drug interaction risk control strategy.

[0033] In another specific embodiment, to make the objectives, technical solutions, and advantages of the present invention clearer, the specific implementation methods of the present invention will be described in detail below with reference to the accompanying drawings. This embodiment takes the recommendation of medication for patients in a clinical setting of an intensive care unit (ICU) as an example to illustrate the method proposed by the present invention in detail.

[0034] Step 1, Data Acquisition and Preprocessing: like Figure 1 As shown, the system first acquires the patient's electronic health record (EHR) data, sourced from publicly available datasets (such as MIMIC-III / IV). A patient's EHR is defined as a longitudinal sequence of their medical visits. Each medical visit Includes a set of diagnostic codes (Based on ICD-9 / 10 standards), a set of surgical codes (Based on ICD-9-CM standard) and a set of drug codes (Based on ATC standard). Data is cleaned, standardized, and imputed as necessary, and a learnable initial embedding matrix is ​​defined for each medical code (diagnosis, surgery, drug). Drug interaction (DDI) relationships are established using an adjacency matrix. To indicate, among which It is a collection of drug-related terms.

[0035] The second step is to construct a prospective patient characterization: like Figure 2 As shown, for the first The process of constructing the patient's characterization during the first visit is as follows: 1. Structured coding: For medical visits Diagnostic set in and surgical collection Through the initial embedding matrix Find the vector for each encoding and use masked mean pooling to calculate the initial internal representation. and .

[0036] 2. LLM Semantic Enhancement: To address the problems mentioned in the background section, this application introduces LLM-driven semantic enhancement. This application designs a collaborative prompting strategy for downstream decision-making: Regarding diagnosis: the instruction is for LLM to act as a "clinical information expert," not only to provide a disease definition, but also to proactively clarify its core treatment goals, typical first- and second-line drug classes, key drug contraindications, and related typical surgeries.

[0037] For surgery: Instruct the LLM to act as a "surgical pharmacologist," focusing on the main indications for the surgery (associated diagnosis), key preoperative medication adjustments, and common short-term postoperative medications.

[0038] All prompts force LLM to anchor entities to precise medical codes (code anchoring), ensuring that the generated semantics align with entities within the dataset. The description text generated by LLM is vectorized by a semantic encoder and then weighted and aggregated using self-attention pooling to obtain the external semantic representation. and The calculation process for self-attention pooling is as follows: , in, It is the input encoded vector sequence. and It is a learnable weight matrix. It is attention weight. This represents the hyperbolic tangent activation function. This represents the unnormalized attention score. Represents the normalized exponential function, This represents the vector representation after weighted aggregation. Indicates the first Attention weights for each input vector. Indicates the first One input encoded vector.

[0039] 3. Two-way integration: such as Figure 3 As shown, a bi-directional cross-attention fusion module is used to enable the internal initial representation... With external semantic representation Information is extracted by providing context for each other. First, both are projected onto a unified fusion space: .

[0040] in The linear projection layer representing the original representation. This represents a linear projection layer that enhances the representation.

[0041] 4. Then perform two multi-head cross-attention operations: , , in, This indicates a multi-head attention mechanism. Represents the query matrix. Represents the key matrix. This represents the value matrix. The fusion process employs a gated residual mechanism to dynamically control the intensity of information injection. For example, the update: .

[0042] in It is a gating signal. Indicates splicing, Represents element-wise product. This represents the Sigmoid activation function. The learnable weight matrix representing the gating mechanism. Indicates the fused Characterization. For Performing symmetric operations yields The final fusion characterization is as follows: .

[0043] in, Presentation layer normalization operation.

[0044] The third step is to construct a dual-expert, multimodal drug characterization system: like Figure 3 As shown, this module is designed to provide a complete history of a patient's medications. Create a unified, high-quality context representation : 1. Dual-expert parallel enhancement: Molecular structure expert: Utilizing a self-supervised model pre-trained on massive molecular graph data, this algorithm generates molecular structure embeddings for each drug in historical drug use. These embeddings are then aggregated and fused with the initial embeddings from historical drug use through cross-attention to obtain enhanced molecular features. .

[0045] Semantic experts designed a hierarchical cueing strategy using the WHO-recommended ATC classification system as the semantic framework. This strategy guides LLM (Label Learning Model) to generate structured semantic knowledge for drugs at three levels: anatomy (ATC-1), therapeutics (ATC-2), and pharmacology (ATC-3 to 5). Similarly, the generated semantic representations are fused with the initial embeddings of historical drug use to obtain enhanced semantic features. .

[0046] 2. Hierarchical and Deep Integration: Intermodal interaction: enhancing molecular features and semantic features As two "tokens," they are input into an MLP-Mixer architecture. For example... Figure 4 As shown, MLP-Mixer operates through decoupled token-mixing and channel-mixing MLP blocks: , in, It is the input tensor ( (modalities) Operate along the token dimension, Operate along the feature dimension, This represents the matrix transpose operation.

[0047] Final unification: After deep interaction is completed in the MLP-Mixer, an AttentionGated Fusion mechanism is adopted to provide two refined modal views. Calculate adaptive attention weights and perform a weighted summation accordingly to ultimately generate a single drug history context vector. .

[0048] in, This represents the transpose of the learnable context vector. Indicates the first A view vector for each modality.

[0049] Step 4, context-aware temporal reasoning: like Figure 4 As shown, this step is a specific implementation of the "static anchoring dynamic" concept of the present invention: 1. Context Integration: The single, static global medication history vector generated in step 3 is integrated. Broadcasting is performed along the time dimension to form a matrix that matches the length T of the patient visit sequence. Each of these rows is always equal to .

[0050] 2. Timing coding: such as Figure 5 As shown, at each time step The dynamic clinical characterization of this time step ( and This is concatenated with the corresponding global medication context to form the comprehensive input vector for that time step: , in, This indicates a vector concatenation operation.

[0051] This sequence integrates the global medication context. The input is fed into a bidirectional gated recurrent unit (Bi-GRU) network, which ultimately generates a sequence of hidden states. This serves as a deep encoding of the patient's dynamic health trajectory.

[0052] Step 5, Label-Aware Prediction and Optimization: 1. Label-aware prediction: To achieve accurate multi-label prediction, a label-level attention mechanism is used as the prediction head. It embeds the entire drug vocabulary into a matrix. As a query, retrieve the output sequence of the Bi-GRU. (As Key and Value). For the first in the vocabulary The calculation process for each drug is as follows: , in, It is a medicine Dynamically generated context vectors Let $t$ represent the hidden state vector of the Bi-GRU at time step $t$. Indicates the drug The unnormalized prediction score, This represents a linear transformation layer (fully connected layer).

[0053] 2. Optimization objective: The model's total loss function. It consists of two parts: mission loss A conditional DDI security loss .

[0054] Mission loss The DiceBCELoss is used, and its definition is: , in, This represents the weighting coefficients that balance the cross-entropy loss and the Dice loss. This represents the binary cross-entropy loss function. This represents the Dice loss function.

[0055] DDI security loss The purpose of penalizing pairings in predicted drug combinations that have known interactions is calculated as follows: , in, It is a predictive drug The probability, Indicates the drugs in the DDI adjacency matrix and medicine Does an interaction exist? (1 for existence, 0 for non-existence). This loss is conditional. The final optimization objective is: , in It is an indicator function. It is a hyperparameter that controls the intensity of DDI penalty. This indicates whether the DDI rate of the current prediction result exceeds the preset safety threshold. .

[0056] Through the above steps, this invention constructs an end-to-end drug recommendation model. This model uses LLM as a semantic hub to collaboratively address key challenges in patient representation, drug representation, and temporal reasoning, thereby generating accurate and safe drug recommendations.

[0057] Experimental verification: Dataset: To verify the effectiveness of this invention, the experiments used the MIMIC-III and MIMIC-IV public datasets, which are widely used in clinical research in intensive care units (ICUs). These datasets contain clinical records, physiological monitoring data, laboratory results, and medication records of ICU patients. The data preprocessing workflow is as follows: diagnostic and surgical codes were standardized according to the International Classification of Diseases, Ninth Revision (ICD-9) and Tenth Revision (ICD-10), and medication data were standardized according to ATC Level 3 codes, retaining only the visit records that simultaneously contain disease, surgical, and medication information. Detailed statistical information of the datasets is shown in Table 1 below.

[0058] Table 1. Dataset Statistics Table

[0059] Evaluation indicators: Multiple metrics are used to comprehensively evaluate model performance: Jaccard Similarity Score: Measures the consistency between predicted and actual medication use; a higher score is better. Represents the prediction result. Represents the true label, This represents the total number of times a patient visits a doctor.

[0060] , Drug Interaction Rate (DDI Rate): Measures the frequency of adverse drug interactions in a recommended combination; a lower DDI rate indicates higher safety. It is an indicator function. It is the prior DDI relation matrix.

[0061] , F1 score: The harmonic mean of precision and recall, which comprehensively reflects the accuracy of the model.

[0062] , Implementation details: All experiments were conducted on an NVIDIA GeForce RTX 4060 (16GB) GPU. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. Model hyperparameters were set as follows: embedding dimension and Bi-GRU hidden layer dimension were both 512, the number of Bi-GRU layers was 1, the cross-attention module contained 8 attention heads, and the internal dropout rate was 0.1. Regarding the loss function, the weights of the BCE loss were... The balance factor for DDI loss is 0.5. The value was set to 0.07. Training was performed using the Adam optimizer with a batch size of 256, a learning rate of 4e-5, and a weight decay of 1e-4, for a total of 50 training epochs. During training, the optimal model was saved using both Jaccard similarity and DDI rate as monitoring metrics.

[0063] Benchmark model comparison experiment: To verify the effectiveness and advantages of the model proposed in this study, this application selected several representative benchmark methods in the field of drug recommendation for comparison. These methods cover a variety of technical approaches, from traditional machine learning and deep sequence modeling to graph neural networks in recent years, and can comprehensively reflect the performance of different modeling paradigms in drug recommendation tasks.

[0064] (1) Classical benchmark model.

[0065] Classic benchmark models mainly include LR (Logistic Regression), ECC (Encoder–Decoder Combination), RETAIN, LEAP, GAMENet, and MICRON.

[0066] Among them, LR and ECC are early statistical or shallow learning methods that mainly rely on the structured features in medical records for linear or nonlinear mapping, thus having limitations in capturing complex dependencies. RETAIN and LEAP introduce temporal and attention mechanisms, which can model the dynamic features of the visit sequence to some extent, but still struggle to capture high-order interactions between drugs. GAMENet was the first to introduce graph structures into drug recommendation, fusing historical information through a memory network, which improved accuracy, but its high DDI (drug interaction rate) indicates that it still poses safety risks. The MICRON model improves short-term predictive ability by aggregating patient time series from multiple perspectives, but its overall performance is still limited by its feature representation capabilities.

[0067] As can be seen from the results in Tables 2 and 3, the Jaccard and F1 indices of these classic methods are generally low (MIMIC-III average Jaccard < 0.51, MIMIC-IV average Jaccard < 0.47), and the DDI mostly remains between 0.07 and 0.09, indicating that they have not yet achieved an ideal balance between safety and accuracy.

[0068] (2) Recent benchmark model.

[0069] Recent methods such as DrugRec (TKDE 2024) and BiMoRec (ESW 2025) have introduced stronger multimodal and graph mining capabilities in their model structures. DrugRec delves into the relationship patterns between drugs and diseases through graph neural networks, achieving a Jaccard score of 0.5352 and an F1 score of 0.6890 on MIMIC-III, while reducing the DDI to 0.0684, showing significant improvements compared to traditional models. BiMoRec further employs a bimodal feature alignment mechanism, jointly modeling drug semantic and structural features, increasing the Jaccard score to 0.5422 and the F1 score to 0.6940, outperforming DrugRec and demonstrating that fusing semantic and structural information has a significant benefit for drug recommendation.

[0070] On the MIMIC-IV dataset, both models maintained stable performance (BiMoRec: Jaccard=0.4917, F1=0.6425), but the overall improvement was limited, indicating that the generalization ability under different data sources still needs to be improved.

[0071] (3) The overall performance of the model in this study.

[0072] The multimodal drug recommendation model proposed in this study, after incorporating semantic enhancement, molecular structure features, and dynamic attention mechanisms, achieved state-of-the-art performance on both datasets. On the MIMIC-III dataset, the model achieved a Jaccard index of 0.5539 and an F1 score of 0.7049, both superior to all baseline models; simultaneously, the DDI was controlled at 0.0676, achieving a balance between accuracy and safety. On the MIMIC-IV dataset, the model achieved Jaccard and F1 scores of 0.5224 and 0.6709, respectively, representing improvements of approximately 3–4 percentage points compared to DrugRec and BiMoRec, demonstrating good robustness across datasets.

[0073] Overall, this study represents a significant breakthrough in multimodal feature interaction and clinical semantic modeling, improving prediction accuracy while effectively reducing the risk of drug interactions. Therefore, the proposed model demonstrates high application potential in drug recommendation tasks, balancing clinical interpretability and safety.

[0074] Table 2 Comparison of metrics with the baseline model on the MIMIC-III dataset

[0075] Table 3. Comparison of metrics with the baseline model on the MIMIC-IV dataset.

[0076] Ablation experiment: To verify the specific contributions of each core module in this invention to drug recommendation performance, a systematic ablation experiment was conducted on the model. The experiment involved progressively removing or replacing key components, including the LLM-based DrugSemantic Enhancement module, the Molecular Expert module, the dual-expert collaborative fusion module, the LLM semantic enhancement module for patient representations, and different feature interaction and label modeling mechanisms, to evaluate the impact of each part on the model's accuracy and safety. The experimental results are shown in Table 4.

[0077] Table 4 Comparison of Ablation Experiments

[0078] (1) Complete model.

[0079] The complete model proposed in this invention integrates LLM-driven patient semantic enhancement, drug semantic and molecular dual-expert collaborative modeling, MLP-Mixer modal interaction, and label attention mechanism, achieving optimal overall performance (Jaccard=0.5539, F1=0.7049, DDI=0.0676). While maintaining high accuracy, the model effectively reduces the occurrence rate of high-risk drug combinations, demonstrating excellent safety and robustness.

[0080] (2) Remove drug semantic LLM enhancement.

[0081] When the drug semantic enhancement module was removed, and only molecular structure and patient semantic features were retained, the model's Jaccard score decreased to 0.5447, F1 score decreased to 0.6962, and DDI increased to 0.0729. This indicates that drug semantic information can significantly improve the clinical interpretability of drug representation and plays a crucial role in the model's overall decision-making.

[0082] (3) Removal of drug molecule structure enhancement.

[0083] After removing the molecular structure expert module, the model's Jaccard score decreased to 0.5453, F1 score decreased to 0.6967, and DDI increased to 0.0727. These results indicate that molecular structure layer features can help the model identify implicit interactions between drugs, which is of great significance for improving the safety recommendations.

[0084] (4) Remove the dual-expert module for drugs.

[0085] Simultaneously removing both semantic and molecular expert pathways significantly reduced model performance (Jaccard=0.5384, F1=0.6908), indicating that the synergistic effect of the dual expert structure is crucial for improving the model's representational ability and multimodal fusion effect.

[0086] (5) Remove patient-associated LLM enhancement.

[0087] After removing the LLM-based patient semantic descriptions, the model's overall performance significantly deteriorated (Jaccard=0.5222, F1=0.6761). This indicates that the disease and surgical semantic context provided by LLM can effectively enrich the patient state representation and improve the accuracy of drug decision-making reasoning.

[0088] (6) Remove all LLM enhancement modules.

[0089] After completely removing all LLM semantic enhancements, the model's performance further declined (Jaccard=0.5001, F1=0.6566), indicating that LLM plays a core semantic support role in the system of this invention, and its removal will seriously weaken the model's ability to perceive and reason about clinical semantics.

[0090] (7) Architecture-level ablation experiments.

[0091] Further replacement experiments were conducted on the internal architecture of the model: Simple splicing replaces cross-attention mechanism: After replacing Bi-Directional Cross-Attention with simple splicing operation, Jaccard=0.5443, F1=0.6959, and DDI increased to 0.0759, indicating that the cross-attention mechanism can effectively capture semantic dependencies between heterogeneous modalities.

[0092] Simple splicing to replace the MLP-Mixer module: After replacing MLP-Mixer with simple splicing, Jaccard=0.5437 and F1=0.6953, indicating that MLP-Mixer plays an important role in nonlinear fusion in deep modal interaction.

[0093] Linear layer replacement of label attention mechanism: When a single-layer linear mapping is used to replace the label attention mechanism, the model's Jaccard score drops to 0.5303 and F1 score drops to 0.6833, indicating that the label attention mechanism can effectively capture the fine-grained association between drugs and patient history, and is a key component for improving recommendation accuracy.

[0094] Experimental results fully validate the necessity and synergistic effect of each module in this invention. By introducing a prospective patient representation construction mechanism, this invention enables the model to align the patient's clinical status with potential medication decisions from the initial stage of representation learning. Simultaneously, the innovative dual-expert drug representation module comprehensively and deeply characterizes drugs from both molecular structure and clinical semantic dimensions, and utilizes advanced architectures such as MLP-Mixer to achieve efficient multimodal information fusion. Crucially, the "static-anchored-dynamic" temporal reasoning mechanism proposed in this invention cleverly combines long-term medication patterns with short-term disease changes, significantly improving the clinical logic and robustness of the model's decisions.

[0095] In summary, this invention proposes a multimodal intelligent modeling method for clinical drug recommendation, integrating Large Language Model (LLM) semantic enhancement, molecular structure expert modeling, and dynamic temporal reasoning mechanisms to achieve deep semantic alignment between patient status and drug knowledge. Through a "static-anchored-dynamic" temporal reasoning framework, the model simultaneously captures long-term medication patterns and short-term disease changes, significantly improving the clinical logic and stability of predictions. On the patient side, the LLM semantic enhancement module integrates the semantic context of diagnosis and surgery, resulting in a more comprehensive patient representation. On the drug side, a dual-expert module jointly characterizes drug features at the semantic and molecular levels, achieving efficient fusion through MLP-Mixer and attention gating mechanisms, balancing accuracy and interpretability. The tag attention mechanism further enhances drug-level matching capabilities, strengthening the accuracy and safety of predictions.

[0096] Experimental results show that the present invention achieves state-of-the-art performance on both the MIMIC-III and MIMIC-IV datasets: on MIMIC-III, Jaccard scores reach 0.5539, F1 scores reach 0.7049, and DDI drops to 0.0676; on MIMIC-IV, Jaccard and F1 scores are 0.5224 and 0.6709, respectively, demonstrating good cross-dataset generalization ability. Ablation experiments further validate the synergistic effect of LLM enhancement and the multimodal fusion module. Overall, the present invention significantly improves both the accuracy and safety of drug recommendations, providing a new technical approach for constructing intelligent medication decision-making systems with clinical semantic understanding and reasoning capabilities, and has high practical value and promising prospects for application.

[0097] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A multi-modal drug recommendation method based on a large language model driver, characterized in that, The method comprises the following steps: Obtaining and preprocessing the electronic health record data of the patient, including the diagnosis code, operation code and history drug code of the patient in each visit, and cleaning, standardizing and coding mapping the data to construct an initial learnable embedding vector for each medical entity; Building a prospective patient representation: using a large language model to generate enhanced semantics for diagnosis codes and operation codes for drug treatment, and performing deep alignment of the enhanced semantics and the structured representation extracted from the electronic health record through a bidirectional fusion mechanism to generate a patient representation; Building a double-expert multi-modal drug representation: setting up a molecular structure expert channel and a semantic expert channel in parallel, enhancing the knowledge of the patient's history drug records, and integrating the chemical structure information and clinical semantic information of the drug into a global drug history representation through a hierarchical deep fusion strategy; Context-aware temporal reasoning: taking the global drug representation as a static context, modulating each time step of the dynamic clinical event sequence composed of patient representations of historical visits, and then inputting the modulated sequence into a time series model to generate a context-aware patient health trajectory representation; Generating drug recommendation results: based on the health trajectory representation, using a label-aware prediction module to dynamically calculate the recommendation probability for each candidate drug, and combining a conditional drug interaction risk control strategy to output the final drug combination.

2. The method of claim 1, wherein the method is based on a large language model. The building of the prospective patient representation specifically comprises: Generating an initial structured representation: for the set of diagnosis codes in each visit, converting the discrete codes into continuous vectors through an embedding layer, and using a mask mean pooling method for aggregation to generate an initial diagnosis representation and operation representation; Setting a collaborative prompt strategy for semantic enhancement: setting instructions for drug decision-making to guide the large language model to generate enhanced semantics for diagnoses and operations; wherein, for diagnosis codes, the large language model is guided to generate semantic descriptions of their definitions, core treatment targets, typical drug categories, drug contraindications and related operations; for operation codes, the large language model is guided to generate semantic descriptions of their indications, preoperative drug adjustments and postoperative commonly used drugs; and all medical entities are bound to unique codes in the standard database; Using bidirectional cross-attention for deep fusion: letting the structured representation and the semantic representation be query, key and value to each other, performing bidirectional information query and refinement, and dynamically controlling the information injection intensity through a gated residual network to generate the fused patient representation.

3. The method of claim 2, wherein the method is characterized by, The bidirectional cross-attention for deep fusion specifically comprises: Projecting the initial structured representation and the enhanced semantic representation into a unified fusion space; Performing two multi-head cross-attention operations, making the two representations query, key and value to each other, and performing bidirectional information refinement; Dynamically controlling the information injection intensity through a gated residual network to prevent one side of information from being overwhelmed by the other side, finally adding the refined representations and performing layer normalization to generate the fused patient representation.

4. The method of claim 1, wherein the method is characterized by, The building of the double-expert multi-modal drug representation specifically comprises: The molecular structure expert path is set: a pre-trained molecular model is used to convert the SMILES string of the drug into a molecular structure embedding, which is fused with the initial embedding of the historical medication through a cross-attention mechanism to generate enhanced molecular features. The semantic expert path is set: a hierarchical prompting strategy based on the anatomical-therapeutic-chemical classification system is used to guide the large language model to generate structured semantic knowledge of the drug in the aspects of anatomy, therapy, and pharmacology, and the enhanced semantic features are generated by fusing the initial medication embedding. A two-stage hierarchical fusion strategy is adopted: the enhanced molecular features and the enhanced semantic features are input into the MLP-Mixer architecture for inter-modal interaction, and then the adaptive weights are calculated through the gated attention fusion module to perform weighted summation to generate the global medication history representation.

5. The method of claim 4, wherein the method is characterized by, The setting of the molecular structure expert path specifically includes: A pre-trained molecular model is used to convert the SMILES string of the drug into a high-quality molecular structure embedding. The molecular structure embedding is fused with the initial embedding of the patient's historical medication record through a bidirectional cross-attention mechanism to generate enhanced molecular features that deeply integrate the patient's medication context.

6. The method of claim 4, wherein the method is characterized by, The setting of the semantic expert path specifically includes: A hierarchical prompting strategy based on the anatomical-therapeutic-chemical classification system is used to guide the large language model to generate detailed descriptions of the drug's action organs, therapeutic subgroups, clinical indications, mechanisms of action, and drug interactions in the dimensions of anatomy, therapy, and pharmacology / chemistry. The generated hierarchical semantic embedding is fused with the initial medication embedding through cross-attention to generate enhanced semantic features.

7. The method of claim 4, wherein the method is characterized by, The two-stage hierarchical fusion strategy specifically includes: The enhanced molecular features and the enhanced semantic features are input into the MLP-Mixer architecture to explicitly model the deep nonlinear mapping relationship between the two modalities through token mixing and channel mixing MLP blocks. The interacted modal views are input into the gated attention fusion module, which calculates adaptive weights for each modality through a small attention network and performs weighted summation to generate a unified global medication history representation.

8. The method of claim 1, wherein the method is based on a large language model. The context-aware temporal reasoning specifically includes: Static context broadcasting: the global medication history representation is replicated and expanded along the time dimension to form a static context matrix with the same length as the consultation sequence; Forming a comprehensive input vector: at each time step, the dynamic patient representation at that time step is concatenated with the corresponding vector of the static context matrix to form a comprehensive input vector; Temporal network encoding: the sequence of comprehensive input vectors is input into a bidirectional gated recurrent unit network to encode and generate a context-aware patient health trajectory representation.

9. The method of claim 1, wherein the method is based on a large language model. The generation of drug recommendation results specifically includes: Using label-level attention mechanism as prediction head: the embedding matrix of the drug vocabulary is used as the query matrix, and the health trajectory representation sequence is used as the key and value matrix to calculate the attention score for each candidate drug, generate a dedicated context vector, and calculate the recommendation probability based on it; The model is trained with a composite optimization objective: the task loss uses DiceBCELoss, which is a weighted combination of binary cross-entropy loss and Dice loss; the drug interaction safety loss is a conditional regularization term that is activated only when the predicted drug interaction rate exceeds the clinical threshold, and is obtained by calculating the co-occurrence probability of the predicted drug pair and multiplying it by the prior risk value.

10. A multi-modal drug recommendation system based on a large language model driving, characterized in that, The application of a multi-modal drug recommendation method based on a large language model, comprising: A preprocessing module: obtaining and preprocessing the electronic health record data of the patient, including the diagnosis code, operation code and historical medication code of the patient's previous visit, and cleaning, standardizing and coding mapping the data, constructing an initial learnable embedding vector for each medical entity; A prospective patient representation module: using a large language model to generate enhanced semantics for diagnosis codes and operation codes for drug treatment, and aligning the enhanced semantics with the structured representation extracted from the electronic health record through a bidirectional fusion mechanism to generate patient representation; A double-expert multi-modal drug representation module: parallelly setting a molecular structure expert channel and a semantic expert channel, enhancing the knowledge of the patient's historical medication records, and integrating the chemical structure information and clinical semantic information of the drugs into a global medication history representation through a hierarchical deep fusion strategy; A context-aware temporal reasoning module: taking the global medication representation as a static context, modulating each time step of the dynamic clinical event sequence composed of the patient's historical visit representation, and then inputting the modulated sequence into a temporal model to generate a context-aware patient health trajectory representation; A drug recommendation result generation module: based on the health trajectory representation, using a label-aware prediction module to dynamically calculate the recommendation probability for each candidate drug, and combining a conditional drug interaction risk control strategy to output the final drug combination.

Images