Combined drug recommendation method based on multi-modal alignment

Abstract

The invention provides a combined medicine recommendation method based on multi-modal alignment. A core module of the recommendation method comprises a cloth perception multi-modal medicine alignment module, a time sequence multi-view patient aggregation module and a combined medicine recommendation module. According to the recommendation method, electronic health records EHRs serve as a data source, multi-modal information matched with physical signs of a patient and medicine application characteristics is obtained, effective alignment and fusion are conducted on the multi-modal information through the core module, and suggestion data matched with the multi-modal information and composed of multiple medicines are generated. A DDI loss function and a dynamic weighting strategy are introduced to process conflicts among different drugs in the generated suggestion data so as to output safe suggestion data, and it is avoided that conflicting drugs are recommended in the suggestion data; according to the method, personalized, high-safety, scientific and reasonable drug combination suggestions can be provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medicine, and particularly to a combined drug recommendation method based on multi-modal alignment. BACKGROUND

[0002] Electronic health records (EHRs) as the core data source of this task, gather rich multi-modal information, including text (such as clinical notes, drug instructions), structured data (drug molecular structure), image data (such as X-ray), and signal data (such as sensor monitoring records) and so on. These multi-modal information together constitute a panoramic view of patient conditions and drug characteristics, providing important support for accurate drug use.

[0003] However, when using multi-modal data, existing methods often extract features from each modality independently, lacking explicit alignment and fusion of information from different data sources. This approach can usually only capture common features between modalities, while ignoring the complementary information unique to each modality, resulting in suboptimal drug combinations generated.

[0004] As shown in Figure 1 (a), when modality alignment is not considered, the model relies only on the common "pain relief" attribute of ibuprofen and acetaminophen when analyzing them, such as "relieving pain" in the text description and similar molecular substructures. For patients with both headaches and gastric ulcers, this model ignores the key negative information in the ibuprofen text description that "may cause gastrointestinal ulcers", thus incorrectly recommending ibuprofen, increasing the patient's risk of medication.

[0005] In contrast, as shown in Figure 1 (b), by introducing multi-modal alignment technology, the model can map heterogeneous modalities such as text descriptions and molecular structures to a shared latent space. This process not only captures the "pain relief" commonality between drugs, but also fully utilizes the complementary features of each modality. Therefore, the aligned model can identify and weigh the side effects of ibuprofen, and recommend acetaminophen, which has less gastrointestinal irritation, to patients. At the same time, for patients with allergic arthritis, the model can accurately use the "anti-inflammatory" properties that ibuprofen has and acetaminophen lacks, and recommend ibuprofen.

[0006] This shows that explicit alignment of multi-modal information helps to reveal key associations and complementary relationships between modalities, thus generating drug combinations that take into account both efficacy and safety. SUMMARY

[0007] The application provides a combined drug recommendation method based on multi-modal alignment, aiming at the problem that modal complementary information is easily ignored in existing combined drug recommendation based on multi-modal electronic health records. Figure 2 The application realizes effective alignment and fusion of multi-modal information, thereby providing personalized, safe and scientific combined drug recommendation.

[0008] The application adopts the following technical solutions.

[0009] A combined drug recommendation method based on multi-modal alignment, the core modules of the recommendation method include distribution-aware multi-modal drug alignment module, time-series multi-view patient aggregation module and combined drug recommendation module. Figure 2 As shown in the figure, the recommendation method takes electronic health records (EHRs) as data source, obtains multi-modal information matched with patient signs and drug application characteristics, and effectively aligns and fuses the multi-modal information through the core modules to generate suggestion data composed of multiple drugs matched with the multi-modal information. A dynamic weighting strategy is introduced to handle the conflicts between different drugs in the generated suggestion data, so as to output safe suggestion data and avoid recommending conflicting drugs in the suggestion data.

[0010] As shown in the figure, the recommendation method includes the following steps. Figure 2

[0011] Step one, distribution-aware multi-modal drug alignment.

[0012] Step two, time-series multi-view patient aggregation.

[0013] Step three, generating recommendation data of combined drug use and preventing recommending conflicting drugs in the suggestion data.

[0014] Step one includes the following steps.

[0015] Step 1.1, multi-modal drug embedding, based on multi-modal electronic health records, using the text modal description, identifier ID modal and molecular structure modal of each drug, obtaining the corresponding drug embedding to capture effective drug features and enrich drug embedding.

[0016] Step 1.2, distribution-aware modal alignment, designing a multi-modal transmission mechanism for drugs according to the requirements of distribution alignment, wherein optimal transport (OT) is used to minimize the transmission cost between one modal m and another modal m'; the text embedding X T and molecular structure embedding X S of the drug are aligned to the drug ID embedding X​I shared uniform and comparable latent space to ensure consistency of drug embeddings.

[0017] Step 1.1 is specifically,

[0018] Text modality: BioBERT model pre-trained on large-scale biomedical corpus is selected to utilize its rich domain knowledge and excellent semantic understanding ability to obtain drug text description from DrugBank dataset Efficient encoding for high-quality vector representation:

[0019]

[0020] where is a multi-layer perceptron, matrix Each row of the matrix represents the text embedding of a drug.

[0021] ID modality: a learnable embedding vector is assigned to each drug to capture unique information:

[0022] Molecular structure modality: RDKit tool is used to extract atoms and bonds from SMILES string to construct molecular graph; then graph neural network GNN is used to model the molecular structure graph all the interactions between atoms in the molecule.

[0023] Take ibuprofen as an example, its molecular structure SMILES string is

[0024] CC(C)CC1=CC=C(C=C1)C(C)C(O)=O;

[0025] Through the RDKit tool, the string can be converted into a molecular graph, and the graph neural network (GNN) is used to model the molecular graph to learn the atomic connection relationship and functional group characteristics, thereby obtaining the molecular structure modality representation.

[0026] Each atom is connected by a chemical bond with an initial embedding GNN uses a message passing and aggregation mechanism to capture the complex high-order relationship between atoms, thereby updating the embedding of atom a i :

[0027] where represents the embedding of atom a i at the l-th layer of GNN, represents the embedding of atom a iThe set of neighboring atoms; the MSG function receives information about each neighboring atom, the AGG function aggregates the embeddings of neighboring atoms, and the UPD function updates a based on the aggregated neighboring atom embeddings. i Embedding;

[0028] Inspired by its effectiveness in obtaining long-range interatomic cooperative effects, a graph transformer is used as the GNN in the actual implementation of this invention;

[0029] Then, the molecular graph is processed using the mean pooling function. Atom embedding in These elements are aggregated to form a global embedded structure:

[0030]

[0031] Where L represents the total number of GNN layers. For all Drug molecules are used with the same GNN having shared parameters, and their corresponding molecular embeddings are stored as follows:

[0032]

[0033] Step 1.2 specifically involves: First, calculating the transportation cost using the cosine distance.

[0034]

[0035] in and These are modal X m and X m′ The row vectors; alignment is expressed by the following formula:

[0036]

[0037] in It is the optimal transmission scheme. From arrive The amount of information; and They are respectively the dimensions of and The vector consists of all 1s; and the Sinkhorn algorithm is used to accelerate the calculation of the optimal solution. Then the embedding of mode m is updated; the distribution between modes m and m′ remains consistent:

[0038]

[0039] Step two includes the following steps;

[0040] Step 2.1, Multi-perspective medical record embedding; In order to fully understand the patient's health status, each patient's medical record is coded from multiple perspectives using diagnostic, surgical, and historical medication information from the multimodal electronic health record;

[0041] Step 2.2, Patient temporal embedding, is used to capture the patient's previous diagnosis, treatment process, and medication information. In Step 2.1, two learnable embedding tables are first initialized. and For the multi-hop diagnosis and surgical vector d at the t-th visit t p t Select the corresponding diagnostic and surgical embeddings, and add them together using vector-matrix multiplication to generate the medical diagnosis and surgical embeddings respectively:

[0042] To generate visit embeddings from historical medication views, a cross-attention mechanism is used to fuse data from three modalities. and The characteristics of drug use. The embedded part after polymerization is denoted as:

[0043]

[0044] in Learnable weights The calculation method is as follows:

[0045] Q = H q W q K = H k W k ,

[0046]

[0047] in It is by A matrix formed by stacking the embeddings in the matrix. W q and All weights are learnable; the fused patient visit embedding is as follows:

[0048]

[0049] Used to effectively enhance and capture correlation and complementary information between different modalities;

[0050] Similar to the patient visit embeddings from diagnostic and surgical perspectives, the patient visit embeddings from a historical medication perspective are calculated as follows:

[0051]

[0052] Where mt-1 It is the multi-hop vector of the medications the patient took at the (t-1)th visit, and the multi-view embedding at the tth visit is defined as:

[0053]

[0054] In step 2.2, three independent gated recurrent units (GRUs) are used to capture the patient's previous diagnosis, treatment process, and medication information, expressed as a formula:

[0055]

[0056] The embedding vectors obtained from the three formulas are concatenated to form the final patient embedding vector, expressed as follows:

[0057] Step three, the combination therapy recommendation, includes the following sub-steps: based on the final patient-embedded P T The recommended drug combination is predicted for the patient at their Tth visit, expressed by the formula:

[0058]

[0059] in It is a multilayer perceptron; it is trained using two different loss functions to strike a balance between accuracy and safety in CMR; in the sub-step, the formulas for the two loss functions are as follows:

[0060] Step 3.1 Multi-label prediction loss; Treating the combination drug recommendation task as a multi-label classification task, two commonly used loss functions are employed to enhance the robustness of the results: binary cross-entropy loss and multi-label boundary loss, with the following formulas:

[0061]

[0062] in This is the patient's actual medication usage during their t-th visit;

[0063] The hyperparameter λ is experimentally set to 0.95 by default; it is worth noting that the multi-label residual loss... The margin of the true label should be at least 1 larger than that of other labels to achieve more stable predictions.

[0064] Step 3.2, Drug Interaction DDI Loss; To reduce the predicted incidence of drug interactions in drug combinations, the following loss is minimized:

[0065]

[0066] Where matrix A ijThis indicates that a DDI knowledge base is pre-built, and if there is a drug conflict between drug i and drug j, then A... ij =1, if there is no drug conflict, then A ij =0, This represents the probability of paired drug interactions;

[0067] During the model training process of the DDI knowledge base, The loss function calculates the interaction risk of all drug pairs in the currently recommended drug combination; if the model recommends drugs with conflicting interactions, The loss value will become larger; a huge loss value will serve as a penalty signal, forcing the model to adjust its internal parameters in subsequent optimization processes, reducing the probability of recommending both conflicting drugs at the same time.

[0068] During the training of the DDI knowledge base, accuracy and the incidence of drug interactions typically increase together. This is because drug interactions are common in real-world electronic health record data, and both correct and incorrect predictions of drug combinations can increase the incidence of drug interactions. Therefore, balancing the model's accuracy and safety is crucial for effective combination therapy recommendations.

[0069] In step three, a dynamic weighting strategy is introduced to form the final objective function, expressed by the following formula:

[0070]

[0071] in ∈ represents the preset security DDI threshold.

[0072] This invention proposes a combination drug recommendation method based on multimodal alignment, aiming to address the problem of overlooking modal complementarity information in existing combination drug recommendations based on multimodal electronic health records. The method comprises three core modules: a distributed sensing multimodal drug alignment module, a temporal multi-perspective patient aggregation module, and a combination drug recommendation module. For example... Figure 2 As shown, the present invention achieves effective alignment and fusion of multimodal information through the collaborative work of the above technical solutions, thereby providing personalized, safe and scientifically reasonable combination drug recommendations.

[0073] The advantages of this invention are as follows:

[0074] (1) A distribution-aware multimodal drug alignment module for combination drug recommendation is proposed. To our knowledge, this is the first time that the correlation and complementarity information between modalities has been explicitly captured from the perspective of distribution alignment, thereby achieving accurate and safe combination drug recommendation.

[0075] (2) By combining rich time series of visits with multi-view information from diagnosis, surgery and fusion of historical medications, a comprehensive temporal patient embedding is obtained for drug combination prediction. Attached Figure Description

[0076] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0077] Appendix Figure 1 This is a comparative diagram of drug recommendation methods that do not consider inter-modal alignment and those that align different modalities (in the diagram, (a) the combined drug recommendation is based on extracting features independently for each modality and does not consider inter-modal alignment; in the diagram, (b) aligning different modalities (such as text description and molecular structure) improves the consistency and complementarity between drug modalities).

[0078] Appendix Figure 2 This is a schematic diagram of the overall framework of the present invention (the colored × symbol indicates the removal of drug-drug interaction (DDI) pairs);

[0079] Appendix Figure 3 This is a visualization of the distribution of text and molecular drug embeddings in the MIMIC-III dataset, learned by different models. Detailed Implementation

[0080] As shown in the figure, a combination drug recommendation method based on multimodal alignment is presented. The core modules of the recommendation method include a perceptual multimodal drug alignment module, a temporal multi-view patient aggregation module, and a combination drug recommendation module. Figure 2 As shown, the recommendation method uses electronic health records (EHRs) as the data source to obtain multimodal information that matches patient vital signs and drug suitability characteristics. A core module effectively aligns and fuses this multimodal information to generate recommended data that matches the multimodal information and consists of multiple drugs. A DDI loss function is also introduced. A dynamic weighting strategy is used to handle conflicts between different drugs in the generated recommendation data, so as to output safe recommendation data and avoid recommending conflicting drugs in the recommendation data.

[0081] like Figure 2 As shown, the recommendation method includes the following steps;

[0082] Step 1: Distribution-sensing multimodal drug alignment;

[0083] Step 2: Temporal multi-perspective patient aggregation;

[0084] Step 3: Generate recommended data for combination therapy and prevent conflicting drugs from being recommended in the suggested data.

[0085] Step one includes the following steps;

[0086] Step 1.1: Multimodal drug embedding. Based on multimodal electronic health records, the corresponding drug embedding is obtained by utilizing the text modal description, ID modal of the identifier, and molecular structure modal of each drug to capture effective drug features and enrich the drug embedding.

[0087] Step 1.2: Distribution-aware modal alignment. Design a multimodal drug delivery mechanism according to the distribution alignment requirements, where optimal delivery (OT) is used to minimize the transmission cost between one modality m and another modality m′; embed the drug text into X. T and molecular structure embedding X S Align to embed X with drug ID I In a shared, unified, and comparable potential space, to ensure consistency in drug embedding.

[0088] Step 1.1 specifically involves:

[0089] Text Modality: The BioBERT model, pre-trained on a large-scale biomedical corpus, was selected to leverage its rich domain knowledge and excellent semantic understanding capabilities to analyze drug text descriptions obtained from the DrugBank dataset. Efficient encoding into high-quality vector representations:

[0090]

[0091] in It is a multilayer perceptron, matrix Each line represents a text embedding of a drug;

[0092] ID modality: Assign a learnable embedding vector to each drug to capture unique information:

[0093]

[0094] Molecular structure modality: Molecular graphs were constructed by extracting atoms and bonds from the SMILES string using the RDKit tool; then, graph neural networks (GNNs) were used to model the molecular structure graphs. The interactions between all atoms in the middle;

[0095] Taking ibuprofen as an example, its molecular structure SMILES string is:

[0096] CC(C)CC1=CC=C(C=C1)C(C)C(O)=O;

[0097] The RDKit tool can be used to convert the string into a molecular graph, and then a graph neural network (GNN) can be used to model the molecular graph to learn the atomic connection relationships and functional group features, thereby obtaining a molecular structure modal representation.

[0098] Each atom Through chemical bonds Connected, with initial embedding GNN uses message passing and aggregation mechanisms to capture complex high-order relationships between atoms, thereby updating atom a. i Embedding:

[0099] in Represents atom a i Embedding in the l-th layer of GNN Represents atom a i The set of neighboring atoms; the MSG function receives information about each neighboring atom, the AGG function aggregates the embeddings of neighboring atoms, and the UPD function updates a based on the aggregated neighboring atom embeddings. i Embedding;

[0100] Inspired by its effectiveness in obtaining long-range interatomic cooperative effects, a graph transformer is used as the GNN in the actual implementation of this invention;

[0101] Then, the molecular graph is processed using the mean pooling function. Atom embedding in These elements are aggregated to form a global embedded structure:

[0102]

[0103] Where L represents the total number of GNN layers. For all Drug molecules are used with the same GNN having shared parameters, and their corresponding molecular embeddings are stored as follows:

[0104]

[0105] Step 1.2 specifically involves: First, calculating the transportation cost using the cosine distance.

[0106]

[0107] in and These are modal X m and X m′ The row vectors; alignment is expressed by the following formula:

[0108]

[0109] in It is the optimal transmission scheme. From arrive The amount of information; and They are respectively the dimensions of and The vector consists of all 1s; and the Sinkhorn algorithm is used to accelerate the calculation of the optimal solution. Then the embedding of mode m is updated; the distribution between modes m and m′ remains consistent:

[0110]

[0111] Step two includes the following steps;

[0112] Step 2.1, Multi-perspective medical record embedding; In order to fully understand the patient's health status, each patient's medical record is coded from multiple perspectives using diagnostic, surgical, and historical medication information from the multimodal electronic health record;

[0113] Step 2.2, Patient temporal embedding, is used to capture the patient's previous diagnosis, treatment process, and medication information. In Step 2.1, two learnable embedding tables are first initialized. and For the multi-hop diagnosis and surgical vector d at the t-th visit t p t Select the corresponding diagnostic and surgical embeddings, and add them together using vector-matrix multiplication to generate the medical diagnosis and surgical embeddings respectively:

[0114]

[0115] To generate visit embeddings from historical medication views, a cross-attention mechanism is used to fuse data from three modalities. and The characteristics of drug use. The embedded part after polymerization is denoted as:

[0116]

[0117] in Learnable weights The calculation method is as follows:

[0118] Q = H q W q K = H k W k ,

[0119]

[0120] in It is by A matrix formed by stacking the embeddings in the matrix. W q and All weights are learnable; the fused patient visit embedding is as follows:

[0121]

[0122] Used to effectively enhance and capture correlation and complementary information between different modalities;

[0123] Similar to the patient visit embeddings from diagnostic and surgical perspectives, the patient visit embeddings from a historical medication perspective are calculated as follows:

[0124]

[0125] Where m t-1 It is the multi-hop vector of the medications the patient took at the (t-1)th visit, and the multi-view embedding at the tth visit is defined as:

[0126]

[0127] In step 2.2, three independent gated recurrent units (GRUs) are used to capture the patient's previous diagnosis, treatment process, and medication information, expressed as a formula:

[0128]

[0129] The embedding vectors obtained from the three formulas are concatenated to form the final patient embedding vector, expressed as follows:

[0130] Step three, the combination therapy recommendation, includes the following sub-steps: based on the final patient-embedded P T The recommended drug combination is predicted for the patient at their Tth visit, expressed by the formula:

[0131]

[0132] in It is a multilayer perceptron; it is trained using two different loss functions to strike a balance between accuracy and safety in CMR; in the sub-step, the formulas for the two loss functions are as follows:

[0133] Step 3.1 Multi-label prediction loss; Treating the combination drug recommendation task as a multi-label classification task, two commonly used loss functions are employed to enhance the robustness of the results: binary cross-entropy loss and multi-label boundary loss, with the following formulas:

[0134]

[0135] in This is the patient's actual medication usage during their t-th visit;

[0136] The hyperparameter λ is experimentally set to 0.95 by default; it is worth noting that the multi-label residual loss... The margin of the true label should be at least 1 larger than that of other labels to achieve more stable predictions.

[0137] Step 3.2, Drug Interaction DDI Loss; To reduce the predicted incidence of drug interactions in drug combinations, the following loss is minimized:

[0138]

[0139] Where matrix A ij This indicates that a DDI knowledge base is pre-built, and if there is a drug conflict between drug i and drug j, then A... ij =1, if there is no drug conflict, then A ij =0, This represents the probability of paired drug interactions;

[0140] During the model training process of the DDI knowledge base, The loss function calculates the interaction risk of all drug pairs in the currently recommended drug combination; if the model recommends drugs with conflicting interactions, The loss value will become larger; a huge loss value will serve as a penalty signal, forcing the model to adjust its internal parameters in subsequent optimization processes, reducing the probability of recommending both conflicting drugs at the same time.

[0141] During the training of the DDI knowledge base, accuracy and the incidence of drug interactions typically increase together. This is because drug interactions are common in real-world electronic health record data, and both correct and incorrect predictions of drug combinations can increase the incidence of drug interactions. Therefore, balancing the model's accuracy and safety is crucial for effective combination therapy recommendations.

[0142] In step three, a dynamic weighting strategy is introduced to form the final objective function, expressed by the following formula:

[0143]

[0144] in ∈ represents the preset security DDI threshold.

[0145] In this example, the RDKit tool is used to extract atoms and bonds from the SMILES string to construct a molecular diagram, and then a graph neural network (GNN) is used to model the interactions between all atoms in the molecular structure diagram.

[0146] Taking ibuprofen as an example, its molecular structure SMILES string is CC(C)CC1=CC=C(C=C1)C(C)C(O)=O. Using the RDKit tool, this string can be converted into a molecular graph (as shown in the right figure). Then, a graph neural network (GNN) is used to model the molecular graph, learning the atomic connection relationships and functional group features, thereby obtaining a molecular structure modal representation.

[0147] Molecular structure modalities embody the most fundamental pharmacological information of a drug, providing complementary information to the textual modalities in the model. In this example, the scheme is as follows... Figure 1 As shown, ibuprofen and acetaminophen both share the commonality of "pain relief" in their text descriptions, but their molecular structures are completely different. This difference determines that ibuprofen possesses "anti-inflammatory" properties that acetaminophen lacks, but it may also cause side effects such as "gastrointestinal ulcers." By learning molecular structure diagrams, GNNs can capture these key structural features (such as functional groups) that determine the unique pharmacological effects of drugs, thereby distinguishing drugs with similar text descriptions but significantly different actual pharmacological effects, improving the accuracy of drug recommendations and reducing the risk of potential drug interactions (DDI).

[0148] The scheme described in this example introduces the DDI loss function. Addressing drug interactions is illustrated in steps 3.2 and 3.3 of this protocol:

[0149] A DDI knowledge base is pre-built, represented as a matrix A. ij If there is a drug interaction between drug i and drug j, then A ij =1, if there is no drug conflict, then A ij =0.

[0150] During model training, The loss function calculates the interaction risk of all drug pairs in the currently recommended drug combination. If the model recommends drugs with conflicting effects, The loss value will become very large.

[0151] This huge loss value will serve as a strong penalty signal, forcing the model to adjust its internal parameters in subsequent optimization processes to reduce the probability of recommending both conflicting drugs.

[0152] The solution also introduces a dynamic weighting strategy, which increases the weighting when the predicted DDI occurrence rate exceeds a preset safety threshold. The weight of the loss in the total loss makes the model more focused on "safety" and prioritizes avoiding recommending conflicting drugs.

[0153] Example:

[0154] In this example, to evaluate the performance of the method described in this example on the combination drug recommendation task, the performance was validated on the MIMIC-III and MIMIC-IV datasets. The study included patients with at least two medical visits. The MIMIC-III dataset contained 6350 patients, 15031 medical visits, 1903 diagnoses, 1409 surgeries, and 131 drugs. The MIMIC-IV dataset contained 61264 patients, 163877 medical visits, 2000 diagnoses, 11056 surgeries, and 131 drugs. The training, validation, and test sets were divided into 2 / 3, 1 / 6, and 1 / 6 ratios, respectively, for the same experiments.

[0155] This example uses similarity score (Jaccard), average F1 score (F1), area under the precision-recall curve (PRAUC), drug interaction rate (DDI), and average number of medications used (#Med.) to evaluate the model’s fit with real-world prescriptions.

[0156] This example compares 12 state-of-the-art baselines from three main perspectives: (1) instance-based methods: LR, ECC, LEAP; (2) time-series research methods: RETAIN, MICRON, GAMENet, COGNet, LAMERec; (3) modality-aware methods: SafeDrug, DrugRec, MoleRec, DEPOT.

[0157] To demonstrate the effectiveness of this invention in the combination drug recommendation task, Table 1 shows the Jaccard, F1 score, PRAUC, drug interaction (DDI), and average number of drugs on the MIMIC-III and MIMIC-IV datasets. The true average number of treatments in the test sets of the two datasets are 19.7937 and 11.9788, respectively. Overall, this invention outperforms all baselines on both datasets across all recommendation evaluation metrics (i.e., Jaccard, F1 score, and PRAUC). On the MIMIC-IV dataset, this invention achieves a 2.02% improvement in Jaccard performance compared to the second-best performance (e.g., DEPOT). Instance-based methods are traditional approaches that focus on binary classification for each drug label but are often limited by their inability to capture the complex relationships between patient and drug data. This example significantly outperforms these methods across all metrics, with an average improvement of 11.81%. Temporal methods incorporate continuous patient records but still face challenges in modeling interactions across multiple modalities. This example significantly outperforms these models by better capturing temporal and cross-modal correlations, showing significant improvements on both the Jaccard metric (10.54% improvement on the MIMIC-III dataset) and the F1 metric (6.93% improvement on the MIMIC-III dataset). Among modality-aware methods, DEPOT excels in modeling the molecular structure of drugs, outperforming most baseline methods and highlighting the importance of drug feature modeling.

[0158] Compared to DEPOT, this example captures cross-modal correlations and modal-specific complementary information, further improving recommendation accuracy. Regarding the safety assessment metric DDI, SafeDrug leads with its controllable DDI loss function. However, this invention significantly outperforms SafeDrug across all accuracy metrics (up to a 12.44% improvement on the Jaccard metric on the MIMIC-III dataset), while its average DDI rate is only 0.0016 higher than SafeDrug. DDI is common in real-world electronic health record data. Pursuing the lowest possible DDI rate without considering efficacy can lead to poor drug combination outcomes. Therefore, this performance trade-off reflects the practical balance between predictive accuracy and safety, with the small increase in DDI offset by a significant improvement in recommendation effectiveness.

[0159] To better understand the contributions of modality alignment and modality fusion components, ablation studies were conducted, specifically as follows: removing modality alignment refers to performing modality fusion directly without an alignment strategy; removing modality fusion means directly averaging the drug embeddings after the alignment strategy. As shown in Table 2, compared to removing modality alignment, this invention achieves performance improvements, ranging from 3.44% of PRAUC on the MIMIC-IV dataset to 6.62% of F1 on the MIMIC-IV dataset. These results demonstrate that optimal transfer alignment of multimodal drug embeddings is effective from a distributional perspective. Furthermore, the performance improvement in this example compared to removing modality fusion ranges from 0.70% of PARUC on the MIMIC-IV dataset to 2.99% of DDI on the MIMIC-III dataset.

[0160] To highlight the advantages of this example in multimodal drug integration and recommendation, three example drugs from MIMIC-III are provided, with detailed multimodal information given in Table 3. The distribution of textual and molecular structure modal embeddings is illustrated by removing the modality alignment and integrity methods of this invention. Figure 3 As shown in (a), the text embeddings of drugs A, B, and C highly overlap, making them difficult to distinguish using only text features. This stems from their similarity in explanatory text, indicating that the modality alignment method of this invention has limited alignment capabilities across different modalities, and that relying solely on cross-attention mechanisms is insufficient for effective information fusion. In contrast, Figure 3 (b) demonstrates the effectiveness of the complete method in capturing cross-modal correlations and modality-specific complementary information. For example, textual and molecular features successfully distinguish the different effects of drug A (propofol, used to maintain general anesthesia) and drug C (lorazepam, used to treat panic disorder). Furthermore, the method accurately captures inter-drug relationships, such as the closer embedding distance between drugs B and C in both modalities, consistent with their similar pharmacological effects (e.g., controlling multiple types of epileptic seizures).

[0161] This paper proposes a novel framework for combination drug recommendation using multimodal electronic health records. The framework employs a distribution-aware multimodal drug alignment strategy based on optimal transmission, effectively capturing the correlation and complementarity information between different modalities and learning consistent and accurate drug embeddings. Simultaneously, by aggregating multi-view visit sequences of diagnoses, surgeries, and historical medications, temporal patient embeddings are constructed for drug combination prediction. Experimental results demonstrate that this method achieves significant improvements over existing techniques.

[0162] Table 1: Experimental results on the MIMIC-III and MIMIC-IV datasets. (Best performance is indicated in bold, and second-best performance is indicated by underline.)

[0163]

[0164]

[0165] Table 2: Ablation studies on the MIMIC-III and MIMIC-IV datasets, where "#Med." indicates the average number of drugs.

[0166]

[0167]

[0168] Table 3: Multimodal information for three example drugs on the MIMIC-III dataset. The textual descriptions provide clinical uses, while the molecular structures reflect pharmacological properties, highlighting the complementary information between the multimodal data.

[0169]

Claims

1. A multi-modal alignment based combined drug recommendation method, characterized in that: The core module of the recommendation method comprises a cloth-aware multi-modal drug alignment module, a time-series multi-view patient aggregation module and a combined drug recommendation module; the recommendation method takes electronic health records (EHRs) as a data source, acquires multi-modal information matched with patient signs and drug application characteristics, effectively aligns and fuses the multi-modal information through the core module, generates suggestion data matched with the multi-modal information and composed of multiple drugs, and introduces a DDI loss function and a dynamic weighting strategy to handle conflicts between different drugs in the generated suggestion data, to output safe suggestion data and avoid recommending conflicting drugs in the suggestion data.

2. The combined drug recommendation method based on multi-modal alignment according to claim 1, characterized in that: The recommendation method Comprises the following steps; Step one, distribution perception multi-modal drug alignment; Step two, time sequence multi-view patient aggregation; Step three, generate recommended data of combination drug, and prevent recommending conflicting drugs in the recommended data.

3. The method of claim 2, wherein: Step one comprises the following steps; Step 1.1, multi-modal drug embedding, based on multi-modal electronic health records, using the text modal description, identifier ID modal and molecular structure modal of each drug, obtaining the corresponding drug embedding to capture effective drug features and enrich drug embedding; Step 1.2: Distribution-aware modal alignment. Design a multimodal drug delivery mechanism according to the distribution alignment requirements, where optimal delivery (OT) is used to minimize the transmission cost between one modality m and another modality m'; embed the drug text into X. T and molecular structure embedding X S Align to embed X with drug ID I In a shared, unified, and comparable potential space, to ensure consistency in drug embedding.

4. The method of claim 3, wherein: Step 1.1 specifically, Text modality: Pre-trained BioBERT model is chosen to encode the drug text description obtained from DrugBank dataset Efficient encoding into high-quality vector representations: wherein is a multi-layer perceptron, matrix each row of the matrix represents a textual embedding of a drug; ID modal: assign a learnable embedding vector to each drug to capture unique information: Molecular structure modality: atoms and bonds are extracted from SMILES strings using RDKit tools to construct a molecular graph; then a graph neural network GNN is used to model the molecular structure graph interactions among all atoms in the molecule Through the RDKit tool, the SMILES string is converted into a molecular graph, and then the graph neural network GNN is used to model the molecular graph to learn the atomic connection relationship and functional group features, so as to obtain the molecular structure modal representation; Each atom By chemical bonds Connected, with initial embeddings GNNs employ a message passing and aggregation mechanism to capture complex high-order relationships among atoms, thereby updating the embedding of an atom a i : wherein represents an atom a i at the GNN layer 1, represents an atom a i a set of neighbor atoms of The MSG function receives information of each neighbor atom, the AGG function is used to aggregate the embeddings of neighbor atoms, and the UPD function updates the embedding of a based on the aggregated embeddings of neighbor atoms i ; Adopting graph transformer as GNN; The molecular graph is then aggregated by a mean-pooling function on the atom embeddings to form a global structure embedding:​ where L denotes the total number of GNN layers. For all molecules of a drug, the same GNN with shared parameters is used and their corresponding molecular embeddings are stored as:

5. The method of claim 3, wherein: Step 1.2 specifically: first, calculate the transportation cost by cosine distance: where and are the row vectors of the modalities X m and X m' respectively; the alignment is formulated as follows: where is the optimal transport plan, is the amount of information from to ; and are all-ones vectors of dimensions and respectively; and the Sinkhorn algorithm is employed to speed up the computation of the optimal plan. The embedding of modality m is then updated; the distribution between modalities m and m' remains consistent:

6. The method of claim 2, wherein: Step two comprises the following steps; Step 2.1, multi-view clinic embedding; In order to fully understand the health status of the patient, the diagnosis, surgery and historical drug information of each patient's clinic record is respectively encoded by using the multi-modal electronic health record; Step 2.2, patient time sequence embedding, used to capture the patient's past diagnosis, treatment process and drug information.

7. The method of claim 6, wherein: In step 2.1, two learnable embedding tables are first initialized, and For the multi-hop diagnosis and surgery vector d at the tth visit t , p t , the corresponding diagnosis and surgery embeddings are selected and added by vector matrix multiplication to generate the visit diagnosis and surgery embeddings, respectively, which can be expressed in formulas as: To generate the consultation embedding from the historical medication view, a cross-attention mechanism is utilized to fuse the medication features from the three modalities and ​ The fused and aggregated embedding is denoted as: wherein learnable weights The calculation is as follows: Q = H q W q K = H k W k , wherein is a matrix stacked by embeddings in , W q and are learnable weights; the fused visit embedding is as follows: For effectively enhancing and capturing the correlation and complementary information between different modalities; The calculation method of clinic embedding under the perspective of historical drug use is as follows: where m t-1 is the multi-hop vector of the medication taken by the patient at the visit t-1, and the multi-view visit embedding at visit t is defined as:

8. The method of claim 6, wherein: In step 2.2, three gated recurrent units GRU are used to capture the patient's past diagnosis, treatment process and drug information, which is expressed in the formula as: The embedding vectors obtained from the three formulas are concatenated to form the final patient embedding vector, expressed in a formula as:

9. The method of claim 2, wherein: The combination medication recommendation in Step Three includes the following sub-steps: predicting the drug recommendation combination for the patient at the Tth visit according to the final patient embedding P T , which is expressed in a formula as: wherein is a multi-layer perceptron; trained using two different loss functions to balance accuracy and safety of the CMR; in sub-steps, the two loss functions are formulated as: Step 3.1 multi-label prediction loss; The combined drug recommendation task is regarded as a multi-label classification task, and two commonly used loss functions are adopted to enhance the robustness of the results, namely binary cross entropy loss and multi-label boundary loss, whose formulas are as follows: wherein is the true medication use of the patient at the tth visit; The hyperparameter λ is set by experiment by default; Step 3.2, drug-drug interaction DDI loss; In order to reduce the incidence of drug-drug interaction in the predicted drug combination, minimize the following loss: where matrix A ij represents the pre-constructed DDI knowledge base, A ij = 1 if there is a drug-drug conflict between drug i and drug j, and A ij = 0 if there is no drug-drug conflict, represents the probability of the pair-wise drug interaction; In the model training process of the DDI knowledge base, The loss function calculates the interaction risk of all drug pairs in the current recommended drug combination; if the model recommends drugs with conflicts, The loss value will become larger; with a huge loss value as a punishment signal, the model will adjust its internal parameters in the subsequent optimization process to reduce the probability of recommending conflicting drugs at the same time.

10. The method of claim 9, wherein: In step three, a dynamic weighting strategy is introduced to form the objective function, which is expressed in the formula as follows: wherein ∈ is a pre-set safety DDI threshold.

Images