Knowledge Lenses for Multidimensional Domains

Abstract

The method (600) includes receiving multidimensional health data (300) from at least one data source (202). The multidimensional health data includes unstructured data (300u). The method also includes annotating the unstructured data to generate annotated data (300a), processing the annotated data to obtain training healthcare data (300T), and training a knowledge graph (50) based on the training healthcare data. The method also includes receiving a query (402) requesting information associated with the knowledge graph and retrieving the information requested by the query from the knowledge graph.

Description

[Technical Field]

[0001] The present disclosure relates to knowledge lenses for multidimensional domains. [Background technology]

[0002] Public and private pharmacology data are available from several sources. Some sources may also store adverse event statistics for drugs, drug combinations, and concomitant medications. Pharmacology and adverse event data represent multidimensional health data stored in various sources, are not in a searchable format, and lack the ability to link multidimensional data in a manner suitable for contextual querying.

[0003] Pharmacodynamics describes how a specific therapeutic agent affects a disease, while pharmacokinetics describes how the body processes a drug. While the route of drug intervention is usually well-known, pharmacokinetics must consider the pathways by which the drug itself is metabolized and other pathways through which the drug may unintentionally exert adverse effects. Drug safety is therefore an important aspect to consider in the development of new drugs and combination drug therapies for the treatment of specific diseases. To further enhance the ability to predict drug safety, patients exhibiting certain characteristics may be more susceptible to adverse events when treated with a particular drug, drug class, and / or combination drug therapy, while patients without these characteristics may not be as vulnerable to adverse events. Thus, different subclasses of the population may metabolize drugs differently, providing various potential responses to a drug. This may affect the drug's dosage, safety, efficacy, and usefulness in treating individual patients. Summary of the Invention

[0004] One aspect of the present disclosure provides a computer-implemented method executed on data processing hardware, whereby the data processing hardware performs operations including receiving multidimensional health data from at least one data source, where the multidimensional health data includes unstructured data. The operations also include annotating the unstructured data to generate annotated data, processing the annotated data to obtain training healthcare data, and training a knowledge graph based on the training healthcare data. The operations also include receiving a query requesting information to be associated with the knowledge graph and retrieving the information requested by the query from the knowledge graph.

[0005] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the query includes a natural language query, and obtaining the information requested by the query includes processing the natural language query by performing query interpretation on the natural language query using an inference model to determine a type of information requested by the natural language query, and retrieving information from a knowledge graph based on the type of information requested by the natural language query. In these implementations, the operations may also include using the inference model to generate a natural language summary of the information retrieved from the knowledge graph and providing the natural language summary of the information for output from the user device. Here, the inference model may utilize a large-scale language model to generate the natural language summary of the information. Additionally or alternatively, the inference model may include a neural network model.

[0006] In some examples, the operations also include receiving standard reference data. In these examples, annotating the unstructured data includes annotating the unstructured data based on the standard reference data. In some additional examples, the operations also include receiving concepts defining an ontology for semantically linking the training healthcare data. In these additional examples, training the knowledge graph based on the training healthcare data includes training the knowledge graph based on the training healthcare data with the concepts. The information requested by the query can optionally include information regarding the safety of particular drugs for treating the disease.

[0007] In some implementations, the operations also include executing a knowledge controller configured to display a user interface on a screen of a user device for viewing information retrieved from the knowledge graph. In these implementations, receiving a query can include receiving a query from a user device, where a user inputs the query via the user interface. Additionally or alternatively, the operations can display the knowledge graph in the user interface as an interactive knowledge graph.

[0008] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving multidimensional health data from at least one data source. The multidimensional health data includes unstructured data. The operations also include annotating the unstructured data to generate annotated data, processing the annotated data to obtain training healthcare data, and training a knowledge graph based on the training healthcare data. The operations also include receiving a query requesting information associated with the knowledge graph and retrieving the information requested by the query from the knowledge graph.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the query includes a natural language query, and obtaining the information requested by the query includes processing the natural language query using an inference model, performing query interpretation on the natural language query to determine a type of information requested by the natural language query, and retrieving information from a knowledge graph based on the type of information requested by the natural language query. In these implementations, the operations may also include using the inference model to generate a natural language summary of the information retrieved from the knowledge graph and providing the natural language summary of the information for output from the user device. Here, the inference model may leverage a large-scale language model to generate the natural language summary of the information. Additionally or alternatively, the inference model may include a neural network model.

[0010] In some examples, the operations also include receiving standard reference data. In these examples, annotating the unstructured data includes annotating the unstructured data based on the standard reference data. In some additional examples, the operations also include receiving concepts defining an ontology for semantically linking the training healthcare data. In these additional examples, training the knowledge graph based on the training healthcare data includes training the knowledge graph based on the training healthcare data with the concepts. The information requested by the query may optionally include information regarding the safety of certain drugs for treating the disease.

[0011] In some implementations, the operations also include executing a knowledge controller configured to display, on a screen of the user device, a user interface for viewing information retrieved from the knowledge graph. In these implementations, receiving the query can include receiving the query from the user device, where the user enters the query via the user interface. Additionally or alternatively, the operations can display the knowledge graph in the user interface as an interactive knowledge graph.

[0012] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of a system including a knowledge graph linking multidimensional health data and a user interface for viewing the knowledge graph and / or viewing inferences from the knowledge graph. [Figure 2A] FIG. 2A is a schematic diagram of an example knowledge graph builder for constructing a knowledge graph from the multidimensional health data of FIG. [Figure 2B] FIG. 2B is a schematic diagram of an example knowledge controller that receives multidimensional health data stored in various data sources via data inputs and executes a knowledge graph builder to create the knowledge graph of FIG. 1 . [Figure 3A] Figure 3A is a schematic representation of annotated data for case narratives of clinical trial patients / participants. [Figure 3B] FIG. 3B is a schematic representation of annotated data for drug labels of specific drugs. [Figure 4] FIG. 4 is a schematic diagram of an inference model that receives a query for information and retrieves the information from a knowledge graph. [Figure 5A]FIG. 5A is a schematic diagram of an example of a user interface for presenting data represented by a knowledge graph. [Figure 5B] FIG. 5B is a schematic diagram of an example of an interactive knowledge graph. [Figure 6] FIG. 6 is a flowchart showing an example of the operational configuration of a method for creating a knowledge graph from multidimensional health data and performing inference on the knowledge graph. [Figure 7] FIG. 7 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0014] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0015] Referring to FIG. 1 , in some implementations, the exemplary system 100 includes a knowledge graph 50 linking multidimensional health data 300 and a user interface (UI) 500 for viewing the knowledge graph 50 and / or viewing inferences from the multidimensional health data 300 in the knowledge graph 50. For example, the knowledge graph 50 may provide insights into potential signals represented by the knowledge graph 50 and generate hypotheses / evidence regarding the occurrence of specific adverse events based on pharmacological or biological data. In some examples, an inference model 400 runs on the knowledge graph 50 to ensure the safety of a drug or drug class in the treatment of a disease. As described below, the inference model 400 can receive queries regarding the safety of a drug or drug class in a specific patient population and traverse the knowledge graph 50 to make inferences / predictions about the safety of the drug or drug class in the specific patient population. For example, the inference model 400 can predict adverse events of a specific drug, and more specifically, can predict adverse events of a specific drug in a patient population with specific personality traits. These predictions can include probabilities or likelihoods.

[0016] The inference model 400 may further enable search capabilities via the knowledge graph 50 to gain insight into the information represented by the knowledge graph 50. That is, the inference model 400 may extend the knowledge graph 50 to perform relevant searches across the multimodal data contained in the knowledge graph 50 to obtain statistics, data, inferences, and / or recommendations related to the safety profile of drugs, drug classes, and / or patient populations or subpopulations. In this manner, search results may highlight the similarities between candidate drugs and existing drugs from a safety perspective using various similarity algorithms, including, but not limited to, embedding, sine / cosine / Jaccard similarity, or other types of distance measures between data points in the knowledge graph 50.

[0017] As described below, the generation of and interaction with the knowledge graph 50 provides numerous operational use cases in pharmacovigilance, including understanding overlaps, case management operations, and medical review of case collections as well as individual cases. These insights reduce the complexity, cost, and time of integrating and migrating safety information related to drugs, drug classes, and / or patient populations or subpopulations. The UI 500 provides an extensible framework for the analysis of PV data, enabling prospective views using retrospective data represented by the knowledge graph 50. In this way, concepts of a patient's progression through treatment with a drug or drug class can be generated / predicted, fundamentally changing how adverse events are learned within the system and shifting the focus of the underlying process for understanding safety information from the specific case itself to the patient.

[0018] The system 100 includes a user device 110 associated with a user 102 and communicating with a remote system 130 via a network 120. The user 102 may include, but is not limited to, a research professional, a clinical trial professional, a physician, a healthcare provider, or a patient. The user device 110 may correspond to a computing device such as, but not limited to, a desktop workstation, a laptop workstation, or a mobile computing device (e.g., a smartphone or tablet). The remote system 130 may be a distributed system (e.g., a cloud environment) having scalable / elastic resources 140 including computing resources 142 (e.g., data processing hardware) and storage resources 144 (e.g., memory hardware). The computing resources 142 may include a service abstraction layer and a hypertext transfer protocol wrapper on a server virtual machine instantiated thereon. Thus, the computing resources 142 may be configured to receive a query 402 from the user device 110 and transmit a response (e.g., a knowledge graph 50, a portion of the knowledge graph 50, a prediction inferred from the knowledge graph 50 by an inference model, etc.) to the user device 110.

[0019] In the depicted example, the computing resources 142 manage the storage of the knowledge graph 50 on the storage resources 144. The computing resources 142 may further execute a knowledge controller 150 configured to communicate with the user device 110 and serve as an interface mechanism to enable the user device 110 to build / create, interact with, and perform operations (e.g., read / write) on the knowledge graph 50. Specifically, the knowledge controller 150 may execute a knowledge graph builder 200 to enable input of the multidimensional health data 300 and rules and ontologies for identifying specific concepts / entities within the health data. The knowledge graph builder 200 may build / create the knowledge graph 50. The knowledge graph 50 represents each concept identified within the health data as a node and links related nodes together based on interrelationships between the concepts. Thus, knowledge graph 50 may represent a cluster of cases, each case including a group of related nodes linked together based on interrelationships between the concepts represented by the nodes. For example, a case may include a patient node representing a patient with a medical condition, one or more drug nodes each representing a drug or drug class prescribed to the patient to treat the medical condition, and one or more adverse event nodes each representing an adverse event (AE) experienced by the patient while being prescribed the drug or drug class. As described in more detail below, some cases in knowledge graph 50 may additionally or alternatively include nodes representing other types of concepts of potential interest, as specified by the rules / ontology input into knowledge graph builder 200.

[0020] Once the knowledge graph 50 is constructed by the knowledge graph builder 200, the knowledge controller 150 enables data retrieval of the knowledge graph 50 from the storage resource 144 and displays a UI 500 for viewing the knowledge graph 50 on the screen 116 of the user device 110. The knowledge controller 150 can allow the user 102 to interact with the knowledge graph 50 displayed in the UI 500. For example, the user 102 can select a node of interest to view more detailed information about the selected node. In one example, when the user 102 selects a patient node, the knowledge controller 150 causes the UI 500 to display a pop-up window displaying detailed information about the patient represented by the patient node. The detailed information can include the patient's demographic information (e.g., age, gender, location), biomarkers, diseases, prescribed medications, attending physicians, or other characteristics of the patient. The knowledge controller 150 further enables the user 102 to provide a query 402 to present specific data of interest from the knowledge graph 50. For example, user 102 may provide a single natural language query or multiple individual queries requesting knowledge controller 150 to present cases from knowledge graph 50 that include men between the ages of 50 and 60 who have been prescribed a particular combination of medications. In this example, knowledge controller 150 may update knowledge graph 50 so that only linked nodes for cases that include men between the ages of 50 and 60 who have been prescribed a particular combination of medications are displayed in UI 500, rather than all cases being displayed in UI 500.

[0021] In some implementations, the knowledge controller 150 executes the inference model 400 to make inferences / predictions from the knowledge graph 50 regarding information requested by a query 402 entered by the user 102. For example, the user 102 may enter a natural language query requesting information about the safety of a particular therapeutic drug with respect to a particular patient's personality trait (e.g., males aged 50-60), and the inference model 400 may traverse the knowledge graph 50 to make inferences / predictions about the safety of the therapeutic drug with respect to the personality trait. In some examples, the inference model 400 generates a natural language summary based on the inferences / predictions about the safety of the therapeutic drug with respect to the personality trait specified by the query. In this example, the summary may read, "Males aged 50-60 are likely to experience circulatory collapse if prescribed the therapeutic drug."

[0022] 2A shows a schematic diagram of an example knowledge graph builder 200 used in creating a knowledge graph 50 from multidimensional health data 300 stored in various data sources 202. The knowledge graph builder 200 includes a data input 210 that receives data 300 from the data sources 202. In some examples, a user 102 retrieves the multidimensional health data 300 from the data sources 202 using a knowledge controller 150 by allowing the user 102 to provide criteria for the type of multidimensional health data 300 to be represented by the knowledge graph 50. The knowledge graph builder 200 uses the multidimensional health data 300 as training data for training the knowledge graph 50.

[0023] The illustrated example shows a non-exhaustive list of data sources 202. The data sources 202 may be interchangeably referred to as “data stores 202.” Details of the present disclosure may include other data sources 202 for providing the multidimensional health data 300 in addition to or in place of the data sources 202 shown in the illustrated example. The data sources 202 include a patient information data source, a case narrative data source, a clinical trial study data source, a product information data source, and an adverse event (AE) data source. The patient information data sources may include health data for each patient in the patient corpus. Patients in the corpus may be participants in various clinical trials and / or studies related to the treatment of diseases and / or conditions, and may also be participants to aid in the development of drug therapies for the treatment of these diseases. The patient information for each patient in the patient corpus may include structured and / or unstructured data, including patient notes entered into one or more electronic medical records (EMRs) by research professionals, clinical trial professionals, physicians, and / or healthcare providers. Patient data / records include demographic information for each patient, such as age, sex, ethnicity, height, weight, and BMI (body mass index), as well as genetic data, phenotype, proteome, climate, adverse drug event history, any diseases / symptoms, allergies, previous health conditions, vital signs, recommended treatments, risks, medical history, family health history, test results, current medications, and / or past medications. Source data for adverse drug event history and / or medical history can be obtained by accessing, requesting, or collecting data on patients who have experienced adverse drug reactions and comparing it with data from a broad population control set not taking the drug / drug in question to identify a relationship between the specific reaction and genotype / phenotype. For example, light-skinned individuals (a type of phenotype with a genotypic background) generally burn more easily in the sun and may also be particularly sensitive to certain drugs. Population genetics information can include a variety of sources, such as DNA samples taken directly from individuals with documented adverse drug reactions to certain drugs.

[0024] The patient information for one or more patients in the patient corpus may also include identifiers and details of clinical trials and / or studies (past or current) in which the patient has participated, as well as details of trial outcomes and adverse events experienced by the patient. Patient notes, entered in the form of unstructured data, may contain many strings of characters arranged into sentences. These sentences may be arranged into one or more paragraphs.

[0025] The case narrative data store may include narratives of patients participating in clinical trials or other health research studies. Case narratives may be stored in the form of unstructured data consisting of multiple strings of characters arranged in sentences. An example case narrative is shown in FIG. 3A. As used herein, case narratives illustrating the narrative of a clinical trial are exemplary only; the present disclosure similarly contemplates other types of unstructured data consisting of multiple strings of characters arranged in sentences, such as published literature and scientific research papers. These sentences may be arranged in one or more paragraphs. Each case narrative may comply with regulatory requirements and procedures aimed at reducing the time and cost burden of effective patient safety reporting during all phases of clinical trials, whether conducted on healthy volunteers or patients with the disease / condition being studied. Patient safety narratives provide a complete, clinically relevant chronological account of the progression of events experienced during or immediately following a clinical trial. Case narratives can follow Council of International Organizations of Medical Sciences (CIOMS) forms, Case Report Forms (CRFs), MedWatch forms, Data Clarification Forms (DCFs), and clinical database listings. In some instances, case narrative data stores include Clinical Study Reports (CSRs) that contain short narratives describing each death, each other serious AE, and other significant AEs that are judged to be of particular interest because they are clinically significant. Thus, narratives included in a CRS should include the following: These include the nature, intensity, and outcome of the AE, the clinical course leading up to the AE, timing indications relative to administration of the study drug, relevant laboratory values, actions taken (and timing) with respect to the study drug in relation to the AE, treatments or interventions, autopsy findings (if applicable), the investigator's and sponsor's (if applicable) opinion regarding causality, patient identifier, patient's age and sex, the patient's overall clinical condition if applicable, the disease being treated and duration of the current disease episode, relevant concomitant / previous conditions and details of onset / duration, relevant concomitant / previous medications (e.g., concomitant medication i.e., Conmed) and details of dosage, and the study drug administered (including dosage and duration).The patient safety narrative within or accompanying a CSR describes all relevant events for a single patient, along with any relevant background information detailed above. An individual CSR (ICSR) involves a single patient, one or more identifiable reporters, one or more suspected AEs clinically and temporally related to treatment, and one or more suspected medicinal products. In clinical trials, an individual case is information provided by a primary source that describes a serious adverse event, related or unrelated to the administration of one or more investigational medicinal products to an individual patient at a specific time point. The reported adverse event must be a diagnosis. If a diagnosis has not been made at the time, the case may instead include multiple signs and symptoms, and therefore there are multiple events reported. ICSRs generated postmarketing differ in that multiple event terms may be reported per case. These events must be temporally or clinically related and ordered according to their clinical relevance to the product. That is, a serious, unexpected adverse event is designated the "primary event" for reporting purposes, while nonserious or expected adverse events are ranked lower within the case. Furthermore, in post-marketing ICSRs, all spontaneously reported adverse events are considered drug-related unless otherwise specified by the reporter, whereas in clinical practice, investigators make their own interpretation of causality.

[0026] Clinical trial research data sources may include one or more regulatory information sources that provide accessible information about publicly and / or privately supported clinical trials for a wide range of diseases and conditions. Therefore, clinical trial research data sources may also include one or more web-based resources (e.g., www.clinicaltrials.org) that provide patients, their families, healthcare professionals, researchers, and the general public with easy access to information about publicly and privately supported clinical trials for a wide range of diseases and conditions. Information posted on these web-based resources may be provided and updated by the clinical trial sponsor or principal investigator. Trials are typically submitted (i.e., registered) to the website at the time of initiation, and information on the site is updated throughout the trial. In some cases, trial results may be submitted after the trial is completed. As an example, a clinical trial data source may include "www.clinicaltrials.gov."

[0027] A clinical trial research data store may contain both clinical and post-marketing data on drugs and drug classes used in clinical trials, thereby providing useful safety information throughout the product lifecycle, starting with first use in patients / humans. Some of the information stored in a clinical trial research data store may contain the same, or substantially the same, information as a case narrative data source. Here, a clinical trial research data store contains information about medical research in human volunteers. Most of the records stored in the data source describe clinical trials (also called interventional studies). Clinical trials are research studies in which human volunteers are assigned to an intervention (e.g., a drug, behavior, or procedure) based on a protocol (or plan), and the impact on biomedical or health outcomes is then evaluated. A clinical trial research data store also contains records describing observational studies and programs that provide access to investigational drugs outside of clinical trials (expanded access). The following types of information may be summarized in a clinical trial record: The study includes the disease / condition being studied / treated, the intervention (drug, behavior, or procedure), the name and description of the study, the study design, the treatment or drug combination that is part of the study, concomitant medications, eligibility requirements for study participants, the study location, contact information for the study location, links to related information, a description of study participants (number of participants who started and completed the study and demographic data), the results of the study, and a summary of adverse events. Concomitant medications (also called "conmeds") are other prescription, over-the-counter (OTC), or dietary supplements that study participants take in addition to the drug or drug combination being studied. Conmeds may be used by study subjects for the same indication as the study or for other indications.

[0028] A product information data source includes a corpus of drugs and / or products / devices available to treat various diseases and conditions. It may include real-world data on drug types or classes, metabolic pathways, drug pharmacokinetics, and pharmacodynamics. It may also provide a drug taxonomy that describes drug characteristics such as metabolites, clearance rates, serum peak concentrations, pharmacodynamics, therapeutic categories, and chemical structures, or a method for grouping drugs and examining their relationship with both response and genotype. In some examples, a corpus of available drugs includes drugs and drug combinations for treating specific types of diseases, such as immunotherapeutic drugs available for cancer treatment. For each drug in the corpus of available drugs, a product information data source may provide a corresponding drug label that is used to ensure patient safety by providing healthcare professionals with an overview of the drug's safety and efficacy. In some cases, if the drug is an over-the-counter (OTC) drug, the drug label is targeted to a patient population. However, if the drug is a prescription drug or an investigational drug, the drug label is not targeted to a patient population. This is because the administration of prescription and investigational drugs is always under the supervision of a healthcare practitioner who is licensed to prescribe or otherwise authorize their administration. Generally, the following is an outline of what a drug label should include: a highlighted section providing a concise summary of label information, complete prescribing information, a statement of limitations, the product name, the date of approval in one or more jurisdictions, a boxed warning, recent major changes, indications and use, dosage and administration, dosage form and strength, contraindications, warnings and precautions, side effects, drug interactions, use in specific populations, and a statement of patient counseling information. Figure 3B shows an example of a side effect listed on a drug label for a given drug.Investigational drugs may include one or more of the following: protocol number, generic name, sponsor's name and address, patient identification information, special warnings, investigator's name, study acronym or title, Institutional Review Board (IRB) name for the drug, investigational drug dose / concentration / strength, dosage form (e.g., lyophilized powder, solution, suspension, capsule, tablet, etc.), lot / batch number, expiration date / retest date, and a unique approved standardized identifier that is distinct from other investigational drugs.

[0029] In some examples, product information data sources include publicly available open product labels maintained by the U.S. Food and Drug Administration (FDA). Product information data sources may also include one or more drug code directories, such as the National Drug Code (NDC) directory maintained by the FDA, which contains information on finished, unfinished, and compounded drugs. Here, drug manufacturers / facilities are required to provide the regulatory authority (i.e., FDA) with an up-to-date list of all drugs manufactured, prepared, propagated, compounded, or processed for sale at their facilities. Drugs are identified and reported using a unique three-segment number called the National Drug Code (NDC), which serves as the FDA's drug identifier. The FDA may publish NDC numbers in the NDC directory, which is updated daily. While drug labels may be recorded in an unstructured format, drugs submitted to the FDA for inclusion in the NDC directory are provided in the form of a Structured Product Label (SPL) electronic listing file by the labeler (which may include the manufacturer or entity listed on the product label). The NDC Directory includes product listing data submitted for all finished pharmaceutical products, including prescription drugs, over-the-counter drugs, approved drugs, unapproved drugs, and repackaged and relabeled drugs.

[0030] Additionally, for unfinished drug products, such as investigational drugs being investigated in clinical trials, drug manufacturers that produce active pharmaceutical ingredients are required to submit to the FDA an up-to-date list of all drug products manufactured, prepared, grown, compounded, or processed in their facilities for commercial distribution in the United States. Accordingly, the NDC directory may maintain an unfinished drug product database that includes product listing data submitted for all unfinished drug products, such as active pharmaceutical ingredients, drug products for further processing, and bulk drug substances for compounding. Notably, the resulting knowledge graph 50 can advantageously link a finished drug product to relevant information from when the finished drug product, or at least its drug substance, was in its unfinished stage, allowing users 102 to easily browse (e.g., via interface 500) relevant information related to a drug product at all stages of development.

[0031] Additionally, the product information data source may include information about finished compounded human drug products manufactured by outsourcing facilities that may choose to assign NDC numbers to their products. Such outsourcing facilities may be exempt from drug registration and listing requirements if they meet certain conditions under the law, allowing them to assign NDC numbers to their finished compounded human drug products, but not requiring them to do so. The NDC directory may include only compounded drug products reported under the sales category "Human Drug Products Compounded by Outsourcing Facilities (Exempt from Approval Requirements)" that have been assigned NDC numbers. The product information may include search results containing information reported to the FDA within the past two years. In particular, the annotator 220 may annotate data about unfinished and compounded human drug products obtained from the NDC directory, thereby enabling the data presented in the knowledge graph 50 to be distinguished from finished drug products. This is because simply including a product in the NDC directory does not mean that the FDA has verified the information provided or that the product is FDA approved. In this case, the annotator 220 may view the label / tag in the corresponding Structured Product Label (SPL) when the product is submitted.

[0032] AE report data sources include records of all AE cases reported to one or more regulatory authorities. AE report data sources may include adverse event data provided by pharmaceutical companies, hospitals, physicians, health insurance companies, state, federal, and international organizations, for example. The primary source of data for the pharmaceutical industry is individual adverse events recorded by the safety departments of various pharmaceutical companies. In both cases, source data can focus on clinical trials, postmarketing surveillance, research databases, and so on. The unedited data within each source database is referred to as "verbatim." Clinical trial data available in the literature includes safety data. Other information is collected and accessible from the World Health Organization (WHO), General Practice Research Databases (GPRDs), and others. For example, AE report data sources may include the U.S. Food and Drug Administration's (FDA) Adverse Event Reporting System (FAERS). This system maintains data for use by the pharmaceutical industry, healthcare providers, and the general public, who search for information related to human AEs reported to the FDA by consumers. Thus, AE report data sources may include data on AEs reported to regulatory authorities (e.g., the FDA) for specific drugs or biologics. However, because the reports do not indicate that a specific drug or biological product caused the AE, the data maintained by the AE report data source itself is not an indicator of the drug or biological product's safety profile. Furthermore, the data maintained by the AE report data source may contain duplicate or incomplete reports, and some reports may lack necessary information. Furthermore, the reports merely reflect the AE reporter's observations and opinions and do not establish a causal relationship between the AE and the drug or biological product. Furthermore, the information contained in the reports may not be verified or medically confirmed, and incidence rates cannot be calculated. As described below, building a knowledge graph 50 based on multidimensional health data 300 collected from all of the various data sources 202 in conjunction with the knowledge controller 150 provides the ability to understand drug safety for specific subpopulations and subpopulation characteristics, which is not possible through a simple search of the AE report data source.

[0033] Referring to FIG. 2B, the knowledge controller 150 may be configured to execute instructions to receive multidimensional health data 300 stored in various data sources 202 via data input 210 (FIG. 2A) and execute the knowledge graph builder 200 to create and update the knowledge graph 50. The health data 300 received via the data input 210 may be stored in the memory hardware 114 of the user device 110 and / or the memory hardware 144 of the cloud computing environment 130. In some examples, the multidimensional health data 300 is categorized into one of three categories: (i) disease data 310, (ii) patient data 320, and (iii) medication data 340. These categories of multidimensional health data 300 are exemplary only and may additionally or alternatively include other categories, such as categories representing payer data including claims and prescriptions. The disease data 310 includes a list of diseases and medical conditions, each of which lists one or more treatments 312. These treatments include not only approved drug or drug combination treatments, but also past and current experimental treatments implemented through clinical trials. Patient data 320 is stored as a table containing data permanently associated with individual patients, such as identifying information, demographic information, and multiple sub-tables 322, 324, 326, 328, and 330 linked to the table in a few-to-many relationship. Thus, data associated with each information record in the table of patient data 320 is stored in the various sub-tables corresponding to the record. For example, sub-table 322 might list the patient's permanent (chronic) health conditions, sub-table 324 might list the patient's known allergies, and sub-table 326 might list all medications the patient is currently taking. Subtable 328 may also contain any symptoms or conditions the patient is currently experiencing, which may be populated from adverse events (AEs) reported during a clinical trial, adverse events (AEs) reported by a healthcare provider (HCP), or by matching records (e.g., lab results) listed in Current Test Results subtable 330.

[0034] Therapeutic drug data 340 may be represented by a table containing a schedule of all available therapeutic drugs, drug classes, and drug combinations used to treat a disease. The therapeutic drug data 340 may be indexed to link to multiple subtables 342, 344, 346, and 348. Each drug represented by the therapeutic drug data 340 may be entered with drug information and scaled guidelines. The drug information may include the respective NDC number, drug class, chemical class, biological pathway, metabolite, structure, generic name, and administration method. The scaled guidelines may indicate known health risks and effectiveness in treating the underlying disease / condition. The biological pathway associated with a drug or drug class may indicate which mechanism, such as an enzyme, is activated (i.e., overexpressed / underexpressed) to lead to a specific biological activity. In other words, a drug may target an enzyme beneficial to a specific pathway, but because the pathways overlap, blocking one pathway may enhance another. This phenomenon, known as a signaling cascade, is often seen when targeting pathways in cancer treatment. Therefore, because cancer encompasses multiple pathways, drug combination treatments that target multiple enzymes in a particular pathway are often required. Subtable 342 may include a drug interaction list indicating medications known to interact with the underlying medication. Subtable 344 may indicate available dosages of the underlying medication, and subtable 346 may indicate concomitant medications (e.g., Conmed) that the patient or participant is taking in addition to the underlying drug or drug combination.

[0035] In particular, the multidimensional health data 300 input into the knowledge graph builder 200 via data input 210 includes both unstructured data 300u and structured data 300b. The unstructured data 300u may include multiple strings of characters arranged in sentences. The sentences may be organized into one or more paragraphs. Returning to FIG. 2A , the knowledge graph builder 200 executes an annotator 220 to analyze the unstructured data 300u, extract key terms and information therefrom, and provide annotated data 300a for use in creating the knowledge graph 50. The annotator 220 may execute one or more natural language processing (NLP) models 225, each configured to receive the unstructured data 300u and output corresponding annotated data 300a. Some NLP models may be trained to annotate specific types of unstructured data 300u. In some examples, a special-purpose NLP model is trained to analyze unstructured data 300u related to a case narrative and output annotated data 300a that annotates the case narrative with key terms identified within the case narrative. For example, FIG. 3A shows annotated data 300a for a case narrative about a patient / participant in a clinical trial who collapsed while being treated with an experimental drug in combination with another drug, dexamethasone, for multiple myeloma. In the illustrated example, the NLP model 225 annotates the case narrative to identify and annotate different types of terms 301, 301a-301d. Here, a first term 301a is associated with a description of a specific drug (e.g., dexamethasone) in the case narrative, and a second term 301b is associated with a description of an adverse event (e.g., collapse / collapse, multiple myeloma, hypertension, nausea, headache, dilated pupils, death, cardiac arrest) in the case narrative. The NLP model 225 can identify and annotate other unique types of terms in the case narratives. The same or a different NLP model 225 can be trained to analyze the unstructured data 300u about drug labels and output annotated data 300a that annotates the case narratives with key terms identified in the case narratives.For example, FIG. 3B shows annotated data 300a for a drug label for a particular drug, where the NLP model 225 has annotated each instance of an adverse event listed on the corresponding drug label.

[0036] Returning to FIG. 2A , in some implementations, the annotator 220 receives standard reference data 222, including dictionaries, thesauri, taxonomies, and hierarchical structures used to generate the annotated data 300a from the unstructured data 300u input to the annotator 220. Thus, the standard reference data 222 not only provides terms that the NLP model 225 can use to identify when parsing the unstructured data 300u, but also complements the identified terms with related terms, synonyms, and lexical variants. One example of standard reference data 222 is the Medical Dictionary for Regulatory Activities (MedDRA), which identifies many different adverse events at different hierarchical levels. Here, MedDRA may include five hierarchies, ranging from very specific to very general. The most specific level, called the "Least Level Terms" (LLT), contains over 80,000 terms that parallel how information is communicated and reflect how observations are reported in practice. The next level, called "Preferred Terms" (PTs), contains individual descriptors (single medical concepts) that describe symptoms, signs, disease diagnoses, indications for treatment, tests, surgical or medical procedures, and medical-social or family history characteristics. Each LLT is linked to only one PT, and each PT contains at least one LLT, as well as synonyms and lexical variations of the PT (e.g., abbreviations, different word orders). The next level, called "High Level Terms" (HLTs), groups related PTs based on anatomy, pathology, physiology, etiology, or function. HLTs related to each other by anatomy, pathology, physiology, etiology, or function are linked to "High Level Group Terms" (HLGTs). Finally, MedDRA groups HLGTs into the most general level, called "System Organ Classes" (SOCs). System Organ Classes are grouped by etiology (e.g., infectious or parasitic infections), site of manifestation (e.g., gastrointestinal disorders), or purpose (e.g., surgical or medical procedures). Additionally or alternatively, the standard reference data 222 may include custom data such as rules, terminology, language models, dictionaries, and / or libraries that the annotator 220 uses to analyze the unstructured data 300u received via the data input 210 and annotate the annotated data 300a.Standard reference data 222 such as MedDRA may further characterize reported adverse events by their severity.

[0037] 2A , the knowledge graph builder 200 also includes a converter 230 configured to merge the annotated data 300 a and the structured data 300 b into training healthcare data 300, 300T for training the knowledge graph 50. The training healthcare data 300T may include information about diseases / conditions (e.g., cancer), clinical trial patients and / or participants diagnosed with the disease / condition, therapeutic classes for treating the disease / condition, various therapeutics and drug combinations associated with the therapeutic class prescribed to the patients and / or participants (including both approved and investigational drugs and drug combinations being the subject of research / clinical trials), concomitant medications the patient / participant is taking in addition to the base therapeutic or drug combination, the effectiveness of the therapeutic and drug combination, and adverse events experienced by the patient / participant while taking the therapeutic and drug combination and / or after the patient / participant has stopped taking the therapeutic and drug combination.

[0038] In some examples, the knowledge graph builder 200 trains the knowledge graph 50 using the training healthcare data 300T to provide a drug safety system that can make inferences / predictions about the safety of drugs, drug combinations, and / or drug classes used to treat diseases / conditions. The converter 230 can receive concepts 232 that provide an ontology for training the knowledge graph 50 based on the multidimensional training healthcare data 300T. Specifically, the concepts 232 enable the converter 230 to semantically link the training healthcare data 300T within the knowledge graph 50 to enable contextual queries about the healthcare data 300T. The concepts 232 may include user-specified rules that define nodes related to the treatment of diseases and edges or links that connect the nodes to represent interrelationships (e.g., relationships) between the treatment-related concepts. In some implementations, the knowledge graph 50 generated by the knowledge graph builder 200 is self-forming, meaning that the knowledge graph builder 200 uses the NLP model 225 and standard reference data 222 to identify and create concepts / nodes from the healthcare data 300, without the user having to explicitly provide concepts 232. Continuing with this example, the concepts 232 input into the converter 230 of the knowledge graph builder 200 may define nodes including disease nodes (e.g., cancer or a specific type of cancer, such as melanoma), treatment nodes (e.g., immunotherapy), drug nodes associated with the treatment nodes (which may indicate targeted biological pathways), adverse event (AE) nodes associated with the treatment nodes and drug nodes, biological pathway nodes associated with the drug nodes, and patient / participant nodes associated with the disease, treatment, and drug nodes. The user interfaces 500, 500b in FIG. 5B illustrate an example of a knowledge graph 50 with which the user 102 can interact.

[0039] The resulting knowledge graph 50 represents a model that includes individual concepts (nodes) and predicates that describe the properties and / or relationships between those individual nodes. The knowledge graph's underlying logical structure (e.g., N-order logic) connects the various individual nodes using predicates. The knowledge graph 50 and logical structure combine to form a language that describes facts, concepts, relationships, conclusions, propositions, etc. The knowledge graph 50 and logical structure may be continuously or periodically generated and updated by an artificial intelligence engine (i.e., knowledge graph builder 200) in response to new healthcare data 300 received from data sources 202 at data input 210 (FIG. 2A). The predicates and individual nodes may be generated based on the healthcare data input to the knowledge graph builder 200. Updated or new standard reference data 50 is continually provided to the knowledge graph builder 200, allowing the knowledge graph builder 200 to continually modify the individual elements and predicates represented by the knowledge graph 50.

[0040] The converter 230 of the knowledge graph builder 200 may generate the knowledge graph 50 from the training healthcare data 300T and the concepts 232 by determining semantic relationships for aligning the training healthcare data 300T and the concepts 232. In some examples, the converter 230 uses machine learning techniques to align and integrate the training healthcare data 300T with the concepts 232 to generate the knowledge graph 50. Additionally or alternatively, the converter 230 may use any combination of schema-level matching techniques, instance-level matching techniques, or hybrid matching techniques to align and integrate the training healthcare data 300T with the concepts 232.

[0041] 4 , a user interface 500 executing on the user device 110 allows the user 102 to issue a query 402 to the inference model 400 requesting information associated with the knowledge graph 50. In some examples, the query 402 received from the user device 110 requests the inference model 400 to return safety information associated with a drug, drug class, or other form of treatment (i.e., surgery) used to treat a disease. The user 102 can input the query 402 as a natural language query via the user interface 500, and the inference model 400 is configured to perform query interpretation on the natural language query to determine what type of information the user 102 is requesting from the knowledge graph 50. For example, the natural language query 402 may include the query "return all adverse events reported for drug X in testing," and the inference model 400 can convert the natural language query 402 into a graphical query and leverage the existing structure of the knowledge graph 50 to retrieve the requested information. The natural language query 402 can specify different levels of granularity for the information that the inference model 400 is requested to return from the knowledge graph 50. For example, the natural language query 402 might include the query, "Return all adverse events reported for men ages 40-55 who have been diagnosed with melanoma and treated with investigational drug X." The inference model 400 can return a response 404 conveying the requested information for the user interface 500 to output to the user 102. Here, the user interface 500 can display the response 404 on the display 116 of the user device 110 and / or output synthesized speech from a speaker of the user device 110 conveying the requested information to the user 102.

[0042] In some implementations, the inference model 400 utilizes a large-scale language model (LLM) that leverages data from the knowledge graph 50 for downstream tasks, such as generating a summary of information from the knowledge graph 50 requested by a natural language query 402. Here, the natural language query 402 is provided as a prompt to the LLM 400, which generates a response 404 based on the knowledge graph 50 that conveys the information requested by the prompt query 402. A user can provide the natural language query 402 as a follow-up prompt to the LLM 400 to further refine the previous response 404 output by the LLM 400, providing a conversational interface. In this manner, the user interface and the inference model 400 can provide a conversational assistant function (e.g., a chatbot) that allows a user to interact with the knowledge graph 50 using natural dialogue.

[0043] Additionally or alternatively, the inference model 400 may include a neural network model trained to make predictions by traversing the knowledge graph 50. Here, the user 102 provides a query 402 such as, "Is it safe to take drug X while taking drug Y?" and the inference model 400 can convert the natural language query into a graph query to traverse the knowledge graph 50 and identify adverse event nodes with edges / links connected to drug nodes for drug X and drug Y. The training data used to train the neural network model 400 may include example training queries each paired with the knowledge graph 50 and ground truth adverse event nodes (or other types of nodes of interest in the knowledge graph 50), teaching the neural network model 400 how to convert the training queries into graph queries to traverse the knowledge graph 50 and identify the corresponding ground truth adverse event nodes paired with each training query. Other examples of natural language queries 402 include, "Can drug X cause side effect E in patient B who is taking drug Y?", "What is the risk if patient B takes drug X while taking drug Y?", and "What is the risk if patient B takes drug X while also taking comorbidity C?". The inference model 400 can perform inferences from the data associated with the identified nodes and make predictions regarding the safety of taking drug X while taking drug Y. Because nodes in the knowledge graph 50 may include embeddings in the embedding space, the inference model 400 can make predictions based on the relationships between nodes represented in the embedding space. These inferences may take into account the number of patients taking both drug X and drug Y, as well as the severity of adverse events. The inference may also consider adverse events associated with other drugs with similar properties to drug X and drug Y (e.g., drugs that target similar biological pathways as drug X and drug Y) when performing inferences to predict the safety of taking drug X while taking drug Y. These inferences may also identify unique characteristics of the response 404 output by the inference model 400 .For example, knowledge graph 50 may indicate that patients under 18 years of age treated with both drug X and drug Y are highly likely to experience a particular adverse event, while patients over 60 years of age are unlikely to experience a serious adverse event. Based on these predictions, inference model 400 may generate one or more candidate responses to query 402 and, if desired, score the candidate responses based on knowledge graph 50. Inference model 400 may present the highest-scoring candidate response to the user via UI 500, or may present all or a portion of the highest-scoring candidate responses to the user via UI 500.

[0044] 4 and 5A, in some configurations, the inference model 400 receives a pre-configured query 402 from the user device 110 in response to a user input indication indicating a selection of a menu item, graphical feature, and / or filter option presented in the UI 500. The UI 500, 500a in FIG. 5A may correspond to a dashboard or reporting tool for accessing and viewing information associated with the knowledge graph 50. Additionally, the UI 500 allows the user 102 to enter a natural language query 402 in a text field 502 displayed in the UI 500. Thus, the user 102 can issue the natural language query 402 and then further refine their search for information they want to invoke from the knowledge graph 50 or for the inference model 400 to infer by selecting graphical elements 504, such as menu items, drop-downs, and / or filtering options, displayed in the UI 500 to issue the pre-configured query 402.

[0045] 5A illustrates that UI 500, 500a enables user 102 to interact with knowledge graph 50. It enables user 102 to issue one or more queries 402 specifying information associated with knowledge graph 50 and presents the information associated with knowledge graph 50 specified by query 402. For example, UI 500a can present information retrieved from knowledge graph 50 in a format easy for user 102 to view by entering the information retrieved from knowledge graph 50 into table 520. In the illustrated example, table 520 includes multiple rows associated with each case of a patient / participant prescribed a particular medication (e.g., C5013) and columns containing values ​​retrieved from knowledge graph 50 for various attributes, such as each patient / participant's demographics (e.g., gender / age), the Conmed the patient / participant is taking, the patient / participant's risk / factors, adverse events, drug label, and case narrative. The values ​​entered into each column of table 520 may include information ascertained from nodes in knowledge graph 50. Additionally, some columns may include hyperlinks to information sources that a user 102 can select to be directed to the information source. For example, a user 102 can view the case narrative for each patient / participant by selecting the "View" hyperlink displayed in the "Narrative" column. In this example, UI 500 can display a webpage containing the case narrative. UI 500 can be configured to display the webpage as a pop-up viewer above table 520 so that a user 102 can quickly view the case narrative without leaving table 520.

[0046] Referring to FIG. 5B , in some implementations, UI 500, 500b presents an interactive knowledge graph 50. In the illustrated example, knowledge graph 50 includes a disease node (e.g., cancer or a specific type of cancer) as a root node and treatment nodes 1-3 branching from the disease node, each corresponding to a different type of treatment for treating the disease associated with the disease node. Here, the first treatment node (treatment 1) includes a first type of treatment, such as immunotherapy, the second treatment node (treatment 2) corresponds to a second type of treatment, such as hormone therapy, and the third treatment node (treatment 3) corresponds to a third type of treatment, such as chemotherapy. Although not shown, any one of the treatment nodes may be connected to one or more other disease nodes, indicating that the corresponding type of treatment can be used to treat multiple diseases or multiple different types of diseases. For simplicity, knowledge graph 50 only shows nodes branching from and related to the first treatment node (treatment 1).

[0047] Knowledge graph 50 shows multiple drug nodes (Drug 1, Drug 2, ... Drug N) branching from a first treatment node (Treatment 1). Each of these drug nodes corresponds to a different drug associated with a first type of treatment for treating an underlying disease. The one or more drugs represented by the drug nodes may include an investigational drug evaluated for treating the disease in a clinical trial. Additionally or alternatively, the one or more drugs represented by the drug nodes may include a drug approved by a regulatory authority (e.g., the FDA) as effective for treating the disease. For simplicity, knowledge graph 50 shows only the child nodes branching from and related to the first drug (Drug 1).

[0048] The knowledge graph 50 includes multiple adverse event nodes (AE1, AE2, AE3) branching from a first drug node (Drug 1), each corresponding to a respective adverse event associated with the first drug node (Drug 1) associated with a first type of therapy (Treatment 1) for treating the underlying disease. In some examples, the adverse events represented by the AE nodes include preferred terms (PTs) specified in the MedDRA directory. Upon receiving a user input indication indicating selection of a particular adverse event node displayed in the interactive knowledge graph 500b, the interactive knowledge graph 500b can further present detailed information about the particular adverse event node (e.g., related terms, synonyms, and lexical variants of the PT). Optionally, the knowledge graph 50 can include a pathway node branching from the first drug node (Drug 1) that represents a biological pathway related to the first drug node (Drug 1). Although not shown in the example, additional edges can connect the same pathway node to other drug nodes associated with the same or different therapy nodes in the interactive knowledge graph 50. In some examples, the user 102 provides a refined query 402 (i.e., a natural language query or a pre-defined query) requesting the interactive knowledge graph 50 to selectively display or remove certain types of nodes, such as pathway nodes. Similarly, the refined query 402 can be more specific, and the user 102 can instruct the interactive knowledge graph 50 to display only certain types of nodes that branch off from a particular source node (e.g., query 402 to display only AE nodes that branch off from a particular drug node, but not AE nodes that branch off from other drug nodes). Also, in some examples, the user 102 interacts with the interactive knowledge graph 50 by providing a user input instruction indicating a selection of a particular node in the knowledge graph 50, which causes the interactive knowledge graph 50 to display child nodes that branch off from the particular node selected by the user 102. The interactive knowledge graph 50 can receive the user input instruction through an input device.Input devices include, but are not limited to, touch input if the display 118 includes a touch screen, a mouse or stylus, an image capture device that recognizes gestures or gaze direction, or a voice interface.

[0049] Branching from the first AE node (AE1), the knowledge graph 50 includes a first group of one or more patient nodes (Patient A), each representing a patient / participant who experienced a first adverse event during or after treatment with the drug represented by the first drug node. Branching from the second AE node (AE2), the knowledge graph 50 includes a second group of one or more patient nodes (Patient B), each representing a patient / participant who experienced a second adverse event during or after treatment with the drug represented by the first drug node. The first group of one or more patient nodes (Patient A) also branches from the second AE node (AE2), each representing a patient / participant who experienced both the second adverse event and the first adverse event during or after treatment with the drug represented by the first drug node. In the illustrated example, a group of first patient nodes (Patient A) may form a first cluster (e.g., in the embedding space) based on patients / participants sharing a first characteristic / characteristic, and a group of second patient nodes (Patient B) may form a second cluster (e.g., in the embedding space) based on patients / participants sharing a second characteristic / characteristic different from the first characteristic / characteristic. For example, a first AE node (AE1) may indicate an adverse event of hair loss, a second AE node (AE2) may indicate an adverse event of low blood pressure, and each patient / participant represented by the group of first patient nodes (Patient A) may be female (e.g., the first characteristic / characteristic), and each patient / participant represented by the group of second patient nodes (Patient B) may be male (e.g., the second characteristic / characteristic). Here, the interactive knowledge graph 50 may reveal to the user 102 that women who take a first drug (Drug 1) experience hair loss as a side effect, but men who take the first drug (Drug 1) do not experience hair loss. However, both female patients / participants, represented by the group under the first patient node (Patient A), and male patients / participants, represented by the group under the second patient node (Patient B), who take the first drug (Drug 1), experience hypertension, independent of gender.

[0050] As described above in the preceding paragraph, the knowledge graph builder 200 can construct the knowledge graph 50 by determining the embedding value of each node and presenting the nodes in the embedding space such that nodes that are closer to each other in the embedding space are more related than nodes that are farther from each other in the embedding space. Thus, the length (and optionally the direction) of an edge connecting two nodes can indicate how related the two nodes are to each other. As a non-limiting example, if the training healthcare data 300T indicates that nearly all patients / participants who took a particular drug experienced a particular adverse event, the length of the edge connecting the drug node and the adverse event node will be shorter than if only a small percentage of those patients / participants experienced the particular adverse event. For example, the knowledge graph 50 includes nodes representing diseases, targets, drugs, and input tachygenomic data for understanding adverse events. The knowledge graph 50 leverages machine learning to calculate edges between nodes to aid in predicting potential adverse events.

[0051] Continuing to refer to FIG. 5B , the knowledge graph 50 further includes a third group of two patient nodes (Patient C) branching from the third AE node (AE3), each representing a patient / participant who experienced a third adverse event during or after treatment with the drug represented by the first drug node. In this example, the third adverse event may be a fatal adverse event, such as circulatory collapse, that resulted in the death of both of the patients / participants represented by the third group of two patient nodes (Patient C). Based on the long length of the edge connecting the first drug node (Drug 1) and the third adverse event node (AE3) and the fact that only two patients / participants experienced the adverse event, the interactive knowledge graph 50 presented by the UI 500b in FIG. 5B may determine that the third adverse event (e.g., circulatory collapse) is a rare event that may occur in patients / participants taking the first drug represented by the first drug node (Drug 1). However, UI 500b of FIG. 5B allows user 102 to perform inference on knowledge graph 50 to identify possible causes of a third adverse event. Here, user 102 can issue query 402 requesting inference model 400 to identify common characteristics common to two patients / participants represented in a third group of two patient nodes in interactive knowledge graph 50, but not common to a majority of patients / participants represented in the first and second groups of patient nodes in interactive knowledge graph 50. Inference model 400 can traverse the nodes of interactive knowledge graph 50 and determine that both patients (Patient C) represented in the third group of two patient nodes were taking a first drug and a concomitant medication that was not being taken by either of the other participants / patients represented in the other groups of patient nodes (Patient A and B). Inference model 400 can present a summary of this finding via UI 500b and provide a link to the drug label of the concomitant medication. By checking the medication label, the user 102 can learn that circulatory collapse is a known adverse event of the concomitant medication.As discussed above, a user can issue a natural language query 402 to the inference model 400 via the UI 500b, and the inference model 400 can utilize the LLM 400 to return a response 404 that summarizes the information contained in the interactive knowledge graph 50 in response to the query 402. The response 404 can annotate the summarized information with appropriate links that the user can select to view additional information.

[0052] When the interactive knowledge graph 50 receives a user input indication indicating a node selection, it can present detailed information related to the node. For example, the user 102 can select one of the patient nodes and cause the interactive knowledge graph 50 to present detailed information about the patient represented by the selected patient node. The interactive knowledge graph 50 can display a pop-up window that conveys the detailed information. The detailed information can include demographic information about the patient, details about clinical trials in which the patient participated, the Conmed the patient was taking while taking the first drug, any adverse events the patient experienced, and other information of potential interest available in the knowledge graph 50. The interactive knowledge graph 50 can also further annotate some of the detailed information, such as by providing a hyperlink to the source of the detailed information. For example, the interactive knowledge graph 50 can provide at least one of a hyperlink to a clinical trial in which the patient participated, a hyperlink to the patient's lab results or electronic medical record (EMR), or a hyperlink to the drug label for the first drug and the Conmed the patient was taking while taking the first drug.

[0053] FIG. 6 is a flowchart illustrating example operations of a method 600 for creating a knowledge graph 50 from multidimensional health data 300 and performing inference on the knowledge graph 50. The data processing hardware 142 of FIG. 1 executes instructions stored in the memory hardware 144 of FIG. 1 , causing the data processing hardware 142 to perform the operations of the method 600. At operation 602, the method 600 includes receiving multidimensional health data 300 from at least one data source 202, where the multidimensional health data includes unstructured data 300u. The multidimensional health data 300 may also include structured data 300b. At operation 604, the method 600 includes annotating the unstructured data 300u to generate annotated data 300a and processing the annotated data 300a to obtain training healthcare data 300T.

[0054] At operation 606, the method 600 includes training the knowledge graph 50 with the training healthcare data 300T. At operation 608, the method 600 includes receiving a query 402 requesting information associated with the knowledge graph 50. At operation 610, the method 600 includes retrieving the information requested by the query 402 from the knowledge graph 50. The query 402 is received from a user device 110 associated with the user 102, and the method 600 can send / provide a response 404 to the user device 110 conveying the information retrieved from the knowledge graph 50.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Examples of applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, gaming applications, etc.

[0056] Non-transitory memory is a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use in a computing device. Non-transitory memory is volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs), etc. Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), disk or tape, etc.

[0057] 7 is a schematic diagram of an example computing device 700 that can be used to implement the systems and methods described herein. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0058] Computing device 700 includes a processor 710, memory 720, a storage device 730, a high-speed interface / controller 740 connecting to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connecting to a low-speed bus 770 and storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be implemented on a common motherboard or in other manners as desired. Processor 710 is capable of processing instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730 for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 connected to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as desired. Additionally, multiple computing devices 700 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0059] Memory 720 stores information non-transiently within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory, and read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), or electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0060] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be a floppy disk device, a hard disk device, an optical disk device, a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, a computer program product is embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 720, the storage device 730, or memory on the processor 710.

[0061] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. This assignment of roles is merely exemplary. In some implementations, the high-speed controller 740 is connected to memory 720, to a display 780 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 750 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 760 is connected to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be connected to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or to a network device such as a switch or router (e.g., via a network adapter).

[0062] The computing device 700, as shown, can be implemented in a variety of forms, such as a standard server 700a or multiple such servers 700a, a laptop computer 700b, or part of a rack server system 700c.

[0063] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor (either dedicated or general-purpose) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0064] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0065] The processes and logic flows described herein are executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows may also be executed by special-purpose logic circuitry, such as FPGAs (field-programmable gate arrays) and ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operatively coupled to receive data from, transmit data to, or both. However, a computer need not necessarily include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices. This includes, for example, semiconductor memory devices such as EPROMs, EEPROMs, flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0066] To provide for interaction with a user, one or more aspects of the present disclosure can be implemented on a computer that includes a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to a user, and optionally includes a keyboard and pointing device (e.g., a mouse or trackball) for the user to provide input to the computer. Other types of devices can also be used to provide for interaction with a user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0067] While several implementations have been described above, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (600) running on data processing hardware (142), the data processing hardware (142) comprising: receiving multidimensional health data (300) including unstructured data (300u) from at least one data source (202); annotating the unstructured data (300u) to generate annotated data (300a); processing the annotated data (300a) to obtain training healthcare data (300T); training a knowledge graph (50) based on said training healthcare data (300T); receiving a query (402) requesting information associated with the knowledge graph (50); and retrieving the information requested by the query (402) from the knowledge graph (50); A computer-implemented method (600) for performing operations, comprising:

2. the query (402) comprises a natural language query (402); Obtaining the information requested by the query (402) comprises: processing the natural language query (402) by performing query interpretation on the natural language query (402) using an inference model (400) to determine the type of information requested by the natural language query (402); retrieving said information from said knowledge graph (50) based on the type of information requested by said natural language query (402); Including, The computer-implemented method (600) of claim 1.

3. The operation further comprises: generating a natural language summary of the information retrieved from the knowledge graph using the inference model; providing the natural language summary of the information for output from a user device (110); Including, The computer-implemented method (600) of claim 2.

4. The inference model (400) utilizes a large-scale language model to generate the natural language summary of the information. The computer-implemented method (600) of claim 3.

5. The inference model (400) includes a neural network model. The computer-implemented method (600) of claim 2.

6. The operation further comprises: Receiving standard reference data (222) Including, annotating the unstructured data (300u) includes annotating the unstructured data (300u) based on the standard reference data (222); A computer-implemented method (600) according to any one of claims 1 to 5.

7. The operation further comprises: receiving concepts (232) defining an ontology for semantically linking said training healthcare data (300T); Including, training the knowledge graph (50) based on the training healthcare data (300T) includes training the knowledge graph (50) based on the training healthcare data (300T) using the concepts (232); A computer-implemented method (600) according to any one of claims 1 to 6.

8. The operation further comprises: Executing the knowledge controller (150) Includes The knowledge controller (150) is configured to display a user interface (500) for viewing the information obtained from the knowledge graph (50) on a screen (116) of the user device (110). The computer-implemented method (600) of any one of claims 1 to 7.

9. Receiving the query (402) comprises: receiving the query (402) from the user device (110); The query (402) is input by the user via a user interface (500); The computer-implemented method (600) of claim 8.

10. The operation further comprises: executing a knowledge controller (150), the knowledge controller (150) being configured to display a user interface (500) on a screen (116) of a user device (110); Displaying the knowledge graph (50) as an interactive knowledge graph (50) on the user interface (500); Including, A computer-implemented method (600) according to any one of claims 1 to 9.

11. the information requested by the query (402) includes information regarding the safety of a particular drug for treating a disease; A computer-implemented method (600) according to any one of claims 1 to 10.

12. A system (100), comprising: data processing hardware (142) and memory hardware (144); the memory hardware (144) is in communication with the data processing hardware (142) and stores instructions that, when executed by the data processing hardware (142), cause the data processing hardware (142) to perform operations; The operation is receiving multidimensional health data (300) including unstructured data (300u) from at least one data source (202); annotating the unstructured data (300u) to generate annotated data (300a); processing the annotated data (300a) to obtain training healthcare data (300T); training a knowledge graph (50) based on said training healthcare data (300T); receiving a query (402) requesting information associated with the knowledge graph (50); retrieving the information requested by the query (402) from the knowledge graph (50); Including, System (100).

13. the query (402) comprises a natural language query (402); Obtaining the information requested by the query (402) comprises: processing the natural language query (402) by performing query interpretation on the natural language query (402) using an inference model (400) to determine the type of information requested by the natural language query (402); retrieving said information from said knowledge graph (50) based on the type of information requested by said natural language query (402); Including, The system (100) of claim 12.

14. The operation further comprises: generating a natural language summary of the information retrieved from the knowledge graph using the inference model; providing the natural language summary of the information for output from a user device (110); Including, The system (100) of claim 13.

15. The inference model (400) utilizes a large-scale language model to generate the natural language summary of the information. The system (100) of claim 14.

16. The inference model (400) includes a neural network model. The system (100) of claim 13.

17. The operation further comprises: Receiving standard reference data (222) Including, annotating the unstructured data (300u) includes annotating the unstructured data (300u) based on the standard reference data (222); A system (100) according to any one of claims 12 to 16.

18. The operation further comprises: receiving concepts (232) defining an ontology for semantically linking said training healthcare data (300T); Including, Training the knowledge graph (50) based on the training healthcare data (300T) includes training the knowledge graph (50) based on the training healthcare data (300T) using the concepts (232). Including, A system (100) according to any one of claims 12 to 17.

19. The operation further comprises: Executing the knowledge controller (150) Including, The knowledge controller (150) is configured to display a user interface (500) for viewing the information obtained from the knowledge graph (50) on a screen (116) of the user device (110). A system (100) according to any one of claims 12 to 18.

20. Receiving the query (402) includes receiving the query (402) from the user device (110) input by the user via the user interface (500). Including, 20. The system (100) of claim 19.

21. The operation further comprises: executing a knowledge controller (150), the knowledge controller (150) being configured to display a user interface (500) on a screen (116) of a user device (110); Displaying the knowledge graph (50) as an interactive knowledge graph (50) on the user interface (500); Including, A system (100) according to any one of claims 12 to 20.

22. the information requested by the query (402) includes information regarding the safety of a particular drug for treating a disease; A system (100) according to any one of claims 12 to 21.

Images