A system and method for extracting and interpreting information using a large language model system

Abstract

A large language model system is disclosed. In one arrangement the system comprises: a data processing system configured to implement a first large language model, a second large language model, and a third large language model, wherein the data processing system is configured to: receive inquiry data representing a user inquiry; input at least the inquiry data to the first large language model and receive a first data output as a corresponding output from the first large language model; input at least the first data output and a knowledge graph to the second large language model and receive a second data output as a corresponding output from the second large language model; input at least the first data output, the second data output, and the inquiry data to the third large language model and receive a third data output as a corresponding output from the third large language model; and output the third data output as an answer to the user inquiry.

Description

[0001]A SYSTEM AND METHOD FOR EXTRACTING AND INTERPRETING INFORMATION USING A LARGE LANGUAGE MODEL SYSTEM The present disclosure relates to a system and method that can be used to monitor patients and ensure they receive appropriate care by improving the accuracy of answers generated by large language models (also referred to as LLMs), as well as reducing the chances of generated answers containing false information. Large language models (LLMs), such as ChatGPT and GPT-4 have gained extensive recent attention due to their impressive capabilities in understanding and generating human language. Therefore, the adoption of LLMs to assist physicians in clinical decision-making, as well as decision making in other fields, has received growing research interest in both communities of artificial intelligence and clinical medicine. However, applying LLMs in healthcare remains a challenging undertaking due to the lack of faithfulness and evidence in the generated content. Faithfulness refers to the fact that LLMs tend to produce hallucinations. A hallucination is generated content that looks reasonable but is not based on factual information and knowledge. An example of an hallucination would be an LLM suggesting a drug used to slow the progress of a disease is actually used to treat joint pain. Such hallucinations can be harmful in healthcare decision- making scenarios. The traceability of evidence is another important requirement when using an LLM to assist with decision making. In healthcare, it is crucial for LLMs to show the source of generated content (referred to as evidence), which can be used to explain why the LLMs give such recommendations. However, existing LLMs often lack the capability to provide clear evidence of the source behind the provided information. Clearly identifying a source can also be used to ensure that the generated content is not the result of a hallucination. Such challenges hinder the adoption of LLMs in healthcare settings, as well as other decision-making use cases. Transparency, reliability, and trustworthiness are critical in achieving trust from a user looking to receive advice from an LLM. It is an object of the present disclosure to at least partly address one or more of the challenges discussed above. According to an aspect of the disclosure, there is provided a large language model system, comprising a data processing system configured to implement a first large language model, a second large language model, and a third large language model, wherein the data processing system is configured to: receive inquiry data representing a user inquiry; input at least the inquiry data to the first large language model and receive a first data output as a corresponding output from the first large language model; input at least the first data output and a knowledge graph to the second large language model and receive a second data output as a corresponding output from the second large language model; input at least the first data output, the second data output, and the inquiry data to the third large language model and receive a third data output as a corresponding output from the third large language model; and output the third data output as an answer to the user inquiry. The above system allows for a general purpose LLM to be adapted to a specific domain, as an alternative to training a new model. The knowledge graph and first data output allow for an existing model to be guided to produce the desired third output. Training a model can be very expensive and time consuming, so using an existing model is preferable. Equally, a general model will have a wider variety of available training data, leading to improved performance. For example, the content generated by a general LLM may be more easily interpretable. The use of three specific LLMs allows for each model to be specialised to perform a specific task. The model can therefore generate better results than if only a single model was used. Each LLM can receive more specific and detailed instructions, leading to better performance for that specific task. A single LLM would lead to worse performance due to the increased complexity of instructions that would be provided. Optionally, the first large language model is configured to generate the first data output by identifying and extracting keyword data representing keywords from the inquiry data dependent on whether the keywords are relevant to one or more predetermined target categories. Optionally, the target categories comprise: diseases; symptoms; and drugs. Extracting relevant keywords, such as the diseases, symptoms and drugs mentioned in the user inquiry, allows for these keywords to be provided to later LLMs. The later LLMs can therefore be directed to specific tasks, without first having to identify what information from the user inquiry is relevant. Reducing the complexity of a task provided to an LLM generally leads to improved performance on that task. Optionally, the second large language model is configured to generate the second data output by extracting information from the knowledge graph that is relevant to the keywords represented by the extracted keyword data and providing the extracted information as a part of the second data output. Extracting relevant knowledge from the knowledge graph allows for a general model to perform well on domain specific queries, as the knowledge required to answer the query is extracted from the knowledge base. Providing the extracted knowledge to the third LLM allows it to generate an answer that incorporates the extracted knowledge, rather than using an LLM that is trained on domain specific training data. Optionally, the knowledge graph represents information from one or more databases as nodes and edges, optionally wherein the one or more databases contain medical knowledge. Using a graph of nodes and edges allows for the information contained within the one or more databases to be more easily traversed and accessed. Optionally, each node includes location data representing a location in the one or more databases where the information represented by the node is located, and wherein the second large language model is further configured to include the location data of a node in the second data output if the extracted information from the knowledge graph includes the information represented by that node. Optionally, the third large language model is configured to include the location data corresponding to any extracted information used to generate the third data output within the third data output. Including the location data representing the location in the databases within the output provided to the user allows for the user to then easily verify the authenticity of the information, increasing trust in the output generated by the system. Ensuring that the information is accurate is especially relevant if the answer is going to be used to assist with decision making. For example, in the medical domain, the answer may be used to diagnose or treat a patient. Including the source allows for clinicians to easily evaluate the source of the content, improving the credibility of the model. If the user requires more information on the topic, but is unsure of an exact query, the user is able to easily find the location of the information in the one or more databases. For example, if further background information about a disease that a patient had been diagnosed with was required, the database entry could be easily located and consulted. Optionally, the nodes comprise: disease nodes corresponding to respective diseases; and drug nodes corresponding to respective drugs. Optionally, each disease node is associated with a symptom node embedding obtained by encoding a symptom textual description of one or more symptoms associated with the respective disease, optionally including the primary symptoms of the disease. Optionally, each drug node is associated with a disease node embedding obtained by encoding a disease textual description of one or more diseases treatable by the respective drug, optionally including the primary diseases the drug can treat. Optionally, the edges in the knowledge graph represent similarity between respective pairs of nodes, optionally a cosine similarity, optionally a normalized cosine similarity. The knowledge graph will therefore capture the relationships between diseases, symptoms and drugs. Associating each node with an encoding of the textual information that is relevant to that node, such as the symptoms associated with a disease or the diseases treatable by a drug, allows for the nodes to be compared to determine their similarity. The nodes can be compared based on the similarity of their encodings. Two nodes that have a similar textual description, for example a disease and a drug that both discuss the same symptom, will therefore have a high similarity. The knowledge graph is therefore able to determine the relationships between the nodes and represent it in a form that is more easily traversable than the original textual description would be. Optionally, the edges are limited to include only edges connecting a disease node to a drug node, and a drug node to a disease node. Optionally, the edges from each node are limited to include only the Kth most similar nodes, where K is an integer, optionally equal to or less than 10. Limiting the number of edges reduces the size of the knowledge graph, and so reduces the amount of memory required to store the knowledge graph. As edges between drugs and edges between nodes that are not similar are unlikely to represent useful information, removing these nodes does not impact the usefulness of the knowledge graph. Optionally, the data processing system is further configured to: generate, using the knowledge graph, a set of training data comprising a plurality of tuning prompts; and tune, using the set of training data, the second large language model to adapt the second large language model to a downstream task. Optionally, each of the tuning prompts comprises an example input and a target output, wherein the example input comprises an example user inquiry and keyword data, and the target output comprises the desired output for that specific input. Optionally, the target output comprises a list of knowledge categories indicating what categories of knowledge should be extracted from the knowledge graph, and a list of extracted information that has been extracted from the knowledge graph for each keyword represented in the keyword data, wherein the information includes the knowledge category that each piece of extracted information belongs to. The second LLM is therefore provided with example inputs, and example outputs that should then be generated as a result. Training the second LLM 8 on the set of training data allows for a generic large language model to be adapted to a specific domain and task, improving performance. The inclusion of the knowledge categories indicate what types of knowledge the second LLM should visit based on the input. Additionally, including an example of the extracted information allows for the second LLM to be trained on how to accurately extract information and how it should then be presented in the generated output. Tuning the second LLM with the tuning prompts therefore improves performance and ensures that the answer generated takes the proper form. Optionally, the third large language model is configured to generate the third data output using only the first data output, the second data output, and the user inquiry. Limiting the third large language model to only use the information provided improves faithfulness and / or reduces hallucinations, since only the extracted information, which comes from a verifiable source, is used when generating the third output. The third large language model will not hallucinate information to answer the user inquiry. Optionally, the data processing system is further configured to input a prompt instructing the first large language model, second large language model and / or third large language model, wherein the prompt comprises a chain of sub-prompts representing intermediate steps to be taken to arrive at the first output, second output, and / or third output respectively from the received inputs. Providing a chain of steps to follow, rather than a single prompt, allows for a complex problem to be approached logically. The large language model is guided along the series of steps to be taken to arrive at the answer. In contrast to the traditional single prompt, e.g., ‘Your task is to answer multiple-choice questions’, providing a chain of steps can imitate the human thought process when solving complex problems involving multi-step logical reasoning. As a result, a coherent series of intermediate reasoning steps are provided that can guide the model through the process to get to the final answer step by step. The model therefore outputs a more reasoned, accurate, and logical answer. Embodiments of the disclosure will be further described by way of example only with reference to the accompanying drawings. Fig. 1 is a flow chart illustrating the steps of a method of extracting and interpreting information using three LLMs. Fig. 2 is a diagram illustrating an example configuration of the three LLMs configured to extract information in a medical context, the inputs provided to the LLMs, and the outputs the LLMs produce. Fig. 3 is a diagram of a knowledge graph constructed from several drug, disease and symptom knowledge bases. Fig. 4 is a table showing the performance of different LLM techniques, with a higher value meaning a more accurate performance. Fig. 5 shows graphs demonstrating performance of a system when different percentages of the knowledge base are used. Fig. 6 is a table showing the ability of different LLM techniques to reason, through the use of a zero-shot learning setting. The present disclosure relates to a large language model (LLM) system. Fig. 1 is a flow chart illustrating example steps that the LLM system is configured to perform. Fig. 2 shows a configuration of three LLMs 6, 8, 10 that the LLM system comprises. Fig. 2 also shows inputs provided to the three LLMS 6, 8, 10, and an output provided to fulfil a downstream task 12. The LLM system comprises a data processing system configured to implement the first LLM 6, the second LLM 8, and the third LLM 10. The data processing system may be configured to perform the steps illustrated in Fig. 1. Each LLM 6, 8, 10 is configured to receive one or more inputs and provide one or more outputs. The three LLMs 6, 8, 10 may collaborate with each other to ensure that the generated content is accurate and supported by reliable evidence. An example configuration of the three LLMs 6, 8, 10 is shown in Fig. 2 and described below. In this example configuration, the LLMs 6, 8, 10 are configured to operate in the medical domain. The downstream task 12 in this context may, for example, comprise providing a drug recommendation. LLMs typically rely on a large-scale pre-training strategy. During the pre- training, an LLM may be trained on a massive corpus of open domain text (e.g., Wikipedia articles and / or books etc.) in an unsupervised or self-supervised learning manner. Widely used training objectives are Masked Language Modelling (MLM) and Autoregressive Language Modelling (ALM).Given a sequence of input text, such as medical text, {^^^, ^^ଶ, … , ^^ே} with N words,where ^^^denotes the ^^௧^word in the sequence, MLM would randomly mask out a certain percentage (e.g., 15%) of the input words, obtaining a sequence of masked words andunmasked words (^^^, ^^\^). Then, the LLM is tasked with predicting the randomlymasked words ^^^based on the remaining unmasked words ^^\^. Therefore, the MLM losscan be defined as follows^^ெ^ெ = − ^ log(^^ఏ(^^^ |^^\^)) (1)௪^where p and θ denote the word probability and the model parameters, respectively. The masked tokens are predicted as a classification problem by selecting one word from the vocabulary. As a result, the MLM training objective can force the LLM to accurately understand the context, and reason about the missing information. The MLM training objective is widely used in BERT-like LLMs. Autoregressive Language Modelling (ALM) is another popular training objective widely used in large language models, e.g., ChatGPT and GPT-3. In implementations,given the context of the previous words ^^^:௧ି^ = {^^^, ^^ଶ, … , the LLM is trained topredict the next word ^^௧. Therefore, the ALM loss can be defined as In this way, ALM can enable the LLM to capture the dependencies and context within the text by modelling the conditional probability distribution of each word in the sequence. The ALM has been widely used in GPT-like LLMs to achieve great success. During inference, given the input sequence, e.g., user inquiry, ^^ = ^^^:ே ={^^^, ^^ଶ, … , ^^ே}, an LLM can generate the subsequent words using the following equation: , where ∼ denotes maximum probability sampling from the vocabulary. By iteratively usingthe above equation, the LLM can output a sequence ^^ = {^^^, ^^ଶ, … , ^^ெ} to the inputsequence, where M denotes the length of the output sequence. The output sequence can be taken as a response to the input user inquiry. The three LLMs 6, 8, 10 of the LLM system of the present disclosure may be trained using either of these objectives. Alternatively, a different training objective may be used. The data processing system and a corresponding method of extracting and interpreting information will now be described further with reference to Fig. 1. The processing system is configured to receive inquiry data representing a user inquiry from a user of the system, and thereby implement step S1 of Fig. 1. The inquiry data may relate to the application of drugs and medicine in a healthcare scenario. As an example, the inquiry data may comprise a request for an identification of a drug that can be used to treat identified symptoms. In such a case, the data processing system may recommend the most appropriate drug. The data processing system may take account of a patient’s individual diseases and symptoms. As a further example, the inquiry data may comprise a request for a calculation of a safe dosage for an identified drug. In such a case, the data processing system may recommend a safe dosage. The data processing system may individualise a recommended dosage to the patient making the request to provide maximum efficacy while minimising the occurrence of adverse effects and side effects. As a further example, the inquiry data may comprise a request to identify patterns of adverse reactions associated with use of an identified drug. In such a case, the data processing system may take into account individual characteristics of the patient and properties of the drug to identify patterns of adverse reactions associated with drug use. As a further example, the inquiry data may comprise a request to predict an interaction of two identified drugs when administered in combination. In such a case, the data processing system may use knowledge of different drugs to predict effects of drug combinations. This may provide physicians with information that allows them to better understand interactions between different drugs and to provide guidance for individualised treatment planning. As a further example, the inquiry data may comprise a request to answer a pharmacology related question. In such a case, the data processing system may answer the pharmacology-related question accurately by extracting comprehensive knowledge from existing knowledge databases, which supports decision-making and medication management. The data processing system is configured to input at least the inquiry data to the first LLM 6 and receive a first data output as a corresponding output from the first LLM 6, thereby implementing step S2 of Fig. 1. The first LLM 6 may be configured to generate the first data output by identifying and extracting keyword data from the inquiry data 2. The keyword data is extracted dependent on whether the keywords are relevant to one or more predetermined target categories. The target categories are selected to be specific to an expected domain of the inquiry data 2. The target categories may relate to areas of knowledge that are expected to be useful in answering the user inquiry 2. The first LLM 6 may be referred to as an inquiry analysis LLM 6. In the context of using the method to answer medical related inquiries, the target categories may comprise diseases, symptoms and drugs. As an example, in a case where the user inquiry lists a set of symptoms, and a set of drugs and requests that the best drug to treat the set of symptoms be identified, the first LLM 6 may identify the symptoms and drugs as keywords. Optionally, the data processing system may be further configured to input a task to the first LLM 6. The task may define the target categories. The task may define a format for the first data output. An example of a task provided to the first LLM 6, as well as an example user inquiry and corresponding first data output are shown in Table 1. The first LLM 6 determines what knowledge is required to answer the inquiry. Table 1 Input Task: You are tasked with identifying the drug, symptom, and disease from user inquiry: Drugs: List any. If none, your answer for this section should be ’[]’. Answer should be in form ’[drug a, drug b, ...]’. Symptoms: List any. If none, your answer for this section should be ’[]’. Answer should be in form ’[symptom a, symptom b, ...]’. Diseases: List any. If none, your answer for this section should be ’[]’. Answer should be in form ’[disease a, disease b, ...]’. User Inquiry: Steroid refractory ulcerative colitis is said to be present in a patients who have active disease despite: A:Prednisolone up to 0.5 mg / kg / day over a period of 4 weeks; B: Prednisolone up to 0.75 mg / kg / day over a period of 4 weeks; C: Prednisolone up to 1 mg / kg / day over a period of 4 weeks; D: Prednisolone up to 1.5 mg / kg / day over a period of 4 weeks. Output Drug: [‘Prednisolone’] Symptom: [’Steroid refractory, active disease’] Disease: [’ulcerative colitis ’] A unified prompting method may be used to adapt general purpose LLMs to a specific domain, such as the medical domain. The first LLM 6 may be implemented based on this approach using a general purpose LLM, such as InstructGPT (Ouyang, L. et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. 652 Syst. 35, 27730–27744 (2022)). The unified prompting method enables the general purpose LLM to adapt efficiently to the medical domain. Adapting a general purpose LLM may result for example in improved extraction of relevant medical information from the first LLM 6, which may improve generation of responses that guide subsequent LLMs (e.g., the second LLM 8 and the third LLM 10). Using a general purpose LLM prevents the need to train a specific LLM for the task, which can be both time consuming and expensive. To adapt deep learning models to a specific domain, transfer learning and domain adaptation are both widely used techniques. However, both techniques usually rely on a large volume of downstream data for end-to-end fine-tuning. In embodiments of the present disclosure, a few-shot prompting method is used instead. The few-shot prompting method enables the first LLM 6 to adapt efficiently to specific domains with very limited downstream data, e.g., few-shot examples, which can be viewed as demonstrations of the target downstream task that the first LLM 6 is required to carry out. A few-shot example contains an example input, and the desired output that should be produced for that input. The few-shot examples are then provided to the first LLM 6 along with the current user inquiry, with the few-shot examples illustrating to the first LLM 6 how the task should be performed and what the output should look like. Table 1 shows an example of a few-shot prompt, with the user inquiry and corresponding output forming a single few-shot prompt. A prompt instructing the first LLM 6 what to do may also be provided as input. Using a general purpose LLM and fine tuning it to a specific domain is more efficient than training a new model from scratch. Each few-shot prompt may be formed of an input and an output. The input may comprise a task and a user inquiry. The task may therefore be repeated for each few-shot prompt. Following the above strategy, given few-shot prompts ^^ = ^^^:^ = {^^^, ^^ଶ, … , ^^^}and the current user inquiry ^^ = ^^^:ே = {^^^, ^^ଶ, … , ^^ே}, a full input sequence X of the firstLLM 6 may be provided as a concatenation of a user inquiry and the prompts, e.g.,^^ = ^^^; ^^^ = {^^^, ^^ଶ, … , ^^^; ^^^, ^^ଶ, … , ^^ே} (4)The total length of the input sequence in this case is ^^ + ^^. Then, the output sequence ofthe first LLM 6 can be generated by equation (3). Based on effective few-shot prompting, the first LLM 6 may be configured to use Chain-of-Thought (CoT) prompting to improve the first data output by analysing user inquiries more accurately, for example as described in Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. 654 Syst. 35, 24824–24837 (2022). CoT prompting involves using multiple related sentences to imitate the human thought process and provides the model with a series of intermediate reasoning steps. The first LLM 6 may be configured specifically to identify and extract keywords related to diseases, symptoms, and / or drugs. Extracting such keywords supports accurate answering of inquiries in the medical domain. The second LLM 8 may be provided by training an LLM on a knowledge graph 13, depicted in Fig. 3, constructed from one or more knowledge bases 4. The data processing system is further configured to input at least the first data output and the knowledge graph 13 to the second LLM 8, as shown in step S3 of Fig. 1. The data processing system then receives a second data output as corresponding output from the second LLM 8. The second LLM 8 may be referred to as a knowledge acquisition LLM 8. The second LLM 8 is configured to capture potentially relevant information from the knowledge bases 4. The knowledge bases 4 store information that may be relevant to answering the user inquiry. The one or more knowledge bases 4 may comprise third party and / or publicly available databases, such as: ‘Drugs.com’, the UK National Health Service, PubMed, or any other suitable medical knowledge database. The second LLM 8 provides evidence that can be used by the third LLM 10. The second LLM 8 takes the output of the first LLM 6 as input and provides the real knowledge for the third LLM 10 to accurately answer the user inquiry. Therefore, the objective of the second LLM 8 is to extract appropriate knowledge from the knowledge bases 4. The second LLM 8 may be configured to generate the second data output by extracting information from the knowledge graph 13 that is relevant to the keywords represented by the extracted keyword data and providing the extracted information as a part of the second data output. This extracted information may then be used to assist the third LLM 10 with accurately answering the user inquiry. The knowledge graph 13 is constructed to help the second LLM 8 to better explore the knowledge located within the knowledge bases. The knowledge graph 13 is preconstructed to represent all diseases and drugs stored within the knowledge bases as nodes, and the relationships between them as edges. The knowledge graph 13 may thus represent information from one or more third party and / or publicly available databases, such as one or more databases containing medical knowledge, as mentioned above. A plurality of categories of knowledge may be linked to the nodes. The plurality of categories of knowledge may comprise one or more of the following categories of drug knowledge: drug description and indication; drug dosage recommendation; drug adverse effects; drug toxicity; drug-food interactions; drug-drug interactions; drug pharmacodynamics; and pubmed experimental summaries. The plurality of categories may alternatively or additionally comprise one or more of the following categories of disease knowledge: common symptoms; disease causes; disease diagnosis; disease treatment; and disease complications. The knowledge categories are linked to each node in the graph, such that each drug node 14 may contain information falling within one or more of the drug knowledge categories relating to that specific drug, and each disease node 16 may contain information falling within one or more of the disease knowledge categories relating to that specific disease. Each node may include location data representing a location in the knowledge database where the information represented by the node is located. The second LLM 8 may be further configured to include the location data of a node in the second data output if the extracted information from the knowledge graph 13 includes the information represented by that node. Any information extracted from the knowledge graph 13 will therefore have a reference to the location within the original databases that the information was sourced from. The third LLM 10 may be configured to include the reference corresponding to any extracted information used to generate the third data output within the third data output. Providing a reference to where the information used to answer the user inquiry was originally located allows for the authenticity of the information to be determined by checking the source. Ensuring that information is accurate is especially relevant in the medical domain, where the generated answer may be used to determine how a patient is treated or what they are diagnosed with. Including the source of the evidence used to arrive at the answer can enable clinicians to easily inspect the source of the generated content (i.e., evidence) and evaluate the providence of the presented information, thus significantly improving the credibility of model generated content (e.g., an output provided to the user by the LLM system). Including the source of the knowledge allows the model to generate evidence- traceable content in its answers. Equally, if further information is desired without the user having a specific inquiry in mind, the source of the information can be examined. For example, if further background information was desired for a drug that was being considered for use in a treatment, the reference could be used to further review the drug. The knowledge graph 13 may be a disease-symptom-drug graph wherein the nodes comprise disease nodes 16 corresponding to respective diseases and drug nodes 14 corresponding to respective drugs. The diseases-symptom-drug graph may capture the relationships between drugs, diseases and symptoms. Each disease node 16 may be associated with a symptom node embedding obtained by encoding a symptom textual description of one or more symptoms associated with the respective disease, optionally including the primary symptoms of the disease. The symptom textual description may be extracted from the knowledge bases. Each drug node 14 may be associated with a disease node embedding obtained by encoding a disease textual description of one or more diseases treatable by the respective drug, optionally including the primary diseases the drug can treat. The disease textual description may be extracted from the knowledge bases. The symptom node embedding and the drug node embedding may be obtained by extracting the textual descriptions and generating an embedded form to represent them, such as a vector. The embedded form allows for a node to be compared to another node, to determine the similarity of the two nodes. The MiniLM text encoder (Wang,W. et al. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers (2020)) may be used to obtain the node embeddings. The edges in the knowledge graph may represent similarity between respective pairs of nodes. Optionally the edges may represent a cosine similarity between nodes. Optionally the edges may represent a normalized cosine similarity. As an example, for theset of disease nodes ^^ = {^^^, ^^ଶ, … , ^^^} containing all disease nodes 16 associated withsymptoms extracted from the user inquiry and the set of all drug nodes ^^ … , ^^^},given a disease node 16 ^^^ ∈ ^^ and a drug node 14 ^^^ ∈ ^^, the distance between the twonodes may be calculated using: where the ^·,·^ denotes the cosine similarity of the embedding representation of the nodesand ^^ = 1.0 is a temperature hyperparameter. Therefore, the edge weights are calculatedby the normalised similarity of different nodes. Similarly, given the drug node 14 ^^^ ∈ ^^, the distance between the drug node 14and all the disease nodes ^^ = {^^^, ^^ଶ, … , ^^^} that can be treated by the drug correspondingto the drug node 14 ^^^ can be computed using: The edges generated by the system may be limited to include only edges connecting a disease node 16 to a drug node 14, and a drug node 14 to a disease node 16. Equally, the K most relevant drugs with the top-K similarity may be selected as diseaseproximity nodes ^^^ = {^^^^, ^^^ଶ, … , ^^^^}. The edges of the knowledge graph may then begenerated such that the edges from each node are limited to include only the Kth most similar nodes, where K is an integer, optionally equal to or less than 10, and optionally equal to 5. Limiting the amount of edges to only the most relevant reduces the size of the knowledge graph, meaning that the knowledge graph can be fully explored in a shorter time frame and that less memory is required to store the knowledge graph. Diseases can often be treated with a wide range of drugs. The use of the hyperparameter K to reduce the number of drugs considered further prevents the system from being overwhelmed by a large number of available drugs and enables it to prioritize the most relevant drugs for treatment. When limiting the number of drugs and diseases considered to K, the distancescan be computed using equations (7) and (8):(7)(8) The second LLM 8 may receive as input: a set of node embeddings gd, containing the symptom node embeddings and disease node embeddings associated with disease nodes 16 and / or drug nodes 14 that represent diseases and / or drugs in the keyword data identified and extracted by the first LLM 6; a global graph embedding ga, obtained by averaging all of the node embeddings; a prompt p, that describes how the second LLM 8 should generate the expected output based on the user inquiry; and the inquiry data. The combination of ga, and gdis referred to as g. The global graph embedding may be the average of a set of graph embeddings gi, wherein each node i is represented using the convolution of features of the nodes connected to the ith node. The graph embedding ^^,the task instruction P, and the inquiry data S are combined together as the input, i.e., ^^ ={^^; ^^; ^^} and provide the second LLM 8 with the information required to extract anyrelevant knowledge that may be used in answering the user inquiry. A graph convolution network is used to generate the graph embedding gifor the ithnode. The graph convolution network creates a convolution of all of the nodes connected to the ithnode. The nodes that are considered to be connected may only be the immediate neighbours that have an edge joining the two nodes. Alternatively, two nodes may be considered to be connected if they share a neighbour. Alternatively, two nodes may be considered to be connected if the one node can be reached from the other within a given number of edge transitions. The graph embedding githen represents the ithnode based on the nodes that are connected to the node. The prompt p describes the details of how to accurately generate the expected output based on the user inquiry. The prompt p may instruct the second LLM 8 to first identify which of the plurality of categories of knowledge are needed, and to extract information from the knowledge graph in the identified categories of knowledge. The extracted information may relate to the inquiry data. The prompt p may describe the task as a step by step process. An example of a prompt that could be provided is shown in table 2. Table 2 Instructions Task: You are tasked with extracting the knowledge to answer a medical inquiry accurately. Step 1: Identify the categories of knowledge needed (List the numbers corresponding to the knowledge categories necessary) to answer the inquiry correctly. If none, your answer for this section should be ’[]’. Answer should be in form ’[1, 2, 3, ...]’. The knowledge categories of drugs are: 1. Drug description and indication. 2. Drug dosage recommendation. 3. Drug adverse effect. 4. Drug toxicity. 5. Drug-food interaction. 6. Drug-drug interaction. 7. Drug pharmacodynamics. 8. Pubmed experimental summaries. The knowledge categories of diseases and symptoms are: 1. Common symptoms. 2. Disease causes. 3. Disease diagnosis. 4. Disease treatment. 5. Disease complications. Step 2: Extract the specific knowledge from the identified knowledge categories to answer the inquiry correctly. If none, your answer for this section should be ’[]’. Answer should be in form ’[knowledge a, knowledge b, ...]’. The performance of the second LLM 8 at extracting relevant knowledge from the knowledge graph can be further improved through the use of knowledge-based instruction prompt tuning to adapt the generic large language model used for the second LLM 8 to a specific downstream task. Knowledge-based instruction prompt tuning uses a very small number of tuning prompts, including pairs of input and output, to adapt the LLM to the downstream task by allowing the second LLM 8 to be trained on a number of specific examples. These tuning prompts may be generated using the knowledge graph. An example of the construction of a set of training data used to train the second LLM 8 to improve performance at a downstream task is now described. The data processing system may be further configured to generate, using the knowledge graph, a set of training data. The set of training data may comprise a plurality of tuning prompts. The data processing system may be further configured to tune, using the set of training data, the second LLM 8 to adapt the second LLM 8 to the downstream task. The tuning process may take place before any input is received by the second LLM 8. Each of the tuning prompts may comprise an example input and a target output. The example input and target output represent an example of what the second LLM 8 may receive as input, and the output that should then be generated as a result. Training the second LLM 8 on the set of training data using the Autoregressive Language Modelling training objective allows for a generic large language model to be adapted to a specific domain and task, improving performance. The example input comprises an example user inquiry, as well as a set of keywords extracted by the first LLM. The target output comprises the desired output for that specific example input. Since the objective of the second LLM 8 is to navigate the knowledge graph and extract the final knowledge, the target output may comprise a list of knowledge categories indicating what categories of knowledge should be extracted from the knowledge graph, a list of extracted information that has been extracted from the knowledge graph for each keyword, and the knowledge category that the extracted information belongs to. The knowledge categories indicate what types of knowledge the second LLM 8 should visit. Example knowledge categories include one or more of: Drug description and indication; Drug dosage recommendation; Drug adverse effect; Drug toxicity; Drug-food interaction; Drug-drug interaction; Drug pharmacodynamics; Pubmed experimental summaries, and / or: Common symptoms; Disease causes; Disease diagnosis; Disease treatment; Disease complications. The target output further includes the extracted information and the knowledge category that it belongs to. Including an example of the extracted information allows for the second LLM 8 to be trained on how to accurately extract information and how it should be presented. An example of the inputs and outputs of the second LLM 8 is shown in Table 3. Table 3 Input Instructions User Inquiry: Steroid refractory ulcerative colitis is said to be present in a patient who has active disease despite: A: Prednisolone up to0.5mg / kg / dayover a periodof 4weeks; B: Prednisolone up to0.75mg / kg / dayover a periodof 4weeks; C: Prednisolone up to1mg / kg / dayover a periodof 4weeks; D: Prednisolone up to1.5mg / kg / dayover a periodof 4weeks. Drug: [‘Prednisolone’] Symptom: [’Steroid refractory, active disease’] Disease: [’ulcerative colitis ’] Output Knowledge categories: Drugs[1, 2], Diseases-symptoms[4]. Knowledge Drug 1 (Drug description and indication): Prednisolone: Prednisolone is a corticosteroid that may be used to reduce inflammation and calm down an overactive immune system… Link[https: / / www.drugs.com / prednisolone.html] Knowledge Drug 2 (Drug dosage recommendation): Prednisolone: Applies to the following strengths: tebutate 20mg / mL; sodiumphosphate 20mg / mL; 15mg / 5mL; 5mg; (as sodiumphosphate) 5mg / 5mL; sodiumphosphate 15mg / 5mL ... Link[https: / / www.drugs.com / dosage / prednisolone.html] Knowledge Diseases-symptoms 4 (Disease treatment): Ulcerative Colitis: ... Corticosteroids, such as prednisolone, are an alternative medicine to reduce inflammation They can be used with or instead of 5-ASAs to treat a flareup if 5-ASAs alone are not effective ... Link[https: / / www.nhs.uk / conditions / ulcerative- colitis / treatment / ] The data processing system is further configured to input at least the first data output, the second data output, and the inquiry data to a third LLM 10 and receive a third data output as a corresponding output from the third LLM 10, as shown in step S4 of Fig. 1. The system is then configured to output the third data output as an answer to the user inquiry, as shown in step S5 of Fig. 1. The third LLM 10 may be referred to as an evidence generation LLM. The third LLM 10 is responsible for interacting with the user and is configured to generate a final answer to the inquiry based on the identified evidence extracted by the second LLM 8. An existing LLM, such as the previously mentioned InstructGPT may again form the basis of the third LLM 10. A knowledge-consistency prompting technique may be used to improve the performance of the third LLM 10. As part of the knowledge consistency prompting technique, the third LLM 10 may be configured to generate the third data output using only the first data output, the second data output, and the user inquiry. The prompt provided to the third LLM 10 may further include a ‘to do’ list instructing the third LLM 10 to only use the provided knowledge when answering the user inquiry. The prompt may further include a ‘not to do’ list instructing the LLM not to make assumptions not supported by the provided content and to avoid providing personal opinions, out-of-context knowledge or interpretations. The knowledge-consistency prompting technique improves faithfulness and / or reduces hallucinations, since only the extracted information, which comes from a verifiable source, is used when generating the third output. An example of a knowledge- consistency prompt that could be used is shown in Table 4, which also shows an example of chain-of-thought prompting and evidence-traceable prompting. Table 4 Prompting Strategy Prompts Chain-of-Thought Prompting Task: Your task is to answer multiple-choice questions. Understand the question, analyze it, and provide a concise and accurate answer. Answer Format: Analysis: Provide an analysis that logically leads to the answer based on the relevant information. Final Answer: Provide the final answer, which should be a single letter in the alphabet representing the best option among the multiple choices provided in the question. Knowledge-Consistency Prompting Among the provided choices, choose the one that best fits the criteria below: TO DO: Only use the knowledge provided to answer the inquiry NOT TO DO: 1. Do not make assumptions not supported by the provided content. 2. Avoid providing personal opinions or interpretations. 3. Summarize and interpret the knowledge provided objectively and accurately Evidence-Traceable Prompting When analyzing each choice, include the relevant knowledge relied upon and display its source link (provided as Link[https: / / ...]) to the relevant part of your output. Alternatively, or additionally, the prompt can instruct the third LLM 10 to include within the third output the location data representing the location in the one or more databases for any information used by the third LLM 10. As shown in Table 4, the prompt can also indicate the structure that the location data should be provided in. Including the location data allows for clinicians to easily inspect and evaluate the source of the generated content, significantly improving the credibility of content generated by the model. The data processing system may be further configured to input a prompt instructing the first LLM 6, second LLM 8 and / or third LLM 10, with the prompt comprising a chain of sub-prompts representing intermediate steps to be taken to arrive at the first output, second output, and / or third output respectively from the received inputs. Providing a chain of sub-prompts, also called chain-of-thought prompting, introduces a complete paragraph or multiple related sentences that form a chain of steps to be followed to complete the task. Providing a chain of steps to follow, rather than a single prompt, allows for a complex problem to be approached logically, with the large language model being guided along the series of steps to be taken to arrive at the answer. In contrast to the traditional single prompt, e.g., ‘Your task is to answer multiple-choice questions’, the structure of Chain-of- thought Prompting (CoT) can imitate the human thought process when solving complex problems involving multi-step logical reasoning. As a result, the CoT provides a coherent series of intermediate reasoning steps that can guide the model on how to get to the final answer step by step, allowing the model to output a more accurate answer. Using a CoT technique allows for a more reasoned and logical answer to be provided. Example 1 below shows an example of a user inquiry, and the outputs produced by each of the three LLMs, according to the present disclosure. The user inquiry provides a number of symptoms and a list of possible drugs that could be used to treat them. The first LLM 6 identifies all of the drugs and symptoms mentioned as being keywords, as well as the lack of any mention of a specific disease. The second LLM 8 extracts relevant knowledge relating to the identified keywords, and provides a link to the database that the information was extracted from, allowing for the source of the information to be identified. Finally, the third LLM 10 provides an answer to the user inquiry based on the extracted information, including a link to the source of the knowledge Example 1 User Inquiry (Question): A 29-year-old woman develops painful swelling of both hands. She is also very stiff in the morning. Physical examination reveals involvement of the proximal interphalangeal joints and metacarpophalangeal (MCP) joints. Her RF is positive and ANA is negative. Which of the following medications is most likely to improve her joint pain symptoms? A: D-penicillamine; B: an anti-malarial; C: methotrexate; D: NSAID or aspirin. Answer from LLM system of the present disclosure: D. Output from first LLM: 1. Drug: ['D-penicillamine', 'an anti-malarial', 'methotrexate', 'NSAID or aspirin’] 2. Symptom: ['Painful swelling of both hands, stiffness in the morning, involvement of proximal interphalangeal joints and metacarpophalangeal joints’] 3. Disease: [] Output from second LLM: 1. D-penicillamine: Penicillamine is a chelating (KEE-late-ing) agent that binds to excess copper and removes it from the blood stream. Penicillamine is used to remove excess copper in people with an inherited condition called Wilson's disease. Penicillamine is also used to treat severe rheumatoid arthritis after other medicines have been tried without success. Penicillamine is not approved to treat juvenile rheumatoid arthritis. Link[https: / / www.drugs.com / mtm / penicillamine.html]. 2. Anti-malarial: Hydroxychloroquine is a quinoline medicine used to treat or prevent malaria, a disease caused by parasites that enter the body through the bite of a mosquito. Hydroxychloroquine is also used to treat symptoms of rheumatoid arthritis and discoid or systemic lupus erythematosus. Link[https: / / www.drugs.com / mtm / hydroxychloroquinehtml]. 3. Methotrexate: Methotrexate is an antineoplastic agent used to treat various cancers, severe psoriasis, severe rheumatoid arthritis, and juvenile rheumatoid arthritis. It may not improve pain symptoms in short-term. It helps to slow the progression of rheumatoid arthritis and are key to preventing long-term joint damage but is not commonly used for symptom relief. NSAID is commonly used in combination for short- term symptom relief. It is also indicated for pediatric acute lymphoblastic leukemia and pediatric polyarticular juvenile idiopathic arthritis in oral solution form. Link[https: / / www.drugs.com / methotrexate.html]. 4. NSAID or aspirin: Aspirin, a salicylate, is used for immediate relief of pain, fever, inflammation, arthritis, migraines, and reduce the risk of major adverse cardiovascular events. It provides relief for various symtoms such as the flu, the common cold, neck and back pain, rheumatoid arthritis, bursitis, burns, and various injuries. Link[https: / / www.drugs.com / aspirin.html]. 5. Rheumatoid Arthritis: Rheumatoid arthritis is a long-term condition that causes pain, swelling and stiffness in the joints. The condition usually affects the hands, feet and wrists. Common medications: methotrexate, leflunomide, hydroxychloroquine, sulfasalazine. DMARD such as methotrexate is usually the first medicine given for rheumatoid arthritis, often with another NSAID to relieve any pain. DMARDs help slow the progress of RA, and reduce pain, stiffness, and inflammation, however, they do not provide short-term pain relief and may take several weeks or months to demonstrate a clinical effect. NSAIDs , such as aspirin, ibuprofen, are used to help decrease swelling, pain, and fever and relieve joint pain symptoms instead. Link[https: / / www.nhs.uk / conditions / rheumatoid-arthritis / ] Output from third LLM: The woman in the question presents with painful swelling of both hands, stiffness in the morning, and involvement of the proximal interphalangeal joints and metacarpophalangeal (MCP) joints. These symptoms are consistent with rheumatoid arthritis (RA). The RF is positive, which further supports the diagnosis of RA. The question asks which medication is most likely to improve her joint pain symptoms. A: D-penicillamine is used to treat severe rheumatoid arthritis after other medicines have been tried without success, suggesting it is not the first-line treatment for pain relief in rheumatoid arthritis. More information is available here (https: / / www.drugs.com / mtm / penicillamine.html). B: An anti-malarial such as Hydroxychloroquine is used to treat symptoms of rheumatoid arthritis, but it is not primarily used for pain relief. More information is available here (https: / / www.drugs.com / hydroxychloroquine.html). C: Methotrexate is a first-line treatment for rheumatoid arthritis, but it is used to slow the progression of the disease and does not provide immediate pain relief. More information is available here (https: / / www.drugs.com / methotrexate.html). D: NSAIDs or aspirin are commonly used to provide immediate relief from pain, inflammation, and fever, making them suitable for relieving joint pain symptoms in rheumatoid arthritis. More information is available here (https: / / www.drugs.com / aspirin.html). Final Answer: D The present system may be implemented in multiple configurations. Each of the first LLM 6, second LLM 8 and third LLM 10 may be implemented using a different respective processing unit for each large language model. Alternatively, the three LLMs may be implemented on the same processing unit. Each processing unit may comprise any suitable combination of hardware, firmware and / or software for performing the required functionality, including for example one or more processors, memory, data bus system, storage, network communication interfaces, etc. A method of configuring the above system will now be described. The method comprises configuring three LLMs to act as the first LLM 6, second LLM 8 and third LLM 10. The LLMs can be located on the same system or on separate systems. The LLMs may be general purpose pre-trained LLMs. Alternatively, a set of training data may be used to train each of the three LLMs. The method further comprises generating a first prompt to be provided to the first LLM 6, wherein the first prompt is a set of instructions to extract relevant keywords. Optionally, the prompt includes a series of few-shot prompts comprising an example input and output. The method further includes generating a second prompt to be provided to the second LLM 8, wherein the second prompt is a set of instructions to extract relevant knowledge. Finally, the method includes generating a third prompt to be provided to the third LLM 10, wherein the third prompt is a set of instructions to generate an answer based on provided information. The accuracy of the above described system and method will now be evaluated in comparison to existing state of the art LLMs. The models selected for comparison were Galactica, InstructGPT, Flan-PaLM, Med-PaLM-2, ChatGPT, and GPT-4. The LLM system of the present disclosure is referred to as DrugGPT. The models are evaluated on eight public datasets: • MedQA-USMLE is a multiple-choice dataset, which contains questions and their associated answer candidates. This high-quality dataset is collected from the professional national medical board examination, i.e., United States Medical License Exams (USMLE). During testing, 10,000 questions related to drug recommendation, drug dosage, and pharmacology were randomly selected. The different models were tasked to select the most appropriate answer out of the multiple options provided. The accuracy was calculated to report the performances on the selected categories; • MedMCQA is a dataset collected from real-world high-quality medical entrance exam questions, covering various healthcare topics and medical subjects. For evaluation, 10,000 test samples related to drug recommendation and drug dosage were randomly selected. Each test sample is composed of a question, four choices, and a correct answer. The different models were instructed to select the most appropriate choice with the accuracy of the answer determining the models performance; • MMLU-Medicine is a dataset collected from the MMLU dataset, which is a massive multitask test consisting of multiple-choice questions from various branches of knowledge, i.e., humanities, social sciences, and hard sciences. All samples (508) from the MMLU Professional Medicine and the College Medicine splits were used to evaluate the performances of the selected models. The models were instructed to give the most appropriate answer for each question, with accuracy being calculated based on the number of correct answers. • ChatDoctor is a dataset composed of real-world patient- physician conversations, in which the conversation describes the disease and symptoms, followed by the final drug recommendations given by the physician. All of the 796 test samples were used to evaluate the performance of the selected models. The models were instructed to list all available drugs for the treatment of patients based on their diseases and symptoms. To evaluate the performances of the models, the Recall, Precision, and F1 scores were calculated. • ADE-Corpus-v is a dataset collected from published medical case reports and contains diverse adverse drug reaction data, i.e., adverse drug events (ADEs), which provides relations between drugs and adverse effects. This dataset is used to evaluate the performance of the selected models on tasks relating to an adverse reaction between drugs. Following the conventional ADE evaluations, 5,000 test samples were randomly selected. The models were then instructed to identify the adverse effect mentioned in the input queries. The accuracy was calculated based on the results. • Drug-Effects contains details of various drugs used for diverse medical conditions, e.g., Acne, Cancer, Heart Disease, and their side effects. The dataset is used for evaluating drug adverse reactions. Focusing on common situations, such as a patient having consumed alcohol or being pregnent, 3,000 test samples are randomly selected. The models were then instructed to identify whether the selected drug is proven to have a side effect in the given situation. The models provided answers in the form of two yes / no choices corresponding to the effects. The model’s answer is correct only if the model correctly identifies both effects. • DDI-Corpus is a manually annotated dataset containing 5,028 drug-drug interactions (DDIs), including pharmacokinetic and pharmacodynamic interactions. 1,000 test samples were selected to form the testing set. 500 positive samples were randomly selected involving samples with the most common DDI. 500 negative examples were then constructed that did not have a DDI, as verified by the model. The ratio of positive samples to negative samples is 1:1. The models were instructed to answer ‘yes / no’ for each drug-drug interaction, with the number of correct answers determining the models accuracy. • PubMedQA is a dataset aiming to answer research questions. The dataset is able to verify the reasoning ability of models. For evaluation, 10,000 test samples were randomly selected. The test samples were related to pharmacology so that the performance of pharmacology question answering could be verified. Each test sample consists of a question, a long answer (i.e., conclusion), and a short answer (i.e., ‘yes’ / ‘no’). When evaluating the model, the question was taken as the user inquiry. The models were then required to give an appropriate short answer (yes / no) and related reasons (conclusions). The accuracy was calculated based on the correctness of the given short answers. Five runs were conducted to reduce the randomness of the different models. Fig. 4 shows a table illustrating the performance of the different large language models on the eight public datasets described above. The table shows the mean and standard deviation (SD) of performance, as a percentage, with a higher value corresponding to a more accurate performance. The table shows that DrugGPT consistently achieves a better performance than the alternatives over a wide range of datasets and with fewer parameters. The above approach is far more effective at assisting physicians with drug analysis than existing LLMs, as shown by the increased accuracy, while also providing less deviation in most cases. The system therefore provides the correct answer more consistently than previous techniques. Table 5 Table 5 provides a further comparison between the performance of ChatGPT, GPT- 4, and DrugGPT on drug-drug interaction tasks. The task involved evaluating the therapeutic efficacy of the Moderna COVID-19 vaccine when administered in combination with other drugs. 100 drugs that have an interaction with the Moderna COVID-19 vaccine and 100 drugs that do not have an interaction were randomly identified. The different models were then asked to identify whether the efficacy of the vaccine would be impacted. As shown in Table 5, DrugGPT has a far greater performance than either ChatGPT or GPT-4. As none of the three models had been trained specifically on Covid-19 data, the difference in accuracy illustrates the ability to generalise and transfer the above system to new drugs without the need to retrain the model for that specific drug. Retraining a model is extremely expensive and time consuming, and so a system that can be generalised without requiring any further training is advantageous. Fig. 5 illustrates the robustness of DrugGPT when only a percentage of a knowledge base is available. Even with a heavily reduced knowledge base, the system is able to match and outperform existing models. The data shows that the system is not reliant on having an in-depth and complete knowledge base. The system is therefore able to be applied to domains using only a small-scale, specialised knowledge base, while still achieving a better performance than existing models. Fig. 6 compares the ability of DrugGPT to reason about learned knowledge compared to existing models. The models were provided with a number of zero-shot learning tasks (meaning that no examples of the desired output were provided) to test the model’s ability to reason about drugs using knowledge already learnt by the models. Five runs were again conducted, to reduce randomness. As shown, the DrugGPT system achieves the best results across all tasks with fewer parameters. Additionally, conducting a t-test between the results of the above system and GPT-4 indicate that the results are statistically significant. Table 6 Table 6 illustrates how the system performs without one or more of the large language models. As shown, each model provides an improvement in accuracy to the system overall. The use of the three individual models in combination therefore has a significant effect on performance, enabling the model to better analyse inquiries, captures relevant broad knowledge, and match inquiries and knowledge to generate evidence. The use of the three large language models in combination produces a synergistic effect greater than the use of only one or two of the large language models used separately.

Claims

CLAIMS 1. A large language model system, comprising: a data processing system configured to implement a first large language model, a second large language model, and a third large language model, wherein the data processing system is configured to: receive inquiry data representing a user inquiry; input at least the inquiry data to the first large language model and receive a first data output as a corresponding output from the first large language model; input at least the first data output and a knowledge graph to the second large language model and receive a second data output as a corresponding output from the second large language model; input at least the first data output, the second data output, and the inquiry data to the third large language model and receive a third data output as a corresponding output from the third large language model; and output the third data output as an answer to the user inquiry.

2. The system of claim 1, wherein the first large language model is configured to generate the first data output by identifying and extracting keyword data representing keywords from the inquiry data dependent on whether the keywords are relevant to one or more predetermined target categories.

3. The system of claim 2, wherein the target categories comprise: diseases; symptoms; and drugs.

4. The system of claim 2 or 3, wherein the second large language model is configured to generate the second data output by extracting information from the knowledge graph that is relevant to the keywords represented by the extracted keyword data and providing the extracted information as a part of the second data output.

5. The system of claim 4, wherein the knowledge graph represents information from one or more databases as nodes and edges, optionally wherein the one or more databases contain medical knowledge.

6. The system of claim 5, wherein each node includes location data representing a location in the one or more databases where the information represented by the node is located, and wherein the second large language model is further configured to include the location data of a node in the second data output if the extracted information from the knowledge graph includes the information represented by that node.

7. The system of claim 6, wherein the third large language model is configured to include the location data corresponding to any extracted information used to generate the third data output within the third data output.

8. The system of any of claims 5 to 7, wherein the nodes comprise: disease nodes corresponding to respective diseases; and drug nodes corresponding to respective drugs.

9. The system of claim 8, wherein each disease node is associated with a symptom node embedding obtained by encoding a symptom textual description of one or more symptoms associated with the respective disease, optionally including the primary symptoms of the disease.

10. The system of claim 9, wherein each drug node is associated with a disease node embedding obtained by encoding a disease textual description of one or more diseases treatable by the respective drug, optionally including the primary diseases the drug can treat.

11. The system of claim 10, wherein the edges in the knowledge graph represent similarity between respective pairs of nodes, optionally a cosine similarity, optionally a normalized cosine similarity.

12. The system of claim 11, wherein the edges are limited to include only edges connecting a disease node to a drug node, and a drug node to a disease node.

13. The system of claim 11 or 12, wherein the edges from each node are limited to include only the Kth most similar nodes, where K is an integer, optionally equal to or less than 10.

14. The system of any of claims 10 to 13, wherein a plurality of categories of knowledge are linked to the nodes.

15. The system of claim 14, wherein the plurality of categories of knowledge comprise one or more of the following: drug description and indication; drug dosage recommendation; drug adverse effects; drug toxicity; drug-food interactions; drug-drug interactions; drug pharmacodynamics; and pubmed experimental summaries; and / or common symptoms; disease causes; disease diagnosis; disease treatment; and disease complications.

16. The system of claim 14 or 15, wherein the second large language model is configured to receive as input: a set of node embeddings gd, containing the symptom node embeddings and disease node embeddings associated with disease nodes and / or drug nodes that represent diseases and / or drugs in the keyword data identified and extracted by the first large language model; a global graph embedding ga, obtained by averaging all node embeddings in the knowledge graph; a prompt p that describes how the second large language model should generate the second data output; and the inquiry data.

17. The system of claim 16, wherein the prompt p instructs the second large language model to:identify which of the plurality of categories of knowledge are needed; and extract information from the knowledge graph in the identified categories of knowledge, optionally wherein the extracted information relates to the user inquiry.

18. The system of any of claims 14 to 17, wherein the data processing system is further configured to: generate, using the knowledge graph, a set of training data comprising a plurality of tuning prompts; and tune, using the set of training data, the second large language model to adapt the second large language model to a downstream task.

19. The system of claim 18, wherein each of the tuning prompts comprises an example input and a target output, wherein the example input comprises an example user inquiry and keyword data, and the target output comprises the desired output for that specific input.

20. The system of claim 19, wherein the target output comprises a list of knowledge categories indicating what categories of knowledge should be extracted from the knowledge graph, and a list of extracted information that has been extracted from the knowledge graph for each keyword represented in the keyword data, wherein the information includes the knowledge category that each piece of extracted information belongs to.

21. The system of any preceding claim, wherein the third large language model is configured to generate the third data output using only the first data output, the second data output, and the user inquiry.

22. The system of any preceding claim, wherein the data processing system is further configured to input a prompt instructing the first large language model, second large language model and / or third large language model, wherein the prompt comprises a chain of sub-prompts representing intermediate steps to be taken to arrive at the first output, second output, and / or third output respectively from the received inputs.

23. A computer implemented method for extracting and interpreting information using a large language model system, the large language model system comprising a first large language model, a second large language model and a third large language model, the method comprising: receiving inquiry data representing a user inquiry; providing at least the inquiry data as an input to the first large language model and receiving a first data output as a corresponding output from the first large language model; providing at least the first data output and a knowledge graph as an input to the second large language model and receiving a second data output as a corresponding output from the second large language model; providing at least the first data output, the second data output, and the user inquiry as an input to a third large language model and receiving a third data output as a corresponding output from the third large language model; and outputting the third data output as an answer to the user inquiry.

24. The method of claim 23, wherein the user inquiry relates to diseases, symptoms and / or drugs, optionally wherein the user inquiry represents at least one of: a request for identification of a drug that can be used for the treatment of identified symptoms; a request for the calculation of a safe dosage for an identified drug; a request to identify patterns of adverse reactions associated with use of an identified drug; a request to predict the interaction of two identified drugs administered in combination; and a request to answer a pharmacology related question.

25. A method of configuring the system of claim 1, comprising: configuring three large language models to act as the first large language model, second large language model and third large language model, optionally wherein a set of training data is used to train each of the three large language models; generating a first prompt to be provided to the first large language model, wherein the first prompt is a set of instructions to extract relevant keywords;generating a second prompt to be provided to the second large language model, wherein the second prompt is a set of instructions to extract relevant knowledge; generating a third prompt to be provided to the third large language model, wherein the third prompt is a set of instructions to generate an answer based on provided information; generating one or more few shot prompts containing an example input and the desired output for that input.

Images