Method and system for predicting biological entities

Abstract

A computer-implemented method predicting a biological entity meeting a user- defined biological requirement using a knowledge base, the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a biological requirement for which a biological entity is to be predicted; obtaining, based on the query, a query sentence text describing the biological requirement and including mention of a biological entity, in which the biological entity itself is masked for prediction; selecting a candidate biological entity for the masked biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, wherein the evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the candidate biological entity is the masked biological entity based on the retrieved evidence sentences.

Description

[0001] METHOD AND SYSTEM FOR PREDICTING BIOLOGICAL ENTITIES

[0002] FIELD OF THE INVENTION

[0003] The present application relates to computer implemented techniques for training a machine learning model to predict biological entities meeting user-defined conditions, particularly biological targets, relevant to a user-specified biological context. The application also describes methods of applying the trained model to identify biological entities for use in drug discovery.

[0004] BACKGROUND

[0005] The first stage of the drug discovery process is to identify suitable biological targets that play a role in a disease of interest, where this role has the potential to be modulated by a therapeutic. The term “biological target” encompasses any biological entity that interacts with, and the activity of which is modulated by, a particular therapeutic, such as genes, nucleic acid sequences, proteins, amino acid sequences, protein complexes, or biological pathways for the treatment of diseases.

[0006] In order to identify biological targets that are suitable from among a large number of potential candidates, an assessment of their characteristics against a set of desired criteria must be carried out. Depending on the context and the purpose for which a biological target is required, the desired characteristics may span multiple categories such as ligandability, safety, and therapeutic evidence which must all be taken into account, and as a result the search is required to consider multiple properties of the candidates at once. In traditional approaches, this complex analysis is carried out manually by a scientist reviewing data relating to potential biological targets and sifting the candidates for potential matches to the desired characteristics. In cases involving multiple categories and large numbers of potential biological targets, such manual analysis is time consuming and tends to create delays and increased costs of the process of developing new treatments for diseases. The challenge of discovering and validating targets is reflected in the failure rate of drug candidates in the clinic, where promising treatments fail to show efficacy even in relatively late-stage trials.

[0007] The drive to find higher quality potential drug targets for validation has driven the increasing use of bioinformatics and machine learning techniques to analyse the vast and varied biomedical data sources to improve the accuracy of biologically relevant target predictions and reduce the impact of human error and bias.

[0008] One approach involves extracting data from these data sources to knowledge graphs, which combine various biomedical entities in one network, connected by associated relationships. Large-scale knowledge graphs (KGs) have gained prominence over the past several decades as a means for representing complex structured data at-scale, leading to the development of machine learning models designed to predict new or unknown information from a KG (Ji et al., 2021). A subclass of these models deals with link prediction, i.e. inferring new facts from a given KG consisting of (subject, relation, object) facts (or triples). For instance, a link prediction algorithm might reason from a KG containing the triple (USA, ElectedPresident, JFK) to infer that the triple (JFK, BornlnCountry, USA) also likely exists (i.e. JFK was bom in the country USA).

[0009] In the biomedical domain, entities represented in a knowledge graph can be, for example, molecules, biological functions and diseases or phenotypes. Relationships include molecular interactions, gene-functional associations, and drug-target interactions. The intrinsic hierarchical structure of many biological entity types makes them particularly well suited to exploring via a knowledge graph. Various computational tools and techniques have been developed to probe biomedical knowledge graphs and extrapolate from them to make predictions of suitable biological targets. These knowledge graph tools are among the best performing target ID techniques currently being deployed.

[0010] However, there are significant limitations to these graph-based inference methods. These methods typically comprise two stages: data extraction to the knowledge graph and target prediction from the knowledge graph, both of which are imperfect. Firstly, it is difficult to translate complex, context dependent literature data into relationships in the knowledge graph. Extracting relationships from this rich contextual information in the literature into a compact graph, consisting of highly generalized (but high-confidence) relationships, inevitably results in a loss of information.

[0011] Similarly, it is difficult to translate a complex query, defining a specific biological context of interest, into a graph query. Typically, this is done by choosing a query entity in the graph (a disease or a mechanism) that best matches the disease or mechanism under consideration, then querying for targets that best match that query entity. The translation process has several well-known shortcomings such as: (1) there is potentially no good query entity in the graph to match the user’s requirements; (2) a lack of alignment between the closest targets and the desired properties of predicted targets; (3) no straightforward way to query on multiple entities at once (e.g. a disease and a mechanism in combination).

[0012] This has lead more recently to the application of large language models (LLMs) to predicting biological targets for drug discovery. In particular, advances have been made by training LLMs to predict masked biological entities in sentences extracted from the biomedical literature. Recent research has shown that language models can act as knowledge bases, supplying answers to factual user queries on- demand using only the learned parameters (Petroni, F., et al., Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, 2019). Free text input queries can be input into the trained models to predict biological entities, such as drug targets, having properties defined in the input query. The nature of the text input means more complex biological requirements can be defined, providing a much more flexible approach.

[0013] There are however a number of possible limitations to the LLM approach. Firstly, it is difficult to assess the models reasoning for a prediction to determine, for example, which components of the huge input training data are contributing to the prediction, and whether this matches the user’s intentions. This is a significant concern when using such models to predict biological entities, such as drug targets, when acting on model hypotheses may be costly or risky.

[0014] There is also no way of training the model to value certain types of data within the large text training corpora as being more valuable or predictive than others. Another issue is that LLM-approaches are by their nature limited to text data, such as published biomedical literature, and therefore do not leverage other data modalities. LLM approaches are also computationally expensive and are not well suited to giving well calibrated predictions that would allow all possible candidate entities, for example all genes within the genome, to be ranked against each other. Finally, existing models require re-training to take into account new data or evidence. For example when new published evidence relating to a particular disease or drug is available, with existing models the training corpus must be updated and the model re-trained to make predictions based on it.

[0015] There is a related challenge in how to retrieve information from a large knowledge base to support a hypothesis. The technical task of retrieving relevant items of data within a large multimodal data store is non-trivial. For example, a drug discovery scientist may need to return data that supports the hypothesis that “gene X is a promising drug target for type-l I diabetes. This involves searching different types of data, such as biomedical literature, transcriptomics data, gene expression data and knowledge graph data. There is no efficient and reliable computational technical for searching this type of multimodal data and comparing disparate data sets to return the most relevant items of data to an input query.

[0016] Accordingly, there is a need for an improved technique for identifying suitable biological targets or other biological entities meeting user-defined requirements, or having defined properties, usable for drug discovery. There is a related need for improved data retrieval techniques for searching a knowledge store and outputting data that supports an input user query or hypothesis. SUMMARY OF INVENTION

[0017] In a first aspect of the invention there is provided a computer-implemented method for training a machine learning model to predict a biological entity meeting a user- defined biological requirement using a knowledge base, the method comprising: providing a training knowledge base comprising a corpus of textual data; providing a training data set comprising training query sentences each comprising text including mention of a biological entity, in which the biological entity itself is masked for prediction, where the query sentence has a known correct biological entity; for each training query sentence, selecting a candidate biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, which may be the correct biological entity or another biological entity, wherein the evidence sentences are retrieved based on computing a similarity of the training query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model comprising a machine learning model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict a probability that the candidate biological entity is the correct biological entity.

[0018] Since knowledge retrieval and prediction are separate within the model, the method allows for direct understanding of the evidence used by the model to provide a prediction. This improves over large language model approaches where the model reasons over all training data and it is difficult to disentangle the evidence used to understand why a model is reaching a prediction. This is very important for drug discovery applications where it is essential that it is understood why a particularly biological entity, such as a gene / protein target, is predicted as a good candidate for drug discovery. This structure of the model also has a number of important technical advantages when the methods are deployed as a user tool. In particular, they allow for the user to make the predictions based on their own data provided as the knowledge base. In particular, the retrieving of evidence may be achieved by an encoder trained on large publicly available data, but the inference may be carried out on smaller, proprietary user data sets. This differs to LLM based approaches where the predictive model makes predictions based over the full training data and so it is not possible for a user to readily apply the model to their own datasets. The structure of the model means it is much more computationally efficient that prior art LLM approaches and allows for example predictions, with explanations, across the full genome. The knowledge base can also incorporate multimodal data and so makes improvement over prior art methods that are limited to using text data, allowing for patient level transcriptomics data to be leveraged in the predictions. The model is also tuneable by user feedback on what constitutes good evidence for a particular type of query.

[0019] The model is further usable for retrieving data from a knowledge base that supports an input query. In particular, the evidence sentences that are found to provide the highest probability prediction that the biological entity is the correct biological entity for the input query, are the most relevant items of data from the knowledge base. As defined further below, these items of data, and the source of these items of data, such as a particular journal article, or a transcriptomics data set, may be output to the user, such that the trained model acts as an efficient data retrieval tool for retrieving data and recommending files such as documents and data sets to a user.

[0020] Preferably the biological entity to be predicted comprises a biological target for drug discovery. Preferably the user-defined biological requirement defines one or more properties of the biological target to be identified, such as a required relationship with one or more other biological entities, such as a disease. The methods of the present invention may be particularly optimised for identifying a biological target for drug discovery. In particular the present method allows for efficient computation over a full set of drug targets to determine an optimum drug target based on the input requirement. The method may comprise a subsequent step of determining a drug that interacts with an identified drug target, for example a drug that modulates the functioning of the drug target. In particular the method may comprise selecting a biological target based on the predicted probability and determining a drug (i.e. a therapeutic compound) that interacts with the selected drug target.

[0021] Preferably the training query sentence describes a biological context involving the biological entity, the biological context preferably comprising a relationship with another biological entity. In particular, the training query sentence may be a statement involving the biological entity, mentioning one or more other biological entities. In this way, at inference time, a “biological requirement” can be defined in a similar way, but specifying a required biological context, such as one or more required relationships with one or more other biological entities. The training query sentences preferably comprise sentences extracted from a biomedical text corpus, preferably processed using named entity recognition such that the biological entities are entity-linked to biological entity identifiers. The entity linking can be used for efficient retrieval of relevant sentences from the knowledge base by matching sentences that mention the same biological entity and analysing these for similarity to the query sentence.

[0022] Preferably the method comprises training on positive and negative training examples, wherein training on a positive training example comprises: selecting the correct biological entity as the candidate entity and retrieving from the knowledge base a plurality of evidence sentences each including mention of the correct biological entity; inputting each training query sentence and a plurality of retrieved evidence sentences into the reasoner model, where mention of the correct biological entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict that the masked biological entity is the correct entity; and wherein training on a negative training example comprises: selecting the correct biological entity as the candidate entity and retrieving from the knowledge base a plurality of evidence sentences each including mention of an incorrect biological entity; inputting each training query sentence and a plurality of retrieved evidence sentences into the reasoner model, where mention of the incorrect biological entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict that the masked biological entity is not the correct entity. In particular the training query sentence may be a (possibly N ER-processed) sentence from the literature. For positive training examples, the candidate biological entity is the biological entity that appeared in the original sentence from the literature and the model is trained to output an increased probability. For negative training examples, a biological entity other than the entity appearing in the sentences is chosen. In this way, the model can be trained predict biological entities for inference query sentences that do not have a known biological entity.

[0023] Preferably in the negative training example, the incorrect biological entity is randomly sampled from the full set of possible candidate entities according to its frequency of mention in the literature. This ensures that the model cannot learn based on the frequency with which the entity appears in the literature, and so no literature bias is introduced. In some examples, the method comprises only retrieving evidence sentences deriving from literature sentences having a publication date before the publication date of the query sentence. In this way, the model is trained to predict new relationships and biological information, learning from the previous progression of knowledge in the literature.

[0024] Preferably retrieving a plurality of sentences from the knowledge base comprises: inputting the query sentence into a query encoder trained to output a query embedding; inputting sentences from the knowledge base into an evidence encoder trained to output an evidence embedding; computing the similarity of the query embedding to each evidence embedding and selecting the plurality of evidence sentences with embeddings most similar to the query embedding. Preferably the evidence and query encoder have been trained to form embedding that encode semantic information from the input sentence, so that the meaning of two sentences can be compared. This vector search approach provides a computationally efficient method of retrieving evidence sentences from the literature.

[0025] In other examples, the knowledge base comprises a vector store, in which evidence sentences are stored as evidence embeddings and the method comprises inputting the query sentence into an encoder to form a query embedding and then performing a vector search on the vector store. Preferably the knowledge base comprises a separate vector store for each possible candidate biological entity. In this way, the processing of retrieving evidence is more efficient as the method only requires searching the vector store corresponding to the candidate biological entity. Preferably the vector store comprises a FAISS index and preferably the knowledge base comprises a FAISS index for each candidate biological entity.

[0026] Preferably the query encoder, and preferably the evidence encoder, comprise a transformer encoder. The encoders may be pretrained by masked language modelling. This provides an effective way of training the encoders to learn to create embeddings encoding relevant contextual information and improves the relevance of retrieved information. In some examples the query encoder and evidence encoder may be trained on a retrieval objective using a contrastive loss function. For example, the query encoder, and or evidence encoder, may be trained during the process of training the retriever model on the objective of predicting the correct biological entity. In this way, the encoders are trained to learn embeddings that return relevant evidence usable for the prediction task.

[0027] In some examples the query encoder and evidence encoder comprise a retriever encoder, the retriever encoder comprising an encoder trained by: providing a training data set comprising a plurality of entity-linked text sequences, each text sequence including a mention of a biological entity, where the biological entity is linked to a corresponding biological entity identifier from a set of possible biological entity identifiers; masking the mention of the biological entity within each text sequence; encoding each masked text sequence into an input representation for a machine learning model, the machine learning model comprising the retriever encoder, which is configured for mapping the input representation to an output representation, and a classifier layer that receives the output representation and performs classification into classes corresponding to the set of possible biological entity identifiers; and training the machine learning model to predict the biological entity identifier of the masked biological entity based on the input representation.

[0028] The query encoder and evidence encoder may be optimised for retrieving sentences for predicting a required biological entity type, the encoders optimised by training the encoder to predict the same biological entity type as that sought for the user-defined biological requirement, the biological entity type preferably comprising a biological target. This trains the encoders to learn embeddings that return particularly relevant evidence for the required prediction task.

[0029] Preferably inputting each training query sentence and the corresponding plurality of retrieved evidence sentences into the reasoner model comprises: combining the query embedding with each evidence embedding individually into a queryevidence vector representing how the query sentence relates to the corresponding evidence sentence, so as to create a query-evidence vector for each queryevidence sentence pair; forming an input sequence comprising a sequence of all query-evidence vectors and inputting the input sequence into the reasoner model. “Relates to the corresponding evidence sentence” may mean that the embeddings encode information on the relation between the semantic content of the sentences, for example whether the information is related.

[0030] Preferably the method further comprises encoding metadata associated with each evidence sentence into a metadata vector, wherein the metadata comprise information regarding the source of the evidence sentence; and combining the metadata vector into each corresponding query-evidence vector. Preferably the metadata comprises one or more of: the publication date of the evidence sentence, the paper and / or journal from which the evidence sentence was extracted, the impact factor of the journal from which the evidence sentence was extracted. By encoding information about the source of the evidence into the query-evidence vector, the model is able to use this information in predicting the biological entities. In this way, the model can learn to, for example, place greater significance on more recent publications, or on evidence that comes from a large number of sources rather than a single source.

[0031] Preferably combining the query sentence with each evidence sentence individually into a query-evidence vector representing the representing how the query sentence relates to the corresponding evidence sentence: for each evidence sentence, inputting a query embedding representing the query sentence and an evidence embedding representing the evidence sentence into a respective query-evidence encoder trainable to output the query-evidence vector. The queryevidence encoder may comprise a convolutional network layer, and the method may comprise, for each evidence sentence: stacking the query embedding and the evidence embedding to form a matrix; inputting the matrix into the queryevidence encoder to output the query-evidence vector representing the similarity of the query sentence and the corresponding evidence sentence. In this way, the query-evidence encoder may be trained to learn an embedding that is usable by the reasoner model to provide an improved prediction.

[0032] Preferably the method comprises training the query-evidence encoders and the reasoner model end-to-end to learn to predict a probability that the candidate biological entity is the correct biological entity based on the retrieved evidence sentences. In this way, the models are cooperatively trained to provide improved prediction results.

[0033] The reasoner model preferably comprises an artificial neural network (ANN). More specifically the reasoner model preferably comprises: a Transformer encoder trainable to map the sequence of query-evidence vectors to a sequence of output embeddings; and an output layer trainable to compute the output probability based on the output embeddings. This form of model can assess contextual information, so can use the relationship between information in difference evidence sentences in a combined way to provide the prediction. Preferably the transformer encoder does not use positional embeddings. In this way, the prediction does not depend on the sequence in which the evidence sentences are presented to the model.

[0034] Preferably the method comprises a subsequent step of fine-tuning the reasoner model. Preferably fine-tuning the reasoner model comprises: providing an optimal evidence set, the optimal evidence set comprising a plurality of different evidence statements that optimally indicate that a candidate biological entity meets the biological requirement; for a training query sentence, retrieving two sample sets of evidence, each sample set of evidence comprising a plurality of sentences from the knowledge base; for each sample set of evidence, querying a language model to determine the degree to which the optimal evidence set is supported by the sample set of evidence and accordingly determine which sample set of evidence better supports the optimal evidence set; use the determination to apply a label to the sample sets of evidence, indicating which better supports the optimal evidence set; and train the reasoner model to adapt the output probability based on the applied label. The optimal evidence set may be user defined, for example input in a user interface or selected from a selection of evidence returned by the model.

[0035] The method preferably further comprises training the reasoner model to output an increased probability for a sample evidence set that better supports the optimal evidence set. In some examples each sample set of evidence supports a different candidate biological entity as meeting the biological requirement. For example the method may comprise inputting a user query and two different candidate entities to return to the two evidence sets. The reasoner may be trained during fine-tuning using a margin ranking loss.

[0036] The training query sentence and evidence sentences may be based on sentences extracted from published biomedical literature, wherein the publication date of the evidence sentences are required to be prior to the publication date of the training query sentence. In this way the model learns to predict future literature statements from older literature statements, requiring the model to predict new filings.

[0037] In another aspect of the invention there is provided a computer-implemented method for predicting a biological entity meeting a user-defined biological requirement using a knowledge base, the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a biological requirement for which a biological entity is to be predicted; obtaining, based on the query, a query sentence text describing the biological requirement and including mention of a biological entity, in which the biological entity itself is masked for prediction; selecting a candidate biological entity for the masked biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, wherein the evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the candidate biological entity is the masked biological entity based on the retrieved evidence sentences. The method preferably additionally comprises outputting a candidate biological entity with the highest predicted probability.

[0038] In this way, the user the method provides a prediction based on relevant evidence retrieved from a knowledge base, where the users can examine the specific evidence used and the contribution to the prediction. This allows the user to provide their own inference knowledge base on which a trained encoder can be applied to make predictions based on their own private data. The method provides a computationally efficient method for predicting over all entities within a specific entity type category, such as all genes in genome.

[0039] The method preferably comprises selecting a plurality of candidate biological entities and for each candidate biological entity: retrieving a plurality of evidence sentences from the knowledge base and inputting the query and plurality of evidence sentences into the reasoner model to predict a probability for each candidate entity. The method may comprise retrieving evidence for each possible candidate biological entity, for example each entity within a biological entity type set, for example every gene / protein target (“biological target”) in the human genome. The biological entity preferably comprises biological target, preferably a gene, and the method comprises, for each of a set of total known biological targets within the human genome: retrieving a plurality of evidence sentences from the knowledge base and inputting the query and plurality of evidence sentences into the reasoner model to predict a probability for each candidate biological target (or synonymously “drug target”).

[0040] The biological requirement preferably defines a relationship with another biological entity. The biological requirement may be referred to as a biological context. A biological context may comprise a description involving the biological entity, preferably the biological target. In particular, a “biological context” may be considered a description of the biological entity to be masked and predicted, in relation to one or more additional biological entities. In other words the biological context preferably comprises a sequence of text comprising a mention of a biological entity to be masked and a mention of one or more other biological entities. Preferably the biological context comprises a mention of a biological entity to be masked and predicted, preferably a biological target, and a description of a relationship of the biological entity with one or more additional biological entities. In this way, the model can be trained to predict a masked biological entity from the remaining unmasked biological context of an input query. Preferably each query sentence comprises a mention of a biological entity of interest and one or more additional biological entities, where biological entities preferably comprise one or more of a biological target, a disease, a biological mechanism, a cell type or a tissue type.

[0041] The query sentence preferably includes mention of one or more additional biological entities such as a biological mechanism, a cell type or a tissue type. Each query sequence preferably describes a biological context involving these biological entities, i.e. a statement regarding relationships and / or interactions between the biological entities. Preferably only mentions of a single biological entity are masked, the remainder of the text sequence comprising plain text. In this way, the model is trained to predict the single biological entity based on a representation encoding the remaining plain text of the text sequence.

[0042] Preferably the biological entity to be predicted comprises a biological target and the biological requirement defines a relationship with a disease.

[0043] Preferably the reasoner model is trained as described above under the first aspect.

[0044] Preferably the inference knowledge base is different to the training knowledge base. In this way the model can be trained on publicly available data and then deployed on user data.

[0045] Preferably the method further comprises computing a contribution to the predicted probability of a candidate biological entity of an evidence sentence used in the prediction. In this way the user can determine which evidence is contributing the most to the prediction. The method may comprise computing a Shapley value of an evidence sentence used in the prediction. Preferably this may be done by computing an estimated Shapley value of an evidence sentence used in the prediction, preferably by computing a Shapley Additive exPlanation, SHAP, value.

[0046] The method preferably comprises outputting one or more evidence sentences based on the computed contribution to the predicted probability. The method may comprise outputting one or more evidence sentences having the highest computed contribution. The method may comprise receiving a user specified biological requirement and entity, and outputting the evidence sentences within the knowledge based that best support the biological entity being true of the entity. In this way, the invention may provide a novel information retrieval system that solves the problem of identifying parts of a knowledge base (whether textual or numerical data that has been textualized) that best supports the statement in a computationally efficient manner.

[0047] Preferably, retrieving a plurality of evidence sentences comprises: encoding the query sentence and sentences from the knowledge base into embeddings and computing the similarity of the embeddings to return a plurality of evidence sentences most similar to the query sentence. Preferably retrieving a plurality of evidence sentences comprises: inputting the query sentence and sentences from the knowledge base into a retriever encoder trained to output an embedding representing the input sentence, wherein the retriever encoder preferably comprises a Transformer encoder. Preferably the retriever encoder has been trained by: providing a training data set comprising a plurality of entity-linked text sequences, each text sequence including a mention of a biological entity, where the biological entity is linked to a corresponding biological entity identifier from a set of possible biological entity identifiers; masking the mention of the biological entity within each text sequence; encoding each masked text sequence into an input representation for a machine learning model, the machine learning model comprising the retriever encoder for mapping the input representation to an output representation and a classifier layer that receives the output representation and performs classification into classes corresponding to the set of possible biological entity identifiers; and training the machine learning model to predict the biological entity identifier of the masked biological entity based on the input representation.

[0048] In other examples, rather than encoding the evidence sentences at inference time, the knowledge base may comprise a vector store, storing the embeddings of the evidence sentences. In these examples, the method comprises encoding the input query into a query embedding and searching the vector store for evidence embeddings using the query embedding. In particular a vector search method may be used. The method may comprise computing the similarity of the query embedding and the embeddings stored in the vector store. The knowledge base may comprise a separate vector store for each candidate biological entity. In this way, a vector search need only be varied out on the vector store relating to the candidate entity for which a prediction is to be computed. The vector store may comprise a FAISS index.

[0049] In some examples the method may further comprise prompting a large language model to determine the relevance of the retrieved evidence sentences to the input query and removing one or more evidence sentences from the evidence sentences determined to be least relevant to the query and inputting only the remaining evidence sentences into the reasoner model. The method may comprise masking the evidence sentences determined to be least relevant or replacing the corresponding embedding with a NULL embedding. The inventors have determined that this LLM-in-the-loop auditing process further improves the predictive power of the model in predicting entities that meet the query requirement.

[0050] Providing an inference knowledge base preferably comprises providing an entity- linked biomedical medical text corpus. The method may comprise extracting sentences including mention of a biological entity from the biomedical text corpus (i.e. having an entity-linked entity), and encoding the sentences into evidence embedding. The mention of the biological entity may be removed or masked prior to encoding into an embedding. Preferably providing an inference knowledge base comprises: converting a nontext dataset to a textualised data set and including the textualised data set in the knowledge base. The non-text data set may comprise one of: genetics or genomics data; transcriptomics data; proteomics data; epigenomics data; proteinprotein interaction network data; pre-clinical data; clinical data; knowledge graph data; image data; diagrams and figures; outputs from other models.

[0051] Genetics data may comprise one or more of: somatic mutations; GWAS; Microbiome analysis.

[0052] Transcriptomics may comprise bulk and or single cell flavours and may comprise one or more of: Expression; Differential expression; microRNA-Seq; Coexpression; Differential coexpression.

[0053] Epigenomics may comprise one or more of: Chromatin accessibility assays (ATAC-Seq); Chromatin binding and histone modifications (ChlP-Seq); Chromatin conformation (Hi-C); DNA Methylation.

[0054] Clinical data may comprise data derived from electronic health records.

[0055] Image data may comprise one or more of: Immunohistochemical microscopy (i.e. cell staining); Fluorescence microscopy; Regular microscopy (cell morphology and counting).

[0056] Pre-clinical data may comprise one or more of: Genome-wide CRISPR screening; Results from animal and other model organisms; assay results from cell lines.

[0057] Converting a non-text data set to a textualized data set may comprise converting a component of data within the non-text data set to a text description of the component of data.

[0058] Preferably converting a non-text dataset to a textualized data set comprises: providing a patient-level dataset comprising numerical data, converting one or more components of the data into a corresponding sentence describing the component of the data and including mention of a biological entity. In some examples the component of non-text data may be converted using a machine learning model trained to output a description of the component of data. In other examples the non-text data is converted to text data using a template, where numerical data may be converted to a qualitative description of the numerical data value, such as “low”, “moderate” or “high”.

[0059] Preferably providing an inference knowledge base comprising receiving an imported user data set, wherein the inference knowledge base is different to the training knowledge base. In some examples the inference knowledge base comprises two or more of: published biomedical literature; textualized patient-level datasets, where patient-level datasets preferably comprise genetics or transcriptomics data; textualized knowledge graph data.

[0060] Preferably retrieving a plurality of evidence sentences comprises: inputting the query sentence and sentences from the knowledge base into a retriever encoder trained to output an embedding representing the input sentence, wherein the retriever encoder comprises a Transformer encoder and is trained only on sentences extracted from biomedical literature text.

[0061] In some examples the user query comprises completion of a plurality of input fields, the plurality of input fields preferably comprising a disease of interest and one or more biological entities or relationships, wherein obtaining a query sentence comprises: generating one or more natural language text sequences based on the completed fields using a template. The input fields may comprise a disease of interest, a biological mechanism, and, optionally, one or more of: a tissue type, and a cell type, a trait, biomarker, assay, patient subgroup.

[0062] Preferably the method further comprises displaying, on a user interface, one or more evidence sentences and the predicted probability of the candidate biological entity meeting the biological requirement.

[0063] Preferably the method further comprises receiving a user selection of the displayed evidence sentences and updating the predicted probability based on the user selection. The method may further comprise receiving a user selection of the displayed evidence sentences and re-training the reasoner model to increase the contribution of evidence sentences of the type selected by the user.

[0064] Preferably the method further comprises displaying a contribution of each displayed evidence sentences to the predicted probability, the contribution preferably comprising a computed SHAP value.

[0065] In a further aspect of the invention there is provided a computer-implemented method for training a machine learning model to predict an entity meeting a user- defined requirement using a knowledge base, the method comprising: providing a training knowledge base comprising a corpus of textual data; providing a training data set comprising training query sentences each comprising mention of an entity, in which the entity itself is masked for prediction, where the query sentence has a known correct entity; for each training query sentence, selecting a candidate entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate entity, which may be the correct entity or another entity, wherein the evidence sentences are retrieved based on computing a similarity of the training query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model comprising a machine learning model, where mention of the candidate entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict a probability that the candidate entity is the correct entity.

[0066] Although the method is described herein in relation to predicting a biological entity, and it is specifically configured to provide improvements over the prior art at the specific requirements of biological entity prediction, it may also be generalised and applied to any prediction task involving predicting the probability that a candidate entity meets the requirements of a user query.

[0067] In another aspect of the invention there is provided a method for predicting an entity meeting a user-defined requirement using a knowledge base, the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a requirement for which an entity is to be predicted; obtaining, based on the query, a query sentence text describing the requirement and including mention of an entity, in which the entity itself is masked for prediction; selecting a candidate entity for the masked entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate entity, wherein the evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the candidate entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the candidate entity is the masked entity based on the retrieved evidence sentences.

[0068] The features described herein in relation to the first and second aspect may equally be applied in the more general methods, where reference to a “biological entity” may be replaced with “entity”.

[0069] In another aspect of the invention there is provided a computer-implemented method for retrieving evidence (i.e. data) from a knowledge base that supports a user-defined requirement of an entity (also referred to as a “candidate entity”), the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a requirement for the entity; obtaining, based on the query, a query sentence comprising text describing the requirement and including mention of the entity, in which the entity itself is masked (i.e. masked for prediction); retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the entity, wherein the evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting the query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the entity is the masked entity based on the retrieved evidence sentences; computing a contribution to the predicted probability of each evidence sentence used in the prediction; outputting one or more evidence sentences that have the greatest contribution to the predicted probability (i.e. the predicted probability of the candidate entity).

[0070] The machine learning model preferably comprises a machine learning model, preferably an artificial neural network, for example a transformer model. “Evidence” here is intended to mean data within the knowledge base that supports the user-defined requirement of an entity.

[0071] A further implementation of the invention provides an information retrieval method for returning the most relevant data (which may include textual data, including textualised non-text data) from a corpus (knowledge base) to an input query or statement. The task of retrieving specific data relevant to demonstrating a particular requirement or statement is non-trivial. The present method provides a computationally efficient method for retrieving the most relevant pieces of text or non text data (converted to text) that supports a particular input statement. This is particularly advantageously applied to drug discovery, where it allows for searching over large text and textualized non-text data sets to return the most relevant pieces of data that support the efficacy of a specified drug or drug target.

[0072] As with the other aspects of the invention, the method may comprise selecting one or more candidate biological entities and selecting the plurality of evidence sentences from the knowledge base for each candidate biological entity. The method may then comprise inputting the query sentence and corresponding plurality of retrieved evidence sentences into the reasoner model to predict a probability for each candidate biological entity. The method may comprise, for each candidate entity, computing a contribution to the predicted probability of each evidence sentence used in the prediction; outputting one or more evidence sentences that have the greatest contribution to the predicted probability of the candidate entity. In this way, the method may be used to retrieve data for the query sentence across a plurality of different candidate entities.

[0073] Preferably the method comprises outputting n evidence sentences that have the greatest contribution to the predicted probability. N may be set by user input. In some examples the method may comprise outputting all evidence sentences having a computed contribution above a predetermined threshold.

[0074] Preferably the entity is a biological entity, preferably a drug target. Preferably the user-defined requirement defines a relationship with one or more further biological entities. For example it may define that the drug target has a defined relationship with a disease, for example that it is a promising drug target for targeting a specified disease.

[0075] Preferably the method comprises outputting a relevant component of the data store from which a corresponding retrieved evidence sentence is sourced. In particular the method may comprise outputting a file or group of files from which an output evidence sentence derives. For example the method may comprise outputting a biomedical literature article from which an identified evidence sentence is derived. The file or group of files may comprise a numerical data set, for example an omics data set, from which an evidence sentence is derived. This is in the case in which the knowledge base comprises textualised non-text data in which non-text data, such as numerical data, has been converted to text sentences describing components of the data. In this way, the present invention is usable as a data retrieval tool for retrieving multi-modal data relevant to a user query.

[0076] In particular, as described under the other aspects of the invention, the knowledge base may comprise one or more non-text datasets that have been converted to a textualised data set by converting one or more components of the non-text dataset into a corresponding sentence describing the component of the data and including mention of a biological entity.

[0077] The non-text data set preferably comprises one of: genetics or genomics data; transcriptomics data; gene expression data; differential gene expression data; proteomics data; protein-protein interaction network data; clinical data; knowledge graph data; image data. The method may comprise converting one or more components of the non-text data set into sentences describing the component of the data and including mention of a biological entity. In this way, the trained model of the invention may be used to retrieve numerical data relevant to a text query.

[0078] In this example, computing a contribution to the predicted probability of an evidence sentence comprises: determining the probability of the candidate entity with each evidence sentence masked in turn to determine the relative contribution to the probability of each evidence sentence. Computing a contribution to the predicted probability of an evidence sentence may comprise: computing a Shapley value of the evidence sentence used in the prediction. More specifically, the method may comprise computing an estimated Shapley value of an evidence sentence used in the prediction, preferably by computing a Shapley Additive exPlanation, SHAP, value. The (machine learning) model of this aspect may be trained as defined under the above aspects and described within the present disclosure.

[0079] In another aspect of the invention there is provided a method for training a machine learning model to predict a biological entity, the method comprising: providing a training knowledge base comprising non-text data; performing textual ization on the non-text data by converting components of the data to textualized sentences, the textualized sentences comprising text describing the corresponding component of text data; inputting the text sentences into a machine learning model and training the machine learning model to predict a biological entity based on text input. The inventors have determined that machine learning model taking text as input, such as large language models, may be used with nontext data, such as non-text biomedical data, by first textualizing component of the data to convert it to text. This allows methods, previously only usable on text data to be used on non-text data. In the context of biological entity prediction this allows for data sets such as patient level transcriptomics datasets to be leveraged for improved predictions within the drug discovery pipeline.

[0080] In another aspect of the invention there is provided a method of using a trained machine learning model to predict a biological entity, the method comprising: providing a inference knowledge base comprising non-text data; performing textual ization on the non-text data by converting components of the data to textualized sentences, the textualized sentences comprising text describing the corresponding component of text data; inputting the text sentences into a machine learning model trained to predict a biological entity based on text input.

[0081] In some examples the machine learning model is trained on textualized non-text sentences and sentences extracted from the biomedical literature. Preferably, the machine learning model is trained on sentences extracted from the literature only. The inventors have determined that improved prediction results using textualized non-text data are obtained where the encoders have been trained on text data only (i.e. not textualized non-text, such as numerical, data). This ensures the embeddings of the textualized non-text data are closer to those of related literature sentences, and not placed in a separate area of the embedding space.

[0082] Preferably the method comprises masking a component of the textualized sentence, inputting the masked textualized sentence and training the machine learning model to predict the masked component of the textualized sentence. Preferably the method comprises masking a biological entity mentioned in the textualized sentence, inputting the masked textualized sentence and training the machine learning model to predict the masked biological entity of the textualized sentence.

[0083] Preferably the non-text data set comprises one of: transcriptomics data; knowledge graph data; gene expression data; differential gene expression data; protein-protein interaction network data; image data.

[0084] Preferably the method comprises: providing a patient-level dataset comprising numerical data, converting one or more components of the data into a corresponding sentence describing the component of the data and including mention of a biological entity. Preferably the method further comprises storing the sentence within the knowledge base.

[0085] Preferably the method comprises converting a non-text data item to a textualized sentence by converting a numerical data item to a text description of the data item. Preferably the data item is a gene expression measurement and the text description is a description of the relative level of expression, for example low, moderate or high.

[0086] Preferably the non-text data comprises tabulated data, with a plurality of items of the data corresponding to different biological entities, wherein performing textual ization on the non-text data by converting components of the data to textualized sentences, the textualized sentences comprising text describing the corresponding component of text data comprises extracting components of the tabulated data to a template sentence describing a relationship between the biological entities, wherein the biological entity preferably comprise one or more of a gene, a tissue type, a disease, a cell type, a mechanism, a direction, a trait.

[0087] Preferably the output sentence is input into a large language model prompted to provide re-worded sentences and the re-worded sentences are used in a machine learning model for predicting a biological entity.

[0088] The term “sentence” as used herein may be replaced with “text sequence”. It may comprise a partial sentence, single sentence or multiple sentence text sequence.

[0089] An embedding as used herein preferably comprises a feature vector, i.e. a vector encoding important distinguishing attributes of the input data. The term embedding is used interchangeably with the term representation and vector. Preferably an embedding captures meaningful structure of the input by placing meaningfully similar inputs close together in the embedding space. An embedding can be learned and reused across models or at different stages of training.

[0090] Preferably a biological entity comprises one or more of a disease, gene, protein, compound, chemical, drug, biological pathway, biological process or mechanism, anatomical region or entity, tissue, cell-line, or cell type. In particular, a biological entity may be a biological entity type of a type defined above, where each specific biological entity is associated with a unique biological entity identifier from a limited set of biological entity identifiers. The biological entity identifier may be used to provide faster retrieval of relevance sentences may splitting the evidence base based on which entities are included in each sentence. Each biological entity type, for example a biological target, disease or mechanism, may have a limited set of biological entities, each with a unique entity identifier.

[0091] Preferably text is processed by the models by performing tokenisation on the text sequence to form a sequence of word-piece tokens representing the text sequence, where the masked biological entity is represented by a mask token. In other examples, other quantised representations or token types may be used to represent the text of the input sequence. In this way, the plain text input sequence is converted into a format suitable for processing by the model.

[0092] Preferably the machine learning model comprises a language model, where the language model preferably comprises a machine learning model for learning representations of text. The machine learning model preferably comprises an encoder for mapping the input representation to an output representation, where the output representation is a transformed representation of the input representation. Preferably the encoder comprises a plurality of layers and the output representation is formed in the final layer of the model. Preferably the model additionally comprises a classifier layer that receives the output representation and performs classification into classes corresponding to the set of possible biological entity identifiers. In particular, all or part of the output representation may be input to the classifier layer which provides a prediction.

[0093] In a further related aspect of the invention, there is provided a computer- implemented method for identifying biological entities for drug discovery, the method comprising: obtaining a first training dataset comprising a plurality of nontextual data items; training a first machine learning, ML, model, using the first training dataset, to convert each non-textual data item into a text data item comprising at least one sentence that summarises information contained in the non-textual data item; and training a second ML model, using the text data items, to identify biological entities for drug discovery.

[0094] Obtaining a first training dataset may comprise obtaining non-textual data items comprising tabular patient-level experimental data. The non-textual data items may comprise patient data, for example omics, genetics or sequencing data. For example, the first ML model may generate at least one sentence (also referred to herein as a text sequence) that describes an experimental result. The non-textual data items may comprise knowledge graph data. The first ML model may generate at least one sentence describing a relationship between two biological entities within a knowledge graph. The first ML model may convert a triple, such as a (subject, verb, object) triple to a sentence. The method may comprise training the machine learning model using text sequences extracted from a biomedical text corpus and text sentences generated from a non-text data source. The method may further comprise obtaining a set of sentence templates, wherein training the first ML model comprises using the first training dataset and the set of sentence templates. The method may further comprise using a generative model to generate augmented sentence templates to the set of sentence templates, wherein training the first ML model comprises using the first training dataset, the set of sentence templates, and the set of generated augmented sentence templates.

[0095] Using a generative model may comprise using one-shot or few-shot prompting of the generative model to generate augmented sentence templates. Training the first ML model may comprise training the first ML model to: generate a knowledge graph for a non-textual data item, and use the knowledge graph to convert the non-textual data item into a text data item. Training the second ML model may comprise: obtaining a second training dataset comprising a plurality of entity- linked text sequences, each text sequence including a mention of a biological entity, where the biological entity is linked to a corresponding biological entity identifier from a set of possible biological entity identifiers; masking the mention of the biological entity within each text sequence; encoding each masked text sequence into an input representation for the second ML model; and training the second ML model, using the input representations and the text data items, to predict the biological entity identifier of the masked biological entity based on the input representation.

[0096] In a further aspect of the invention there is provided a system comprising a processor configured to perform the method of any appended claim. In a further aspect of the invention there is provided a computer program comprising instructions that, when executed by a computer, cause the computer to perform a method of any appended claim.

[0097] BRIEF DESCRIPTION OF DRAWINGS

[0098] Figure 1 A illustrates a model architecture used in the method of training and using a machine learning model to identify biological entities according to the present invention;

[0099] Figure 1 B illustrates a model architecture used in the method of training and using a machine learning model to identify biological entities according to the present invention;

[0100] Figure 2 illustrates a user interface for implementing a method of identifying biological entities according to the present invention;

[0101] Figure 3 illustrates a method of training a machine learning model to identify biological entities according to the present invention;

[0102] Figure 4 illustrates a method of training an encoder for use the present invention;

[0103] Figure 5 illustrates a method of converting non-text data to text data for use in a method according to the present invention;

[0104] Figure 6a and 6b illustrate a method of training and using a machine learning model to identify biological entities according to the present invention;

[0105] Figure 7 illustrates a system for identifying biological entities for drug discovery according to the present invention;

[0106] DETAILED DESCRIPTION

[0107] Overview The present invention relates to computer-implemented methods and system for making predictions using a knowledge base. More specifically, it is configured to predict the probability of an answer to a user query based on evidence available in the knowledge base. In its broadest implementation it may be applied to any question-answer task, or any task involving the prediction of the likelihood that a statement is true, based on a knowledge base. However, it may be particularly configured to answering biological questions based on a biomedical knowledge base, for example determining biological targets for a particular disease. Several examples of the invention are described below with a focus on a specific exemplary application of identifying new drug targets for drug discovery.

[0108] Biological targets (also referred to gene / protein targets, “protein-coding gene entities” or simply “genes”) encompass any biological component that plays a role in disease processes and may be modulated with a therapeutic compound. The term “biological target” used herein encompasses, for example, genes, nucleic acid sequences, proteins, amino acid sequences, protein complexes, or biological pathways. Although the present invention builds on recent advances in remote field of language modelling, these have been adapted specifically to the problem of biological entity (and more specifically biological target) prediction. This task involves utilising trained language models and an inference corpus to infer candidate targets given a particular biomedical context. This task involves a number of domain-specific challenges over simply predicting masked words given context, a common application of language models.

[0109] The technical challenges include (1) the varying and inconsistent terminology used to refer to biological entities in the literature, (2) the significant word distribution shift from general domain corpora to biomedical literature, (3) the fact that target identification requires identification and ranking of a plurality of targets for further assessment, rather than simply identifying a best-fit word as in general word slot-filling, (4) the lack of hard negatives in the training data to train the model, given it cannot be certain that other targets, other than the specific mention of a target in a training example, do not play a role in the disease in question; (5) the fact the knowledge base is incomplete and the problem involves extrapolating to predict new unknown targets for a specific disease; (6) the need to use multimodal data, including text data, such as a biomedical literature corpus, and multimodal data such as gene expression data and clinical trial data.

[0110] Existing large language model (LLM) -based techniques for predicting biological entities or the relationships between biological entities have made a number of advantages over knowledge graph-based methods. However there are a number of draw backs associated with this type of technique, including (1) the fact that it is difficult to interpret why the model is making a prediction and which aspects of the training data it is relying on to compute the probability of a prediction, (2) such models are large, complex and computationally expensive, reducing the ease with which they can be applied to many important problems particularly in drug discovery, (3) they are only usable on text data and (4) they must be re-trained to introduce new evidence or data.

[0111] The present invention improves on these LLM techniques by separating the knowledge and reasoning part of the model. Figure 1A schematically illustrates an example model architecture usable in the computer implemented methods of the present invention. Figure 1 B schematically illustrates a further example of a model architecture, with corresponding components of the models of Figure 1A and 1 B having the same reference numerals. The model 1 comprises a knowledge retrieval module 110 and a reasoning module 120. The knowledge retrieval module 110 is configured to retrieve relevant evidence from a knowledge base, such as a corpus of textual evidence 111 and the reasoning model 120 is configured to make a prediction based on the evidence retrieved from the knowledge base 111 . This separation of knowledge and reasoning tasks is a core aspect of the invention and allows for increased understanding of why the model is making a prediction, improved computational efficiency since the reasoner model is reasoning across a limited number of evidence sentences rather than an entire text corpus and also provides implementation advantages since a user can apply the model to a new knowledge base, such as their own proprietary datasets, without the model being originally trained on these. The model allows for evidence-based predictions in which the evidence used for the predictions is explicit and can be analysed by the user and explored to understand the importance of each piece of evidence in reaching a prediction outcome. The model can use multimodal data imported into the knowledge base to make the predictions and is tuneable based on user feedback and what constitutes good evidence to make conclusion on a particular type of query. This structure of model can predict, with explanations, efficiently and cheaply, for example providing predictions for candidate biological targets across the full genome.

[0112] The model predicts based on the combination of the query statement and a number of pieces of textual evidence, retrieved from a larger corpus by a retriever model i.e.. output = f( (g(q, ei), g(q, e2), g(q, e3) ... g(q, en) ) where q is the textual query, ei ... enare textual evidence statements, output is a predicted probability (or probability logit) and both f and g are functions, potentially learnable via backpropagation. Thus the model is semi-parametric and the output is a function of the available evidence corpus. By iteratively excluding different combinations of evidence, it is possible to approximate each evidence text’s impact on the final score using game theory approaches such as Shapley values. The model can further be trained to learn that one combination of query and evidence is superior to another, e.g. f( (g(q1 , e11), > , g(q1 , e13) ... g(q1 , e1 n) ) > f( (g(q2, e21), > , g(q2, e23) ... g(q2, e2n) ) where the inequality is learned using a standard pairwise ranking loss such as a margin loss. Using this approach, the model can be finetuned on examples of superior or inferior evidence to reason from evidence according to the approach desired by the end user.

[0113] As shown in Figures 1A and 1 B, the model takes as input an input query 112 defined by the user, which defines a biological requirement for which a biological entity is to be predicted. The biological requirement may be for example a required relationship with another biological entity, for example requesting a biological target which is a promising target for treatment of a particular specified disease of interest. The query 112 takes the form of a text sentence describing the biological requirement and including mention of the biological entity to be predicted. The biological entity to be predicted is masked in the user query 112 such that it is not observable by the model during training or inference. An example of such a user query could be "[target] reduces liver fibrosis in NASH. Here, [target] indicates an unknown biological target which the user wishes to determine. The remainder of the user query defines the biological requirement for which a biological entity is to be predicted, i.e. in this case the biological target must be one for which there is direct or indirect evidence in the knowledge base that it reduces liver fibrosis in NASH.

[0114] The query 112 is input into the knowledge module 110, which is in general a pretrained fixed part of the model, in contrast to the reasoning module 120 which is trained following the method according to the present invention set out below. The knowledge retrieval model 110 works by selecting a candidate biological entity 113, where the candidate biological entity 113 is a possible biological entity which could complete the query 112. These are fed into retriever model 116 which is trained to retrieve sentences from the corpus of textual evidence (the knowledge base) 111 on the basis of a semantic similarity to the input query 112. A plurality of evidence sentences most similar to the input query 112 are then selected as the evidence sentences to be used by the reasoner 112.

[0115] The evidence sentence and the query sentence are input (as their corresponding embeddings 117, 118) into the reasoner model 120, which is trained in such a way as to output computed probability that the candidate target 113 replaces the masked entity in the query 112, i.e. the prediction that the statement of the input enquiry in relation to the candidate target is true. In this way, as described above, the method involves retrieving evidence sentences that support a candidate biological entity meeting the requirement defined by the input query 112 and then inputting those evidence sentences into a machine learning model (the reasoner model 122) in order to output a prediction of the likelihood of the input statement.

[0116] The candidate entity 113 may be defined by the user. For example, the user may input a statement regarding a particular biological entity, such as “target X reduces liver fibrosis in NASH” and the model may compute a probability that the statement is true. More preferably, the user input is a query with a masked biological entity to be determined by the model and the model predicts over a plurality of possible candidate biological entities to rank the candidate biological entities by the predicted probability that they meet the requirement defined in the input query 112. The input query may preferably be in the form of a cloze-style statement, or converted into a cloze-style statement, (e.g. [MASK] is a promising drug target for the treatment of osteoporosis).

[0117] Figures 1A and 1 B show a number of more specific preferable features of the model usable in the methods of the present invention. The knowledge or “retriever” module 110 is preferably based around a pre-trained vector search model. In particular, the retriever 116 functions by encoding the query 112 and sentences from the knowledge base 111 into embeddings and determining the similarity of the embeddings in order to retrieve the most relevant sentences from the knowledge base 111 using the reasoning module. The encoders may take number of different forms that will be described in more detail below, but preferably they are based around the transformer architecture and are pretrained to encode an input sentence into an embedding that encodes information regarding the semantic meaning of the sentence. For example, the encoder may be trained using a masked language modelling (MLM) objective so as to learn contextual information in an input query which is usable to determine the semantic meaning of the input sentence.

[0118] The encoder may be, for example, further trained on the task that supports the objective for which the model will be used at inference. For example, where the model is used to identify biological targets the query encoder may have been pretrained to on a target prediction task. For example it may have been trained by inputting sentences from the biomedical literature including mention of a biological target, masking the mention of the biological target and training the encoder to learn embeddings usable to predict the biological target. In this way the encoder is specifically configured to encode an input sentence into an embedding which stores relevant information for a target ID task. A suitable type of encoder is that used in the model described in W02023 / 089304, and further details on the training process are provided below. All of the entity’s evidence passages may be encoded prior to inference, using the same encoder, and stored in a FAISS search index, n evidence passages with greatest relevance to the query are then retrieved at inference time. The relevance may be determined by computing a similarity metric between the embeddings, for example by computing cosine similarity.

[0119] As described above, the knowledge / retriever module 110 is fixed during training of the reasoning model 120 and acts to identify relevant sentences from the knowledge base 111 and pass these as input into the reason model 120. As shown in Figure 1 , embeddings 118 of the evidence sentences retrieved by the retriever model 116 are each input together with the query embedding formed by the querying encoder 114 into a respective query-evidence encoder 121. The queryevidence encoder is configured to learn a vector representing the similarity between the query and the evidence sentence and each query-evidence embedding is fed as a sequence into the reasoner model 112. Figure 1 B illustrates a preferable query-evidence encoder, in which the query / evidence pair embeddings are layer-normalised and then each pair is combined at corresponding dimensions into a single embedding 119 using convolutional layers. More specifically, the query-evidence encoder independently combines each of the k evidence embeddings with the query. It stacks the evidence with the query to generate a tensor of size [2, h]; it then compresses the tensor into a vector of size [1 , h] using convolutional layers. The convolutional layers have a filter size of [2, 1] across each embedding dimension h, encoding the relationship between the query and evidence in each dimension. This single query-evidence embedding 119 is then input into the reasoner model 122.

[0120] The reasoner model 122 comprises a neural network 123 followed by a linear layer 124 and sigmoid to obtain the binary probability for the input candidate entity 113. The process is iterated over each candidate entity to obtain a binary probability for all candidate entities. The neural network preferably comprises a transformer architecture but does not include sequence embeddings so as to not place any weight on the order in which the evidence sentences are input into the reasoner model 112. The reasoner model may preferably comprise a Set transformer 122 (Lee, J.,et al. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 3744-3753, 2019), as shown in Figure 1 B. The reasoner model 112 maps the sequence of query evidence factors to a sequence of output vectors which are fed into an output layer 124 which map these output vectors to a prediction of whether the candidate biological target 113 completes the input query 112. The reasoner model and query evidence encoders are trained together end-to-end, as will be described in more detail below. The reasoner model is structured so as to look at how the query sentence is related to the evidence sentences and use this to predict the probability that the given candidate biological entity is an answer to the input query sentence.

[0121] Figure 1b also illustrates the explain module 140, which computes Shapley values 142f or each query-evidence pair (corresponding to each piece of evidence). The output of the model is therefore binary probabilities determined independently for each entity as described above, in order to provide a ranked list of the biological targets (or other entities). The computed Shapley values attribute model predictions back to the evidence passages providing an explanation of the model’s prediction. The Shapley values may be computed by masking individual pieces of evidence to evaluate how they contribute to the prediction.

[0122] Multiple methods exist for rapidly approximating Shapley values on deep learning features (Lundberg & Lee, 2017). Defining each of the k inputs to the evidence combiner as a distinct feature gives a relatively small feature space. As a result, it is tractable to use a simple permutation sampling approach to approximate Shapley values. In one example, during training, query-evidence features are replaced at random with a learned NULL embedding. In addition to acting as a regularizer (akin to dropout), introducing the NULL embedding during training ensures that the model can handle missing features robustly when estimating Shapley values. For each training example, a dropout rate may be sampled in Uniform(0, 1) to ensure that the model observes both high-evidence and low- evidence scenarios. Figure 2 illustrates an example of the output of the model of Figure 1 to the user through a user interface 200. The user interface shows the query "[target] reduces liver fibrosis in NASH", with a candidate target (target X) completing the query. The user is shown the returned evidence sentences 202A, 202B, 202C and 202D and their individual contributions to the total predicted probability 204Athat "target X" is a correct answer to the requirement defined in the displayed input query 201 . In this way, the user can directly see the evidence that the model is using to provide the prediction, providing much greater transparency and understanding than existing large language model techniques. As will be described in more detail, it also allows the user to assess the relative weights of the evidence sentences 202 to the prediction and fine tune the model.

[0123] Training

[0124] The model is trained by providing a training knowledge base comprising corpus of textual data and providing a training dataset comprising training query sentences, each comprising text describing biological entity in which a mention the biological entity itself is masked for prediction by the model, where the query sentence has a known correct biological entity. The training query sentences may be any sentences from the literature that have a known biological entity and describe some property of the biological entity, such as a relationship with one or more other biological entities. In other words, the training corpus comprises textual passages each containing at least one named entity from the answer set (for example each gene target to be assessed and ranked). For each passage, the span of every occurrence of the named entity is replaced by a [MASK] token. When multiple unique answers occur, a new passage is generated with each masked in turn while the others appear as plain text. In this case, the query embedding comprises the mean over the output embeddings of each sentence (i.e. corresponding to the [MASK] token in each alternative sentence). Positive and negative training examples are created by using the correct biological entity and incorrect biological entities as the input candidate biological entity 113 respectively. In the specific case of drug target identification, the answer set (i.e. the set of query targets 113) consists of 19,176 protein-coding gene entities, also referred to collectively herein as Genes.

[0125] For each combination of the training query sentence and the candidate entity (which may be a correct or incorrect biological entity), the training query sentence is encoded with the trained query encoder to form a query embedding 117. The query embedding is compared by the retriever 116 to embeddings of sentences from the knowledge base 111 including mention of the same candidate (i.e. correct / incorrect) biological entity. In some examples, the query and knowledge base may comprise entity-linked sentences and the entity linking is used to identify appropriate sentences from the knowledge base. For example sentences are retrieved that include the same entity and these sentences are encoded to be compared by the retriever model.

[0126] The query embedding and each sentence embedding may be compared by computing a similarity metric, for example the cosine similarity between the embeddings. Either a given number, N, of the most similar evidence sentences, or a number of evidence sentences with a similarity above a threshold are then passed in the form of their embeddings 118 to the reasoner module 120, together with the query embedding 117. As described above, the query encoder 114, evidence encoder 115 and retriever 116 of the retrieval module 110 remain fixed during training of the reasoner 122 and query-evidence encoders 121 of the reasoning module 122. The query and evidence encoder could be any model for retrieving relevant text passages from a corpus, for example IF / IDF, BM25, BERT, or pubmedBERT encoder. In particular the query and evidence encoder preferably comprises a language model configured to encoder input text into text embeddings, where semantically similar text sequences are embedded closer together in the embedding space. Specific ways in which the query and evidence encoders 114, 115 may be trained and optimised for retrieving evidence are described below.

[0127] The encoder may be used to embed each of the evidence sentences ahead of time, such that the evidence sentences are stored as embeddings. The embeddings may be portioned according to the entity that they mention to create a distinct FAISS search index (Johnson, J., Douze, M., and J'egou, H. Billionscale similarity search with GPUs. IEEE Transactions on Big Data, 7 (3):535-547, 2019) for each.

[0128] The query-evidence encoders 121 and reasoner module 122 are preferably trained end-to-end to learn to predict that an input incorrect biological entity 113 is not the answer to the query (does not replace the masked entity in the input query 112) and therefore output a low probability as the prediction 130, and that the input correct biological entity 113 (i.e. the biological entity from the original sentences extracted from the literature as a training example) is the correct entity that replaces the masked entity in the input query, assigning a high probability.

[0129] Each query-evidence encoder takes as input the query embedding 117 and an evidence embedding 118, both vectors of size [1 ,d], and outputs a single vector (or query-evidence embedding) also of size [1 ,d]. The query-evidence encoder is therefore configured to combine the query embedding 117 and an evidence embedding 118 into a single vector. It does so in a way that encodes a measure of similarity between the input embeddings 117, 118. Various possible types of encoder may be used to achieve this. In one example, the query-evidence encoder could compute the Hadamard product of the embeddings to provide a vector giving a measure of the similarity. In particular, if the query and evidence encoders have been trained in such a way to encode semantic information in the input sentences as will be described below, then the Hadamard product gives a good measure of the semantic similarity of the sentences.

[0130] In other preferable examples the query-evidence encoder may comprise a convolutional network which provides output a vector representing the similarity of the inputs. In particular, by stacking q and e together into a matrix of size [2, d] and learning convolutional filters of size [2, 1] across the different elements of the matrix during training, the output vector provides a good measure of similarity usable by the reasoner module. The convolutional network gives more capacity than the Hadamard approach, while still being simple (avoiding overfitting to the query) and enforcing the inductive bias that corresponding elements of q and e correspond to similarity. The vectors (“query-evidence embeddings”) output by the query-evidence encoders are formed into an input sequence that is input into the reasoner model 122.

[0131] The reasoner model 122 comprises a machine learning model that takes the sequence of query-evidence embeddings as input and transforms these through the network to a sequence of output embeddings that are usable to generate a prediction that the input candidate biological entity 113 is the correct answer to the query statement 112. The reasoner model 122 preferably comprises a series of neural network layers for transforming the sequence of input embeddings through the network to a sequence of output embeddings. The reasoner model is preferably a transformer-based model. Since there is no inherent ordering between the different query-evidence embeddings, a set transformer may be used. All query-evidence embeddings are combined into a single set of output embeddings that is converted to a probability by one or more output layers. The output layers preferably comprise an output layer that projects each of the N output embeddings to a single scalar per output (N scalars). These scalars may be summed to get a single scalar final score and applying a sigmoid to output a probability based on that score. The output layer there is configured for binary classification, true / false, on whether the candidate entity 113 is the correct masked entity in the training query.

[0132] During training the training query 112 is therefore fed into the query encoder. For a positive training example, the input candidate entity 113 is the original entity from the sentence used for the training query 112 (i.e. the original sentence extracted from the literature), the embedding of the retrieved evidence sentences and the query embedding are fed to each query-evidence encoder, and the output vectors are fed to the reasoner model 122. The query-evidence encoders 121 and reasoner model 122 are trained end to end so that the query evidence encoder 121 learn to form embeddings usable by the reasoner model to output an improved prediction, while the reasoner model learns to transform the input queryevidence embeddings to output embeddings usable to provide an improved prediction. The reasoner 120 is trained with a binary cross entropy loss to differentiate the positive examples from negative examples. For each candidate entity query pair, the reasoner receives the top k evidence passages fetched by the Retriever from the retrieval corpus.

[0133] For negative examples, the input candidate entity 113 is an entity other than the correct original entity appearing in the sentence used for the training query. The incorrect candidate entities are randomly sampled from the set of possible candidate entities (i.e. the set of all possible biological targets in the case of predicting a target as the masked entity). In some examples, the incorrect candidate entities are randomly sampled according its frequency of mention in the literature, so that so well-studied targets come up equally often in negative and positive examples. This ensures that the model can’t rely on how well-studied a target is as a signal. In other examples, the candidate entities may be sampled uniformly and a post-hoc frequency bias correction applied.

[0134] Many potential answer sets, particularly those in drug target identification, can suffer from high class imbalance; i.e. some targets are significantly more well- studied than others. As a result, the output of a learned model p(a q) can be strongly correlated with the prior pCaJ. While bias can be informative (e.g. reflecting the fact that some targets are involved in more diseases than others) it can also be obstructive (e.g. reflecting publishing trends rather than underlying biology). To flexibly control for bias, it is possible to use a corrected output: lnp (a g) - c ■ InpCaJ, where C E [0, 1] and p(af) is computed from the frequency of the answer a, in the training corpus. When c = 1 this reflects a pointwise mutual information (PM I) score. The parameter c can be varied to adjust the level of compensation for the imbalance in mentions across entities in the literature.

[0135] The reasoner model preferably comprises a Transformer model, but without positional embeddings so that no weight is placed on the order of evidence input into the model. The reasoner model may be based on the BERT architecture, where instead of the sequence of token embeddings taken as input the model takes a sequence of query-evidence embeddings as input. The model would therefore look similar to the model shown in Figure 4 to illustrate training of the query encoder, but instead of the work tokens as input at 403 it is the query- evidence embeddings. In some examples the reasoner model may comprise a Set Transformer (Lee, Juho, et al. "Set transformer: A framework for attentionbased permutation-invariant neural networks." International conference on machine learning. PMLR, 2019. arXiv: 1810.00825).

[0136] In some examples the model can be trained whilst requiring that evidence sentences retrieved during training are from the literature sentences published prior to the literature sentence used as the query sentence. In this way, the model is encouraged to extrapolate from the evidence it receives to predict new answers, such as new drug targets for use in drug discovery.

[0137] Fine-tuning training

[0138] Because the model of the present invention reasons explicitly on evidence sentences, it is possible to fine-tune the model in the way that it reasons on the evidence sentences. In particular, using a fine-tuning process, the model can be trained to place a greater weight on certain types or combination of evidence to form its predictions. In the context of predicting biological entities for drug discovery, the model can be fine tuned to focus on specific evidence combinations that match drug discovery goals.

[0139] The fine-tuning approach involves telling the model what constitutes an optimal set of evidence for the prediction task so that, after fine-tuning, the model will provide increased prediction probabilities for entities supported by evidence sets closer to the optimal set of evidence.

[0140] The optimal set of evidence (also referred to as the Constitution) may comprise a number of evidence statements that drug discovery experts look for in determining a promising drug target, for example involving biological relevance, novelty, expression, and complementarity. Using this optimal evidence set, an LLM may be prompted to rank two sets of evidence retrieved by the retrieval module 110 of the present invention. This ranking may be used to provide a label indicating which is the better evidence set and this label can be used to train the reasoner model to rank the better sets of evidence higher than the worse sets of evidence, for example using a margin ranking loss.

[0141] An examples of the fine-tuning set up is shown in Figure 3. This involves using a sample query 301 and inputting into the retriever module with two difference candidate entities 313a, 313b, to retrieve two corresponding set of evidence 318a, 318b. The sample sets of evidence 313a, 313b are then assessed relative to the optimal set of evidence - the “Constitution”.

[0142] The constitution may be a set of statements, determined by experts, or an automated model, as those that need to be fulfilled for an entity to meet a particular requirement. For example, in the case of query relating to identifying biological targets for a disease of interest, and associated with a tissue of interest, cell type of interest and mechanism of interest, the optimal evidence statements could be, purely as an example:

[0143] - Disease relevance and novelty o (S1) Target X is causally associated with the disease o (S2) Target X causally associated with other related diseases or traits o (S3) Target X is not already a well known potential target for the disease (i.e. Target X could be novel for the disease)

[0144] - Mechanism relevance o (S4) Target X has a critical role in the mechanism of interest o (S5) Target X plays a role in other related mechanisms o (S6) Target X is part of one or more signalling or metabolic pathways relevant to the mechanism

[0145] Tissue and cell type expression o (S7) Target X is expressed in the tissue of interest o (S8) Target X is expressed in the cell type of interest o (S9) Target X is differentially expressed between patients with the disease and control individuals

[0146] The better evidence coalition is the one that most convincingly and comprehensively supports each of these 9 statements (S1-S9).

[0147] A sample user query may then be used to fine-tune the model, for example the user query sentence may be formed based on a user input specifying a desire to find therapeutic targets Parkinson’s disease, that are associated with the mechanism neuroinflammation-induced cell death, the tissue: dopaminergic neurons, and cell type: substantia nigra.

[0148] A query sentence may be formed based on the specification of these biological entities and input into the model with two different candidate targets to retrieve two difference evidence coalitions. The retriever module may return the follow for each respective query:

[0149] Evidence coalition for target A o A1 - These results indicate that target X plays an important role in mediating microglial activities and suggest that target X reduction may be self-defence of microglia against inflammatory insults. o A2 - Results suggested that target X significantly ameliorates PD-related phenotypes including extending lifespan via degradation of a-syn, which suggested a pivotal role of DCF1 in the onset and development of neurodegenerative diseases. o A3 - We also nominate the key microglial gene and clopidogrel target target X as a candidate driver of a PD GWAS signal. o A4 - This makes target X a prospective target for novel therapies, aimed to modulate the inflammatory diseases and neurogenesis / neurodegeneration. o A5 - Furthermore, target X inhibition potentiated glial activity to scavenge glutamate toxicity and prevented glial cell senescence and mitochondrial damage, which have recently been reported as critical pathologic features of and therapeutic targets in Parkinson disease (PD) and Alzheimer disease (AD). o A6 - Similar to the pathology observed in MS, ALS and AD, the protein levels of target X were increased in the post-mortem substantia nigra of individuals with PD compared with controls. o A7 - Knockdown of target X suppresses autophagy in macrophage. o A8 - Target X inhibitor ameliorates the MPP+ / MPTP-induced Parkinson's disease through the ASK1 / JNK signalling pathway. o A9 - Inhibition of target X kinase activity by necrostatin-1 (Nec-1 ) protected PC12 cells from cell death induced by 6-hydroxydopamine, a mitochondrial respiration inhibitor used to induce a model of PD, suggesting that target X activation might be involved in the loss of dopaminergic neurons in PD.

[0150] Evidence coalition for Target B

[0151] B1 - One recent work noted that target X deficiency attenuated microglia activation during neuroinflammation, but the underlying mechanism was unknown.

[0152] B2 - We hypothesized that target X may alleviate PD symptoms by degrading a- syn protein.

[0153] B3 - Despite the uniqueness of the locus structure, we will focus on target X as the primary candidate nominated by the PD GWAS browser data sets. B4 - Based on our in vitro data, we next sought to determine whether the pharmacologic inhibition of target X could forestall the degeneration of mDA neurons and associated behavioural deficits in PD.

[0154] B5 - The main purpose of this study was to test whether target X inhibition in glial cells could be used to develop a therapy for PD.

[0155] B6 - What is more, a recent report indicated that target X is upregulated in the substantia nigra of 1-methyl-4-phenyl-12,3,6-tetrahydropyridine (MPTP)-induced PD mice

[0156] B7 - However, the underlying mechanisms responsible for the neuroprotective role of target X in PD are still unclear.

[0157] B8 - Given the intimate link between PINK1 and mitophagy, we reasoned that target X-mediated stabilization of PINK1 may cause mitophagy during ECM detachment.

[0158] B9 - However, by examining the accumulation of newly synthesized protein through shut down of proteasomal protein degradation with MG132, the protein synthesis rate detected was significantly suppressed in target X knockdown cells.

[0159] A LLM is then prompted to output whether the optimal evidence statements are supported or not supported by the returned evidence as follows:

[0160] Step 1 : Group A evaluation

[0161] 51 - Supported: A2 and A5 suggest that Target X ameliorates PD-related phenotypes and has a role in the pathology of PD.

[0162] 52 - Supported: A5 mentions Target X's involvement in Alzheimer's disease (AD) as well.

[0163] S3 - Supported: A3 nominates Target X as a candidate driver of a PD GWAS signal, implying it is not a well-known target. 54 - Supported: A1 and A5 indicate that Target X has a critical role in mediating microglial activities and modulating inflammatory diseases.

[0164] 55 - Supported: A7 mentions Target X's role in autophagy in macrophages, which is a related mechanism.

[0165] 56 - Supported: A8 shows that Target X is part of the ASK1 / JNK signalling pathway relevant to the mechanism.

[0166] 57 - Supported: A6 states that Target X protein levels were increased in the substantia nigra of individuals with PD.

[0167] 58 - Supported: A9 suggests that Target X activation might be involved in the loss of dopaminergic neurons in PD.

[0168] 59 - Supported: A6 shows differential expression of Target X between patients with PD and controls.

[0169] Step 2: Group B evaluation

[0170] 51 - Supported: B1 and B2 suggest that Target X deficiency attenuates microglia activation and may alleviate PD symptoms.

[0171] 52 - Not supported: Group B sentences do not mention Target X's association with other related diseases or traits.

[0172] 53 - Supported: B3 focuses on Target X as the primary candidate nominated by the PD GWAS browser data sets, implying it is not a well-known target.

[0173] 54 - Supported: B1 and B5 indicate that Target X has a role in microglia activation and could be used to develop a therapy for PD.

[0174] S5 - Not supported: Group B sentences do not mention Target X's role in other related mechanisms. S6 - Not supported: Group B sentences do not mention Target X being part of any signalling or metabolic pathways.

[0175] 57 - Supported: B6 mentions that Target X is upregulated in the substantia nigra of MPTP-induced PD mice.

[0176] 58 - Not supported: Group B sentences do not mention Target X's expression in dopaminergic neurons.

[0177] 59 - Not supported: Group B sentences do not mention differential expression of Target X between patients with PD and controls.

[0178] Step 3

[0179] Conclude that Group A most convincingly and comprehensively supports the evaluation statements (S1-S9) as it addresses all nine statements, while Group B only supports five of them. On the basis of this a binary label may be applied to indicate that the group A evidence is stronger and the model may be trained as shown in Figure 3 using a loss function based on the label, for example a margin ranking loss. In this way, the reasoner and query-evidence encoders are trained to recognise stronger combinations of evidence and provide an increased probability prediction forthose types of evidence. The example of target prediction has been described but this approach could equally be applied to other predictive tasks by determining an optimal evidence set for those predictions.

[0180] Here the fine-tuning has been described as an offline pre-training step ahead of use for inference. However, the fine-tuning functionality could be employed in a user application based on the model. For example, the user may be presented with returned evidence based on a query in a user interface as shown in Figure 2. The user interface may be configured to receive a user selection of evidence statements 202 displayed on the user interface, for example by selecting or deselecting the evidence using button 205. The selection of user evidence statement may be used to fine tune the model to provide greater weight to this combination of evidence. In this way, during use the model may be fine-tuned based on user feedback to adapt to user indications of what constitutes the best evidence. The model may therefore be continually trained based on manual feedback during deployment.

[0181] Inference

[0182] At inference time a user inputs a query defining a requirement for which a biological entity is sought. The requirement (or “biological requirement”, “biological question” or “biological context”) defines one or more conditions or properties of the entity that should be met, for example “a promising target fortype-ll diabetes" or

[0183] “targets that improve neuronal health by boosting lysosomal function in lysosomal storage diseases (with the disease model of neuronal ceroid lipofuscinosis, CLN3) by stimulating lysosome biogenesis in cortical neurons, as indicated by decreasing substrate accumulation, or by regulating autophagy, as measured by a change in pH un U2OS cells".

[0184] The trained model then outputs a probability that a selected candidate entity meets the requirement. The candidate entity could be selected by the user to provide a probability specifically for one candidate entity of interest or, more usually, the probabilities can be calculated for a number or all possible candidate entities, such as all genes within the genome. In particular the model may be configured to run a large number of queries simultaneously to return a probability for every possible biological entity of the type indicated in the query. In the case of predicting a biological target the model may run 20,000 queries simultaneously - one for every gene in the genome. The results may be output to the user a ranked list of biological entities based on the computer probability for each.

[0185] The query may be input in a number of different ways, such as by writing a free text query or selecting biological entities of interest as described below, but in all cases is preferably converted to a text statement about a biological entity, in which the actual mention of the entity is masked to the model, such as [MASK] is a promising target for type-ll diabetes. This query sentence is then encoded into a query embedding, as described under the training procedure (since the knowledge retrieval module 110 is the same for training and inference) which is compared to sentence embeddings from the knowledge base to return evidence sentence embeddings for use in the trained reasoner model to output a probability for one or more candidate entities 113 to give a measure of the likelihood that they meet the requirement of the query.

[0186] The user query may be referred to as a “biological question” or “biological requirement”, which species a particular biological context relating to a disease for which a biological target is required. The term “biological context” again is intended to refer to a description of one or more biological processes or mechanisms relating to a disease of interest in which, in the case of prediction, an unknown target plays a role. It can be considered a statement of biological properties or criteria that a biological target is desired to have. The query could be entered as free text, for example:

[0187] / s it possible to find targets that improve neuronal health by boosting lysosomal function in lysosomal storage diseases (with the disease model of neuronal ceroid lipofuscinosis, CLN3) by stimulating lysosome biogenesis in cortical neurons, as indicated by decreasing substrate accumulation, or by regulating autophagy, as measured by a change in pH un U2OS cells?

[0188] This biological query specifies a disease of interest (lysosomal storage diseases) and a number of additional parameters relating to the biological context including cell types, tissue type and biological mechanisms. By defining a biological context in this way, using free text, a user is able to specify the exact requirements for the biological target, in an unrestricted degree of precision. This demonstrates the improvement of the present method over common existing target ID techniques using knowledge graphs, in which only a limited number of biological entities can be defined in an input query. The user may enter the user query using a text input box in the user interface as described below.

[0189] The user query may be converted to define it in terms of a cloze-style statement regarding a biological target in which the mention of the biological target is masked. In particular, in this example, the user query is framed in the form of a question and the natural language text sequence is rephrased slightly in the form of a statement about an unknown target, such as

[0190] [MASK] is a promising therapeutic target to improve neuronal health by boosting lysosomal function in lysosomal storage diseases (with the disease model of neuronal ceroid lipofuscinosis, CLN3) by stimulating lysosome biogenesis in cortical neurons, as indicated by decreasing substrate accumulation, or by regulating autophagy, as measured by a change in pH un U2OS cells.

[0191] Although not essential, the reframing of the biological question into the input sequence brings the format of the input sequence closer to the training examples, to frame the question as a statement relating to a hypothetical target (where the mention of the target itself is masked).

[0192] In other examples, a user may just specify the important biological entities (for example the disease of interest, a biological mechanism, cell type and tissue type) and an initial module of the model may be configured to automatically generate a natural language text sequence, in the style of the training examples, based on the input biological entities. In particular, the user may enter a disease one or more predicate mechanism pairs, a tissue and a cell type. There could be more or fewer fields or the user may be able to select the number of fields they complete and the query text sequences are generated based on just the completed fields.

[0193] The method may use one or more templates to create natural language text sequences of a style similar to the sentences selected from the biomedical text corpus on which the model is trained. This presents a more efficient, user-friendly method in which the user need only identify particular biological entities of interest, and the required input text sequences are generated automatically for input into the model.

[0194] In some examples, the knowledge base 111 used for inference may differ to the knowledge base 111 used for training. For example, the model may be trained on publicly available data such as a large corpus of text data extracted from the biomedical literature. The trained model may be provided as a tool to users in which at inference time, they may use their own knowledge base to made predictions. This is important as often user’s may have proprietary or use-case specific data that they wish to use for inference and the current model allows them to deploy the model on their own data, without having to release this for training. This is an important distinction over LLM-based techniques, where the models make predictions based on the training data as a whole. With the current model architecture, the model can be asked to make predictions based on a different knowledge base to that used for training, allowing better data security and meaning users do not need to release valuable proprietary data.

[0195] The model may further include an additional evidence contribution module configured to calculate a contribution of each evidence sentences to the output predicted probability. The method may comprise, for each evidence sentence calculating a contribution to the predict probability of a candidate biological entity. The calculated contribution may be based on Shapley values, using any known method of estimating Shapley values, such as using Shapley Additive exPlanation (SHAP) value or using the Data Shapley method (Ghorbani, Amirata, and James Zou. "Data shapley: Equitable valuation of data for machine learning." International conference on machine learning. PMLR, 2019. arXiv: 1904.02868). The computed contributions may be displayed to a user to allow a user to assess the relative contribution of different evidence sentences.

[0196] For example, the evidence sentences as their relative contributions may be output as shown in Figure 2, where the retrieved evidence sentences 202 for an input query 201 are displayed, illustrating their contribution to the prediction probability 204. Unlike LLM approaches, the present method allow for direct assessment of the contribution of evidence to a prediction.

[0197] Another important feature of the method is that it can be applied to multimodal data sets, rather than purely text data. In particular, the knowledge base may comprise different types of data, rather than purely literature derived text sentences. For example, the knowledge base may comprise textualized non-text data, such as patient-level transcriptomics data that has been processed to convert it to text sentences retrievable for use by the reasoner model 122.

[0198] Transcriptomics (omics) data includes tissue expression and differential expression data, where tissue expression data shows how much mRNA, for a given gene, is expressed in the desired tissue or cell and differential expression data shows whether there is more or less mRNA, for a given gene, expressed in the disease or healthy state. Transcriptomics data is very important for drug discovery as differentially expressed genes may be causal in the disease aetiology and knowing a gene is expressed in the target tissue or cell increases confidence in it as a drug target. This type of substantially numerical data is not possible to leverage with existing LLM-based techniques, but the present invention can utilise this type of data by applying a textualization process to convert it to text and then storing the text sentences in the knowledge base 111 to be used for predictions of biological entities, for example Target ID for drug discovery.

[0199] The textualization method comprises providing a non-text dataset such as a transcriptomics data set and converting a portion of the data to a descriptive text statement describing the data. For the example of transcriptomics data, this comprises setting thresholds defining different expression categories, such as “lowly expressed”, “moderately expressed” and “highly expressed” and assigning the categories to each data item based on the thresholds, so for a sample where a very high reading of expression was taken this would be assigned the “highly expressed” category. The sample can then be converted to a sentence describing the sample result, for example “A1 BG was highly expressed in the skeletal muscle myoblasts within calf muscle in Duchenne muscular dystrophy”.

[0200] T ranscriptomics data sets will generally comprise tabular data, giving for example, the cell type, tissue, disease state, gene name and expression measurement. The method preferably comprises converting this tabular data to a text sentence using a template, for example [gene] was [lowly / moderately / highly expressed] in [cell type] within [tissue type] in [disease]. Where there are a large number of samples, the method preferably involves taking the consensus expression category for each unique gene / disease / tissue / cell type combination and using this category in the textualized sentence.

[0201] An example method is shown in Figure 5. At step 501 a dataset of transcriptomics data is provided. At step 502 unnecessary or unwanted information is removed. At step 503 samples are grouped by unique combination of project, gene, disease, tissue, cell type. At step 504 for each unique group of gene, disease, tissue, cell type, samples are taken from the project with the highest number of samples. At step 505 the samples are assigned an expression category and at step 506 the consensus expression category for each unique gene, disease, tissue cell type combination is taken and the data is substituted into templates 507. The templates 507 may be augmented by inputting the templated sentences into a large language model prompted to provide to output alternative or reworded sentences describing the same relationship between the relevant biological entities. For example the large language model may be prompted to provide more “literature like” sentences, for example by prompting the model to reword the sentences in a style consistent with the biomedical literature. This may be involve 0 shot or few shot prompts, in the latter cases providing the model with examples of literature sentences, prompting it to replicate the style. Template augmentation can create a diverse set of textual isation templates that retain the same meaning, using zero-shot or few-shot prompting of a generative large language model, to prevent an ML model overfitting to repetitive surface form components of any single template.

[0202] Although this method is described for tissue expression data it can equally be applied to any other type of data that can be described as a text sentence. In one example it may be applied to differential expression data by initially taking differential expression database, comprising a plurality of datasets, grouping these by unique disease-tissue-cell type contexts and taking the top N comparison I Ds for each based on the amount of control and case samples. The data can then be binned into categories based on relative gene expression thresholds. For example the data can be binned into “lowly up / down regulated”, “moderately up / down regulated” and “highly up / down regulated”. This binning process could be case on the magnitude and direction of fold change. For example lowly up / down regulated could be assigned to data where 1 .0 < Iog2fc < 1.5, moderately up / down regulated could be assigned to data where 1.5 < Iog2fc < 2.0 and highly regulated could be assigned to data where Iog2fc > 2.0.

[0203] After binning the data using this process, the data can be substituted into sentence templates based on the assigned categories, for example “[MASK] is (magnitude) (direction_of_regulation) in the (tissue) (cell_type) of a patient with (disease). Again, these templates sentences can be augmented using a LLM to provide more literature like sentences. In this way, differential expression data (and analogously other types of numerical I tabulated data) can be used in the methods for predicting biological entities described herein.

[0204] The method of textualizing non-text data can also be applied to knowledge graph data, to convert relationships between entities to a sentence describing the relationship, protein-protein interaction data and image data. Also, although it has been described using a simple template approach, in other examples textualization can be achieved by training a machine learning model to convert a specific data type to text sentences.

[0205] Using this method, a user may utilise their own multimodal data sets for inference. It is not necessary that the model has been trained on multimodal data, given the templates convert the data to a format similar to literature style sentences which the model has been trained on. However in some example, training comprises training on textualized non text data, such as textualized transcriptomics data. Furthermore, improved results are actually achieved where the query encoder and sentence encoder have been pretrained on literature only data, so that the textualized data embeddings are closer to the literature derived sentence embeddings.

[0206] As shown in the example user interface of Figure 2, the different data modalities may be retrieved and displayed to the user, with the corresponding contribution to the total predicted probability 204a. Figure 2 shows two evidence states 202a and 202b from the biomedical literature, including links for the user to access the original document. The user interface also displays a statement 202c comprising a textualized result from patient level datasets as well as an evidence statement 202d comprising a co-operating combination from multiple sources, here a knowledge graph derived statement and a statement derived from another data source.

[0207] Retriever encoder training

[0208] As described above the retriever module 110 comprises a query encoder 114 that encodes the query 112 as an embedding 117 and an evidence encoder 115 that encodes sentences from the knowledge base 111 as evidence embeddings 118. The encoders are preferably configured such that similar embeddings represent similar sematic content such that the retrieved evidence is relevant to the query. The encoder could take any pre-trained language model encoder that is intended to encode an embedding with the semantic content of the sentence, for example a pre-trained BERT encoder, or a BioBERT encoder to give the encoder greater biomedical domain understanding. Preferably the encoders are such that evidence sentences that indirectly support the query statement are retrieved. That is, they are not just linguistically or semantically similar but they contain information that indirectly supports the statement of the query sentence. For example an evidence sentence that provides evidence relating to an related underlying mechanism of a disease specified in the query would provide useful evidence and so should be encoded in a similar area of the embedding space. One way of achieving this is to pre-train the query and evidence encoder on an objective that supports the deployed use of the model. For example, if the model is primarily to be used for target-ID the encoder could be pre-trained on a target- ID task. Away of preparing a suitable encoder is described in W02023 / 089304.

[0209] A training method for a query / evidence encoder is illustrated in Figure 4. Figure 4 schematically illustrates a possible architecture of a machine learning model 400, including a transformer encoder 408 that can be trained using the illustrated method and then extracted for use as the query and evidence encoder of Figure 1 . The input to the model is a text sequence 401 describing a particular biological context including a mention of a biological target 402 which is masked to the model for prediction. In the example of Figure 4 the input sequence 401 is “[X] is a promising target for type-ll diabetes".

[0210] The first stage is to encode the text sequence 401 into an appropriate input representation for the machine learning model 400. In this case the machine learning model is based on the Transformer architecture (Vaswani A et al. "Attention is all you need." Advances in neural information processing systems 30 2017) and comprises a transformer encoder stack 408 which is configured to map the input representation of the text sequence to an output representation which is useable by a classification layer and softmax 409 to predict the unique entity identifiers 411 of the potential biological targets 412. The transformer encoder 408 may be pre-trained and initialised with the pre-trained model weights, which are then further fine-tuned by the training method illustrated in Figure 4. However, given the present training method may utilise readily available large biomedical training data sets, pre-training is not essential and the inventors have identified that the model actually achieves increased performance by training solely based on the classification training task described below.

[0211] The transformer encoder stack 408 may be based on the BERT model architecture (Devlin, Jacob et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv: 1810.04805 2018), or, more preferably, the BioBERT encoder (Lee, Jinhyuk et al. "BioBERT: a pre-trained biomedical language representation model for biomedical text mining." Bioinformatics 36.42020). As described above, these models may be pre-trained using general (i.e. non-biomedical data sets) or they may be trained solely on biomedical data.

[0212] During training, sequences of text from an entity linked biomedical training corpus are prepared, where each comprises a mention of a biological target. During training, the mention of the biological target is masked and the masked text sequence input into the model 400. The text sequence is input by encoding the masked text sequence into an input representation. In this example, as with the processing of text data in the model of Figure 1 , encoding involves performing tokenisation on the text sequence to encode the text sequence as a sequence of word piece tokens representing the sentence or multiple sentences of detect sequence. Any suitable tokenisation may be performed but in the present example BERT word-piece tokenisation is used to convert text sequence to wordpiece tokens 404. The masked biological target 402 in the text sequence 401 is replaced with the mask token 405 in the token sequence 403. As in the BERT architecture, each token sequence may start with the special token [CLS] 406 denoting the start of a text sequence. Optionally, the word-piece tokens may be combined with positional embeddings 407. The positional embeddings encode the position of the corresponding word piece token in the input sequence.

[0213] The word piece tokens 403 and the positional embeddings 407 may simply be summed to form the input representation of the input text sequence 401. The input representation of the input sequence 401 is then fed to the transformer encoder stack 408 which maps the input representation through the network layers to a transformed output representation, which is fed to the classification layer 409. The classification layer 409 is trained to output a probability 410 for each of the possible unique biological target identifiers 411 , which are each associated with a specific biological target 412. In particular, the dot product between the final MASK embedding and each of the possible biological entities identifiers is computed and passed to a softmax. Preferably, during training the transformer encoder stack 408 and the classification layer 409 are trained together such that the transformer encoder stack 408 learns representations which are useable by the classification layer 409 to predict a probability distribution cross the set of unique biological target identifiers 411.

[0214] In this way, the model is trained to learn from the training data set to form embeddings that encode relevant information to predicting biological targets. The trained encoder can then be extracted as using as the query and evidence encoder in the retriever module of Figure 1 .

[0215] In other examples the query encoder 114 and evidence encoder 115 may be further fine tuned on the entity prediction task illustrated in Figure 1. In particular, the encoders 114, 115 may be trained using the task of providing evidence sentences that provide increase the prediction accuracy on the task of identifying whether a candidate entity 113 is the correct or incorrect entity within input training query statements, as described above for the training of the query-evidence encoder 121 and reasoner model 122. The training process may be exactly as described above, where the query and evidence encoders 114, 115 may be trained at the same time as the query-evidence encoder 121 and reasoner model 122. Alternatively, the reasoner 122 and query evidence encoder 121 may be trained first and then fixed and then the query encoder and evidence encoder 114, 115 may be trained on the objective of predicting the correct biological entity.

[0216] Method and System

[0217] Figure 6A illustrates a method for training a machine learning model to predict a biological entity meeting a user-defined biological requirement using a knowledge base according to the present invention.

[0218] Step 611 comprises providing a training knowledge base comprising a corpus of textual data;

[0219] Step 612 comprises providing a training data set comprising training query sentences each comprising text describing a biological entity, in which a mention of the biological entity itself is masked for prediction, where the query sentence has a known correct biological entity;

[0220] Step 613 comprises, for each training query sentence, selecting a candidate biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, which may be the correct biological entity or another biological entity, wherein the evidence sentences are retrieved based on computing a similarity of the training query sentence to sentences within the knowledge base;

[0221] Step 614 comprises inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model comprising a machine learning model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences; and Step 615 comprises training the reasoner model to predict a probability that the candidate biological entity is the correct biological entity.

[0222] Figure 6B illustrates a method for predicting a biological entity meeting a user- defined biological requirement using a knowledge base according to the present invention.

[0223] Step 621 comprises providing an inference knowledge base comprising a corpus of textual data

[0224] Step 622 comprises receiving a user query defining a biological requirement for which a biological entity is to be predicted;

[0225] Step 623 comprises obtaining, based on the query, a query sentence text describing the biological requirement and including mention of a biological entity, in which the biological entity itself is masked for prediction;

[0226] Step 624 comprises selecting a candidate biological entity for the masked biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, wherein the evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base;

[0227] Step 625 comprises inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the candidate biological entity is the masked biological entity based on the retrieved evidence sentences, data.

[0228] Figure 7 illustrates a system 700 suitable for implementing a computer implemented method according to the present invention. System 700 includes a processor 702 configured to perform the method as described above. In the example of Figure 7 the system includes a user device 701 providing a user interface for a user to input a query defining a biological requirement for which a biological entity is sought. The user query is processed according to the method as described above using processor 702 and the results displaced on the user interface of the user device 701 , for example as shown in Figure 2. The processor 702 may be provided within the user device or locally connected to the user device. In this example, the processor 702 is provided remotely from the user device 701 . The processor 702 may be distributed across a network of servers. The system may further comprise a memory 703. The memory may store the knowledge base used by the model to make predictions. In one example the trained model may be provided at a server 702 and the memory 703 may be local to the user device 701 , where data in the knowledge base may be provided to the trained model to make predictions. In this arrangement, a user may use the trained model to make predictions based on confidential data stored in the local memory 703.

[0229] In the embodiments, examples, and aspects of the invention as described above such as process(es), method(s), system(s) may be implemented on and / or comprise one or more cloud platforms, one or more server(s) or computing system(s) or device(s). A server may comprise a single server or network of servers, the cloud platform may include a plurality of servers or network of servers. In some examples the functionality of the server and / or cloud platform may be provided by a network of servers distributed across a geographical area, such as a worldwide distributed network of servers, and a user may be connected to an appropriate one of the network of servers based upon a user location and the like.

[0230] Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium or non-transitory computer-readable medium. Computer-readable media may include, for example, computer-readable storage media. Computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. A computer-readable storage media can be any available storage media that may be accessed by a computer.

[0231] Although illustrated as a single system, it is to be understood that the computing device of Figure 7 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device. Although illustrated as a local device it will be appreciated that the computing device may be located remotely and accessed via a network or other communication link (for example using a communication interface).

[0232] Further possible implementation features

[0233] Although in the examples described above the retriever module 110 is illustrated in Figure 1 as accessing a corpus of textual evidence from which sentences are encoded by the evidence encoder during use of the model, in other examples the evidence may be stored as embeddings. In this case, the corpus of textual evidence and evidence encoder may be replaced by a “vector store” storing the embeddings of evidence to be retrieved for reasoning. In this way, the embedding does not need to be computed when running the model.

[0234] In particular the evidence sentence embeddings may be stored in per-gene FAISS indices (Johnson, Jeff, Matthijs Douze, and Herve Jegou. "Billion-scale similarity search with gpus." IEEE Transactions on Big Data 7.3 (2019): 535-547). In this way, the candidate biological entity (i.e. the selected gene I target for which a probability is to be computed) may be used to first identify the relevant FAISS indices and then these used for a the vector similarity search.

[0235] This option opens up the possibility of storing other types of data in the vector store, other than text or textualized data. In particular, other types of data such as images may be stored as embeddings, suitable for being compared to an input query to determine their relevance. At inference time, if an image is retrieved as a supporting piece of evidence for the query statement, the image may be returned to the user and displayed in the user interface alongside the pieces of text evidence from the literature or textualized non-text data. For this to function, an encoder must be trained that can take different types of data as input and encode them to the same embedding space.

[0236] In the above examples the user query comprises a single query sentence, used to retrieve evidence and predict a candidate. However, the model may be configured to take multiple different query sentences and their associated most similar evidence as input to the reasoner model. For example in the context of a user providing a template input, in which the query is provided as a number of selected biological entities. The method may comprise generating a plurality of different input query sentences based on the user-indicated biological entities. For example the same “biological requirement” may be expressed in a different of different sentences with differing wording. Each of the different query sentences may be input into the model together.

[0237] In a further example of the invention evidence metadata information may also be provided to the reasoner model. This may be achieved in a number of different ways. In a particularly preferable example, metadata associated with each evidence sentence may be encoded into a vector and combined with the corresponding query-evidence embedding when input into the reasoner model. The metadata may comprise information regarding the source of the evidence sentence. For example it may include one or more of: the publication date or year of publication of the document from which the evidence sentence is taken, information identifying the journal or specific paper, or author of the paper from which the evidence sentence was taken, or the impact factor of the journal from which the paper was taken. The publication date may be provided as the difference between the year of the query and the year of the evidence sentences. This meta data can be encoded into a vector representation, for example using trigonometric encoding as is conventionally used for positional encodings in transformers. The vector representations may be summed with each of the respective query-evidence vectors outputted from the query-evidence encoder. When providing the model with access to the evidence metadata in this way, during training the model may learn to use the meta data to improve predictions. For example, the model can learn to pay attention to evidence nearer in time to the query sentence or the model can be trained to preference evidence corroborated across multiple sources or authors, rather than lots of pieces of evidence from the same paper, or from journals with high impact factors.

[0238] The evidence metadata may be stored with each corresponding evidence sentence (or evidence sentence embedding). The metadata could be encoded and stored in the vector store (for example a FAISS vector store). In another example the metadata may be stored in a separate database such that it can be mapped to the evidence embeddings in the vector store. In particular the evidence metadata may be linked with each corresponding evidence sentence in a relational database. The method may further provide on the fly pre-filtering at inference time of the evidence, for example based on a specified data range, or selecting only a specific data type such as textualized numerical data, such as textualized omics data. This can enable users to interactively compare how inclusion or exclusion of certain subsets of evidence affects model predictions. For example it would allow a user to assess how predictions might change over time as more recent papers are introduced.

[0239] The metadata may be incorporated in other ways, that don’t necessarily involve encoding it into its own vector. In one example the meta data may be pre-pended to every sentence, for example to pre-pend the year of publication to every sentence in the corpus so it’s part of the text of the evidence. The current date could then be input, for example by pre-pending to the query sentence, to bias towards recency.

[0240] The models described in the above examples are based on the Transformer architecture (Vaswani A et al. "Attention is all you need." Advances in neural information processing systems 30 (2017)), or more specifically BERT models (Devlin J et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv: 1810.04805 (2018)). However, they could equally be implanted by other models, such as advances on the these models, suitable for learning embeddings for encoding information to make them usable for the prediction tasks described herein. Although the method of the invention have been described in relation to prediction of biomedical entities, the model could equally be generalised to predicting any type of entity, or masked word of an input sentence, based on a provided multimodal knowledge base. The methods can be applied to any binary prediction task or ranking task in which it is important to have predictions directly explained in terms of a set of supporting evidence.

[0241] Specific implementation example: Data retrieval and file recommendation

[0242] As described above, the methods and systems of the present invention may be used to implement a data retrieval tool and a file recommendation tool.

[0243] The machine learning models may be trained as described above. In particular, the retrieval model may be trained as above to retrieve evidence sentences from the knowledge base based on computing a similarity between the embedding of an input query sentence and the embeddings of sentences within the knowledge base. The knowledge base can includes biomedical literature data, but also nontext data such as numerical omics, genetics and knowledge graph data that has been converted to text data using the methods described above. In some cases the sentences within the knowledge base may be stored as their embeddings within a vector store for efficient retrieval. This is particularly efficient as different vector stores can be created for different candidate entities, entity linked within the evidence sentences.

[0244] The reasoner model, comprising an artificial neural network such as a transformer encoder plus a linear projection layer, is trained as described above to output a predicted probability that a query statement is true based purely on the retrieved evidence sentences. An embedding of the query statement is combined with each evidence sentence embedding to form a plurality of query-evidence embeddings which serve as the input to the reasoner model.

[0245] When using for inference, the user may input a query for which the entity is not known, such as [MASK] is a promising drug target for type-l I diabetes. The method then involves retrieving evidence sentences for each of a plurality of candidate entities and using the corresponding retrieved evidence sentences to predict a probability for each candidate entity. This can be used, for example, to score the entire human genome, returning the genes that are more likely to be a promising drug target (by calculating the probability that each target entity fills the masked token in the input).

[0246] It follows that the same model is also a data retrieval tool configured to return evidence for each entity that best supports that entity filling the masked slot. As described above, the contribution of each evidence sentence to the predicted probability can be calculated (for example using Shapley values) to output the selection of evidence sentences that provide the greatest contribution to the predicted probability. In this way, the model is usable to return sentences from the knowledge base that best support a given input statement. Unlike prior art methods that simply use cosine similarity between embeddings of an input sentences and sentences within the knowledge base, the present invention uses a two stage process in which evidence sentences are initially selected by comparing embeddings, but then these sentences are input into the trained reasoner model to select only those that best support the input query, where the reasoner model has been specifically trained to predict that a statement is correct (based on predicting the linked biological entity is correct).

[0247] The method therefore provides a powerful tool for searching large knowledge bases to retrieve data that is most relevant to an input query. Furthermore, the methods are appliable across multi-modal knowledge bases, since numerical data such as omics data can be converted to text sentences (then encoded into embeddings in a vector store) to be searched over in addition to text data such as biomedical literature.

[0248] The method can be extended to output a file from the data store from which the evidence sentences derive. For example, where the evidence sentence is derived from the biomedical literature, the journal article from which it was extracted may be output to the user so that the user can review it to confirm the relevance of the content to the input query. Where it relates to gene expression data, the full gene expression data set can be output so that the user can review this. In this way, the method serves to output recommended files based on the relevance of their content to supporting the input statement. This is particularly relevant to challenging data analysis tasks such as in drug discovery where multi-modal data relevant to a hypothesis must be found and reviewed, but is extendable to any task involving searching over a knowledge base to return data relevant to a user query.

[0249] Specific implementation example: Predicting clinical trials

[0250] The inventors have applied the methods and systems described herein to demonstrate that the model can predict future clinical trial outcomes based on published scientific literature, outperforming an industry-standard genetics-based approach. In particular the model may be applied to retrospectively predicting success or failure in historical clinical trial data based on evidence published before the trials, using the disease indication and drug target (gene).

[0251] The average cost of drug development is $2.6 billion, representing an enormous societal burden. Half of drugs fail to show efficacy when tested in human populations, after the greatest costs of development have been accrued. As described throughout the present disclosure, the methods and systems of the current invention are usable for predicting drug targets for specified diseases, thereby offering the potential to provide significant advantages and efficiencies in the drug discovery process. To demonstrate this, the model has been tested at predicting clinical trial outcomes, i.e. predicting gene-disease pairs that have succeeded at clinical trials. A benchmark of gene-disease pairs (therapeutic hypotheses) that have succeeded and failed clinical trials (as per Nelson et al. The support of human genetic evidence for approved drug indications. Nature genetics, 47(8):856-860, 2015.), using the PharmaProjects database (Citeline). This benchmark focuses on in vivo efficacy of therapeutic hypotheses as demonstrated by the transition of drugs from Phase ll / lll clinical trials to regulatory approval.

[0252] Genetics data is widely acknowledged to predict successful clinical development (Nelson et al., 2015). The most recently published PharmaProjects aligned genetics dataset (Minikel, E. V., et al Refining the impact of genetic evidence on clinical success. medRxiv, pp. 2023-06, 2023) was used as a competitive baseline. In order to validate the Clinical Trial Prediction data, the published result that the probability of clinical success of therapeutic hypotheses supported by genetics evidence is approximately double the probability without supporting genetics evidence was corroborated (Relative Success: 1.92; 95% Cl (1.64, 1.77)).

[0253] To avoid temporal leakage due to the reporting of clinical trial results in the literature, the Reasoner and Retriever models were trained with pre-2005 literature data and filtered drugs with dated evidence that was prior to 2005.

[0254] To predict on a therapeutic hypothesis the following query template was used: “[MASK] is a promising drug target for the treatment of {DISEASE}.”, substituting {DISEASE} with the disease annotated in PharmaProjects.

[0255] The performance of the model as a multi-modal reasoning system was assessed by generating a sentence for every row in the genetics baseline, with the simple template: “[MASK] is genetically associated with {MeSH name}’’, where the MeSH (Medical Subject Headings) name supplied in the Minikel et al. 2023 data had been programmatically reformatted to better align with how indications are written in the biomedical literature. The corpus of 77,735 genetics sentences was made available to the retriever model alone and in combination with the (year-split) biomedical literature.

[0256] A core set of results of the present methods predicting on the Clinical Trial Outcomes dataset are shown in Table 1 below, the model referred to as “R2E”, with further results across temporal training, bias correction, and method of multimodality. Previous analyses restrict clinical trial outcomes to those diseases with genetics data (Minikel et al., 2023). Performance against all diseases in the Clinical Trial Outcomes data is shown because the literature is not restricted to these diseases. The table shows AUROC for the model retrieving from literature- alone, genetics-alone, or both; in comparison to baselines. Numbers in brackets represent the 0.05 and 0.95 quantiles of 10,000 bootstrapped AUROC calculations.

[0257] Table 1

[0258] When reasoning from templated genetics evidence, the model displays improved performance over the genetics industry standard baseline, when leveraging the same data. The AUROC across the full distribution of the genetics baseline is low, contrary to the high reported Relative Success which is derived from few (554) high confidence predictions. The model further demonstrated an additional improvement when reasoning from the full biomedical literature as opposed to the templated genetics evidence alone.

[0259] The application of the model to predicting clinical trial outcomes was extended further by implementing LLM-in-the-loop auditing of the retrieved evidence sentences. In particular the method may comprise prompting an LLM to annotate the relevance of the retrieved evidence sentences to the query. Evidence sentences deemed as irrelevant by the LLM may be excluded by replacing with the NULL embedding. The final row of table 1 shows the results for the LLM audited evidence, showing a further improvement in the model’s ability to predict clinical trial outcomes. In particular, removing irrelevant evidence sentences from the coalitions used for predictions resulted in a significant increase in AUROC. In total, 527 / 809 high Shapley value genetics sentences were considered relevant and 268 / 527 of these related to a disease that was neither a substring of nor contained the Clinical Outcomes disease, indicating that the model is able to leverage evidence from related traits without needing to apply ontology-similarity metrics

[0260] The parameters used for training the model to provide the above demonstration of drug target prediction against clinical trials are as follows. All sentences were tokenized, and then truncated and padded to a length of 128, using the same vocabulary as PubMedBERT (Gu, Y, et al Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1- 23, 2021). Pre-processing of training examples for both Retriever and Reasoner training was done using Apache Spark (Zaharia, M. et al. Apache spark: A unified engine for big data processing. Commun. ACM, 59(11):56-65, 2016). Both the Retriever and Reasoner were trained independently, for 10 epochs on a single Tesla V100 GPU, with a total training time of approximately 1 week.

[0261] The final Retriever architecture, consisted of a scaled down version of PubMedBERT( Gu et al., 2021) trained from scratch on the retriever encoder training task described above, with 4 hidden layers, 4 attention heads, an intermediate size of 512, a hidden size of 256, and total size of 10 million parameters. Final Retriever training used a batch size of 512, a categorical crossentropy loss, and an AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 0.0001 and no weight decay.

[0262] The final query-evidence encoder component of the Reasoner architecture consisted of a layer normalisation across all concatenated query / evidence pairs, then two convld layers each with kernel size of 1 (first layer: 2 input channels, 8 output channels; second layer: 8 input channels, 1 output channel) across each query / evidence pair individually. The final evidence combiner component of the R2E Reasoner architecture consisted of a set transformer (Lee et al., 2019) over all query-evidence embeddings returning a single embedding, followed by a linear layer and sigmoid to output a binary probability. The set transformer had 4 heads, 2 induced set attention blocks with 32 inducing points for the encoder, and a pooling by multihead attention followed by two set attention blocks in the decoder. The Reasoner had a total size of 2 million parameters. After freezing the Retriever weights, the final Reasoner training used a batch size of 2048, binary crossentropy loss, and AdamW optimizer with a learning rate of 0.0001 and weight decay of 0.001. For both training and inference, 64 evidence sentences were retrieved for a given query. A post-hoc frequency bias correction factor of 0.5 was used for the R2E-Corrected variant, the results for which are in the table above.

Claims

CLAIMS1. A computer-implemented method for training a machine learning model to predict a biological entity meeting a user-defined biological requirement using a knowledge base, the method comprising: providing a training knowledge base comprising a corpus of textual data; providing a training data set comprising training query sentences each comprising text including mention of a biological entity, in which the biological entity itself is masked for prediction, where the query sentence has a known correct biological entity; for each training query sentence, selecting a candidate biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, which may be the correct biological entity or another biological entity, wherein the evidence sentences are retrieved based on computing a similarity of the training query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model comprising a machine learning model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict a probability that the candidate biological entity is the correct biological entity.

2. The computer-implemented method of any preceding claim wherein the biological entity to be predicted comprises a biological target for drug discovery.

3. The computer-implemented method of any preceding claim wherein the query sentence describes a biological context involving the biological entity, the biological context preferably comprising a relationship with another biological entity.

4. The computer-implemented method of any preceding claim comprising training on positive and negative training examples, wherein training on a positive training example comprises:selecting the correct biological entity as the candidate entity and retrieving from the knowledge base a plurality of evidence sentences each including mention of the correct biological entity; inputting each training query sentence and a plurality of retrieved evidence sentences into the reasoner model, where mention of the correct biological entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict that the masked biological entity is the correct entity; and wherein training on a negative training example comprises: selecting the correct biological entity as the candidate entity and retrieving from the knowledge base a plurality of evidence sentences each including mention of an incorrect biological entity; inputting each training query sentence and a plurality of retrieved evidence sentences into the reasoner model, where mention of the incorrect biological entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict that the masked biological entity is not the correct entity.

5. The computer-implemented method of claim 4 wherein, in the negative training example, the incorrect biological entity is randomly sampled from the full set of possible candidate entities according to its frequency of mention in the literature.

6. The computer-implemented method of any preceding claim wherein retrieving a plurality of sentences from the knowledge base comprises: inputting the query sentence into a query encoder trained to output a query embedding; inputting sentences from the knowledge base into an evidence encoder trained to output an evidence embedding; computing the similarity of the query embedding to each evidence embedding and selecting the plurality of evidence sentences with embeddings most similar to the query embedding.

7. The computer-implemented method of claim 6 wherein the query encoder and evidence encoder each comprise a transformer encoder.

8. The computer-implemented method of claim 7 wherein the encoders are pretrained by masked language modelling.

9. The computer-implemented method of claim 6 or claim 7 wherein the encoders are trained on a retrieval objective using a contrastive loss function.

10. The computer-implemented method of claim 6 to 9 wherein the query encoder and evidence encoder comprise a retriever encoder, the retriever encoder comprising an encoder trained by: providing a training data set comprising a plurality of entity-linked text sequences, each text sequence including a mention of a biological entity, where the biological entity is linked to a corresponding biological entity identifier from a set of possible biological entity identifiers; masking the mention of the biological entity within each text sequence; encoding each masked text sequence into an input representation for a machine learning model, the machine learning model comprising the retriever encoder, which is configured for mapping the input representation to an output representation, and a classifier layer that receives the output representation and performs classification into classes corresponding to the set of possible biological entity identifiers; and training the machine learning model to predict the biological entity identifier of the masked biological entity based on the input representation.

11. The computer-implemented method of claim 10 wherein the query encoder and evidence encoder are optimised for retrieving sentences for predicting a required biological entity type, the encoders optimised by training the encoder using the method of claim 10 to predict the same biological entity type as that sought for the user-defined biological requirement, the biological entity type preferably comprising a biological target.

12. The computer-implemented method of any of claims 6 to 11 wherein inputting each training query sentence and the corresponding plurality of retrieved evidence sentences into the reasoner model comprises:combining the query embedding with each evidence embedding individually into a query-evidence vector representing how the query sentence relates to the corresponding evidence sentence, so as to create a query-evidence vector for each query-evidence sentence pair; forming an input sequence comprising a sequence of all query-evidence vectors and inputting the input sequence into the reasoner model.

13. The computer-implemented method of claim 12 further comprising encoding metadata associated with each evidence sentence into a metadata vector, wherein the metadata comprise information regarding the source of the evidence sentence; and combining the metadata vector into each corresponding query-evidence vector.

14. The computer-implemented method of claim 13 wherein the metadata comprises one or more of: the publication date of the evidence sentence, the paper and / or journal from which the evidence sentence was extracted, the impact factor of the journal from which the evidence sentence was extracted.

15. The computer-implemented method of any of claims 12 to 14 wherein combining the query sentence with each evidence sentence individually into a query-evidence vector representing the representing how the query sentence relates to the corresponding evidence sentence comprises: for each evidence sentence, inputting a query embedding representing the query sentence and an evidence embedding representing the evidence sentence into a respective query-evidence encoder trainable to output the queryevidence vector.

16. The computer-implemented method of claim 15 wherein the queryevidence encoder comprises a convolutional network layer, and the method comprises, for each evidence sentence: stacking the query embedding and the evidence embedding to form a matrix;inputting the matrix into the query-evidence encoder to output the queryevidence vector representing the similarity of the query sentence and the corresponding evidence sentence.

17. The computer-implemented method of claim 15 or 16 where the method comprises training the query-evidence encoders and the reasoner model end-to- end to learn to predict a probability that the candidate biological entity is the correct biological entity based on the retrieved evidence sentences.

18. The computer-implemented method of any of claims 13 to 17 where the reasoner model comprises: a Transformer encoder trainable to map the sequence of query-evidence vectors to a sequence of output embeddings; and an output layer trainable to compute the output probability based on the output embeddings.

19. The computer-implemented method of claim 18 wherein the transformer encoder does not use positional embeddings such that the order of the queryevidence vectors does not impact the output probability.

20. The computer-implemented method of any preceding claim further comprising a subsequent step of fine-tuning the reasoner model, where fine- tuning the reasoner model comprises: providing an optimal evidence set, the optimal evidence set comprising a plurality of different evidence statements that optimally indicate that a candidate biological entity meets the biological requirement; for a training query sentence, retrieving two sample sets of evidence, each sample set of evidence comprising a plurality of sentences from the knowledge base; for each sample set of evidence, querying a language model to determine the degree to which the optimal evidence set is supported by the sample set of evidence and accordingly determine which sample set of evidence better supports the optimal evidence set;use the determination to apply a label to the sample sets of evidence, indicating which better supports the optimal evidence set; and train the reasoner model to adapt the output probability based on the applied label.

21. The computer-implemented method of claim 20 further comprising training the reasoner model to output an increased probability for a sample evidence set that better supports the optimal evidence set.

22. The computer-implemented method of claim 20 or 21 wherein each sample set of evidence supports a different candidate biological entity as meeting the biological requirement23. The computer-implemented method of any of claims 20 to 22 wherein the reasoner is trained using a margin ranking loss.

24. The computer-implemented method of any preceding claim wherein the training query sentence and evidence sentences are based on sentences extracted from published biomedical literature, wherein the publication date of the evidence sentences are required to be prior to the publication date of the training query sentence.

25. A computer-implemented method for predicting a biological entity meeting a user-defined biological requirement using a knowledge base, the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a biological requirement for which a biological entity is to be predicted; obtaining, based on the query, a query sentence comprising text describing the biological requirement and including mention of a biological entity, in which the biological entity itself is masked for prediction; selecting a candidate biological entity for the masked biological entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate biological entity, whereinthe evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the candidate biological entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the candidate biological entity is the masked biological entity based on the retrieved evidence sentences.

26. The computer-implemented method of claim 25 wherein the biological entity to be predicted comprises a biological target, preferably a gene, and the method further comprises determining a drug that interacts with a predicted biological target.

27. The computer-implemented method of claim 25 or claim 26 comprising selecting a plurality of candidate biological entities and for each candidate biological entity: retrieving a plurality of evidence sentences from the knowledge base and inputting the query and plurality of evidence sentences into the reasoner model to predict a probability for each candidate entity.

28. The computer-implemented method of claim 27, wherein the biological entity comprises a drug target, preferably a gene, and the method comprises, for each of a set of total known drug targets within the human genome: retrieving a plurality of evidence sentences from the knowledge base and inputting the query and plurality of evidence sentences into the reasoner model to predict a probability for each candidate drug target.

29. The computer-implemented method of any of claims 25 to 28 wherein the biological requirement defines a relationship with another biological entity.

30. The computer-implemented method of any of claims 25 to 29 wherein biological entity to be predicted comprises a biological target and the biological requirement defines a relationship with a disease.31 . The computer-implemented method of any of claims 25 to 30 wherein the reasoner model has been trained according to any of claims 1 to 24.

32. The computer-implemented method of claim 31 wherein the inference knowledge base is different to the training knowledge base.

33. The computer-implemented method of any of claims 25 to 32 further comprising: computing a contribution to the predicted probability of a candidate biological entity of an evidence sentence used in the prediction.

34. The computer-implemented method of claim 33 further comprising: computing a Shapley value of an evidence sentence used in the prediction.

35. The computer-implemented method of claim 34 further comprising: computing an estimated Shapley value of an evidence sentence used in the prediction, preferably by computing a Shapley Additive exPlanation, SHAP, value.

36. The computer-implemented method of any of claims 33 to 35 further comprising: outputting one or more evidence sentences having the highest contribution to the predicted probability.

37. The computer-implemented method of any of claims 25 to 36 wherein retrieving a plurality of evidence sentences comprises: encoding the query sentence and sentences from the knowledge base into embeddings and computing the similarity of the embeddings to return a plurality of evidence sentences most similar to the query sentence.

38. The computer-implemented method of claim 37 wherein retrieving a plurality of evidence sentences comprises: inputting the query sentence and sentences from the knowledge base into a retriever encoder trained to output an embedding representing the input sentence, wherein the retriever encoder comprises a Transformer encoder.

39. The computer-implemented method of claim 38 wherein the retriever encoder has been trained by: providing a training data set comprising a plurality of entity-linked text sequences, each text sequence including a mention of a biological entity, where the biological entity is linked to a corresponding biological entity identifier from a set of possible biological entity identifiers; masking the mention of the biological entity within each text sequence; encoding each masked text sequence into an input representation for a machine learning model, the machine learning model comprising the retriever encoder for mapping the input representation to an output representation and a classifier layer that receives the output representation and performs classification into classes corresponding to the set of possible biological entity identifiers; and training the machine learning model to predict the biological entity identifier of the masked biological entity based on the input representation.

40. The computer-implemented method of any of claims 25 to 39 wherein providing an inference knowledge base comprises: providing an entity-linked biomedical medical text corpus.

41. The computer-implemented method of any of claims 25 to 40 wherein providing an inference knowledge base comprises: converting a non-text dataset to a textualised data set and including the textualised data set in the knowledge base.

42. The computer-implemented method of claim 41 wherein the non-text data set comprises one of: genetics or genomics data; transcriptomics data; gene expression data; differential gene expression data; proteomics data; protein-protein interaction network data; clinical data; knowledge graph data; image data.

43. The computer-implemented method of claim 41 or 42 wherein converting a non-text dataset to a textualized data set comprises: providing a patient-level dataset comprising numerical data, converting one or more components of the data into a corresponding sentence describing the component of the data and including mention of a biological entity.

44. The computer-implemented method of any of claims 25 to 43 wherein providing an inference knowledge base comprises receiving an imported user data set, wherein the inference knowledge base is different to the training knowledge base.

45. The computer-implemented method of any of claims 41 to 44 wherein the inference knowledge base comprises two or more of: published biomedical literature; textualized patient-level datasets, where patient-level datasets preferably comprise genetics or transcriptomics data; textualized knowledge graph data;46. The computer-implemented method of claim 45 wherein retrieving a plurality of evidence sentences comprises: inputting the query sentence and sentences from the knowledge base into a retriever encoder trained to output an embedding representing the input sentence, wherein the retriever encoder comprises a T ransformer encoder and is trained only on sentences extracted from biomedical literature text.

47. The computer-implemented method of any of claims 25 to 46 wherein the knowledge base comprises a vector store, in which evidence sentences are stored as evidence embeddings, wherein the method comprises: inputting the query sentence into an encoder to form a query embedding and then performing a vector search on the vector store.

48. The computer-implemented method of claim 47 wherein the knowledge base comprises a separate vector store for each candidate biological entity.

49. The computer-implemented method of any of claims 25 to 48 wherein the user query comprises completion of a plurality of input fields, the plurality of input fields comprising a disease of interest and one or more biological entities or relationships, wherein obtaining a query sentence comprises: generating one or more natural language text sequences based on the completed fields using a template.

50. The computer implemented method of claim 49 wherein the input fields comprise a disease of interest, a biological mechanism, and, optionally, one or more of: a tissue type, and a cell type, a trait, biomarker, assay, patient subgroup.51 . The computer implemented method of any of claims 25 to 50 the method further comprising: displaying, on a user interface, one or more evidence sentences and the predicted probability of the candidate biological entity meeting the biological requirement.

52. The computer implemented method of claim 51 the method further comprising: receiving a user selection of the displayed evidence sentences and updating the predicted probability based on the user selection.

53. The computer implemented method of claim 51 or claim 52 the method further comprising: receiving a user selection of the displayed evidence sentences and retraining the reasoner model to increase the contribution of evidence sentences of the type selected by the user.

54. The computer implemented method of claim 52 or 53 the method further comprising: displaying a contribution of each displayed evidence sentences to the predicted probability, the contribution preferably comprising a computed SHAP value.

55. A computer-implemented method for training a machine learning model to predict an entity meeting a user-defined requirement using a knowledge base, the method comprising: providing a training knowledge base comprising a corpus of textual data; providing a training data set comprising training query sentences each comprising mention of an entity, in which the entity itself is masked for prediction, where the query sentence has a known correct entity; for each training query sentence, selecting a candidate entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate entity, which may be the correct entity or another entity, wherein the evidence sentences are retrieved based on computing a similarity of the training query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model comprising a machine learning model, where mention of the candidate entity is masked in the query sentence and evidence sentences; and training the reasoner model to predict a probability that the candidate entity is the correct entity.

56. A computer-implemented method for predicting an entity meeting a user- defined requirement using a knowledge base, the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a requirement for which an entity is to be predicted; obtaining, based on the query, a query sentence text describing the requirement and including mention of an entity, in which the entity itself is masked for prediction; selecting a candidate entity for the masked entity and retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the candidate entity, wherein the evidence sentences areretrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting each training query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the candidate entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the candidate entity is the masked entity based on the retrieved evidence sentences.

57. A computer-implemented method for retrieving data from a knowledge base that supports a user-defined requirement of an entity, the method comprising: providing an inference knowledge base comprising a corpus of textual data; receiving a user query defining a requirement for the entity; obtaining, based on the query, a query sentence comprising text describing the requirement and including mention of the entity, in which the entity itself is masked; retrieving a plurality of evidence sentences from the knowledge base, each evidence sentence including mention of the entity, wherein the evidence sentences are retrieved based on computing a similarity of the query sentence to sentences within the knowledge base; inputting the query sentence and a plurality of retrieved evidence sentences into a reasoner model, where mention of the entity is masked in the query sentence and evidence sentences, the reasoner model trained to predict a probability that the entity is the masked entity based on the retrieved evidence sentences; computing a contribution to the predicted probability of each evidence sentence used in the prediction; outputting one or more evidence sentences that have the greatest contribution to the predicted probability.

58. The computer-implemented method of claim 57 wherein the reasoner model has been trained according to any of claims 1 to 24.

59. The computer-implemented method of claim 57 or 58 further comprising: outputting a file from the knowledge base from which an output evidence sentence derives.

60. The computer-implemented method of any of claims 57 to 59 wherein the knowledge base comprises multi-modal data.61 . The computer-implemented method of any of claims 57 to 60 wherein the entity comprises a biological entity and the knowledge base comprises entity- linked sentences derived from two or more of: published biomedical literature; textualized patient-level datasets, where patient-level datasets preferably comprise genetics or transcriptomics data; textualized knowledge graph data62. The computer-implemented method of any of claims 57 to 61 wherein computing a contribution to the predicted probability of an evidence sentence comprises: determining the probability of the candidate entity with each evidence sentence masked in turn to determine the relative contribution to the probability of each evidence sentence.

63. The computer-implemented method of any of claims 57 to 62 wherein computing a contribution to the predicted probability of an evidence sentence comprises: computing a Shapley value of the evidence sentence used in the prediction.

64. The computer-implemented method of claim 63 further comprising: computing an estimated Shapley value of an evidence sentence used in the prediction, preferably by computing a Shapley Additive exPlanation, SHAP, value.

65. A system comprising a processor configured to perform the method of any preceding claim.

66. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any preceding claim.

Images