Dynamic knowledge retrieval in predictive models

Abstract

A computer-implemented method, comprising: receiving input data associated with a target case; applying a trained neural embedding module to learn embeddings representing the input data; constructing, based on the learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between the learned embeddings and embeddings of the external knowledge items; executing the at least one retrieval query on at least one external data source, to retrieve the one or more relevant external knowledge items; generating an aggregated embeddings representation of the input data, and an aggregated embeddings representation of the one or more relevant external knowledge items; and inferencing a prediction module on the generated aggregated embeddings of the input data and the one or more relevant external knowledge items, to output a target prediction associated with the target case.

Description

DYNAMIC KNOWLEDGE RETRIEVAL IN PREDICTIVE MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application Ser. No. 63 / 528,554, filed July 24, 2023, entitled “DYNAMIC KNOWLEDGE RETRIEVAL IN PREDICTIVE MODELS,” the contents of which are incorporated herein by reference in their entirety.FIELD OF THE INVENTION

[0002] This invention relates to the field of machine learning.BACKGROUND

[0003] Predicting the medical outcomes of hospitalized patients holds the promise of enhancing clinical decision making, with respect to clinical triage, workflow optimization, and hospital resource management. With the advent of electronic health records (EHRs), more clinical data is available to train machine learning models for outcome prediction. In particular, language models pretrained on biomedical and / or clinical text are demonstrating increasing proficiency when fine-tuned for predicting specific outcomes, such as mortality, length of hospitalization, readmission, sepsis, prolonged mechanical ventilation, and diagnostic coding. Such predictive models approaches have incorporated pretraining over a general corpora of clinical notes, as well as general biomedical literature. Increasingly, models have leveraged unstructured text from clinical notes, since they can contain key information for outcome prediction.

[0004] Similarly, language models can use descriptions of the entities for general biomedical and chemical prediction tasks, e.g., predicting molecule toxicity, side-effects or therapeutic properties, or drug-drug interactions and synergistic properties, or material properties such as thermoelectric properties and material functions.

[0005] However, the accuracy and performance of prediction models can benefit from dynamic retrieval of scientific literature and other external sources of scientific knowledge, that is relevant to a particular target case or patient.

[0006] The foregoing examples of the related art and limitations related therewith are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent to those of skill in the art upon a reading of the specification and a study of the figures.SUMMARY OF THE INVENTION

[0007] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods which are meant to be exemplary and illustrative, not limiting in scope.

[0008] There is provided, in an embodiment, a system comprising at least one hardware processor; and a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to: receive input data associated with a target case, apply a trained neural embedding module to learn embeddings representing the input data, construct, based on the learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between the learned embeddings and embeddings of the external knowledge items, execute the at least one retrieval query on at least one external data source, to retrieve the one or more relevant external knowledge items, generate an aggregated embeddings representation of the input data, and an aggregated embeddings representation of the one or more relevant external knowledge items, and inference a prediction module on the generated aggregated embeddings of the input data and the one or more relevant external knowledge items, to output a target prediction associated with the target case.

[0009] There is also provided, in an embodiment, a computer -implemented method, comprising: receiving input data associated with a target case; applying a trained neural embedding module to learn embeddings representing the input data; constructing, based on the learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between the learned embeddings and embeddings of the external knowledge items; executing the at least one retrieval query on at least one external data source, to retrieve the one or more relevant external knowledge items; generating an aggregated embeddings representation of the input data, and an aggregated embeddings representation of the one or more relevant external knowledge items; and inferencing a prediction module on the generatedaggregated embeddings of the input data and the one or more relevant external knowledge items, to output a target prediction associated with the target case.

[0010] There is further provided, in an embodiment, a computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to receive input data associated with a target case; apply a trained neural embedding module to learn embeddings representing the input data; construct, based on the learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between the learned embeddings and embeddings of the external knowledge items; execute the at least one retrieval query on at least one external data source, to retrieve the one or more relevant external knowledge items; generate an aggregated embeddings representation of the input data, and an aggregated embeddings representation of the one or more relevant external knowledge items; and inference a prediction module on the generated aggregated embeddings of the input data and the one or more relevant external knowledge items, to output a target prediction associated with the target case.

[0011] In some embodiments, the target case is a patient, the input data comprises health records and / or clinical data associated with the patient, and the target prediction represents a clinical outcome or a clinical intervention suggestion with respect to the patient.

[0012] In some embodiments, the target case is one or more molecules or chemical compositions; the input data comprises data with respect to the one or more molecules or chemical compositions; and the target prediction represents chemical, material and / or therapeutic properties of the one or more molecules or chemical compositions.

[0013] In some embodiments, the trained neural embedding module is a pre-trained transformer neural network.

[0014] In some embodiments, the input data comprises data in at least one of the following modalities: tabular data, natural language data, genomic data, proteomic data, image data, video data, and / or audio data.

[0015] In some embodiments, the trained neural embedding module comprises a dedicated embedding function for at least one of the data modalities.

[0016] In some embodiments, the similarity is determined based on one of the following methods: a dot product operation, a Euclidean distance function, weighted average of encoder representations, or an Optimal Transport mechanism for a sparse matching between sets of vectors.

[0017] In some embodiments, the one or more relevant external knowledge items are selected from at least one of the following categories of knowledge items: scientific papers, medical studies, clinical trials, patents, entries in a biomedical or chemical knowledge base, and knowledge graphs.

[0018] In some embodiments, the prediction module applies a cross -attention transformer which jointly encodes the aggregated embeddings of the input data and the one or more relevant external knowledge items, into a joint encoding, and wherein the joint encoding is fed into a loss function which outputs the target prediction.

[0019] In some embodiments, the target prediction classifies the aggregated embeddings of the input data and the one or more relevant external knowledge items into one of a set of class labels.

[0020] In some embodiments, the class labels indicate a binary variable.

[0021] In some embodiments, the class labels indicate a discrete variable on a scale.

[0022] In some embodiments, the target prediction comprises an indication of a contribution of each of the one or more relevant external knowledge items to the target prediction.

[0023] In some embodiments, the indication comprises, with respect to the one or more relevant external knowledge items comprising textual data, an indication of a contribution of specific parts within each of the one or more relevant external knowledge item.

[0024] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the figures and by study of the following detailed description.BRIEF DESCRIPTION OF THE FIGURES

[0025] The present invention will be understood and appreciated more comprehensively from the following detailed description taken in conjunction with the appended drawings in which:

[0026] Figs. 1A-1B depict an overview of various predictive models, in accordance with some embodiments of the present invention.

[0027] Fig. 1C is a block diagram of an exemplary system for realizing a predictive model in the medical, clinical, and / or general scientific domains, configured to perform a specified prediction task, wherein the predictive model is trained on input data augmented by case-specific knowledge that is automatically retrieved from external sources and databases, in accordance with some embodiments of the present invention.

[0028] Fig. 2 illustrates the functional steps in a method for training a predictive model in the medical, clinical, and / or general scientific domains to perform a specified prediction task, wherein the predictive model is trained on input data augmented by case- specific knowledge that is automatically retrieved from external sources and databases, in accordance with some embodiments of the present disclosure.

[0029] Fig. 3A illustrates an overview of a pipeline for training a predictive model in the medical, clinical, and / or general scientific domains to perform a specified prediction task, wherein the predictive model is trained on input data augmented by case -specific knowledge that is automatically retrieved from external sources and databases, in accordance with some embodiments of the present disclosure.

[0030] Fig. 3B illustrates an embedder module, in accordance with some embodiments of the present invention.DETAILED DESCRIPTION

[0031] Disclosed herein is a technique, embodied in a system, method, and computer program product, for improving the accuracy of a task- specific predictive model, by training the model on data augmented by automatic ally -retrieved case-relevant knowledge from external scientific literature. In some embodiments, the present technique provides for a predictive model in the medical, clinical, and / or general scientific domains, configured to perform a specified prediction task, wherein thepredictive model is trained on input data augmented by case-specific knowledge that is automatically retrieved from external sources and databases.

[0032] In some embodiments, a predictive model of the present technique may be configured to output a prediction with respect to a target case in the clinical or general scientific domains. For example, as can be seen in Fig. 1A, the present predictive model may be configured to predict patient clinical outcome, such as a specific diagnosis (e.g., cardiovascular failure or stroke), or the likelihood of mortality, length of hospitalization, readmission, developing sepsis, and / or the need for prolonged mechanical ventilation. In such cases, the predictive model may be trained to perform a specified prediction task (e.g., likelihood of in-hospital mortality), based on data of a particular patient, such as patient electronic health record (EHR), patient clinical or admission notes, and the like. In some embodiments, the training data may be augmented with automatic ally -retrieved case-specific knowledge from external data sources, such as medical publications and related scientific literature. In some embodiments, the predictive model in then configured to output a prediction with respect to the particular patient, based on the augmented training data.

[0033] In another example, as can be seen in Fig. IB, the present model may be configured to predict biomedical and / or related properties associated with a molecule, a chemical composition, cell or tissue types, a DNA sequence, genetic mutations, and the like. The predicted properties may include therapy response, interactions and synergies with other chemical compositions, adverse effects in patients, toxicity, conductivity, thermoelectric properties, and the like. In such cases, the predictive model may be trained to perform a specified prediction task (e.g., molecule toxicity), based on data associated with a particular molecule, which may be augmented with automatically-retrieved casespecific knowledge from external data sources, such as medical publications and related scientific literature. In some embodiments, the predictive model in then configured to output a prediction with respect to the particular molecule, based on the augmented training data.

[0034] Reference is made to Fig. 1C, which is a block diagram of an exemplary system 100 for realizing a predictive model in the medical, clinical, and / or general scientific domains, configured to perform a specified prediction task, wherein the predictive model is trained on input data augmented by case-specific knowledge that is automatically retrieved from external sources and databases.

[0035] In some embodiments, system 100 may comprise a hardware processor 102, and a random-access memory (RAM) 104, and / or one or more non-transitory computer- readable storage device 106. In some embodiments, system 100 may store in storage device 106 software instructions or components configured to operate a processing unit (also ‘hardware processor,’ ‘CPU,’ ‘quantum computer processor,’ or simply ‘processor’), such as hardware processor 102. In some embodiments, the software components may include an operating system, including various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0036] The software instructions and / or components operating hardware processor 102 may comprise an embedder module 106a, a knowledge retrieval module 106b, an aggregator module 106c, and / or a prediction module 106d.

[0037] In some embodiments, system 100 may further comprise a user interface 108 comprising, e.g., a display monitor for displaying images, a control panel for controlling system 100, and a speaker for providing audio feedback.

[0038] System 100 as described herein is only an exemplary embodiment of the present invention, and in practice may be implemented in hardware only, software only, or a combination of both hardware and software. System 100 may have more or fewer components and modules than shown, may combine two or more of the components, or may have a different configuration or arrangement of the components. System 100 may include any additional component enabling it to function as an operable computer system, such as a motherboard, data busses, power supply, a network interface card, a display, an input device (e.g., keyboard, pointing device, touch-sensitive display), etc. (not shown). Components of system 100 may be co-located or distributed, or the system may be configured to run as one or more cloud computing ‘instances,’ ‘containers,’ ‘virtual machines,’ or other types of encapsulated software applications, as known in the art.

[0039] The instructions of system 100 will now be discussed with reference to the flowchart of Fig. 2 which illustrates the functional steps in a method 200 for training a predictive model in the medical, clinical, and / or general scientific domains to perform a specified prediction task, wherein the predictive model is trained on input dataaugmented by case-specific knowledge that is automatically retrieved from external sources and databases, according to some embodiments of the present disclosure.

[0040] The various steps of method 200 will be described with continuous reference to exemplary system 100 shown in Fig. 1C, and to the block diagram of Fig. 3A, which provides an overview of a pipeline for training a predictive model in the medical, clinical, and / or general scientific domains to perform a specified prediction task, wherein the predictive model is trained on input data augmented by case-specific knowledge that is automatically retrieved from external sources and databases, according to some embodiments of the present disclosure.

[0041] The various steps of method 200 may either be performed in the order they are presented or in a different order (or even in parallel), as long as the order allows for a necessary input to a certain step to be obtained from an output of an earlier step. In addition, the steps of method 200 may be performed automatically (e.g., by system 100 of Fig. 1C), unless specifically stated otherwise. In addition, the steps of method 200 are set forth for exemplary purposes, and it is expected that modification to the flow chart is normally required to accommodate various network configurations and network carrier business policies.

[0042] Method 200 begins in step 202, wherein system 100 may receive, as input, a dataset comprising a plurality of datapoints associated with a target prediction case.

[0043] As noted above, in one example, the target case may be associated with predicting a clinical outcome in a patient, such as a specific diagnosis (e.g., cardiovascular failure or stroke), or the likelihood of mortality, length of hospitalization, readmission, developing sepsis, the need for prolonged mechanical ventilation, etc. In such cases, the input data may comprise data of a particular patient, such as patient electronic health record (EHR), patient admission notes, and the like.

[0044] In some embodiments, input data may comprise data of a particular patient, including, but not limited to:Patient demographics, medical history, medication and allergies, immunization status,laboratory test results, radiology images, vital signs, and / or personal statistics.

[0045] In another example, the target case may be associated with predicting biomedical and / or related properties associated with a molecule, a chemical composition, cell or tissue types, a DNA sequence, a genetic mutation, and the like. The predicted properties may include therapy response, interactions and synergies with other chemical compositions, adverse effects in patients, toxicity, conductivity, thermoelectric properties, and the like. In such cases, the input data may comprise data associated with the particular molecule, chemical composition, cell or tissue type, genetic mutation, etc., as the case may be.

[0046] In some embodiments, the input data received in step 202 may comprise at least one of the following data modalities:Structured tabular data: Structured data may include quantitative data presented in a tabular format, e.g., in rows and columns. In the case of patient medical data, for example, structured data may include clinical or lab results, and the like.Unstructured natural language data: Unstructured natural language data may comprise natural language text or entries, such as scientific papers, articles, clinical notes, patents, etc.Image data.Audio data.Video data.

[0047] In step 204, the instructions of embedder module 106a may cause system 100 to generate embeddings of the input data received in step 202.

[0048] In some embodiments, the embeddings are numerical vectors that encode the meaning of input data (e.g., a word, a sentence, a whole document, an image, etc.) in such a way that datapoints that are closer in the vector space are expected to be similar in meaning.

[0049] In some embodiments, embedder module 106a may be configured to generate embeddings which represent input data at different resolution levels. For example, embedder module 106a may be configured to generate embeddings which represent each word in a document (e.g., a scientific paper or a clinical note), each sentence in the document, each section in the document, or the entire document.

[0050] In some embodiments, embedder module 106a may employ any one or more suitable embedding functions known in the art to perform step 204. In some embodiments, as can be seen in Fig. 3B, embedder module 106a may employ dedicated embedding functions for each data type or modality (e.g., structured tabular, natural language data, images, video, audio, etc.). In other embodiments, embedder module 106a may employ only one type of embedding function, e.g., Data2Text, to transform all input data modalities into textual representation.

[0051] For example, embedder module 106a may employ a transformer neural network, which may be pre-trained on scientific literature and clinical texts.

[0052] In some embodiments, embedder module 106a may be configured to embed image data a visual transformer (ViT) model, which may be pre-trained on scientific and / or medical images.

[0053] Optionally, embedder module 106a may employ a multimodal transformer model for embedding text and image data, which may be pre-trained with a contrastive learning objective over pairs of images and image captions.

[0054] In some embodiments, embedder module 106a may be configured to transform input tabular data into textual data via a suitable model, such as Data2Text, which generates text structured data. For example, tabular information, such as lab results, may be transformed into textual input. In another embodiment, embedder module 106a may be configured to generate embeddings of structures data using a dedicated transformer architecture, such as TabNet.

[0055] With reference back to Fig. 2, in step 206, the instructions of knowledge retrieval module 106b may cause system 100 to generate one or more retrieval queries, based, at least in part, on the embeddings generated in step 204.

[0056] In some embodiments, knowledge retrieval module 106b may be configured to use the embeddings generated in step 204 to generate one or more query vectors, e.g., via a fully-connected layer. In some embodiments, the query vectors generated from theembeddings representing the input data may be used to retrieve relevant external data (e.g., scientific and medical paper, articles, patents, etc.) from external sources and databases (such as scientific papers or patent databases).

[0057] In one embodiment, each embedding vector is used separately to generate a query vector for retrieval of external data. In other embodiments, different embeddings may be combined via a cross-attention mechanism into a single retrieval query.

[0058] In one embodiment, the embeddings and respective query vectors may be pretrained, based on supervised data collected to match input data and external data objects, e.g., match patients with papers, patients with images, drugs with papers, drug-drug interactions with papers, etc. In some embodiments, the pre-training uses a contrastive loss objective, and may be initialized with weights from previously pre-trained models or random weights.

[0059] In step 208, the instructions of knowledge retrieval module 106b may cause system 100 to use the one or more retrieval queries constructed in step 206, to retrieve relevant external data (e.g., scientific and medical paper, articles, patents, etc.), from external sources and databases (such as scientific papers or patent databases).

[0060] Accordingly, in some embodiments, the instructions of knowledge retrieval module 106b may cause system 100 to execute the one or more retrieval queries constructed in step 206 one or more external data sources and databases (such as scientific papers or patent databases), to retrieve relevant external data or knowledge items (e.g., scientific and medical paper, articles, patents, etc.).

[0061] In some embodiments, the retrieval is based on a similarity between the embeddings of the input data generated in step 204 and embeddings of the external knowledge items. In some embodiments, the retrieval is based on matching the embeddings of the input data generated in step 204, to corresponding embeddings of external data items, to determine relevance of external data. In some embodiments, the embeddings of the external data items may be generated in a manner similar to the embeddings of the input data generated in step 204.

[0062] In some embodiments, relevance may be determined, e.g., based on a distance function, such as a dot product operation or Euclidean distance; via weighted versions where weights are learned by knowledge retrieval module 106b; or via an Optimal Transport mechanism for a sparse matching between sets of vectors.

[0063] In some embodiments, the input data may comprise two or more discrete objects (e.g., a set of drugs for which the goal is to predict drug synergy). In such cases, each of the objects may be embedded as described with reference to step 204, wherein external data retrieval may be performed independently for each of the individual objects. In addition, a set-focused retrieval mechanism may be employed, of example, which is configured to retrieve external data relevant to two or more of the individual objects in the set (e.g., a scientific paper matching multiple drugs). This may be achieved by a scoring function that rewards multiple matches. In another embodiment, an average embedding of all individual objects (and each modality therein) may be computed, and this embedding is fed into the retrieval query. In another embodiment, each object’s embedding is updated via a cross-attention mechanism, wherein attention is computed across objects; these updated embeddings are then fed into the retrieval query. In another embodiment, the Data2Text module is applied over a set of object representations to create set-focused textual queries in addition to individual-focused queries. The individual and set-oriented retrieval results are then pooled together into a retrieval union set.

[0064] In step 210, the instructions of aggregator module 106c may cause system 100 to aggregate the various data embeddings generated in step 204 with respect to the input data, and to aggregate the various data embeddings generated in step 208 with respect to the retrieved relevant external data items, respectively.

[0065] In some embodiments, aggregator module 106c may employ a cross-attention mechanism across data types and modalities. For example, at a first stage, aggregator module 106c may aggregate textual information via a multi-document summarization mechanism, e.g., using a pre-trained summarizer language model. As a next stage, aggregator module 106c may then employ a cross-modal transformer mechanism to synthesize an aggregate the various data modalities. In some embodiments, aggregator module 106c may then aggregate and jointly encode the input data and external data.

[0066] In another embodiment, aggregation is based on an identity function, which retains the set of raw retrieved embeddings, to enable assigning specific weights to specific parts (e.g., specific sentences, or spans of text within / across sentences, or specific image segments) within each retrieved external datapoint.

[0067] In step 212, the instructions of prediction module 106d may cause system 100 to output a target prediction associate with the target case for which input data was received in step 202, based on the aggregated data embeddings generated in step 204 with respect to the input data, and the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items.

[0068] In some embodiments, as noted above, the target case may be a patient, the input data may comprise medical and / or clinical data of the patient, and the target prediction may represent a clinical outcome associated with the patient, such as a specific diagnosis (e.g., cardiovascular failure or stroke), or the likelihood of mortality, length of hospitalization, readmission, developing sepsis, the need for prolonged mechanical ventilation, etc.

[0069] In another example, the target case may be associated with predicting biomedical and / or related properties associated with a molecule, a chemical composition, cell or tissue types, a DNA sequence, a genetic mutation, and the like. The predicted properties may include therapy response, interactions and synergies with other chemical compositions, adverse effects in patients, toxicity, conductivity, thermoelectric properties, and the like. In such cases, the input data may comprise data associated with the particular molecule, chemical composition, cell or tissue type, genetic mutation, etc., as the case may be.

[0070] In some embodiments, the prediction may represent a classification task which aims to assign a correct class label to the aggregated data embeddings generated in step 204 with respect to the input data, and the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items.

[0071] In some embodiments, the classification task may be a binary classification task, wherein class labels may indicate a binary variable or value (e.g., 0 / 1, yes / no). For example, in the case of a clinical outcome prediction associated with the need for prolonged mechanical ventilation of a patient, a value of ‘ 1’ or ‘yes’ may indicate that the patient will require ventilation for more than 7 days, while a value of ‘0’ or ‘no’ may indicate that the patient will not require ventilation for more than 7 days. Similarly, in the case of a clinical outcome prediction associated in-hospital mortality of the patient, a value of ‘ 1 ’ or ‘yes’ may indicate that the patient will not survive their current admission,while a value of ‘0’ or ‘no’ may indicate that the patient will survive their current admission.

[0072] In other cases, the classification task may be a non-binary classification task, wherein class labels may indicate a discrete values on a scale of, e.g., 1-5 or any other suitable scale. For example, in the case of a clinical outcome prediction associated with the length of stay of a patient in a hospital, class labels may indicate three of more discrete categories based on predicted length of stay (e.g., less than 3 days, between 3-7 days, between 1-2 weeks, and more than 2 weeks).

[0073] In some embodiments, the instructions of prediction module 106d may cause system 100 to apply a cross-attention transformer which jointly encodes the aggregated data embeddings generated in step 204 with respect to the input data, and the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items.

[0074] In some embodiments, the encoded output may be reduced to the dimensionality of the target number of predicted classes (e.g., binary yes / no prediction of size two or a set of three or more discrete categories), and passed through a SoftMax layer fed into an objective / loss function. In some embodiments, the goal of prediction module 106d is to compute an aggregate representation from the aggregated data embeddings generated in step 204 with respect to the input data, and the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items, and then predict the target outcome of interest using this aggregate representation.

[0075] In some embodiments, the prediction comprises an indication of a contribution of each of the external data items to the prediction. In some embodiments, the indication comprises, with respect to each external data item, indication of a contribution of specific parts within said external knowledge item. For example, in the case of textual data, specific words, sentences, sections, etc. In the case of image data, specific segments, etc.

[0076] In some embodiments, the output may be reduced to the dimensionality of the target number of predicted classes (e.g., binary classes or a discrete set of classes). The result is passed through a SoftMax layer and fed into a loss function.

[0077] In some embodiments, prediction module 106d may be trained based on a weighted cross-entropy loss function to handle class imbalance. Accordingly, given adataset with N total examples, c classes and examples in class i, class weights are computed as follows:... NVE = — . c-nj

[0078] In some embodiments, the loss function may be based on the average of one or more of the following terms:A cross-entropy term between predicted SoftMax scores and the ground truth label, based on output of the aggregated data embeddings generated in step 204 with respect to the input data.A cross-entropy term between predicted SoftMax scores and the ground truth label based on output of the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items. The purpose of this loss term is to encourage predictive external data representations.A cross-entropy term between predicted SoftMax scores and the ground truth label based on output of the joint encoding of both the aggregated data embeddings generated in step 204 with respect to the input data, and the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items.A diversity term that penalizes Euclidean similarity between the aggregated data embeddings generated in step 204 with respect to the input data, and the aggregated data embeddings generated in step 208 with respect to the retrieved relevant external data items, wherein the Euclidean similarity is passed via a sigmoid normalization function.

[0079] In one embodiment, natural language human feedback may be collected with respect to the retrieved external data, comprising, e.g., positive aspects to be enhanced, and / or negative aspects to be downgraded. In some embodiments, for each feedback in the negative set, a loss term may be added to the loss function, wherein the loss penalizes retrieved aspect embeddings (papers, images, parts therein) that have high similarity to the feedback. In some embodiments, for each feedback in the positive set, a loss term may be added to the loss function which penalizes retrieved aspect embeddings (papers, images, parts therein) that have low similarity (e.g., based on distance) to the feedback.

[0080] As part of training, both input data and external data embeddings may be used. In one embodiment, external data embeddings remain fixed and are updated every K iterations, for efficiency purposes. In another embodiment, the external data embeddings are initialized with pre-trained models and remain fixed, wherein only internal data embeddings are tuned.

[0081] In some embodiments, an alternating optimization scheme may be employed, wherein for K iterations / epochs, the parameters of the input data vector aggregator and of the input data embeddings remain fixed, and only the parameters of the retrieval query embeddings are trained. In subsequent K iterations / epochs, the parameters of the retrieval function are trained and the input data embeddings remain fixed. In one embodiment, training is performed in two stages, wherein the, at a first stage, training is performed with input data only, and then, at a second stage, training also includes external data.

[0082] As part of training, different input data types or modalities may be randomly masked out in the transformer layers, for the purpose of encouraging each data modality to contain predictive information and enable prediction on instances with partially - missing data (e.g., patients with only tabular data).

[0083] In some embodiments, one or more reinforcement learning algorithms may be used, for optimizing the loss function, such as the REINFORCE or A3C algorithms.

[0084] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0085] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), aportable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire. Rather, the computer readable storage medium is a non-transient (i.e., not- volatile) medium.

[0086] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0087] Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logiccircuitry, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention. In some embodiments, electronic circuitry including, for example, an application- specific integrated circuit (ASIC), may be incorporate the computer readable program instructions already at time of fabrication, such that the ASIC is configured to execute these instructions without programming.

[0088] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0089] These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0090] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0091] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware -based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0092] In the description and claims, each of the terms “substantially,” “essentially,” and forms thereof, when describing a numerical value, means up to a 20% deviation (namely, ±20%) from that value. Similarly, when such a term describes a numerical range, it means up to a 20% broader range - 10% over that explicit range and 10% below it).

[0093] In the description, any given numerical range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range, such that each such subrange and individual numerical value constitutes an embodiment of the invention. This applies regardless of the breadth of the range. For example, description of a range of integers from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., as well as individual numbers within that range, for example, 1, 4, and 6. Similarly, description of a range of fractions, for example from 0.6 to 1.1, should be considered to have specifically disclosed subranges such as from 0.6 to 0.9, from 0.7 to 1.1, from 0.9 to 1, from 0.8 to 0.9, from 0.6 to 1.1, from 1 to 1.1 etc., as well as individual numbers within that range, for example 0.7, 1, and 1.1.

[0094] The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the explicit descriptions. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0095] In the description and claims of the application, each of the words “comprise,” “include,” and “have,” as well as forms thereof, are not necessarily limited to members in a list with which the words may be associated.

[0096] Where there are inconsistencies between the description and any document incorporated by reference or otherwise relied upon, it is intended that the present description controls.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: at least one hardware processor; and a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to: receive input data associated with a target case, apply a trained neural embedding module to learn embeddings representing said input data, construct, based on said learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between said learned embeddings and embeddings of said external knowledge items, execute said at least one retrieval query on at least one external data source, to retrieve said one or more relevant external knowledge items, generate an aggregated embeddings representation of said input data, and an aggregated embeddings representation of said one or more relevant external knowledge items, and inference a prediction module on said generated aggregated embeddings of said input data and said one or more relevant external knowledge items, to output a target prediction associated with said target case.

2. The system of claim 1, wherein said target case is a patient, said input data comprises health records and / or clinical data associated with said patient, and said target prediction represents a clinical outcome or a clinical intervention suggestion with respect to said patient.

3. The system of claim 1, wherein said target case is one or more molecules or chemical compositions; said input data comprises data with respect to said one or more molecules or chemical compositions; and said target prediction represents chemical, material and / or therapeutic properties of said one or more molecules or chemical compositions.

4. The system of any one of claims 1-3, wherein said trained neural embedding module is a pre-trained transformer neural network.

5. The system of any one of claims 1-4, wherein said input data comprises data in at least one of the following modalities: tabular data, natural language data, genomic data, proteomic data, image data, video data, and / or audio data.

6. The system of claim 5, wherein said trained neural embedding module comprises a dedicated embedding function for at least one of said data modalities.

7. The system of any one of claims 1-6, wherein said similarity is determined based on one of the following methods: a dot product operation, a Euclidean distance function, weighted average of encoder representations, or an Optimal Transport mechanism for a sparse matching between sets of vectors.

8. The system of any one of claims 1-7, wherein said one or more relevant external knowledge items are selected from at least one of the following categories of knowledge items: scientific papers, medical studies, clinical trials, patents, entries in a biomedical or chemical knowledge base, and knowledge graphs.

9. The system of any one of claims 1-8, wherein said prediction module applies a cross-attention transformer which jointly encodes said aggregated embeddings of said input data and said one or more relevant external knowledge items, into a joint encoding, and wherein said joint encoding is fed into a loss function which outputs said target prediction.

10. The system of any one of claims 1-9, wherein said target prediction classifies said aggregated embeddings of said input data and said one or more relevant external knowledge items into one of a set of class labels.

11. The system of claim 10, wherein said class labels indicate a binary variable.

12. The system of claim 10, wherein said class labels indicate a discrete variable on a scale.

13. The system of any one of claims 1-12, wherein said target prediction comprises an indication of a contribution of each of said one or more relevant external knowledge items to said target prediction.

14. The system of claim 13, wherein said indication comprises, with respect to said one or more relevant external knowledge items comprising textual data, an indication of a contribution of specific parts within each of said one or more relevant external knowledge item.

15. A computer-implemented method, comprising: receiving input data associated with a target case; applying a trained neural embedding module to learn embeddings representing said input data; constructing, based on said learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between said learned embeddings and embeddings of said external knowledge items; executing said at least one retrieval query on at least one external data source, to retrieve said one or more relevant external knowledge items; generating an aggregated embeddings representation of said input data, and an aggregated embeddings representation of said one or more relevant external knowledge items; and inferencing a prediction module on said generated aggregated embeddings of said input data and said one or more relevant external knowledge items, to output a target prediction associated with said target case.

16. The computer-implemented method of claim 15, wherein said target case is a patient, said input data comprises health records and / or clinical data associated with said patient, and said target prediction represents a clinical outcome or a clinical intervention suggestion with respect to said patient.

17. The computer-implemented method of claim 15, wherein said target case is one or more molecules or chemical compositions; said input data comprises data with respect to said one or more molecules or chemical compositions; and said target prediction representschemical, material and / or therapeutic properties of said one or more molecules or chemical compositions.

18. The computer-implemented method of any one of claims 15-17, wherein said trained neural embedding module is a pre-trained transformer neural network.

19. The computer-implemented method of any one of claims 15-18, wherein said input data comprises data in at least one of the following modalities: tabular data, natural language data, genomic data, proteomic data, image data, video data, and / or audio data.

20. The computer-implemented method of claim 19, wherein said trained neural embedding module comprises a dedicated embedding function for at least one of said data modalities.

21. The computer-implemented method of any one of claims 15-20, wherein said similarity is determined based on one of the following methods: a dot product operation, a Euclidean distance function, weighted average of encoder representations, or an Optimal Transport mechanism for a sparse matching between sets of vectors.

22. The computer-implemented method of any one of claims 15-21, wherein said one or more relevant external knowledge items are selected from at least one of the following categories of knowledge items: scientific papers, medical studies, clinical trials, patents, entries in a biomedical or chemical knowledge base, and knowledge graphs.

23. The computer-implemented method of any one of claims 15-22, wherein said prediction module applies a cross-attention transformer which jointly encodes said aggregated embeddings of said input data and said one or more relevant external knowledge items, into a joint encoding, and wherein said joint encoding is fed into a loss function which outputs said target prediction.

24. The computer-implemented method of any one of claims 15-23, wherein said target prediction classifies said aggregated embeddings of said input data and said one or more relevant external knowledge items into one of a set of class labels.

25. The computer-implemented method of claim 24, wherein said class labels indicate a binary variable.

26. The computer-implemented method of claim 24, wherein said class labels indicate a discrete variable on a scale.

27. The computer-implemented method of any one of claims 15-26, wherein said target prediction comprises an indication of a contribution of each of said one or more relevant external knowledge items to said target prediction.

28. The computer-implemented method of claim 27, wherein said indication comprises, with respect to said one or more relevant external knowledge items comprising textual data, an indication of a contribution of specific parts within each of said one or more relevant external knowledge item.

29. A computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to: receive input data associated with a target case; apply a trained neural embedding module to learn embeddings representing said input data; construct, based on said learned embeddings, at least one retrieval query vector configured to identify one or more relevant external knowledge items, based on a similarity between said learned embeddings and embeddings of said external knowledge items; execute said at least one retrieval query on at least one external data source, to retrieve said one or more relevant external knowledge items; generate an aggregated embeddings representation of said input data, and an aggregated embeddings representation of said one or more relevant external knowledge items; and inference a prediction module on said generated aggregated embeddings of said input data and said one or more relevant external knowledge items, to output a target prediction associated with said target case.

30. The computer program product of claim 29, wherein said target case is a patient, said input data comprises health records and / or clinical data associated with said patient, and said target prediction represents a clinical outcome or a clinical intervention suggestion with respect to said patient.

31. The computer program product of claim 29, wherein said target case is one or more molecules or chemical compositions; said input data comprises data with respect to said one or more molecules or chemical compositions; and said target prediction represents chemical, material and / or therapeutic properties of said one or more molecules or chemical compositions.

32. The computer program product of any one of claims 29-31, wherein said trained neural embedding module is a pre -trained transformer neural network.

33. The computer program product of any one of claims 29-32, wherein said input data comprises data in at least one of the following modalities: tabular data, natural language data, genomic data, proteomic data, image data, video data, and / or audio data.

34. The computer program product of claim 33, wherein said trained neural embedding module comprises a dedicated embedding function for at least one of said data modalities.

35. The computer program product of any one of claims 29-34, wherein said similarity is determined based on one of the following methods: a dot product operation, a Euclidean distance function, weighted average of encoder representations, or an Optimal Transport mechanism for a sparse matching between sets of vectors.

36. The computer program product of any one of claims 29-35, wherein said one or more relevant external knowledge items are selected from at least one of the following categories of knowledge items: scientific papers, medical studies, clinical trials, patents, entries in a biomedical or chemical knowledge base, and knowledge graphs.

37. The computer program product of any one of claims 29-36, wherein said prediction module applies a cross-attention transformer which jointly encodes said aggregated embeddings of said input data and said one or more relevant external knowledge items, into a joint encoding, and wherein said joint encoding is fed into a loss function which outputs said target prediction.

38. The computer program product of any one of claims 29-37, wherein said target prediction classifies said aggregated embeddings of said input data and said one or more relevant external knowledge items into one of a set of class labels.

39. The computer program product of claim 38, wherein said class labels indicate a binary variable.

40. The computer program product of claim 38, wherein said class labels indicate a discrete variable on a scale.

41. The computer program product of any one of claims 29-40, wherein said target prediction comprises an indication of a contribution of each of said one or more relevant external knowledge items to said target prediction.

42. The computer program product of claim 41, wherein said indication comprises, with respect to said one or more relevant external knowledge items comprising textual data, an indication of a contribution of specific parts within each of said one or more relevant external knowledge item.

Images