Clinical outcome prediction by applying machine learning model to clinical data

Abstract

A method (1200) includes receiving a clinical data table (201) of a patient. A clinical data table stores clinical data associated with a patient in a tabular form. The method further includes extracting one or more classified features (202) and one or more contiguous features (204) from the clinical data table; and determining (182) one or more predicted clinical outcomes of the patient based on the one or more classified features and the one or more contiguous features extracted from the clinical data table using a clinical prediction model (150). The method further includes providing the one or more predicted clinical outcomes of the patient for output from a client device (110) associated with the user.

Description

Technical Field

[0001] This disclosure relates to clinical outcome prediction by applying machine learning models to clinical data. Background Art

[0002] Clinical prediction models play a key role in routine clinical care by predicting outcomes for patients receiving therapies to treat various medical conditions. In addition to informing professionals, patients, and family members about the risk of outcomes associated with a therapy, outcomes predicted by clinical prediction models provide the ability to infer accurate predictions of treatment response and disease progression based on a patient's clinical characteristics and molecular profile. Furthermore, professionals can develop precision medicine for ultimate use in the medical decision-making process to provide personalized treatment decisions for treating a patient's medical condition, taking into account the risk of outcomes given the patient's clinical characteristics and molecular profile. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method executed on data processing hardware, the method causing the data processing hardware to perform operations comprising: receiving a clinical data table of patient clinical data associated with a patient in tabular form; and extracting one or more categorical features and one or more continuous features from the clinical data table. The operations further comprise: determining one or more predicted clinical outcomes for the patient based on the one or more categorical features and one or more continuous features extracted from the clinical data table using a clinical prediction model. The operations further comprise: providing the one or more predicted clinical outcomes for the patient for output from a client device associated with a user.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the clinical prediction model is executed on data processing hardware and includes a clinical table multi-head attention model. Here, the clinical table multi-head attention model includes a categorical feature encoder, a continuous feature encoder, a connector, a multi-head attention network, and a fully connected feedforward network. The categorical feature encoder is configured to receive as input each of one or more categorical features extracted from the clinical data table and generate as output a corresponding categorical embedding for each categorical feature. The continuous feature encoder is configured to receive as input each of one or more continuous features extracted from the clinical data table and generate as output a corresponding continuous feature embedding for each categorical feature. The connector is configured to concatenate the one or more categorical feature embeddings and the one or more continuous feature embeddings to form a set of parameter embeddings. The multi-head attention network is configured to receive as input each parameter embedding in the set of parameter embeddings formed by the connector and generate as output a corresponding context embedding for each parameter embedding in the set of parameter embeddings. The fully connected feedforward network is configured to receive as input the context embedding generated as output from the multi-head attention network and predict one or more clinical outcomes for the patient as output. In these implementations, the multi-head attention network comprises a stack of N layers, each comprising a multi-head attention mechanism, which may comprise a Transformer layer. Each transformer layer may comprise a normalization layer, a masked multi-head attention layer, a residual connection, and a feedforward layer. In addition, one or more clinical outcomes predicted for a patient include multiple clinical outcomes predicted for a patient, and the fully connected feedforward network comprises multiple heads, each of which is configured to receive as input a context embedding generated as an output from the multi-head attention network and predict a corresponding one of the multiple clinical outcomes for the patient as output. The clinical table multi-head attention model is trained via multi-task learning to jointly teach the clinical table multi-head attention model to learn how to predict multiple clinical outcomes for the patient.

[0005] In some examples, the clinical prediction model is executed on data processing hardware and includes a large language model. In these examples, the operation may further include serializing one or more classification features and one or more continuous features extracted from the clinical data table into an input text sequence. Here, determining one or more predicted clinical outcomes of the patient includes processing the input text sequence using a large language model to generate one or more predicted clinical outcomes. The large language model may include a pre-trained large language model and fine-tuned using few-shot learning. Additionally or alternatively, the large language model may include a domain-specific large language model pre-trained on vocabulary and / or grammar associated with a specific domain. For example, a specific domain may include medical terminology.

[0006] The one or more predicted clinical outcomes may include at least one of overall survival, progression-free survival, or best overall response. Additionally or alternatively, the one or more predicted clinical outcomes may include at least one of a recommended treatment or a prognostic biomarker score.

[0007] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware, the memory hardware communicating with the data processing hardware and storing instructions that cause the data processing hardware to perform operations, the operations comprising: receiving a clinical data table of patient clinical data associated with a patient in tabular form; and extracting one or more categorical features and one or more continuous features from the clinical data table. The operations further comprise: determining one or more predicted clinical outcomes for the patient based on the one or more categorical features and one or more continuous features extracted from the clinical data table using a clinical prediction model. The operations further comprise: providing the one or more predicted clinical outcomes for the patient for output from a client device associated with a user.

[0008] This aspect may include one or more of the following optional features. In some implementations, the clinical prediction model is executed on data processing hardware and includes a clinical table multi-head attention model. Here, the clinical table multi-head attention model includes a categorical feature encoder, a continuous feature encoder, a connector, a multi-head attention network, and a fully connected feedforward network. The categorical feature encoder is configured to receive each of one or more categorical features extracted from the clinical data table as input and generate a corresponding categorical embedding for each categorical feature as output. The continuous feature encoder is configured to receive each of one or more continuous features extracted from the clinical data table as input and generate a corresponding continuous feature embedding for each categorical feature as output. The connector is configured to concatenate the one or more categorical feature embeddings and the one or more continuous feature embeddings to form a set of parameter embeddings. The multi-head attention network is configured to receive each parameter embedding in the set of parameter embeddings formed by the connector as input and generate a corresponding context embedding for each parameter embedding in the set of parameter embeddings as output. The fully connected feedforward network is configured to receive as input the context embedding generated as output from the multi-head attention network and predict one or more clinical outcomes for the patient as output. In these implementations, the multi-head attention network comprises a stack of N layers, each comprising a multi-head attention mechanism, which may comprise a Transformer layer. Each transformer layer may comprise a normalization layer, a masked multi-head attention layer, a residual connection, and a feedforward layer. In addition, one or more clinical outcomes predicted for a patient include multiple clinical outcomes predicted for a patient, and the fully connected feedforward network comprises multiple heads, each of which is configured to receive as input a context embedding generated as an output from the multi-head attention network and predict a corresponding one of the multiple clinical outcomes for the patient as output. The clinical table multi-head attention model is trained via multi-task learning to jointly teach the clinical table multi-head attention model to learn how to predict multiple clinical outcomes for the patient.

[0009] In some examples, the clinical prediction model is executed on data processing hardware and includes a large language model. In these examples, the operation may further include serializing one or more classification features and one or more continuous features extracted from the clinical data table into an input text sequence. Here, determining one or more predicted clinical outcomes of the patient includes processing the input text sequence using a large language model to generate one or more predicted clinical outcomes. The large language model may include a pre-trained large language model and fine-tuned using few-shot learning. Additionally or alternatively, the large language model may include a domain-specific large language model pre-trained on vocabulary and / or grammar associated with a specific domain. For example, a specific domain may include medical terminology.

[0010] The one or more predicted clinical outcomes may include at least one of overall survival, progression-free survival, or best overall response. Additionally or alternatively, the one or more predicted clinical outcomes may include at least one of a recommended treatment or a prognostic biomarker score.

[0011] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an example system for using a clinical prediction model to predict one or more clinical outcomes from a table of patient clinical data.

[0013] Figure 2A and Figure 2B is a diagram of an example training process for training a Clinical Tabular Transformer (ClinTaT) model to predict clinical outcomes for patients based on tabular clinical data.

[0014] Figure 3 is a diagram of an example Transformer layer.

[0015] Figure 4 Figure 1 is a diagram of an example training process for fine-tuning a large pre-trained language model to predict clinical outcomes for patients.

[0016] Figures 5A-5D Shown are example plots depicting the performance of the ClinTaT model in predicting clinical outcomes across multiple cancer types.

[0017] Figures 6A-6C Example plots are shown each depicting the performance of the ClinTaT model in predicting clinical outcomes of overall survival and progression-free survival across multiple cancer types.

[0018] Figure 7 is a table depicting the comparison of the area under the curve (AUC) performance between the ClinTaT model and other baseline models in predicting treatment response.

[0019] Figure 8 is a table depicting the comparison of C-index performance between the ClinTaT model and other baseline models in predicting overall survival.

[0020] Figure 9 is a table depicting the comparison of C-index performance between the ClinTaT model and other baseline models in the prediction of progression-free survival.

[0021] Figure 10This is a table depicting the area under the few-shot learning curve (AUC) performance of various large language models pre-trained on different training corpora.

[0022] Figure 11 Table evaluating the performance changes using different encoder networks stacked on top of various pre-trained large language models used for fine-tuning treatment response prediction.

[0023] Figure 12 is a flow chart of an example arrangement of operations of a method for predicting one or more clinical outcomes from tabulated patient data.

[0024] Figure 13 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0025] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Clinical prediction models play a key role in routine clinical care by predicting outcomes for patients receiving therapies to treat various medical conditions. In addition to informing professionals, patients, and family members about the risk of outcomes associated with a therapy, outcomes predicted by clinical prediction models provide the ability to infer accurate predictions of treatment response and disease progression based on a patient's clinical characteristics and molecular profile. Furthermore, professionals can develop precision medicine for ultimate use in the medical decision-making process to provide personalized treatment decisions for treating a patient's medical condition, taking into account the risk of outcomes given the patient's clinical characteristics and molecular profile.

[0027] Although machine learning is commonly used to train conventional clinical prediction models, due to the fact that the vast majority of clinical data required for training are stored in tabular form (i.e., clinical data are stored in tables), machine learning methods are generally limited to tree-based aggregation models, such as decision trees. Compared to tree-based aggregation models, deep neural networks (DNNs) offer many advantages, such as the ability to train the resulting model end-to-end, utilizing unlabeled / unsupervised training data, being highly robust to both missing and noisy data features, and generally providing better interoperability. DNNs employing multi-head attention mechanisms (e.g., such as Transformers) have revolutionized the fields of natural language processing and computer vision. However, these models lack the compatibility of training and interpreting data in tabular form.

[0028] The implementation method of this paper relates to a clinical table transformer (ClinTaT) model for predicting a patient's clinical outcome based on tabular clinical data. Advantageously, the ClinTAT model provides a tabular data modeling architecture using a self-attention mechanism. The examples in this paper depict Transformer as the type of self-attention mechanism adopted by the ClinTAT model, however, other types of self-attention mechanisms can be adopted, such as, but not limited to, Conformer and lightweight convolutional neural networks. The ClinTaT model is particularly effective in modeling continuous features other than categorical features extracted from a patient's clinical data table, so that the continuous features participate in the self-attention modeling process and are not dominated by the categorical features during training. This aspect is particularly important in clinical prediction outcome settings, where continuous features represent key information, such as patient age, patient body mass index (BMI), laboratory results / readings, and other variables representing the patient's medical profile, for predicting one or more outcomes related to the patient's medical condition and the corresponding therapy the patient is receiving to treat the patient's medical condition. The implementation further includes applying multi-task learning techniques to train the ClinTaT model on multiple loss targets to teach the ClinTaT model to learn how to predict multiple clinical outcomes of the patient based on the patient's clinical data, wherein the clinical data is represented by continuous features and categorical features extracted from the clinical data table. Example outcomes that the ClinTaT model can be trained to learn via multi-task learning may include overall survival (OS) (i.e., in months), progression-free survival (PFS) (in months), and best overall response (BOR). The ClinTaT model can be similarly trained to learn how to predict other outcomes, which may include but are not limited to laboratory results, treatment decisions, disease predictions, drug safety decisions / scores, etc.

[0029] Another implementation involves using a large language model (LLM) to predict a patient's clinical outcome based on an input text sequence serialized from tabular clinical data. Advantageously, when available training samples are limited, the LLM is able to provide in-depth context-based learning capabilities by leveraging knowledge from other resources to downstream tasks with minimal tuning. In contrast, the robustness / accuracy of the ClinTaT model is improved when more supervised training samples of clinical data tables containing patients labeled with clinical outcomes are available. Therefore, although the performance of the ClinTaT model may become degraded due to the low inductive bias inherent in long-range dependency modeling when labeled training samples are scarce, the LLM is able to achieve context-based learning capabilities through few-shot learning techniques when only a small amount of training samples are available. The use of LLM is particularly beneficial for predicting clinical outcomes in the field of rare diseases, where historical patient records are limited and therefore generally insufficient to train the ClinTAT model. More specifically, these additional implementations involve utilizing LLMs that have been pre-trained on natural language text in the medical field and then fine-tuned via few-shot learning: by conditioning a domain-specific pre-trained LLM on available input text sequences serialized from tabular clinical data for predicting specific clinical outcomes. Using a domain-specific LLM allows smaller LLMs to be used for few-shot learning, thereby reducing processing / memory requirements and training time to fine-tune the LLM via few-shot learning to predict one or more different clinical outcomes. Example outcomes that an LLM model can be trained to learn via multiple loss objectives can include overall survival (OS) (i.e., in months), progression-free survival (PFS) (in months), and best overall response (BOR). The LLM can be similarly trained to learn how to predict other outcomes, which can include but are not limited to laboratory results, treatment decisions, disease predictions, drug safety decisions / scores, etc.

[0030] refer to Figure 1In some implementations, the system 100 includes a client device 110 that inputs a clinical data table 201 into a clinical prediction model 150 to predict one or more clinical outcomes 182 for a patient based on the clinical data table 201. The client device 110 is associated with a user 10, such as a healthcare professional (HCP), who can communicate with a remote system 140 via a network 130. The remote system 140 can be a distributed system (e.g., a cloud environment) with scalable / elastic resources 142. The resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some implementations, the remote system 140 executes a clinical prediction application 160 configured to execute the clinical prediction model 150. Here, the client device 110 can access the application 160 running on the remote system 140 and input the clinical data table 201 into the clinical prediction model 150 via a graphical user interface (GUI) executed on the client device 110. The client device 110 may additionally or alternatively execute the application 160 to enable the ability to run the clinical prediction model 150 on the client device 110 for predicting the clinical outcome 182 .

[0031] One or more clinical outcomes 182 predicted by the clinical prediction model 150 can inform the patient, healthcare provider and / or the patient's relatives to make better testing and treatment decisions for a specific health condition diagnosed, or to develop risk stratification for a treatment trial. For example, a patient associated with the clinical data table 201 may have metastatic bladder cancer and is receiving (or planning to receive) immunotherapy to produce antibodies against programmed death protein-1 / programmed death protein ligand 1 (PD-1 / PD-L1) as a form of treatment. In this example, the clinical data table 201 includes columns of data related to the patient, such as the type of medical condition (e.g., bladder cancer) the patient is diagnosed with, the class of immunotherapy drugs (e.g., PDL1), patient characteristics / demographics, laboratory results, imaging tests, and patient medical history. Patient characteristics / demographics may include the patient's age (e.g., 74 years old), gender, race, ethnicity, height, weight, body mass index (BMI), etc. Laboratory results may include columns for specific laboratory tests and values ​​indicating the results of the laboratory tests. For example, the column "Albumin" in the data table 201 indicates that the patient's laboratory result albumin (i.e., the level of the biomarker albumin) is equal to "4.1". The clinical data table 201 may include many additional laboratory results, such as hemoglobin and PD-L1 expression measured on the tumor. For example, PD-L1 expression can form the basis of whether immunotherapy is even effective, because a person whose tumor expresses PD-L1 in one percent (1%) or more of the tumor cells may be more likely to have a durable response than a person whose PD-L1 expression is less than one percent (1%). However, because there may be other factors that people with low PD-L1 expression may still respond significantly to immunotherapy, one or more clinical outcomes 182 predicted by the clinical prediction model 150 can help identify whether an individual will respond to immunotherapy. The patient history can indicate other details about the patient, such as whether the patient smokes, whether he drinks alcohol, and whether the patient has been diagnosed with and / or is being treated for other diseases / medical conditions.

[0032] In the example shown, the clinical prediction model 150 corresponds to a cancer prognosis prediction model whose task is to predict clinical outcomes 182 of overall survival (OS), progression-free survival (PFS), and best overall response (BOR). After the clinical prediction model 150 generates / predicts the clinical outcomes 182, the output module 190 can provide one or more clinical outcomes 182 for output from the client device 110. In the example shown, the client device 110 receives the clinical outcomes 182 from the clinical prediction model 150, and the GUI executed on the client device 110 displays the clinical outcomes 182 on the screen 114 of the client device 110. The GUI can also present category features and clinical features from the clinical data table 201 associated with the patient for display on the screen 114. The output module 190 can also store one or more clinical outcomes 182 together with the corresponding clinical data table 201 in the data storage device 180 and / or transmit the clinical outcomes 182 to an agent, institution, or other entity. Notably, and as described with reference to Figure 2A and Figure 2B As described in more detail, the clinical prediction model 150 may be trained using multi-task learning techniques to combine customized loss objectives associated with each of the different clinical outcomes 182 .

[0033] Clinical data table 201 stores the prognostic variables associated with the patient's clinical characteristics and molecular profile in a tabular form. The values ​​of each column can be obtained from various sources and obtained with the patient's consent. The data stored in clinical data table 201 can include both categorical features 202 and continuous features 204. The categorical features 202 in the example shown include "bladder" for the cancer type column and PDL1 for the drug category column. Although not explicitly shown in data table 201, categorical features 202 can additionally include values ​​that have been scored / binned into categories (e.g., high, low, PDL1 positive), such as when a given readout value meets a threshold or falls within a range of values. On the other hand, continuous features 204 are typically related to numerical values ​​such as laboratory result readout, patient age and / or patient BMI / weight. In the example shown, continuous features 204 include "74" for the age column and "4.1" for the albumin column.

[0034] In some examples, the user 10 can select / filter the type of clinical data included in the clinical data table 201 being fed to the clinical prediction model 150. Similarly, the user can obtain a clinical data table 201 for a population of patients that share a particular trait. For example, the user 10 can use the GUI to provide an input requesting a clinical data table 201 for all patients between the ages of 40-50 years old and diagnosed with metastatic prostate cancer. In this scenario, the user 10 can obtain the clinical outcomes 182 for patients in the population of interest who exhibit a particular trait (e.g., 40 to 50 years old and diagnosed with prostate cancer). The user 10 can further insert different values ​​(such as immunotherapy drug class) to observe how the clinical outcome 182 predicted for a given patient changes across different immunotherapy drug classes.

[0035] In some implementations, the user 10 may be coordinating a clinical trial and may compile a clinical data table 201 for a first group of patients / participants belonging to the active control arm (who are receiving conventional / effective treatment used in clinical care), and also compile a clinical data table 201 for a second group of patients / participants who may join the experimental arm to be treated with the target / experimental therapy. This second group may be a simulation group (or virtual patient population) used to test the design of the clinical trial in a simulated manner, thereby significantly reducing patient risk and trial costs. The clinical prediction model 150 can effectively predict the clinical outcomes 182 of these perspective participants in the second group for comparison with the clinical outcomes 182 of the first group of patients / participants in the active control arm. The comparison results can indicate which candidates are suitable for actual participation in the experimental arm. For example, the comparison results may show that the clinical prediction model 150 predicts that the clinical outcomes for women under the age of 50 are undesirable. However, the clinical outcomes 182 predicted for men between the ages of 40 and 50 may indicate a high probability that the target / experimental therapy will be effective for men between the ages of 40 and 50.

[0036] In some implementations, the clinical prediction model 150 includes a clinical table transformer (ClinTaT) model 200 that uses a self-attention mechanism to provide a tabular data modeling architecture. The examples herein depict the Transformer as the type of self-attention mechanism employed by the ClinTAT model 200, however, other types of self-attention mechanisms may be employed, such as, but not limited to, Conformers and lightweight convolutional neural networks. Therefore, the ClinTaT model may also be referred to as a clinical table multi-head attention model. As described below with reference to Figure 2A and Figure 2B Described in more detail, the ClinTaT model is effective in modeling both categorical features 202 and continuous features 204 extracted from a clinical data table 201 of a given patient 10 .

[0037] In other implementations, the clinical prediction model 150 includes a large language model (LLM) 400. The LLM 400 provides the ability to provide deep in-context learning capabilities (even when the number of training data tables is sparse) by leveraging knowledge from other resources to downstream tasks with minimal tuning. While the ClinTaT model 200 is suitable for modeling tabular data, the LLM 400 is trained to learn from input text sequences 402 ( Figure 4 ) predict clinical outcomes. Thus, the LLM 400 is configured to process an input text sequence 402 serialized / converted from the tabular clinical data 201 to generate one or more predicted clinical outcomes 182. The LLM 400 may include a domain-specific LLM pre-trained on a vocabulary / grammar associated with the domain (such as medical terminology). The use of domain-specific LLMs allows smaller LLMs to be used for few-shot learning, whereby the input text sequence 402 serialized from the training data table 40 can be used as the context of a query to predict clinical outcomes 182. Another implementation involves utilizing a large language model (LLM) to predict a patient's clinical outcome based on an input text sequence serialized from the tabular clinical data. Advantageously, when available training samples are limited, the LLM is able to provide deep in-context learning capabilities by leveraging knowledge from other resources to downstream tasks with minimal tuning.

[0038] The training network 50 is trained on a set of training data tables 40, 40a-n, each of which is associated with a corresponding training patient and includes prognostic variables related to the clinical characteristics and molecular profile of the corresponding training patient in tabular form. When the training network 50 is training the ClinTaT model 200, the extractor 55 extracts categorical features 202 and continuous features 204 from each training data table 40 and provides the extracted categorical features 202 and continuous features 204 to the training network 50 for use in training the ClinTaT model.

[0039] Now refer to Figure 2A and Figure 2B The training network 50 trains the ClinTaT model 200 to learn how to extract the M classification features (x cat_1 、x cat_2 ,…x cat_m )202 and C continuous features (x cont_1 、x cont_2 ,…x cont_c) 204 predict one or more clinical outcomes 182. Each training clinical data table 40 also includes one or more tags 60, each of which is related to the corresponding clinical outcome 182 that the model 200 is being trained to learn how to predict. These tags include the clinical outcomes actually obtained / recorded by the training patients associated with the corresponding table 40. The training patients are completely anonymized. In the example shown, the model 200 is being trained to learn how to predict the clinical outcomes 182 including overall survival (OS) (in months), progression-free survival (PFS) (in months) and best overall response (BOR), and the training tags 60 include the corresponding columns of OS (in months), PFS (in months) and BOS (in months) in the table 40. As will become clear, the tags 60 operate as the training target of the loss module 240 to predict the output from the ClinTAT model 200 during training. The architecture of the ClinTaT model is particularly effective in modeling continuous features in addition to categorical features extracted from patient clinical data tables, so that the continuous features participate in the self-attention modeling process and are not dominated by categorical features during training. This aspect is particularly useful in clinical outcome prediction settings, where continuous features represent key information such as patient age, patient body mass index (BMI), laboratory results / readings, and other variables representing the patient's medical profile for predicting one or more outcomes related to the patient's medical condition and the corresponding therapy the patient is receiving to treat the patient's medical condition.

[0040] The ClinTaT model 200 includes a categorical feature encoder 210, a continuous feature encoder 214, a connector 220, a multi-head attention network 400 (e.g., a stack of N Transformer layers), and a fully connected feedforward network 230. For each training data table 40, the categorical feature encoder 210 is the corresponding categorical feature x cat_i 202 Generate classification embedding cat_e φ (x cat_i ) 212 as output. In some examples, the classification feature encoder 210 includes a lookup table for the embedding of each possible classification feature 202 represented by a column of the training data table 40. Specifically, for each classification feature (column) i, ​​the encoder 210 may include an embedding lookup table where i∈{1,2,...,m}. For i The i-th classification feature of the category, embedding table With (d i +1) embeddings, where the extra embedding corresponds to the missing value. Encoded value x i =j∈[0,1,2,..,di] is embedded as in and Column-specific and unique identifier Distinguish the categories in column i from the categories in the other columns. The dimension l of is a hyperparameter. Therefore, the categorical feature encoder 212 outputs a set of categorical embeddings E(x cat )={cat_e φ (x cat_1 ),....,cat_e φ (x cat_m )}, each classification embedding represents a corresponding classification feature in the classification features 202 extracted from the corresponding training data table 40.

[0041] For each training data table 40, the continuous feature encoder 214 generates the corresponding continuous feature xx cont_i 204 Generate continuous embedding cont_e φ (x cont_i ) 216 as output. While the categorical feature encoder 210 includes a dictionary-like lookup table of embeddings, where different numbers of tokens representing categorical features correspond to embeddings, the continuous feature encoder 214 applies a linear neural network layer to multiply the numbers representing corresponding categorical features 204 (e.g., "74" for the age column or "4.1" for the albumin lab result column) to obtain a corresponding sequence of embeddings corresponding to the categorical features 204. By doing so, the continuous embeddings 216 can participate in the self-attention modeling process so that the corresponding continuous features 204 are not dominated by the categorical features 202 during training. Notably, the continuous embeddings 216 generated for each corresponding continuous feature 204 can also include a column-specific and unique identifier The identifier distinguishes the value of the continuous feature 204 represented in column i from the values ​​represented in other columns in the training data table 40. Therefore, the continuous feature encoder 214 outputs a set of continuous embeddings E(x cont )={cont_e φ (x cont_1 ),....,cont_e φ (x cont_c )}, each embedding represents a corresponding continuous feature in the continuous features 204.

[0042] Connector 220 connects the set of classification embeddings E(x cat ) and the set of continuous embeddings E(x cont ), to form a set of parameter embeddings E φ 222, among which E φ ={cat_e φ (x cat_1 ),....,cat_e φ (x cat_m)}+{cont_e φ (x cont_1 ),....,cont_e φ (x cont_c )}. The multi-head attention network 300 may include a stack of N layers, each of which includes a corresponding multi-head attention mechanism ( Figure 3 )306. The multi-head attention network 300 is embedded from the set E of parameters φ 222 receives each corresponding parameter embedding, which includes the corresponding classification feature x cat_i The classification embedding cat_e output from the classification feature encoder 210 φ (x cat_i )212 or for the corresponding classification feature x cont_i 204 continuous embedding cont_e output from the continuous feature encoder 214 φ (x cont_i ) 216. The multi-head attention network 300 is configured to continuously aggregate the set E from the parameter embedding φ The context of other embeddings 212, 216 in each category embeds cat_e φ (x cat_i )212 and each continuous embedding cont_e φ (x cont_i ) 216 generates / transforms the corresponding context embedding 350. The multi-head attention network 300 represented by a stack of N layers (e.g., a stack of N Transformer layers) can be represented as a function f θ , the function embeds the set E of parameters φ The operation is performed and the corresponding context embedding 350 is returned as {h1,...,h m ,h m+1 ,...,h c}, where for i∈{1,…,m,m+1,…c}, h i h i ∈R d It is worth noting that m+1 represents the first continuous feature (x cont_1 )204. Figure 2A As depicted in the context embedding 350{h1,...,h m ,h m+1 ,...,h c}Form a context vector of dimension (dx(m+c)), which is provided as input to a fully connected feed-forward network 230, which may include a multi-layer perceptron (MLP).

[0043] Figure 3 3 shows an example transformer layer 300 among the multiple transformer layers when the multi-head attention network 300 includes multiple transformer layers. As mentioned above, Figure 2A and Figure 2B The multi-head attention network 300 is not limited to transformer layers, but may instead include conformer layers, lightweight convolutional layers, or other networks that employ a multi-head attention mechanism. Figure 3 In the example, the initial transformer layer 300 is embedded from the set of parameters E φ (x cat +x cont ) receives the corresponding parameter embedding 222, which includes the corresponding classification feature x cat_i The classification embedding cat_e output from the classification feature encoder 210 φ (x cat_i )212 or for the corresponding classification feature x cont_i 204 continuous embedding cont_e output from the continuous feature encoder 214 φ (x cont_i ) 216, and generates a corresponding output representation / embedding 350 received as input by the next transformer layer 300. That is, each transformer layer 300 after the initial transformer layer 300 may receive as output an input embedding 350 corresponding to the output representation / embedding generated as output by the immediately preceding transformer layer 300. The final transformer layer 300 (e.g., the last transformer layer in the stack of transformer layers 300) performs the training by successively aggregating the set of parameter embeddings E from the input layer 300. φ The context of other embeddings 212, 216 in each category embeds cat_e φ (x cat_i )212 and each continuous embedding cont_e φ (x cont_i )216 generates / transforms the corresponding context embedding 350, where E φ ={cat_e φ (x cat_1 ),....,cat_e φ (x cat_m ),cont_e φ (x cont_1 ),....,cont_e φ (x cont_c )}.

[0044] Each transformer layer 300 of the multi-head attention network includes a normalization layer 304, a masked multi-head attention layer 306, a residual connection 308, and a feed-forward layer 312. The masked multi-head attention layer 306 provides a flexible way to control the amount of context used by the model 200. Specifically, after the normalization layer 304 normalizes the input parameter embedding 222, the masked multi-head attention layer 306 projects the input to the values ​​of all heads. Thereafter, the masked multi-head layer 306 can mask the attention scores of the current parameter embedding 222 to produce a set E of parameter embeddings. φ The weighted averages of all heads are then concatenated and passed to the dense layer 2 316, where the residual connection 314 is added to the normalized input and output of the dense layer 316 to form the final output of the multi-head attention layer 306. The residual connection 308 is added to the output of the normalization layer 304 through the adder 330 and provided as input to the corresponding one of the masked multi-head attention layer 306 or the feedforward layer 312.

[0045] The feed-forward layer 312 applies a normalization layer 304, followed by a dense layer 1 320, a rectified linear layer (ReLu) 318, and a dense layer 2 316. The ReLu 318 is used as an activation on the output of the dense layer 1 320. As in the multi-head attention layer 406, a residual connection 314 from the output of the normalization layer 404 can be added to the output of the dense layer 2 316 via an adder 330.

[0046] Return Reference Figure 2A Based on the set 350 of context embeddings for each corresponding training data table 40, the fully connected feed-forward network 230 generates one or more predicted clinical outcomes 182 as output, and the loss module 240 generates a training loss 290 based on the one or more predicted clinical outcomes 182 and one or more corresponding training labels 60 of the training data table. Here, the training labels 60 may include corresponding columns for OS (in months), PFS (in months), and BOS (in months) in table 40, and operate as training targets for the training network 50 to teach the ClinTaT model 200 to predict clinical outcomes 182 from the tabulated data 202, 204. Thus, the training network 50 trains the ClinTaT model 200 via supervised learning by updating the parameters of the ClinTaT model 200 based on the training loss 290 obtained for each training data table 40. For example, the parameters / weights of the categorical feature encoder 210, the continuous feature encoder 214, the multi-head attention network 300, and the fully connected feed-forward network 230.

[0047] Figure 2BThe training network 50 is shown to apply multi-task learning techniques to train the ClinTaT model 200 on multiple loss objectives to teach the ClinTaT model 200 to learn how to predict multiple clinical outcomes 182, 182a-c. Continuing with the example, the multiple clinical outcomes 182 that the ClinTaT model can be trained to learn via multi-task learning may include overall survival (OS) (i.e., in months), progression-free survival (PFS) (in months), and best overall response (BOR). The ClinTaT model can be similarly trained to learn how to predict other outcomes, which may include but are not limited to laboratory results, treatment decisions, etc.

[0048] In the example shown, the fully connected feed-forward network 230 includes a plurality of heads 232a-c, each of which is trained to generate a contextual embedding 350 {h1, ..., h} based on the corresponding categorical features 202 and continuous features 204 generated by the multi-head attention network 300 for each training data table 40. m ,h m+1 ,...,h c} to output / generate a corresponding clinical outcome of the plurality of clinical outcomes 182. For example, the OS header 232a is configured to output the predicted overall survival 182a as a value representing a number of months, the PFS header 232b is configured to output the predicted progression-free survival as a value representing a number of months, and the BOR header 232c is configured to output the value of the predicted best overall response. Each header 232a-c can include a single linear projection layer corresponding to the corresponding clinical outcome 182 being predicted by the header.

[0049] The loss module 240 includes a plurality of sub-loss modules 242a-c, each associated with a corresponding one of the headers 232a-c. Each sub-loss module 242 is configured to determine a corresponding sub-loss 244, 244a-c based on the corresponding clinical outcome 182 predicted by the corresponding header 232a-c and the training labels 60a-c associated with the corresponding clinical outcome 182. For example, the sub-loss module 242a may correspond to an OS loss module, which determines an OS loss 244a based on the predicted overall survival 182a output from the OS header 232a and the training labels 60a associated with the true value that the OS header 232a is learning to predict. In some examples, the OS loss 244a includes a Cox proportional hazards (CPH) loss function. The sub-loss module 242b may correspond to a PFS loss module, which determines a PFS loss 244b based on the predicted PFS 182b output from the PFS header 232b and the training labels 60b associated with the true value that the PFS header 232b is learning to predict. In some examples, the PFS loss 244b includes a Cox proportional hazards (CPH) loss function. The sub-loss module 242c can correspond to a BOR loss module that determines the BOR loss 244c based on the predicted BOR 182c output from the BOR head 232c and the training labels 60c related to the true value that the BOR head 232c is learning to predict. In some examples, the BOR loss 244c includes a cross-entropy loss function. Thus, the sub-loss 244 determined by the sub-loss module 242 provides a multi-loss objective for enabling the ClinTaT model 200 to predict multiple endpoints (i.e., clinical outcomes 182), and thereby introduces an inductive bias to allow the model 200 to prefer some predictions over other predictions to achieve better generalization.

[0050] Continue to refer to Figure 2B The loss module 240 includes a combined loss module 248, which is formed by summing different sub-losses 244 each associated with a corresponding one of the different clinical outcomes 182 predicted by the fully connected feed-forward network 230 into a unified loss target L f The total loss represented by L is 290 to provide a joint learning paradigm. f It can be expressed as follows. Where I represents the total number of tasks, and α i represents the soft weight of any task i.

[0051] After the ClinTaT model 200 is trained, the trained ClinTaT model 200 can be used as a clinical prediction model 150 for predicting one or more clinical outcomes 182 from the clinical data table 201 of the corresponding patient. Figure 1 , refer to Figure 2A and Figure 2B , the extractor 55 can extract all categorical features 202 and continuous features 204 from the clinical data table 201. Thereafter, the categorical feature encoder 210 can generate categorical embeddings 212 from the categorical features 202, and the continuous feature encoder 214 can generate continuous embeddings 216 from the continuous features 204. The concatenator 220 can then concatenate the categorical embeddings 212 and the continuous embeddings 216 to form a corresponding set E of parameter embeddings that are input to the multi-head attention network 300. φ 222. The multi-head attention network 300 can apply the function f θ In the parameter embedding set E φ The above operations are performed and the corresponding context embedding 350 is returned as input to the fully connected feed-forward network 230. Based on the context embedding 350, the feed-forward network 230 can predict one or more clinical outcomes 182 as output. Continuing with the example, the feed-forward network 230 can include an OS header 232a that outputs a predicted OS 182a based on the clinical data table 201, a PFS header 232b that outputs a predicted PFS 182b based on the clinical data table 201, and a BOR header 232c that outputs a predicted BOR 182c. Figure 1 As shown in , the output module 190 can present the predicted OS 182a as “overall survival -9.1 months”, the predicted PFS 182b as “progression-free survival -3.1 months”, and the predicted BOR 182c as “best overall response -0” for display on the screen 114 of the client device 110.

[0052] Figure 4 The training network 50 is shown training the LLM 400 on the training data table 40 to teach the LLM to learn how to predict one or more clinical outcomes 182. As previously described, when the available training data (e.g., the training data table) is sparse or insufficient to train the ClinTaT model 200, the training network 50 can train the LLM 400 to be used as the clinical prediction model 150 because the LLM is able to provide deep in-context learning capabilities by leveraging knowledge from other resources to downstream tasks with minimal tuning when available training samples are limited.

[0053] The training network 50 applies serialization 410 to serialize the tabular clinical data stored in each training data table 40 into a corresponding input text sequence 402. Here, the input text sequence 402 serialized from the features in the columns of each training data table 40 includes a sequence of natural language tokens (e.g., words / word fragments) that the LLM is able to understand and encode. In some examples, the serialization 410 applies a manual serialization template to the features of each column. For example, the manual serialization template may include "{attribute} is {value}". In the example shown, for the first training data table 40a having values ​​in columns representing attributes of cancer type, drug class, age, and albumin, the manual serialization template applied by the serialization 410 produces the input text sequence 402 "The patient has been diagnosed with bladder cancer. The age is 74 years old. The albumin is 4.1. The drug class is PD1 / PDL1.". Other attributes (such as any prognostic or predictive biomarkers) may also be used with the methods and systems described herein.

[0054] An input text sequence 402 serialized from each training data table 40 is provided as input to an LLM 400 for predicting one or more clinical outcomes 182 therefrom. Implementations herein relate to an LLM 400 comprising a pre-trained LLM and a training network 50 that performs few-shot learning using the input text sequence 402 serialized from each training data table 40 as context for predicting one or more clinical outcomes 192. The pre-trained LLM 400 may include a domain-specific LLM pre-trained on a vocabulary / grammar associated with a domain (such as medical terminology). Using a domain-specific LLM allows for a smaller LLM to be used for few-shot learning to fine-tune the LLM to predict clinical outcomes, whereby the input text sequence 402 serialized from the training data table 40 can be used as context for a query to predict clinical outcomes 182. In some examples, the training tokens 60 are serialized into corresponding natural language queries that are provided to the LLM 400 to predict clinical outcomes, such that the input text sequence 402 serialized from the training data table 40 can be used as context for a natural language query to predict clinical outcomes. Additionally, using few-shot learning to fine-tune LLMs allows for smaller LLMs with reduced processing / memory requirements and improved latency to make robust and accurate clinical predictions compared to domain-independent LLMs containing billions of more parameters.

[0055] Pre-trained LLMs can include the Bidirectional Encoder Representations from Transformers (BERT) model or domain-specific LLMs pre-trained on clinical and / or biomedical corpora. Example domain-specific LLMs can include, but are not limited to, BioBERT, ClinicalBERT, SciBERT, PubMedBERT.

[0056] Continue to refer to Figure 4 , the training network 50 can also apply an encoder network 248 stacked on the LLM 400 for fine-tuning the LLM 400 on a multi-loss objective to teach the LLM 400 to predict clinical outcomes 182, 182a-c. In some examples, the encoder network 248 includes a single linear layer or a stack of multi-head attention layers. For example, the multi-head attention layer can include a Transformer layer or a Conformer layer. In some examples, the LLM 400 receives one or more clinical outcomes to be predicted by the LLM 400 as a requested query. The encoder network 248 can receive the embedding output from the LLM 400 and predict one or more clinical outcomes 182 as an output, and the loss module 240 generates a training loss 290 based on the one or more predicted clinical outcomes 182 and one or more corresponding training labels 60 for the training data table. Here, the training tokens 60 may include corresponding columns for OS (in months), PFS (in months), and BOS (in months) in Table 40, and operate as training objectives for the training network 50 to fine-tune the LLM 400 and the encoder network 248 to predict the clinical outcome 182 from the input text sequence 402. During fine-tuning, the parameters of the pre-trained LLM 400 may remain fixed / frozen, while the parameters of the encoder network 248 are tuned / updated based on the training loss 290.

[0057] Figures 5A-6C Shown are example drawings depicting the performance of the ClinTAT model 200 in predicting clinical outcomes. In these embodiments, the ClinTaT model 200 is trained on training data samples obtained from a comprehensive curated cohort (MSK-IMPACT) at Memorial Sloan Kettering Cancer Center (MSKCC), with 1,479 patients across 16 different cancer types receiving immune checkpoint blockade (ICB) treatment, wherein patients are divided into responders (R) or non-responders (NR) to treatment (PD-1 / PD-L1 inhibitors, CTLA-4 blocking or combination) based on the best overall response on solid tumor efficacy evaluation criteria (RECIST) v1.1 or imaging. Up to 16 biological features are collected for each patient, including genomic, molecular, clinical and demographic variables (i.e., represented as corresponding categorical features 202 or continuous features 204). The training set includes a training data table 40 for 1,184 patients, and the test set includes a clinical data table for 295 patients. The evaluation objectives were to predict clinical outcome to immunotherapy (binary classification) and both overall survival and progression-free survival (regression) in the test data across different cancer types.

[0058] Figures 5A-5DExample plots 500a-d are shown, each depicting the performance of the ClinTaT model 200 in predicting clinical outcomes across multiple cancer types. The y-axis of each plot 500a-d represents the true positive rate, while the x-axis of each plot represents the false positive rate. Each plot provides a sample of the true positive rate in melanoma ( Figure 5A Drawing 500a), non-small cell lung cancer (NSCLC) ( Figure 5B Drawing 500b), other cancer types ( Figure 5C Drawing 500c) and pancreatic cancer ( Figure 5D In each of the drawings 500d), the comparison of the prediction performance of MSK-IMPACT between ClinTAT and other baselines in terms of receiver operating characteristic (ROC) curves and area under the curve (AUC). The ROC curve is calculated using the response probability calculated by transformer and other baselines. Other baselines include logistic regression (LR), random forest (RF) and XgBoost models. Compared with conventional machine learning models such as LR, RF and XgBoost, ClinTAT achieves superior performance on the test set in predicting responders and non-responders across cancer types, as indicated by the AUC in each of the drawings 500a-d. The results show that the self-attention mechanism for long-range dependency modeling contributes to the overall prediction performance to varying degrees. Figure 7 Table 1 depicts the AUC performance of ClinTaT and other baselines on the prediction of treatment response to MSK_IMPACT, where each column reports the k-sample performance for different values ​​of k. Notably, ClinTaT outperforms other baselines when all training samples are used, however, the advantage is less significant when fewer training samples are used.

[0059] Figures 6A-6C Example plots 600a-c are shown, each depicting the performance of ClinTaT in predicting the clinical outcomes of OS and PFS for test data across multiple cancer types. The y-axis of each plot 600a-c represents the probability of survival, and the x-axis of each plot 600a-c represents the overall survival (months). Plots 600a-c each depict a comparison between true responders (GT-R), predicted responders (Pred-R), true non-responders (GT-NR), and predicted non-responders (Pred-NR). Plots 600a-c reveal the performance of ClinTaT across melanoma ( Figure 6A Drawing 600a), non-small cell lung cancer (NSCLC) ( Figure 6B Drawing 600b) and pancreatic cancer ( Figure 6CFigure 600c shows the difference in overall survival between predicted responders and non-responders for various cancer types. In particular, for predicted non-responders, the predicted survival curve fits the true value curve almost perfectly, indicating that the transformer tends to underestimate the response probability to some extent.

[0060] To test whether the ClinTaT model 200 could also predict overall survival (OS) before administration of immunotherapy, the concordance index (C-index) for OS and PFS was calculated, which ranges between 0 and 1 (0.5 is random performance). Figure 8 Table 2 and Figure 9 Table 3 indicates that the C-index predicted by ClinTaT is significantly higher than those generated by other baselines (Table 2, pancreatic cancer C-index: ClinTaT is 0.724 compared with Xg-Boost is 0.688 compared with random forest is 0.682, p < 0.05; Table 3, pancreatic cancer C-index: ClinTaT is 0.684 compared with XgBoost is 0.671 compared with random forest is 0.666, p < 0.05). These results show that transformers can accurately predict response, OS and PFS before administering immunotherapy.

[0061] Figure 10 Table 4 shows the performance of different BERT LLMs, which are pre-trained on different resource corpora and then consist of a single linear layer as the encoding network 248 ( Figure 4 ) for fine-tuning using only the [cls] token on the MSK_IMPACT test data (averaged over three seeds). PubMedBERT outperforms all other variants and the baseline transformer across all k-sample settings, with an average improvement of over 5%. In the very few-sample setting (4 samples), language model fine-tuning shows significant improvement over the baseline (Table 4, 9.4%), indicating the benefit of LLM in the ability to transfer knowledge to downstream tasks when samples are insufficient. Moreover, the results indicate that the sampling efficiency of using the embedding output from LLM is highly dependent on domain knowledge. For example, SciBERT performs worse than BioBERT and ClinicalBERT because SciBERT is pre-trained on all 1.14M articles of Semantic Scholar for more general scientific knowledge learning.

[0062] In contrast, BioBERT and ClinicalBERT are pre-trained on more domain-specific corpora, such as PubMed, PMC, and clinical MIMIC III notes (available at mimic.mit.edu). Our initial speculation is that domain-specific knowledge transfer can be superior when the pre-training corpus is deep enough. However, the generalization capabilities learned by domain-agnostic models are also effective in scenarios where the source knowledge is neither domain-independent nor strictly domain-specific.

[0063] Although all the results in Table 4 are generated by adding a single linear layer on top of LLM for fine-tuning, Figure 11 Table 5 uses different encoder networks 238 ( Figure 4 ) to evaluate the performance change. The transformer in Table 5 only includes the transformer encoder with a depth of six layers and a dimension of 768. The results indicate that adding computational complexity to the LLM can still improve the semantic representation learning of clinical features, as the transformer architecture performs better than the shallow linear layer.

[0064] A software application (i.e., a software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0065] Non-transitory memory can be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0066] Figure 12 A flowchart of an example arrangement of operations of a method 1200 for predicting clinical outcomes from patient data stored in tabular form is provided. The method 1200 may be based on data stored in memory hardware 1320 ( Figure 13 ) on the data processing hardware 310 to perform an operation. Figure 13 ). The data processing hardware 1310 and the memory hardware 1320 may include the data processing hardware 144 and the memory hardware 146 of the remote system 140. Additionally or alternatively, the data processing hardware 130 and the memory hardware 1320 may reside on the client device 110.

[0067] At operation 1202 , method 1200 includes receiving a clinical data table 201 of a patient. Here, clinical data table 201 stores clinical data associated with the patient in a tabular form. At operation 1204 , method 1200 includes extracting one or more categorical features 202 and one or more continuous features 204 from clinical data table 201 .

[0068] At operation 1206, the method 1200 includes determining, using the clinical prediction model 150, one or more predicted clinical outcomes 182 for the patient based on the one or more categorical features 202 and the one or more continuous features 204 extracted from the clinical data table 201. At operation 1208, the method 1200 includes providing the one or more predicted clinical outcomes 182 for the patient for output from the client device 110 associated with the user 10.

[0069] Figure 13 1 is a schematic diagram of an example computing device 1300 that can be used to implement the systems and methods described in this document. Computing device 1300 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit implementations of the inventions described and / or claimed in this document.

[0070] Computing device 1300 includes a processor 1310, a memory 1320, a storage device 1330, a high-speed interface / controller 1340 connected to memory 1320 and a high-speed expansion port 1350, and a low-speed interface / controller 1360 connected to a low-speed bus 1370 and storage device 1330. Each of components 1310, 1320, 1330, 1340, 1350, and 1360 is interconnected using various buses and can be mounted on a common motherboard or otherwise installed where appropriate. Processor 1310 can process instructions for execution within computing device 1300, including instructions stored in memory 1320 or on storage device 1330, to display graphical information for a graphical user interface (GUI) on an external input / output device (such as a display 1380 coupled to high-speed interface 1340). In other implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple types of memory, where appropriate. Furthermore, multiple computing devices 1300 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0071] Memory 1320 stores information non-transitorily within computing device 1300. Memory 1320 may be a computer-readable medium, one or more volatile memory units, or one or more non-volatile memory units. Non-transitory memory 1320 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 1300. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0072] The storage device 1330 can provide mass storage for the computing device 1300. In some implementations, the storage device 1330 is a computer-readable medium. In various implementations, the storage device 1330 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or a device array (including a device in a storage area network or other configuration). In another implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as a memory 1320, a storage device 1330, or a memory on a processor 1310.

[0073] The high-speed controller 1340 manages bandwidth-intensive operations for the computing device 1300, while the low-speed controller 1360 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 1340 is coupled to the memory 1320, the display 1380 (e.g., via a graphics processor or accelerator), and to the high-speed expansion ports 1350 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 1360 is coupled to the storage device 1330 and the low-speed expansion ports 1390. The low-speed expansion ports 1390, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, via a network adapter.

[0074] As shown in the figures, computing device 1300 can be implemented in many different forms. For example, it can be implemented as a standard server 1300a or multiple implementations in a group of such servers 1300a, as a laptop computer 1300b, or as part of a rack server system 1300c.

[0075] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which can be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0076] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural programming language and / or an object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0077] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), and the one or more programmable processors execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system (for example, FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). By way of example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or random access memory or both. The basic elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices (for example, magnetic disks, magneto-optical disks, or optical disks) for storing data or be operatively coupled to the one or more large-capacity storage devices to receive data from it or to transmit data to it or both. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0078] To provide for interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, voice input, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user (e.g., by sending a web page to a web browser in response to a request received from a web browser on a user's client device).

[0079] Many implementations have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of this disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (1200) executed on data processing hardware (1310), the method causing the data processing hardware (1310) to perform operations comprising: receiving a clinical data table (201) of a patient, wherein the clinical data table (201) stores clinical data associated with the patient in a tabular form; Extracting one or more categorical features (202) and one or more continuous features (204) from the clinical data table (201); determining one or more predicted clinical outcomes (182) for the patient based on the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) using a clinical prediction model (150); and The one or more predicted clinical outcomes (182) for the patient are provided for output from a client device (110) associated with a user.

2. The computer-implemented method (1200) of claim 1, wherein: The clinical prediction model (150) is executed on the data processing hardware (1310) and includes a clinical table multi-head attention model (200) include: A classification feature encoder (210), wherein the classification feature encoder is configured to: receiving as input each classification feature (204) of the one or more classification features (202) extracted from the clinical data table (201); and Generating a corresponding classification embedding (212) for each classification feature (204) as output; A continuous feature encoder (214), the continuous feature encoder being configured to: receiving as input each of the one or more continuous features (204) extracted from the clinical data table (201); and Generating a corresponding continuous feature embedding (216) for each continuous feature (202) as output; a concatenator (220) configured to concatenate the one or more categorical feature embeddings (212) and the one or more continuous feature embeddings (216) to form a set of parameter embeddings (222); A multi-head attention network, which is configured as follows: receiving as input each parameter embedding (222) in the set of parameter embeddings formed by the concatenator (220); and generating as output a corresponding context embedding (350) for each parameter embedding (222) in the set of parameter embeddings (222); as well as A fully connected feedforward network (230), wherein the fully connected feedforward network is configured to: receiving as input the contextual embedding (350) generated as output from the multi-head attention network; and The one or more clinical outcomes (182) for the patient are predicted as output.

3. The computer-implemented method (1200) of claim 2, wherein: The multi-head attention network comprises a stack of N layers (300), each of which comprises a multi-head attention mechanism.

4. The computer-implemented method (1200) of claim 2 or 3, wherein: The multi-head attention network comprises a stack of N Transformer layers (300).

5. The computer-implemented method (1200) of claim 4, wherein: Each Transformer layer in the stack of N Transformer layers (300) includes a normalization layer (304), a masked multi-head attention layer, a residual connection (314) and a feed-forward layer (312).

6. The computer-implemented method (1200) of any one of claims 2-5, wherein: The one or more clinical outcomes (182) predicted for the patient include a plurality of clinical outcomes (182) predicted for the patient; and The fully connected feed-forward network includes a plurality of heads (232), each of the plurality of heads being configured to: receiving as input the contextual embedding (350) generated as output from the multi-head attention network; and A corresponding one of the plurality of clinical outcomes (182) for the patient is predicted as an output.

7. The computer-implemented method (1200) of claim 6, wherein: The clinical form multi-headed attention model (200) is trained via multi-task learning to jointly teach the clinical form multi-headed attention model (200) to learn how to predict the multiple clinical outcomes (182) of the patient.

8. The computer-implemented method (1200) of claim 1, wherein: The clinical prediction model (150) is executed on the data processing hardware (1310) and includes a large language model (400).

9. The computer-implemented method (1200) of claim 8, wherein: The operations further include: Serializing the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) into an input text sequence (402), Wherein, determining the one or more predicted clinical outcomes (182) for the patient includes processing the input text sequence (402) using the large language model (400) to generate the one or more predicted clinical outcomes (182).

10. The computer-implemented method (1200) of claim 8 or 9, wherein: The large language model (400) includes a pre-trained large language model (400) and is fine-tuned using few-shot learning.

11. The computer-implemented method (1200) of any one of claims 8-10, wherein: The large language model (400) includes a domain-specific large language model (400) pre-trained on vocabulary and / or grammar associated with a specific domain.

12. The computer-implemented method (1200) of claim 11, wherein: The specific fields include medical terminology.

13. The computer-implemented method (1200) of any one of claims 1-12, wherein: The one or more predicted clinical outcomes (182) include at least one of overall survival, progression-free survival, or best overall response.

14. The computer-implemented method (1200) of any one of claims 1-13, wherein: The one or more predicted clinical outcomes (182) include at least one of a recommended treatment or a prognostic biomarker score.

15. A system (100) comprising: Data processing hardware (1310); as well as Memory hardware (1320) in communication with the data processing hardware (1310), the memory hardware (1320) storing instructions that, when executed on the data processing hardware (1310), cause the data processing hardware (1310) to perform operations comprising: receiving a clinical data table (201) of a patient, wherein the clinical data table (201) stores clinical data associated with the patient in a tabular form; Extracting one or more categorical features (202) and one or more continuous features (204) from the clinical data table (201); determining one or more predicted clinical outcomes (182) for the patient based on the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) using a clinical prediction model (150); and The one or more predicted clinical outcomes (182) for the patient are provided for output from a client device (110) associated with a user.

16. The system (100) of claim 15, wherein: The clinical prediction model (150) is executed on the data processing hardware (1310) and includes a clinical table multi-head attention model (200) include: A classification feature encoder (210), wherein the classification feature encoder is configured to: receiving as input each of the one or more classification features (202) extracted from the clinical data table (201); and Generating a corresponding classification embedding (212) for each classification feature (204) as output; A continuous feature encoder (214), the continuous feature encoder being configured to: receiving as input each continuous feature (204) of the one or more continuous features (402) extracted from the clinical data table (201); and Generating a corresponding continuous feature embedding (216) for each continuous feature (204) as output; a concatenator (220) configured to concatenate the one or more categorical feature embeddings (212) and the one or more continuous feature embeddings (216) to form a set of parameter embeddings (222); A multi-head attention network, which is configured as follows: receiving as input each parameter embedding (222) in the set of parameter embeddings (222) formed by the concatenator (220); and generating as output a corresponding context embedding (350) for each parameter embedding (222) in the set of parameter embeddings (222); as well as A fully connected feedforward network (230), wherein the fully connected feedforward network is configured to: receiving as input the contextual embedding (350) generated as output from the multi-head attention network; and The one or more clinical outcomes (182) for the patient are predicted as output.

17. The system (100) of claim 16, wherein: The multi-head attention network comprises a stack of N layers, each of which includes a multi-head attention mechanism.

18. The system (100) according to claim 16 or 17, wherein: The multi-head attention network consists of a stack of N Transformer layers.

19. The system (100) of claim 18, wherein: Each Transformer layer in the stack of N Transformer layers includes a normalization layer (304), a masked multi-head attention layer, a residual connection (314), and a feed-forward layer (312).

20. The system (100) of claim 16, wherein: The one or more clinical outcomes (182) predicted for the patient include a plurality of clinical outcomes (182) predicted for the patient; and The fully connected feed-forward network includes a plurality of heads (232), each of the plurality of heads being configured to: receiving as input the contextual embedding (350) generated as output from the multi-head attention network; and A corresponding one of the plurality of clinical outcomes (182) for the patient is predicted as an output.

21. The system (100) of claim 20, wherein: The clinical form multi-head attention model is trained via multi-task learning to jointly teach the clinical form multi-head attention model to learn how to predict the multiple clinical outcomes of the patient (182).

22. The system (100) of claim 15, wherein: The clinical prediction model (150) is executed on the data processing hardware (1310) and includes a large language model (400).

23. The system (100) of claim 22, wherein: The operations further include: Serializing the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) into an input text sequence (402), Wherein, determining the one or more predicted clinical outcomes (182) for the patient includes processing the input text sequence (402) using the large language model to generate the one or more predicted clinical outcomes (182).

24. The system (100) according to claim 22 or 23, wherein: The large language model includes a pre-trained large language model and is fine-tuned using few-shot learning.

25. The system (100) according to any one of claims 22 to 24, wherein: The large language model includes a domain-specific large language model pre-trained on vocabulary and / or grammar associated with a specific domain.

26. The system (100) of claim 25, wherein: The specific fields include medical terminology.

27. The system (100) according to any one of claims 15-26, wherein The one or more predicted clinical outcomes (182) include at least one of overall survival, progression-free survival, or best overall response.

28. The system (100) according to any one of claims 15 to 27, wherein: The one or more predicted clinical outcomes (182) include at least one of a recommended treatment or a prognostic biomarker score.

Images