Predicting clinical outcomes by applying machine learning models to clinical outcomes

Abstract

The method (1200) includes receiving a patient clinical data table (201). The clinical data table stores clinical data associated with the patient in tabular form. The method also includes extracting one or more categorical features (202) and one or more continuous features (204) from the clinical data table and determining, using a clinical prediction model (150), one or more predicted clinical outcomes (182) for the patient based on the one or more categorical features and the one or more continuous features extracted from the clinical data table. The method also includes providing the one or more predicted clinical outcomes (182) for the patient for output from a client device (110) associated with a user.

Description

[Technical Field]

[0001] The present disclosure relates to predicting clinical outcomes by applying machine learning models to clinical outcomes. [Background technology]

[0002] Clinical prediction models play an important role in traditional clinical care by predicting outcomes for patients undergoing treatment to treat various medical conditions. In addition to informing professionals, patients, and families about the risk of treatment outcomes, outcomes predicted by clinical prediction models provide the ability to infer accurate treatment response and disease progression predictions based on the patient's clinical characteristics and molecular profile. As a result, professionals can develop accurate medicines for ultimate use in the medical decision-making process to provide individualized treatment decisions for treating patients' medical conditions, taking into account the patient's clinical characteristics and molecular profile and the risk of outcomes. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method executed on the data processing hardware that causes the data processing hardware to perform operations including receiving a clinical data table of a patient's clinical data associated with the patient in tabular form and extracting one or more categorical features and one or more continuous features from the clinical data table. The operations also include using a clinical prediction model to determine one or more predicted clinical outcomes for the patient based on the one or more categorical features and one or more continuous features extracted from the clinical data table. The operations also include providing the one or more predicted clinical outcomes for the patient for output from a client device associated with a user.

[0004] Embodiments of the present disclosure may include one or more of the following optional features: In some embodiments, the clinical prediction model is executed on data processing hardware and includes a clinical tabular multi-head attention model, where the clinical tabular multi-head attention model includes a categorical feature encoder, a continuous feature encoder, a concatenator, a multi-head attention network, and a fully connected feedforward network. The categorical feature encoder is configured to receive as input each categorical feature of the one or more categorical features extracted from the clinical data table and generate as output a corresponding category embedding for each categorical feature. The continuous feature encoder is configured to receive as input each continuous feature of the one or more continuous features extracted from the clinical data table and generate as output a corresponding continuous feature embedding for each continuous feature. The concatenator is configured to concatenate the one or more categorical feature embeddings and the one or more continuous feature embeddings to form a set of parametric embeddings. The multi-head attention network is configured to receive as input each parametric embedding in the set of parametric embeddings formed by the concatenator and generate as output a corresponding context embedding for each parametric embedding in the set of parametric embeddings. The fully connected feedforward network is configured to receive as input the context embeddings generated as output from the multi-head attention network and predict one or more clinical outcomes for the patient as output. In these embodiments, the multi-head attention network includes a stack of N layers, each including a multi-head attention mechanism that may include a transformer layer. Each transformer layer may include a normalization layer, a masked multi-head attention layer, a residual connection, and a feedforward layer.Further, the one or more clinical outcomes predicted for the patient include a plurality of clinical outcomes predicted for the patient, and the fully connected feedforward network includes a plurality of heads each configured to receive as input the context embeddings generated as output from the multi-head attention network and to predict as output a respective one of the plurality of clinical outcomes for the patient. The clinical tabular multi-head attention model is trained via multi-task learning to jointly train the clinical tabular multi-head attention model to learn how to predict the plurality of clinical outcomes for the patient.

[0005] In some examples, the clinical prediction model is executed on data processing hardware and includes a large-scale language model. In these examples, the operations may further include serializing one or more categorical features and one or more continuous features extracted from the clinical data table into an input text sequence. Here, determining one or more predicted clinical outcomes for the patient includes processing the input text sequence using the large-scale language model to generate one or more predicted clinical outcomes. The large-scale language model may include a pre-trained large-scale language model and is fine-tuned using several rounds of training. Additionally or alternatively, the large-scale language model may include a domain-specific large-scale language model pre-trained with vocabulary and / or syntax associated with a particular domain. For example, the particular domain may include medical terminology.

[0006] The one or more predicted clinical outcomes may include at least one of overall survival, progression-free survival, or best overall response. Additionally or alternatively, the one or more predicted clinical outcomes may include at least one of recommended treatment or prognostic biomarker score.

[0007] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware storing instructions that cause the data processing hardware to perform operations including receiving a clinical data table of a patient's clinical data associated with the patient in tabular form and extracting one or more categorical features and one or more continuous features from the clinical data table. The operations also include using a clinical prediction model to determine one or more predicted clinical outcomes for the patient based on the one or more categorical features and the one or more continuous features extracted from the clinical data table. The operations also include providing the one or more predicted clinical outcomes for the patient for output from a client device associated with a user.

[0008] This aspect may include one or more of the following optional features: In some embodiments, the clinical prediction model is implemented in data processing hardware and includes a clinical tabular multi-head attention model, where the clinical tabular multi-head attention model includes a categorical feature encoder, a continuous feature encoder, a concatenator, a multi-head attention network, and a fully connected feedforward network. The categorical feature encoder is configured to receive as input each categorical feature of the one or more categorical features extracted from the clinical data table and generate as output a corresponding category embedding for each categorical feature. The continuous feature encoder is configured to receive as input each continuous feature of the one or more continuous features extracted from the clinical data table and generate as output a corresponding continuous feature embedding for each continuous feature. The concatenator is configured to concatenate the one or more categorical feature embeddings and the one or more continuous feature embeddings to form a set of parametric embeddings. The multi-head attention network is configured to receive as input each parametric embedding in the set of parametric embeddings formed by the concatenator and generate as output a corresponding context embedding for each parametric embedding in the set of parametric embeddings. The fully connected feedforward network is configured to receive as input the context embeddings generated as output from the multi-head attention network and predict one or more clinical outcomes for the patient as output. In these embodiments, the multi-head attention network includes a stack of N layers, each including a multi-head attention mechanism that may include a transformer layer. Each transformer layer may include a normalization layer, a masked multi-head attention layer, a residual connection, and a feedforward layer.Further, the one or more clinical outcomes predicted for the patient include a plurality of clinical outcomes predicted for the patient, and the fully connected feedforward network includes a plurality of heads each configured to receive as input the context embeddings generated as output from the multi-head attention network and to predict as output a respective one of the plurality of clinical outcomes for the patient. The clinical tabular multi-head attention model is trained via multi-task learning to jointly train the clinical tabular multi-head attention model to learn how to predict the plurality of clinical outcomes for the patient.

[0009] In some examples, the clinical prediction model is executed on data processing hardware and includes a large-scale language model. In these examples, the operations may further include serializing one or more categorical features and one or more continuous features extracted from the clinical data table into an input text sequence. Here, determining one or more predicted clinical outcomes for the patient includes processing the input text sequence using the large-scale language model to generate one or more predicted clinical outcomes. The large-scale language model may include a pre-trained large-scale language model and is fine-tuned using several rounds of training. Additionally or alternatively, the large-scale language model may include a domain-specific large-scale language model pre-trained with vocabulary and / or syntax associated with a particular domain. For example, the particular domain may include medical terminology.

[0010] The one or more predicted clinical outcomes may include at least one of overall survival, progression-free survival, or best overall response. Additionally or alternatively, the one or more predicted clinical outcomes may include at least one of recommended treatment or prognostic biomarker score.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an example system that uses a clinical prediction model to predict one or more clinical outcomes from a patient's clinical data table. [Figure 2A] FIG. 2A is a schematic diagram of an example training process for training a Clinical Tabular Transformer (ClinTaT) model to predict patient clinical outcomes based on tabular clinical data. [Figure 2B] FIG. 2B is a schematic diagram of an example training process for training a Clinical Tabular Transformer (ClinTaT) model to predict patient clinical outcomes based on tabular clinical data. [Figure 3] FIG. 3 is a schematic diagram of an example transformer layer. [Figure 4] FIG. 4 is a schematic diagram of an example training process for fine-tuning a pre-trained large-scale language model to predict patient clinical outcomes. [Figure 5A] FIG. 5A shows an example plot illustrating the performance of the ClinTaT model to predict clinical outcomes across multiple cancer types. [Figure 5B] FIG. 5B shows example plots illustrating the performance of the ClinTaT model in predicting clinical outcomes across multiple cancer types. [Figure 5C] FIG. 5C shows example plots illustrating the performance of the ClinTaT model in predicting clinical outcomes across multiple cancer types. [Figure 5D] FIG. 5D shows example plots illustrating the performance of the ClinTaT model to predict clinical outcomes across multiple cancer types. [Figure 6A] FIG. 6A shows example plots illustrating the performance of the ClinTaT model to predict clinical outcomes of overall survival and progression-free survival, respectively, across multiple cancer types. [Figure 6B] FIG. 6B shows example plots illustrating the performance of the ClinTaT model to predict clinical outcomes of overall survival and progression-free survival, respectively, across multiple cancer types. [Figure 6C]FIG. 6C shows example plots illustrating the performance of the ClinTaT model to predict clinical outcomes of overall survival and progression-free survival, respectively, across multiple cancer types. [Figure 7] FIG. 7 is a table showing a comparison of the area under the curve (AUC) performance in predicting treatment response between the ClinTaT model and other baseline models. [Figure 8] FIG. 8 is a table showing a comparison of the performance of the C-statistic (C-index) for predicting overall survival between the ClinTaT model and other baseline models. [Figure 9] FIG. 9 is a table showing a comparison of the performance of the C-statistics for predicting progression-free survival between the ClinTaT model and other baseline models. [Figure 10] Figure 10 is a table showing the area under the curve (AUC) performance over several training iterations of various large-scale language models pre-trained on different training corpora. [Figure 11] Figure 11 is a table assessing performance changes using different encoder networks stacked on top of various pre-trained large-scale language models to fine-tune treatment response prediction. [Figure 12] FIG. 12 is a flowchart of an example arrangement of operations for a method of predicting one or more clinical outcomes from tabulated patient data. [Figure 13] FIG. 13 is a schematic diagram of an example computing device that may be used to implement the systems and methods described herein.

[0013] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0014] Clinical prediction models play an important role in traditional clinical care by predicting outcomes for patients undergoing treatment to treat various medical conditions. In addition to informing professionals, patients, and families about the risk of treatment outcomes, outcomes predicted by clinical prediction models provide the ability to infer accurate treatment response and disease progression predictions based on the patient's clinical characteristics and molecular profile. As a result, professionals can develop accurate medicines for ultimate use in the medical decision-making process to provide individualized treatment decisions for treating patients' medical conditions, taking into account the patient's clinical characteristics and molecular profile and the risk of outcomes.

[0015] While machine learning has gained popularity for its use in training traditional clinical prediction models, machine learning approaches are typically limited to tree-based ensemble models such as decision trees because the overwhelming majority of clinical data required for training is stored in tabular format (i.e., clinical data is stored in tables). In contrast to tree-based ensemble models, deep neural networks (DNNs) offer many advantages, including the ability to train the resulting models end-to-end, leverage unlabeled / unsupervised training data, be highly robust to both missing values ​​and noisy data features, and generally offer better interoperability. DNNs, with their use of multi-head attention mechanisms (e.g., transformers), are revolutionizing the fields of natural language processing and computer vision. However, these models are not compatible for training and interpreting tabular data.

[0016] Embodiments herein are directed to a Clinical Tabular Transformer (ClinTAT) model for predicting patient clinical outcomes based on tabular clinical data. The ClinTAT model advantageously uses a self-attention mechanism to provide a tabular data modeling architecture. While examples herein illustrate a Transformer as the type of self-attention mechanism used in the ClinTAT model, other types of self-attention mechanisms may be used, such as, but not limited to, a Conformer and a lightweight convolutional neural network. The ClinTAT model is particularly effective at modeling continuous features extracted from patient clinical data tables in addition to categorical features, where the continuous features are included in the self-attention modeling process and are not dominated by the categorical features during training. This aspect is particularly important in a clinical prediction outcome setting, where continuous features represent important information such as the patient's age, the patient's body mass index (BMI), test results / readouts, and other variables representing the patient's medical profile for use in predicting one or more outcomes related to the patient's medical condition and the treatments the patient receives to treat the corresponding patient condition. Embodiments further include applying multi-task learning techniques to train a ClinTaT model with multiple loss objectives to train a ClinTaT model to learn how to predict multiple clinical outcomes for a patient based on the patient's clinical data represented by continuous and categorical features extracted from a clinical data table. Examples of outcomes that a ClinTaT model may be trained to learn via multi-task learning may include overall survival (OS) (in months), progression-free survival (PFS) (in months), and best overall response (BOR). A ClinTaT model may similarly be trained to learn how to predict other outcomes, which may include, but are not limited to, test results, treatment decisions, disease predictions, drug safety decisions / scores, etc.

[0017] Additional embodiments are directed to utilizing large-scale language models (LLMs) to predict patient clinical outcomes based on input text sequences serialized from tabular clinical data. LLMs can advantageously provide deep in-context learning capabilities when available training samples are limited by utilizing knowledge from other resources for downstream tasks with minimal adjustments. In contrast, the robustness / accuracy of ClinTaT models improves when more supervised training samples are available, including patient clinical data tables labeled with clinical outcomes. Thus, while the performance of ClinTaT models can degrade when labeled training samples are scarce due to the low induction bias inherent in long-range dependency modeling, LLMs can achieve in-context learning capabilities through several learning techniques even when only a small number of training samples are available. The use of LLMs can be particularly useful for predicting clinical outcomes in rare disease areas, where past patient records are limited and therefore it is generally inefficient to train ClinTaT models. More specifically, these additional embodiments are directed to leveraging LLMs that have been pre-trained on natural language text in the medical domain and fine-tuned through several training rounds by conditioning the domain-specific pre-trained LLMs with available input text sequences serialized from tabular clinical data to predict specific clinical outcomes. The use of domain-specific LLMs allows smaller LLMs to be utilized for training rounds, thereby reducing the processing / memory requirements and training time of fine-tuning the LLM to predict one or more additional clinical outcomes through several training rounds. Examples of outcomes that an LLM model may be trained to learn through multiple loss objectives may include overall survival (OS) (in months), progression-free survival (PFS) (in months), and best overall response (BOR). LLMs may also be trained to learn how to predict other outcomes, which may include, but are not limited to, test results, treatment decisions, disease predictions, drug safety decisions / scores, etc.

[0018] Referring to FIG. 1 , in some embodiments, a system 100 includes a client device 110 that inputs a clinical data table 201 into a clinical prediction model 150 for predicting one or more clinical outcomes 182 for a patient based on the clinical data table 201. The client device 110 is associated with a user 10, such as a healthcare professional (HCP), that can communicate with a remote system 140 via a network 130. The remote system 140 may be a distributed system (e.g., a cloud environment) having scalable / flexible resources 142. The resources 142 include computational resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some embodiments, the remote system 140 executes a clinical prediction application 160 configured to execute the clinical prediction model 150. Here, the client device 110 may access the application 160 executing on the remote system 140 and input the clinical data table 201 into the clinical prediction model 150 via a graphic user interface (GUI) executing on the client device 110. The client device 110 may additionally or alternatively execute an application 160 to implement the capability to run the clinical prediction model 150 on the client device 110 to predict a clinical outcome 182 .

[0019] The clinical outcomes 182 predicted by the clinical prediction model 150 may be communicated to patients, healthcare providers, and / or relatives of the patient to make more appropriate testing and treatment decisions or to perform risk stratification for therapeutic trials if a particular health condition is diagnosed. For example, a patient associated with clinical data table 201 may have metastatic bladder cancer and may be receiving (or planning to receive) immunotherapy to generate antibodies against programmed cell death-1 / programmed cell death ligand 1 (PD-1 / PD-L1) as a form of treatment. In this example, clinical data table 201 includes various data columns about the patient, such as the type of condition the patient was diagnosed with (e.g., bladder cancer), the class of immunotherapy drug (e.g., PDL1), patient characteristics / demographics, laboratory results, imaging studies, and the patient's medical history. The patient characteristics / demographics may include the patient's age (e.g., 74), gender, race, ethnicity, height, weight, body mass index (BMI), etc. The laboratory results may include columns of specific laboratory tests and values ​​indicative of the results of the laboratory tests. For example, the "Albumin" column in data table 201 indicates that the patient's test result for albumin (i.e., the biomarker albumin value) is equal to "4.1." Clinical data table 201 may include a number of additional test results, such as hemoglobin measured in the tumor and PD-L1 expression. For example, PD-L1 expression may be a basis for whether immunotherapy will be effective, as individuals with tumors that express PD-L1 in one percent (1%) or more of their tumor cells are more likely to have a durable response than individuals with PD-L1 expression below one percent (1%). However, there may be other factors that cause individuals with low PD-L1 expression to still respond significantly better to immunotherapy, and therefore, clinical outcome 182 predicted by clinical prediction model 150 may help identify whether an individual will or will not respond to immunotherapy. The patient's medical history may indicate other details about the patient, such as whether the patient smokes or drinks alcohol, as well as other diseases / conditions for which the patient has been diagnosed and / or is being treated.

[0020] In the illustrated example, clinical prediction model 150 corresponds to a cancer prognosis model that predicts clinical outcomes 182 of overall survival (OS), progression-free survival (PFS), and best overall response (BOR). After clinical prediction model 150 generates / predicts clinical outcomes 182, output module 190 may provide clinical outcomes 182 for output from client device 110. In the illustrated example, client device 110 receives clinical outcomes 182 from clinical prediction model 150, and a GUI executing on client device 110 displays clinical outcomes 182 on screen 114 of client device 110. The GUI may also represent categories and clinical features from clinical data tables 201 associated with the patient for display on screen 114. Output module 190 may also store clinical outcomes 182 along with corresponding clinical data tables 201 in data store 180 and / or transmit clinical outcomes 182 to an institution, organization, or other entity. In particular, as described in more detail with reference to FIGS. 2A and 2B, the clinical prediction model 150 may be trained using multi-task learning techniques to combine customized loss targets associated with each of the different clinical outcomes 182.

[0021] Clinical data table 201 stores prognostic variables related to a patient's clinical features and molecular profile in a tabular format. Values ​​for various columns may be obtained from various sources with patient consent. Data stored in clinical data table 201 may include both categorical features 202 and continuous features 204. In the example shown, categorical features 202 include "bladder" in the cancer type column and PDL1 in the drug class column. Although not explicitly shown in data table 201, categorical features 202 may additionally include values ​​scored / categorized into categories (e.g., high, low, PDL1 positive), such as when a given reading meets a threshold or falls within a range of values. Continuous features 204, on the other hand, generally relate to numerical values ​​such as lab result readings, the patient's age, and / or the patient's BMI / weight. In the example shown, continuous features 204 include "74" in the age column and "4.1" in the albumin column.

[0022] In some examples, user 10 may select / filter the type of clinical data included in clinical data table 201 to be fed into clinical prediction model 150. Similarly, user 10 may retrieve clinical data table 201 for a population of patients sharing a particular characteristic. For example, user 10 may use the GUI to provide input requesting clinical data table 201 for all patients diagnosed with metastatic prostate cancer between the ages of 40 and 50. In this scenario, user 10 may retrieve clinical outcomes 182 for patients in the population of interest that exhibit a particular characteristic (e.g., diagnosed with prostate cancer between the ages of 40 and 50). User 10 may further plug in different values, such as immunotherapeutic drug class, to see how the predicted clinical outcome 182 for a particular patient varies across different immunotherapeutic drug classes.

[0023] In some embodiments, user 10 may coordinate a clinical trial and compile clinical data tables 201 for a first group of patients / participants in an active comparator group receiving conventional / effective treatments used in clinical care, and for a second group of patients / participants who may participate in an experimental group to be treated with a targeted / experimental therapy. This second group is a simulated group (or virtual patient population) for simulating clinical trial designs, significantly reducing patient risk and trial costs. Clinical prediction model 150 can effectively predict clinical outcomes 182 for participants from these perspectives in the second group to compare with clinical outcomes 182 for the first group of patients / participants in the active comparator group. The comparison results may indicate which candidates are suitable for actual participation in the experimental group. For example, it may be revealed that clinical prediction model 150 predicts unfavorable clinical outcomes for women under 50 years old. However, predicted clinical outcomes 182 for men between the ages of 40 and 50 may indicate that the targeted / experimental therapy is more likely to be effective for men between the ages of 40 and 50.

[0024] In some embodiments, the clinical prediction model 150 includes a Clinical Tabular Transformer (ClinTaT) model 200 that uses a self-attention mechanism to provide a tabular data modeling architecture. While the examples herein illustrate a transformer as the type of self-attention mechanism used by the ClinTaT model 200, other types of self-attention mechanisms, such as, but not limited to, conformers and lightweight convolutional neural networks, may also be used. Thus, the ClinTaT model may be referred to as a clinical tabular multi-head attention model. As described in more detail below with reference to FIGS. 2A and 2B , the ClinTaT model is effective at modeling both categorical features 202 and continuous features 204 extracted from a clinical data table 201 for a particular patient 10.

[0025] In another embodiment, the clinical prediction model 150 includes a large-scale language model (LLM) 400. The LLM 400 provides the ability to provide deep in-context learning capabilities (even when the number of training data tables is sparse) by leveraging knowledge from other resources for downstream tasks with minimal adjustments. While the ClinTaT model 200 is suited for tabular data modeling, the LLM 400 is trained to predict clinical outcomes from input text sequences 402 ( FIG. 4 ) serialized from tabular clinical data 201. Thus, the LLM 400 is configured to process input text sequences 402 serialized / converted from tabular clinical data 201 to generate one or more predicted clinical outcomes 182. The LLM 400 may include a domain-specific LLM pre-trained with vocabulary / syntax associated with a domain, such as medical terminology. The use of a domain-specific LLM allows a smaller LLM to be utilized for training several times, whereby an input text sequence 402 serialized from a training data table 400 may be used as a query context to predict clinical outcomes 182. Additional embodiments are directed to leveraging large-scale language models (LLMs) to predict patient clinical outcomes based on input text sequences serialized from tabular clinical data. LLMs can advantageously provide deep in-context learning capabilities when available training samples are limited by leveraging knowledge from other resources for downstream tasks with minimal adaptation.

[0026] The training network 50 is trained with a set of training data tables 40, 40a-n, each associated with a respective training patient and each containing, in tabular form, prognostic variables related to the clinical features and molecular profile of each training patient. When the training network 50 is training the ClinTaT model 200, the extractor 55 extracts categorical features 202 and continuous features 204 from each training data table 40 and provides the extracted categorical and continuous features 202, 204 to the training network 50 for training the ClinTaT model.

[0027] 2A and 2B, the training network 50 uses M categorical features (x) extracted from each training clinical data table 40. cat-_1 ,x cat-_2 ,...,x cat-_m ) 202 and C continuous features (x cont-_1 ,x cont-_2 ,...,x cont-_c ) 204, the ClinTAT model 200 is trained to learn how to predict one or more clinical outcomes 182. Each training clinical data table 40 also includes one or more labels 60, each associated with a corresponding clinical outcome 182 that the model 200 is trained to learn how to predict. These labels include the actual obtained / recorded clinical outcomes of the training patients with which the corresponding table 40 is associated. The training patients are fully anonymized. In the example shown, the model 200 is trained to learn how to predict clinical outcomes 182 including overall survival (OS) (in months), progression-free survival (PFS) (in months), and best overall response (BOR), and the training labels 60 include the corresponding columns in table 40 for OS (in months), PFS (in months), and BOS (in months). As will become apparent, the labels 60 serve as training targets for the loss module 240 to predict the output from the ClinTAT model 200 during training. The ClinTaT model architecture is particularly effective for modeling continuous features extracted from patient clinical data tables in addition to categorical features, where the continuous features are included in the self-attention modeling process and are not dominated by the categorical features during training. This aspect is particularly useful in a clinical prediction outcome setting, where the continuous features represent important information such as the patient's age, the patient's body mass index (BMI), test results / readouts, and other variables representing the patient's medical profile for use in predicting one or more outcomes related to the patient's medical condition and the treatment the patient will receive to treat the corresponding patient condition.

[0028] The ClinTaT model 200 includes a categorical feature encoder 210, a continuous feature encoder 214, a concatenator 220, a multi-head attention network 400 (e.g., a stack of N transformer layers), and a fully connected feedforward network 230. For each training data table 40, the categorical feature encoder 210 generates a corresponding categorical feature x cat_i 202 category embeddings cat_e φ (x cat_i ) 212 as output. In some examples, the categorical feature encoder 210 includes a lookup table of embeddings of the various possible categorical features 202, represented by the columns of the training data table 40. Specifically, for each categorical feature (column) i, ​​the encoder 210 generates an embedding lookup table eφ for i∈{1, 2, ..., m}. i (.) may be included. i For the i-th categorical feature with class, the embedding table eφ i (.) indicates that additional filling corresponds to missing values ​​(d i +1) the encoded value x i =j∈[0,1,2,...,d i ] is embedded in eφ i (j)=[cφ i ,wφ ij ] and cφ i ∈Rl and wφ ij ∈R d-1 A unique identifier cφ is a column specific identifier. i ∈Rl distinguishes the classes in column i from the classes in other columns. i The dimension of and l are hyperparameters. Therefore, the categorical feature encoder 212 generates categorical embeddings E(x cat )={cat_e φ (x cat_1 ),....,cat_e φ (x cat_m )}.

[0029] For each training data table 40, the continuous feature encoder 214 generates a corresponding continuous feature x cont_i 204 continuous embedding cont_e φ (x cont_i ) 216 as an output. The categorical feature encoder 210 includes a dictionary-style lookup table of embeddings, where various numbers of tokens representing categorical features correspond to embeddings, and the continuous feature encoder 214 applies a linear neural network layer to multiply numbers representing corresponding categorical features 204 (e.g., “74” in the age column, “4.1” in the albumin test result column) to obtain each sequence of embeddings corresponding to the categorical features 204. By doing this, the continuous embeddings 216 can be included in the self-attention modeling process, and the corresponding continuous features 204 are not dominated by the categorical features 202 during training. In particular, the continuous embeddings 216 generated for each corresponding continuous feature 204 include a column-specific unique identifier cφ that distinguishes the value of the continuous feature 204 represented in column i from values ​​represented in other columns in the training data table 40. i ∈Rl. Thus, the continuous feature encoder 214 generates continuous embeddings E(x cont )={cont_e φ (x cont_1 ),....,cont_e φ (x cont_c )}.

[0030] The coupler 220 is a parametric embedding E φ To form the set of 222, we use the categorical embedding E(x cat ) and the set of continuous embeddings E(x cont ) set, where E φ ={cat_e φ (x cat_1 ),....,cat_e φ (x cat_m )}+{cont_e φ (x cont_1 ),....,cont_e φ (x cont_c)}. The multi-head attention network 300 calculates the corresponding category feature x cat_i The category embeddings cat_e output from the category feature encoder 210 φ (x cat_i )212, or the corresponding categorical feature x cont_i The continuous embedding cont_e output from the continuous feature encoder 214 of 204 φ (x cont_i )216, including any of the parametric embedding E φ 222. The multi-head attention network 300 receives the corresponding parametric embedding E φ Each category embedding cat_e is computed through successive aggregation of contexts from other embeddings 212, 216 in the set φ (x cat_i )212 and each continuous embedding cont_e φ (x cont_i ) 216 is configured to generate / transform the corresponding context embedding 350. The multi-head attention network 300, represented by a stack of N layers (e.g., a stack of N transformer layers), generates / transforms the parametric embedding E φ Operates on the set {h1,...,h m ,h m+1 ,...,h c}, and returns the corresponding context embedding 350 as θ where i∈{1,...,m,m+1,...c}, and h i ∈R d In particular, m+1 is the first continuous feature (x cont_1 ) 204. As shown in FIG. 2A, the context embedding 350 {h1,...,h m ,h m+1 ,...,h c} forms a context vector of dimension (dx(m+c)) that is provided as input to a fully connected feedforward network 230, which may include a multi-layer perceptron (MLP).

[0031] 3 shows an example of a transformer layer 300 among the multiple transformer layers when the multi-head attention network 300 includes multiple transformer layers. As mentioned above, the multi-head attention network 300 of FIGS. 2A and 2B is not limited to transformer layers, but may instead include conformer layers, lightweight convolutional layers, or other networks that use multi-head attention mechanisms. In FIG. 3, an initial transformer layer 300 is configured to generate a corresponding categorical feature x cat_i The category embeddings cat_e output from the category feature encoder 210 φ (x cat_i )212, or the corresponding categorical feature x cont_i The continuous embedding cont_e output from the continuous feature encoder 214 of 204 φ (x cont_i )216, including any of the parametric embedding E φ 222 and generates a corresponding output representation / embedding 350 received as input by the next transformer layer 300. This means that each transformer layer 300 after an initial transformer layer 300 may receive an input embedding 350 that corresponds to the output representation / embedding produced as output by the immediately preceding transformer layer 300. The final transformer layer 300 (e.g., the last transformer layer in a stack of transformer layers 300) may then generate a parametric embedding E φ Each category embedding cat_e is computed through successive aggregation of contexts from other embeddings 212, 216 in the set φ (x cat_i )212 and each continuous embedding cont_e φ (x cont_i ) 216 to generate / transform the corresponding context embedding 350, where E φ ={cat_e φ (x cat_1 ),....,cat_e φ (x cat_m)},{cont_e φ (x cont_1 ),....,cont_e φ (x cont_c )}.

[0032] Each transformer layer 300 of the multi-head attention network includes a normalization layer 304, a masked multi-head attention layer 306, a residual connection 308, and a feedforward layer 312. The masked multi-head attention layer 306 provides a flexible way to control the amount of context used by the model 200. Specifically, after the normalization layer 304 normalizes the input parametric embedding 222, the masked multi-head attention layer 306 projects the input to the values ​​of all heads. The masked multi-head layer 306 then normalizes the parametric embedding E φ The attention scores may be masked to the current parametric embedding 222 to generate an output conditioned on the set of . The weight-averaged values ​​of all heads are then concatenated and passed to fully connected layer 2 316, where the residual connections 314 are added to the normalized input and the output of the fully connected layer 316 to form the final output of the multi-head attention layer 306. The residual connections 308 are added to the output of the normalization layer 304 by adder 330 and provided as input to a respective one of the masked multi-head attention layer 306 or feedforward layer 312.

[0033] The feedforward layer 312 applies a normalization layer 304, followed by a fully connected layer 1 320, a rectified linear layer (ReLu) 318, and a fully connected layer 2 316. The ReLu 318 is used as the activation for the fully connected layer 1 320. Similar to the multi-head attention layer 406, the residual connection 314 of the output from the normalization layer 404 may be added to the output of the fully connected layer 2 316 by an adder 330.

[0034] 2A , based on the set of context embeddings 350 for each corresponding training data table 40, the fully connected feedforward network 230 generates one or more predicted clinical outcomes 182 as output, and the loss module 240 generates training losses 290 based on the predicted clinical outcomes 182 and the corresponding training labels 60 for the training data tables. Here, the training labels 60 may include corresponding columns in the tables 40 for OS (in months), PFS (in months), and BOS (in months), and may serve as training targets for the training network 50 to train the ClinTaT model 200 to predict the clinical outcomes 182 from the tabulated data 202, 204. Thus, the training network 50 trains the ClinTaT model 200 via supervised learning by updating the parameters of the ClinTaT model 200 based on the training losses 290 obtained for each training data table 40. For example, the parameters / weights of the categorical feature encoder 210, the continuous feature encoder 214, the multi-head attention network 300, and the fully connected feedforward network 230.

[0035] 2B illustrates how the training network 50 applies multi-task learning techniques to train the ClinTaT model 200 with multiple loss objectives to train the ClinTaT model 200 to learn how to predict multiple clinical outcomes 182, 182ac. Continuing the example, the multiple clinical outcomes 182 that the ClinTaT model may be trained to learn via multi-task learning may include overall survival (OS) (in months), progression-free survival (PFS) (in months), and best overall response (BOR). The ClinTaT model may similarly be trained to learn how to predict other outcomes, which may include, but are not limited to, test results, treatment decisions, etc.

[0036] In the example shown, the fully connected feedforward network 230 computes the context embeddings 350 {h1,...,h2,...} generated by the multi-head attention network 300 of the corresponding categorical and continuous features 202, 204 of each training data table 40. m ,h m+1 ,...,h c}, and outputs / generates a respective one of a plurality of clinical outcomes 182. For example, the OS head 232a is configured to output predicted overall survival 182a as a value representing the number of months, the PFS head 232b is configured to output predicted progression-free survival as a value representing the number of months, and the BOR head 232c is configured to output a predicted best overall response value. Each head 232a-c may include a single linear projection layer corresponding to the respective clinical outcome 182 that the head predicts.

[0037] The loss module 240 includes multiple sub-loss modules 242 a-c, each associated with a respective one of the heads 232 a-c. Each sub-loss module 242 is configured to determine a respective sub-loss 244, 244 a-c based on the corresponding clinical outcome 182 predicted by the respective head 232 a-c and the training labels 60 a-c associated with the corresponding clinical outcome 182. For example, the sub-loss module 242 a may correspond to an OS loss module that determines an OS loss 244 a based on the predicted overall survival 182 a output from the OS head 232 a and the training labels 60 a associated with the ground truth values ​​that the OS head 232 a is learning to predict. In some examples, the OS loss 244 a includes a Cox proportional hazards (CPH) loss function. The sub-loss module 242b may correspond to an OS loss module that determines a PFS loss 244b based on the predicted PFS 182b output from the PFS head 232b and training labels 60b related to the ground truth values ​​that the PFS head 232b is learning to predict. In some examples, the PFS loss 244b includes a Cox proportional hazards (CPH) loss function. The sub-loss module 242c may correspond to a BOR loss module that determines a BOR loss 244c based on the predicted BOR 182c output from the BOR head 232c and training labels 60c related to the ground truth values ​​that the BOR head 232c is learning to predict. In some examples, the BOR loss 244c includes a cross-entropy loss function. Thus, the sub-losses 244 determined by the sub-loss module 242 provide a multi-loss objective to enable the ClinTaT model 200 to predict multiple endpoints (i.e., clinical outcomes 182), and subsequently introduces an inductive bias such that the model 200 favors some predictions over others, leading to better generalization.

[0038] Continuing with reference to FIG. 2B, the loss module 240 determines the unified loss target L fThe network 230 includes a combined loss module 248 to provide a collaborative learning paradigm by summing the different sub-losses 244, each associated with a respective one of the different clinical outcomes 182 predicted by the fully connected feedforward network 230, into a total loss 290 represented by L. f is expressed as follows: where I denotes the total number of tasks, and αi denotes the soft weight of any task i.

number

[0039] After the ClinTaT model 200 is trained, the trained ClinTaT model 200 may be used as a clinical prediction model 150 to predict one or more clinical outcomes 182 from a corresponding patient's clinical data table 201. Referring back to FIG. 1 with reference to FIGS. 2A and 2B, the extractor 55 may extract all categorical features 202 and continuous features 204 from the clinical data table 201. Then, the categorical feature encoder 210 may generate category embeddings 212 from the categorical features 202, while the continuous feature encoder 214 may generate continuous embeddings 216 from the continuous features 204. The concatenator 220 combines the parametric embeddings E that are input to the multi-head attention network 300. φ The category and continuous embeddings 212, 216 may be concatenated to form a corresponding set of 222. The multi-head attention network 300 uses the parametric embedding E φ To operate on a set of functions f θmay be applied and return a corresponding context embedding 350 that is input to the fully connected feedforward network 230. Based on the context embedding 350, the feedforward network 230 may predict one or more clinical outcomes 182 as output. Continuing the example, the feedforward network 230 may include an OS head 232a that outputs a predicted OS 182a based on the clinical data table 201, a PFS head 232b that outputs a predicted PFS 182b based on the clinical data table 201, and a BOR head 232c that outputs a predicted BOR 182c. As shown in FIG. 1 , the output module 190 may display the predicted OS 182a as “overall survival −9.1 months,” the predicted PFS 182b as “progression-free survival −3.1 months,” and the predicted BOR 182c as “best overall response −0” for display on the screen 114 of the client device 110.

[0040] 4 shows a training network 50 that trains an LLM 400 with a training data table 40 to train the LLM to learn how to predict one or more clinical outcomes 182. As described above, an LLM can provide deep in-context learning capabilities when available training examples are limited by utilizing knowledge from other resources for downstream tasks with minimal adaptation, so the training network 50 may train an LLM 400 for use as a clinical prediction model 150 when available training data (e.g., a training data table) is sparse or insufficient to train a ClinTaT model 200.

[0041] The training network 50 applies serialization 410 to serialize the tabular clinical data stored in each training data table 40 into a corresponding input text sequence 402. Here, the input text sequence 402, serialized from features in the columns of each training data table 40, includes a sequence of natural language tokens (e.g., words / word fragments) that the LLM can understand and encode. In some examples, serialization 410 applies a manual serialization template to the features of each column. For example, the manual serialization template may include "{attribute} is {value}." In the illustrated example, for a first training data table 40a having values ​​in columns representing attributes of cancer type, drug class, age, and albumin, the manual serialization template applied by serialization 410 generates the input text sequence 402: "Patient diagnosed with bladder cancer. Age is 74. Albumin is 4.1. Drug class is PD1 / PDL1." Other attributes, such as prognostic or predictive biomarkers, may also be used in the methods and systems described herein.

[0042] The serialized input text sequences 402 from the respective training data tables 40 are provided as inputs to the LLM 400 for predicting one or more clinical outcomes 182. Embodiments herein are directed to an LLM 400 including a pre-trained LLM and training network 50 that performs several rounds of training using the serialized input text sequences 402 from the respective training data tables 40 as a context for predicting one or more clinical outcomes 182. The pre-trained LLM 400 may include a domain-specific LLM pre-trained with vocabulary / syntax associated with a domain, such as medical terminology. The use of a domain-specific LLM allows a smaller LLM to be utilized for several rounds of training to fine-tune the LLM for predicting clinical outcomes, whereby the serialized input text sequences 402 from the training data tables 40 may be used as a query context for predicting clinical outcomes 182. In some examples, the training labels 60 are serialized into corresponding natural language queries that are provided to the LLM 400 to predict clinical outcomes, such that the serialized input text sequence 402 from the training data table 40 can be used as the context for the natural language query to predict clinical outcomes. Additionally, using several rounds of learning to fine-tune the LLM allows for smaller LLMs that require reduced processing / memory requirements and improved latency to make robust and accurate clinical predictions compared to domain-independent LLMs that contain billions or more parameters.

[0043] The pre-trained LLM may include a Bidirectional Encoder Representation from Transformers (BERT) model, or a domain-specific LLM pre-trained on clinical and / or biomedical corpora. Examples of domain-specific LLMs may include, but are not limited to, BioBERT, ClinicalBERT, SciBERT, and PubMedBERT.

[0044] Continuing with reference to FIG. 4 , the training network 50 may apply an encoder network 248 stacked on the LLM 400 to fine-tune the LLM 400 with multiple loss objectives to train the LLM 400 to predict clinical outcomes 182, 182ac. In some examples, the encoder network 248 includes a single linear layer or a stack of multi-head attention layers. For example, the multi-head attention layer may include a transformer layer or a conformer layer. In some examples, the LLM 400 receives one or more clinical outcomes for the LLM 400 to predict as a request query. The encoder network 248 may receive the embeddings output from the LLM 400 and predict one or more clinical outcomes 182 as outputs, and the loss module 240 generates a training loss 290 based on the predicted clinical outcomes 182 and corresponding training labels 60 in the training data table. Here, the training labels 60 may include corresponding columns in the OS (in months), PFS (in months), and BOS (in months) tables 40 and may serve as training targets for the training network 50 for pre-tuning the LLM 400 and the encoder network 248 for predicting clinical outcomes 182 from input text sequences 402. During fine-tuning, the parameters of the pre-trained LLM 400 may remain fixed / frozen, while the parameters of the encoder network 248 may be adjusted / updated based on the training loss 290.

[0045] Figures 5A-6C show example plots illustrating the performance of the ClinTAT model 200 in predicting clinical outcomes. In these examples, the ClinTAT model 200 was trained on training data samples acquired by Memorial Sloan Kettering Cancer Center (MSKCC) from a comprehensively curated cohort (MSK-IMPACT) of 1,479 patients treated with immune checkpoint blockade (ICB) across 16 different cancer types. Patients were either responders (R) or non-responders (NR) to treatment (PD-1 / PD-L1 inhibitors, CTLA-4 inhibitors, or their combination) based on Response Evaluation Criteria for Solid Tumors (RECIST) v1.1 or best overall response by imaging. Sixteen biological features were collected for each patient, including genomic, molecular, clinical, and demographic variables (i.e., represented as corresponding categorical and continuous features 202, 204). The training set contains training data tables 40 of 1184 patients, and the test set contains clinical data tables of 295 patients. The evaluation goal is to predict clinical outcomes for immunotherapy (binary classification) and both overall survival and progression-free survival (regression) in test data across different cancer types.

[0046] Figures 5A-5D show example plots 500a-d, respectively, illustrating the performance of the ClinTAT model 200 in predicting clinical outcomes across multiple cancer types. The y-axis of each plot 500a-d represents the true positive rate, and the x-axis of each plot represents the false positive rate. Each plot provides a comparison of predictive performance with MSK-IMPACT in terms of receiver operating characteristic (ROC) curves and area under the curve (AUC) between ClinTAT and other baselines for melanoma (plot 500a in Figure 5A), non-small cell lung cancer (NSCLC) (plot 500b in Figure 5B), other cancer types (plot 500c in Figure 5C), and pancreatic cancer (plot 500d in Figure 5D). The ROC curves were calculated using response probabilities calculated by Transformer and other baselines. The other baselines include logistic regression (LR), random forest (RF), and XgBoost models. As shown by the AUC in each of plots 500a-d, ClinTaT achieved superior performance on the test set in predicting responders and non-responders across cancer types compared to traditional machine learning models such as LR, RF, and XgBoost. This result suggests that the self-attention mechanism of long-term dependency modeling contributed to overall predictive performance to varying degrees. Table 1 in Figure 7 shows the AUC performance of ClinTaT and other baselines in treatment response prediction on MSK-IMPACT, with each column representing the k-shot performance for various values ​​of k. Notably, ClinTaT outperforms the other baselines when all training samples are used, while not significantly degrading when fewer training samples are used.

[0047] Figures 6A-6C show example plots 600a-c, respectively, illustrating the performance of ClinTaT in predicting clinical outcomes for OS and PFS across multiple cancer types in the study data. The y-axis of each plot 600a-c shows the probability of survival, and the x-axis of each plot 600a-c shows overall survival (months). Plots 600a-c plot comparisons between ground truth responders (GT-R), predicted responders (Pred-R), ground truth non-responders (GT-NR), and predicted non-responders (Pred-NR), respectively. Plots 600a-c reveal that the Transformer predicted the difference in overall survival between responders and non-responders across various cancer types: melanoma (plot 600a in Figure 6A), non-small cell lung cancer (NSCLC) (plot 600b in Figure 6B), and pancreatic cancer (plot 600c in Figure 6C). In particular for predicted non-responders, we show that the predicted survival curves nearly perfectly fit the ground truth ones, indicating that the transformer tends to underestimate response probabilities to some extent.

[0048] To test whether the ClinTaT model 200 could also predict overall survival (OS) before administering immunotherapy, concordance indices (C-statistics) were calculated for OS and PFS, ranging between 0 and 1 (0.5 being random performance). Table 2 in Figure 8 and Table 3 in Figure 9 show that the C-statistics for ClinTaT predictions were significantly larger than those generated by the other baselines. (In Table 2, the C-statistics for pancreatic cancer were 0.724 for ClinTaT, 0.688 for XgBoost, and 0.682 for Random Forest, p<0.05; in Table 3, the C-statistics for pancreatic cancer were 0.684 for ClinTaT, 0.671 for XgBoost, and 0.666 for Random Forest, p<0.05.) These results demonstrate that the Transformer can accurately predict response, i.e., OS and PFS, before administering immunotherapy.

[0049] Table 4 in Figure 10 shows the performance of various BERT LLMs pre-trained on different resource corpora following a single linear layer as the encoder network 248 (Figure 4) for fine-tuning using only the [cls] token on the MSK_IMPACT test data (averaged over three seeds). PubMedBERT outperforms all other variants and the baseline transformer by an average improvement of more than 5% across all k-shot settings. In the very few-shot setting (4 samples), fine-tuning the language model shows a significant improvement over the baseline (Table 4, 9.4%), demonstrating the benefit of LLMs' ability to transfer knowledge to downstream tasks when samples are scarce. The results also show that the sample efficiency of using embeddings output from LLMs strongly depends on domain knowledge. For example, SciBERT's performance is worse than that of BioBERT when SciBERT is pre-trained on all 1.14 million articles from SemanticScala for more general scientific knowledge learning.

[0050] In contrast, BioBERT and ClinicalBERT were pre-trained on more domain-specific corpora, such as PubMed, PMC, and clinical MIMIC III notes (available at mimic.mit.edu). Our preliminary speculation is that domain-specific knowledge transfer may be superior when the pre-training corpus is deep enough. However, the generalization ability learned by domain-independent models also works under scenarios where the resource knowledge is neither domain-independent nor morally domain-specific.

[0051] While all of the results in Table 4 are generated by adding a single linear layer on top of the LLM for fine-tuning, Table 5 in Figure 11 evaluates the change in performance using a different encoder network 238 (Figure 4). The transformer in Figure 5 contains only a 6-layer deep transform encoder with a dimension of 768. This result indicates that adding computational complexity to the LLM can improve semantic representation learning of clinical features, as the transformer architecture performs better than shallow linear layers.

[0052] Software architecture (i.e., software resources) may refer to computer software that causes a computing device to perform tasks. In some examples, software applications may be referred to as "applications," "apps," or "programs." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0053] Non-transitory memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently based on use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), disk, or tape.

[0054] 12 provides a flowchart of an example arrangement of operations for a method 1200 for predicting clinical outcomes from patient data stored in tabular format. The method 1200 may be executed by data processing hardware 1310 based on instructions stored in memory hardware 1320 (FIG. 13) that cause the data processing hardware 1310 to perform operations. The data processing hardware 1310 and memory hardware 1320 may include data processing hardware 144 and memory hardware 146 of the remote system 140. Additionally or alternatively, the data processing hardware 1310 and memory hardware 1320 may be included in the client device 110.

[0055] At act 1202, the method 1200 includes receiving a patient clinical data table 201, where the clinical data table 201 stores clinical data associated with the patient in tabular form, and at act 1204, the method 1200 includes extracting one or more categorical features 202 and one or more continuous features 204 from the clinical data table 201.

[0056] At act 1206, the method 1200 includes using the clinical prediction model 150 to determine one or more predicted clinical outcomes 182 for the patient based on the one or more categorical features 202 and the one or more continuous features 204 extracted from the clinical data table 201. At act 1208, the method 1200 includes providing the one or more predicted clinical outcomes 182 for the patient for output from the client device 110 associated with the user 10.

[0057] 13 is a schematic diagram of an example computing device that may be used to implement the systems and methods described herein. Computing device 1300 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, connections and relationships shown herein, and their functionality, are for illustrative purposes only and are not intended to limit the practice of the invention(s) described and / or claimed herein.

[0058] Computing device 1300 includes processor 1310, memory 1320, storage device 1330, high-speed interface / controller 1340 connecting to memory 1320 and high-speed expansion port 1350, and low-speed interface / controller 1360 connecting to low-speed bus 1370 and storage device 1330. Each of components 1310, 1320, 1330, 1340, 1350, and 1360 may be interconnected using various buses, implemented on a common motherboard, or in any other suitable manner. Processor 1310 can process instructions for execution within computing device 1300, including instructions stored in memory 1320 or storage device 1330 for displaying graphical information for a graphic user interface (GUI) on an external input / output device, such as display 1380, coupled to high-speed interface 1349. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as needed. Additionally, multiple computing devices 1300 may be connected, with each device providing a portion of the required operations (eg, a bank of servers, a group of blade servers, or a multi-processor system, etc.).

[0059] Memory 1320 stores information non-transiently within computing device 1300. Memory 1320 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transient memory 1320 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by the computing device. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), disk, or tape.

[0060] Storage device 1330 can provide mass storage for computing device 1300. In some embodiments, storage device 1330 is a computer-readable medium. In various embodiments, storage device 1330 may be a floppy disk device, a hard disk device, an optical disk device, a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 1320, storage device 1330, or memory of processor 1310.

[0061] High-speed controller 1340 manages bandwidth-intensive operations of computing device 1300, while low-speed controller 1360 manages less bandwidth-intensive operations. This division of roles is merely exemplary. In some embodiments, high-speed controller 1340 is coupled to memory 1320, display 1380 (e.g., via a graphics processor or accelerator), and high-speed expansion port 1350, which may accept various expansion cards (not shown). In some embodiments, low-speed controller 1360 is coupled to storage device 1330 and low-speed expansion port 1390. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router via a network adapter.

[0062] The computing device 1300 may be implemented in a number of different forms, as shown in the figure, such as a standard server 1300a or a cluster of such servers 1300a multiple times, a laptop computer 1300b, or as part of a rack server system 1300c.

[0063] Various embodiments of the methods and techniques described herein may be implemented in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be specialized or general-purpose, and may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor that may be coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0064] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0065] The processes and logic flows described in this detailed description may be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be executed by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from, transmit data to, or both. A computer need not, however, have such devices. Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile media and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0066] To provide for user interaction, one or more aspects of the present disclosure may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to a user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may be used to provide for user interaction. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0067] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (1200) executed on data processing hardware (1310), the data processing hardware (1310) comprising: receiving a patient clinical data table (201), the clinical data table (201) storing clinical data associated with the patient in a tabular format; extracting one or more categorical features (202) and one or more continuous features (204) from the clinical data table (201); determining one or more predicted clinical outcomes (182) for the patient based on the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) using a clinical prediction model (150); providing the one or more predicted clinical outcomes (182) for the patient for output from a client device (110) associated with a user; A method (1200) for causing an action to be performed, the action including:

2. the clinical prediction model (150) is executed on the data processing hardware (1310) and includes a clinical tabular multi-head attention model (200); The clinical tabular multi-head attention model (200) comprises: receiving as input a categorical feature (204) for each of the one or more categorical features (202) extracted from the clinical data table (201); to generate as output the corresponding category embedding (212) for each category feature (204), a categorical feature encoder (210) configured; receiving as input each continuous feature of the one or more continuous features (204) extracted from the clinical data table (201); so as to generate as output a corresponding continuous feature embedding (216) for each continuous feature (202), a continuous feature encoder (214) configured as: a concatenator (220) configured to concatenate the one or more categorical feature embeddings (212) and the one or more continuous feature embeddings (216) to form a set of parametric embeddings (222); receiving as input each parametric embedding (222) in the set of parametric embeddings formed by the concatenator (220); so as to generate as output a corresponding context embedding (350) for each parametric embedding (222) in the set of parametric embeddings (222); a multi-head attention network consisting of receiving as input the context embeddings (350) generated as output from the multi-head attention network; predicting, as an output, the one or more clinical outcomes (182) for the patient; a fully connected feedforward network consisting of Including, 12. The method (1200) of claim 1.

3. The multi-head attention network includes a stack of N layers (300), each of which comprises a multi-head attention mechanism. The method (1200) of claim 2.

4. The multi-head attention network includes a stack of N transformer layers (300). The method (1200) of claim 2 or 3.

5. Each transformer layer in the stack of N transformer layers (300) includes a normalization layer (304), a masked multi-head attention layer, a residual connection (314), and a feedforward layer (312). The method (1200) of claim 4.

6. the one or more predicted clinical outcomes (182) for the patient include a plurality of predicted clinical outcomes (182) for the patient; The fully connected feedforward network is receiving as input the context embeddings (350) generated as output from the multi-head attention network; predicting, as an output, each one of the plurality of clinical outcomes (182) for the patient; a plurality of heads (232) each configured as The method (1200) of any one of claims 2 to 5.

7. the clinical tabular multi-head attention model (200) is trained via multi-task learning to jointly train the clinical tabular multi-head attention model (200) to learn how to predict the multiple clinical outcomes (182) of the patient; The method (1200) of claim 6.

8. the clinical prediction model (150) executes on the data processing hardware (1310) and includes a large-scale language model (400); 12. The method (1200) of claim 1.

9. The operation further comprises: serializing the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) into an input text sequence (402); determining the one or more predicted clinical outcomes (182) for the patient includes processing the input text sequence (402) using the large-scale language model (400) to generate the one or more predicted clinical outcomes (182); The method (1200) of claim 8.

10. The large-scale language model (400) includes a pre-trained large-scale language model (400) that is fine-tuned using several rounds of learning.

10. The method (1200) of claim 8 or 9.

11. The large-scale language model (400) includes a domain-specific large-scale language model (400) pre-trained with vocabulary and / or syntax associated with a particular domain; The method (1200) of any one of claims 8 to 10.

12. The specific domain includes medical terms. The method (1200) of claim 11.

13. the one or more predicted clinical outcomes (182) include at least one of overall survival, progression-free survival, or best overall response; The method (1200) of any of claims 1 to 12.

14. the one or more predicted clinical outcomes (182) include at least one of a recommended treatment or a prognostic biomarker score; The method (1200) of any of claims 1 to 13.

15. Data processing hardware (1310); memory hardware (1320) in communication with the data processing hardware (1310); Including, The memory hardware (1320), when executed on the data processing hardware (1310), causes the data processing hardware (1310) to: receiving a patient clinical data table (201), the clinical data table (201) storing clinical data associated with the patient in a tabular format; extracting one or more categorical features (202) and one or more continuous features (204) from the clinical data table (201); determining one or more predicted clinical outcomes (182) for the patient based on the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) using a clinical prediction model (150); providing the one or more predicted clinical outcomes (182) for the patient for output from a client device (110) associated with a user; storing instructions for performing operations including System (100).

16. the clinical prediction model (150) is executed on the data processing hardware (1310) and includes a clinical tabular multi-head attention model (200); The clinical tabular multi-head attention model (200) receiving as input a categorical feature (204) for each of the one or more categorical features (202) extracted from the clinical data table (201); to generate as output the corresponding category embedding (212) for each category feature (204), a categorical feature encoder (210) configured; receiving as input each continuous feature of the one or more continuous features (204) extracted from the clinical data table (201); so as to generate as output a corresponding continuous feature embedding (216) for each continuous feature (202), a continuous feature encoder (214) configured as: a concatenator (220) configured to concatenate the one or more categorical feature embeddings (212) and the one or more continuous feature embeddings (216) to form a set of parametric embeddings (222); receiving as input each parametric embedding (222) in the set of parametric embeddings formed by the concatenator (220); so as to generate as output a corresponding context embedding (350) for each parametric embedding (222) in the set of parametric embeddings (222); a multi-head attention network consisting of receiving as input the context embeddings (350) generated as output from the multi-head attention network; predicting, as an output, the one or more clinical outcomes (182) for the patient; a fully connected feedforward network consisting of Including, The system (100) of claim 15.

17. The multi-head attention network includes a stack of N layers, each of which comprises a multi-head attention mechanism. The system (100) of claim 16.

18. The multi-head attention network includes a stack of N transformer layers.

18. The system (100) of claim 16 or 17.

19. Each transformer layer in the stack of N transformer layers includes a normalization layer (304), a masked multi-head attention layer, a residual connection (314), and a feedforward layer (312).

20. The system (100) of claim 18.

20. the one or more predicted clinical outcomes (182) for the patient include a plurality of predicted clinical outcomes (182) for the patient; The fully connected feedforward network is receiving as input the context embeddings (350) generated as output from the multi-head attention network; predicting, as an output, each one of the plurality of clinical outcomes (182) for the patient; a plurality of heads (232) each configured as The system (100) of claim 16.

21. the clinical tabular multi-head attention model is trained via multi-task learning to jointly train the clinical tabular multi-head attention model to learn how to predict the plurality of clinical outcomes (182) for the patient. The system (100) of claim 20.

22. the clinical prediction model (150) executes on the data processing hardware (1310) and includes a large-scale language model (400); The system (100) of claim 15.

23. The operation further comprises: serializing the one or more categorical features (202) and the one or more continuous features (204) extracted from the clinical data table (201) into an input text sequence (402); determining the one or more predicted clinical outcomes (182) for the patient includes processing the input text sequence (402) using the large-scale language model to generate the one or more predicted clinical outcomes (182); 23. The system (100) of claim 22.

24. The large-scale language model includes a pre-trained large-scale language model (400) that is fine-tuned using several rounds of learning.

24. The system (100) of claim 22 or 23.

25. the large-scale language model comprises a domain-specific large-scale language model pre-trained with vocabulary and / or syntax associated with a particular domain; The system (100) of any of claims 22 to 24.

26. The specific domain includes medical terms.

26. The system (100) of claim 25.

27. the one or more predicted clinical outcomes (182) include at least one of overall survival, progression-free survival, or best overall response; The system (100) of any of claims 15 to 26.

28. the one or more predicted clinical outcomes (182) include at least one of a recommended treatment or a prognostic biomarker score; The system (100) of any of claims 15 to 27.

Images