Healthcare modeling

Abstract

A method and system for determining of a next health event data from a health information dataset based on a machine learning model and a method for training said machine learning model, wherein the machine learning model is based on a transformer and comprises an additional output head for predicting a time to a next token, wherein the transformer comprises causal attention masks, wherein the health information dataset comprises a sequence of health event data and corresponding time data, and wherein the method comprises: a) tokenizing and embedding the sequence of health event data; embedding the corresponding time data based on an age-dependent encoding; combining the embedding of the corresponding time data and the embedding of the tokenized health event data into an input embedding; providing the input embedding to a trained transformer; wherein inside the transformer only information looking back not forward is processed based on the causal attention masks; receiving an output embedding from the trained transformer; determining a next token from the output embedding; determining a respective time-to-the- next-token from the output embedding based on the additional output head; and determining the next health event data from the next token and the time-to-the-next-token.

Description

[0001] HEALTHCARE MODELING

[0002] The present disclosure relates to a method and system for determining of a next health event data from a health information dataset based on a machine learning model. Furthermore the present disclosure relates to a method for training said machine learning model.

[0003] The progression of human disease across age is characterized by periods of health, episodes of acute illness and also chronic debilitation, often manifesting as clusters of co-morbidity. Patterns of multi-morbidity affect individuals unevenly and have been associated with lifestyle, heritable traits and socioeconomic status, see [Ref 1] to [Ref 3],

[0004] Understanding each individual’s multi-morbidity risks is important to tailor healthcare decisions, motivate lifestyle changes or direct entrance into screening programs as is the case for cancer, see [Ref 4] and [Ref 5], Critically, health cannot only be understood by the presentation of individual diagnoses but rather in the context of an individual’s co-morbidities and their evolution over time.

[0005] While a wide range of prediction algorithms exist for specific diseases from cardiovascular disease to cancer, see [Ref 6] to [Ref 8]., few algorithms are capable of predicting the full spectrum of human disease, which recognizes more than 1 ,000 diagnoses at the top level of the ICD-10 (international classification of diseases, tenth revision) coding system.

[0006] Learning and predicting patterns of disease progression is also important in populations that are ageing and which exhibit shifts in their underlying demographic’s morbidities. For example, it has been predicted that globally the number of cancer diagnoses will increase 77% by 2050, see [Ref 9], or that in the UK the number of working-age persons with major illnesses, including depression, asthma, diabetes, cardiovascular disease, cancer or dementia, will increase from 3 to 3.7 million by 2040, see [Ref 10],

[0007] Modelling the expected burden of disease is thus critical for healthcare and economic planning and moreover the continual tracking of disease occurrence along with its likely future prevalence within population groups promotes a more informed health system.

[0008] Recent developments in artificial intelligence (Al) may help overcome some methodological limitations of multi-morbidity modeling which have so far proved difficult to overcome, see [Ref 11], Aside from the great number of diagnoses, these include challenges in modeling temporal dependencies among previous events, the inclusion of potentially diverse prognostically relevant data and also the statistical calibration of predictions.

[0009] Large language models (LLMs) , see [Ref 12] to [Ref 15]., a subfield of Al that enables chatbots such as ChatGPT, see [Ref 16] and [Ref 17]., to model language as a sequence of word fragments (so-called tokens). Generated token by token, the new text is based on all preceding text - and with enough training, the statistical dependencies among these tokens prove sufficient to produce context-aware and even conversational text, which is often indistinguishable from that of a human counterpart.

[0010] The analogy between large language and disease progression modeling - which also entails recognition of past events and exploitation of their mutual dependencies to predict the future sequence of morbidity - has recently inspired a series of new Al models.

[0011] For example, BERT-based models, see [Ref 18] to [Ref 21], have been developed for specific prediction tasks. Transformer models trained on electronic health records have been used for predicting diseases such as pancreatic cancer, see [Ref 22], self-harm, see [Ref 21] and stroke, see [Ref 20], as well as non- clinical parameters such as self-esteem , see [Ref 23],

[0012] However, despite promising proof of concepts, see [Ref 24], the potential for comprehensive and generative modeling of multi-morbidity has not been fully assessed yet.

[0013] According to embodiments the present disclosure, attention-based transformer models, similar to LLMs, can be extended to achieve the task of learning lifetime health trajectories and accurately predict future disease rates of more than 1 ,000 diseases simultaneously based on prior health diagnosis, lifestyle factors and further informative data.

[0014] The embodiments of the present disclosure relate to an extended model, termed Delphi, which is trained on population wide cohorts (either a sample of a population and up to and including an entire population). One example of such training is the training on data from 400,000 participants of the UK Biobank, a population scale research cohort from the UK.

[0015] A preferred embodiment according to the present disclosure is an extended model, Delphi, trained on the UK Biobank data and comprising 2.2 million parameters. The latter model is also referred to as Delphi-2M in the disclosure hereinbelow

[0016] It is also noted that the model according to the present disclosure may also be trained on any other data set (such as the Danish registries) which consists of [person, disease code, age]-type data points.

[0017] The vocabulary of the model includes 1 ,256 ICD10 top level codes, as well as sex, body mass, smoking, alcohol consumption and death. Other tokens or clinical systems could have been used. In particular, finer grained classes may be used, e.g. E45.1 , E45.2,etc. However, the examples of this disclosure omit the subclasses after the period, i.e., only use E45, because the training data in the UK Biobank is currently too limited. In general it is noted that the vocabulary size is a model parameter, which may be chosen as discussed below. The model can handle many more codes as discussed hereinbelow- easily 100k - the choice in the examples below is data limited.

[0018] The analyses not only reveal that Delphi-2M provides accurate instantaneous multistate rates but also highlights how elements of past data influence the subsequent rates of disease and enables sampling of future courses of health and disease at any point throughout an individual's life course.

[0019] The predictions remain accurate even if evaluated on an external dataset from Denmark without any additional finetuning. Delphi can provide competitive individual level prediction of multiple outcomes simultaneously and can model the overall expected evolution of a population from a specific health history starting point.

[0020] It is noted that according to an embodiment of the present disclosure, a prediction of full trajectories from specific health history starting points can be achieved using Delphi’s generative methods, including the likely sequence of “health data events” and their timing in one or more individuals from a population.

[0021] The approach can also be used to derive synthetic datasets of any size that mirror the population characteristics of the training set but do not contain any real individuals. These synthetic datasets can be used to train new Delphi models that reflect health relevant data distributions from the training population but have never seen any real data, which can in turn be used to evaluate population risks.

[0022] In addition, the internal model of Delphi is partly interpretable and can provide a framework for future research into healthcare trajectories.

[0023] Recent large language models, LLMs, such as GPT, Gemini or LLaMa, are showing remarkable abilities to generate context-dependent textual output in a great variety of languages. Initially designed for language modelling, they have been expanded to various other domains, such as images, video, code generation and autonomous LLM agents.

[0024] LLM-powered chat assistants such as ChatGPT and Bard are used by millions of users. Such diverse tasks are fulfilled despite their architectural simplicity as autoregressive models based on transformer blocks. In abstract terms, LLMs express words as numerical tokens and exploit non-linear dependencies among preceding tokens to predict future tokens successively.

[0025] In healthcare, one often seeks to infer the future health of an individual given their past medical history, current conditions, and potential treatment. Conceptually, these predictive tasks are similar to the statistical nature of LLM-based natural language processing, which learns contextual and grammatical logic to provide a meaningful continuation of previous inputs.

[0026] While there is a long history of time-to-event and multistate analyses for medical inference and predictive purposes, these statistically rigorous algorithms can typically model only a relatively small number of outcomes. More recently, classical machine learning approaches, but also attention-based recurrent neural networks and applications of BERT-based models have been developed for specific prediction tasks.

[0027] The present disclosure aims at overcoming or reducing certain shortcomings of the prior art as will be apparent from the description below.

[0028] In the present disclosure it is demonstrated that GPT-based models can achieve the task of learning health trajectories and accurately predict future disease rates, also referred to as disease incidence of a cohort, of more than 1 ,000 diseases based on prior health, lifestyle and further data.

[0029] It is noted that the GPT model is disclosed, e.g., in https: / / doi.org / 10.48550 / arXiv.2303.08774. This document and the GPT model therein are herewith incorporated by reference.

[0030] The model according to the present disclosure is termed Delphi (Delphi Large Predictive Health Inference).

[0031] In an embodiment of the present disclosure, a GPT-2 model is extended to reflect continuous time sequences and fit this model, termed Delphi-2M, to data from 400,000 participants of the UK Biobank, a research cohort.

[0032] Thus, the present disclosure may provide generalizing predictive health models to simultaneously represent multiple diseases, looking into a more distant future, and predicting entire health trajectories. A similar approach that allows LLMs to generate long sequences of natural language tokens auto reg ressively may be used in the disease progression setting to predict the sequence of future health conditions.

[0033] The extensions of the embodiments of the present disclosure may include a cosine embedding of time, temporal causal attention masks, and an exponential waiting time head, which equip Delphi-2M to infer statistically calibrated rates of disease incidence.

[0034] Furthermore, according to the present disclosure, training data from the UK Biobank is used and investigations revealed similar scaling laws as observed for LLMs and an optimal model size of 2M parameters. It is, however, appreciated that the present disclosure is not limited to the exact training data and its origin.

[0035] The analysis also indicates that larger models trained on bigger data sets are poised to achieve greater accuracy.

[0036] The invention is set out in the appended set of claims. Any references in the following description to embodiments, objects, aspects and / or examples which are not covered by the appended claims are considered as not being part of the present invention. According to an embodiment of the present disclosure, there is provided a method for determining of a next health event data from a health information dataset based on a machine learning model, wherein the machine learning model is based on a transformer (and therefore may be termed a transformer model) and comprises an additional output head for predicting a time to a next token, wherein the transformer comprises causal attention masks, wherein the health information dataset comprises a sequence of health event data and corresponding time data, and wherein the method comprises: a) tokenizing and embedding the sequence of health event data; b) embedding the corresponding time data based on an age-dependent encoding; c) combining the embedding of the corresponding time data and the embedding of the tokenized health event data into an input embedding; d) providing the input embedding to a trained transformer; wherein inside the transformer only information looking back not forward is processed based on the causal attention masks; e) receiving an output embedding from the trained transformer; f) determining a next token from the output embedding; g) determining a respective time-to-the-next-token from the output embedding based on the additional output head; and h) determining the next health event data from the next token and the time-to-the-next- token.

[0037] According to the present disclosure, the “health event data” are understood to mean data on any health relevant event on a health trajectory, such as an event selected from medical diagnoses (preferably in the form of a diagnosis codes, such as ICD10 codes), baseline measurements (sex, height, weight, BMI, etc.), lifestyle data (e.g. BMI, smoking and alcohol consumption), prescription records, blood tests (e.g. blood I biomarker assays), genomic data, richer metabolomic information, and data from wearables. In a preferred embodiment, the health event data refers to a medical diagnoses and the timing thereof.

[0038] In an embodiment of the present disclosure, the transformer is a GPT-based transformer.

[0039] In an embodiment of the present disclosure, a weight tying between the token embeddings and logit weights is used.

[0040] In an embodiment of the present disclosure, the tokenizing is based on a token vocabulary consisting of one or more of: i) a diagnosis code token set, preferably n=1257 different ICD-10 level 3 disease tokens; ii) a lifestyle token set, preferably n=9 tokens for alcohol, smoking and BMI, each represented by 3 different levels; iii) a sex token set, preferably n=2 tokens for sex; iv) a healthy padding token, preferably n=1 no-event padding token; v) a non-informative padding token at the beginning or end of the sequence of health data, preferably n=1 additional non-informative padding token.

[0041] In an embodiment of the present disclosure, during the tokenizing step healthy padding tokens are added to the sequence of health data, and wherein preferably the healthy padding tokens are randomly added at a predetermined rate, preferably at a rate of 1 in 5 years

[0042] In an embodiment of the present disclosure, the no-event padding tokens are randomly added at a predetermined rate, preferably at a rate of 1 / 5 years.

[0043] In an embodiment of the present disclosure, the age-dependent encoding is based on sine and cosine base functions.

[0044] It is noted that the term sine and cosine base functions should be understood, in an embodiment, as a functional definition, i.e. the term also encompassed similar implementations, with essentially the same computational function, e.g. rotary position embedding.

[0045] According to an embodiment of the present disclosure, there is provided a method for training a transformer model for prediction of a next health event data from a health information dataset, preferably for a method according to any one of the preceding embodiments, based on a health information database, comprising the steps of: a) retrieving a plurality of health data sets and corresponding time information data from the health information database; c) splitting the data points into training data points, preferably 80% thereof, and validation data points, preferably 20% thereof; d) training the transformer model by stochastic gradient optimization based on the training data points and validation data points, wherein during the training the total loss of the transformer model is the sum of a next disease event prediction loss function based on softmax and a time to next event loss function based on an exponential loss function, defined by: loss = loss_j + loss_T .

[0046] In an embodiment of the present disclosure, loss_j is defined as loss_j = - log p ( j ) = -c ross_ent ropy( logits , tokens ) , and wherein loss_T is defined as loss_T = - log p (T) = - ( logsumexp ( logits ) - sum( exp ( logits ) ) * T) . In an embodiment of the present disclosure, the transformer model is trained using an Adam optimizer, preferably for 200k iterations, preferably for a batch size of 128, preferably decaying the learning rate using a cosine function from 6e-4 to 6e-5.

[0047] In an embodiment of the present disclosure, the method is comprising iteratively performing the steps of the method according to any one of the preceding aspects and embodiments for determining a plurality of next health event data as the health trajectory prediction; and outputting the health trajectory prediction.

[0048] According to an embodiment of the present disclosure, there is provided a data processing system comprising means for carrying out the steps of the method according to any one of the preceding embodiments.

[0049] According to an embodiment of the present disclosure, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any one of the preceding embodiments.

[0050] According to an embodiment of the present disclosure, there is provided a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of the preceding embodiments.

[0051] Brief description of the Drawings

[0052] Embodiments of the present disclosure will now be described by way of example only and with reference to the following accompanying drawings.

[0053] Fig. 1 shows a block diagram of the model according to an embodiment of the present disclosure;

[0054] Fig. 2 shows a bock diagram of the training according to the present disclosure;

[0055] Fig. 3 shows the inference based on one individuum according to an embodiment of the present disclosure;

[0056] Fig. 4 shoes an illustration of a token sequence according to an embodiment of the present disclosure;

[0057] Fig. 5 shows an illustration of the training data according to an embodiment of the present disclosure;

[0058] Fig 6 shows the accuracy of predicted time-to-event based on an embodiment of the present disclosure;

[0059] Fig. 7 shows the scaling laws according to an embodiment of the present disclosure.

[0060] Fig. 8a to 8c show the results of a parameter screen according to an embodiment of the present disclosure;

[0061] Fig. 9 shows the ablation results measured by the cross-entropy differences relative to an age and sex based baseline for an embodiment of the present disclosure; Fig. 10a to Fig. 10e show results of an ablation study for an embodiment of the present disclosure;

[0062] Fig. 11 a to Figi 1d show a comparison of the accuracy of an embodiment of the present disclosure in predicting diverse disease outcomes in the validation cohort compared to the sex and age-stratified incidence as an epidemiological baseline;

[0063] Fig. 12a to Fig. 12c show longitudinal validation results on UK Biobank data for an embodiment of the present disclosure;

[0064] Fig. 13a to Fig. 13e show sampling health trajectories according to an embodiment of the present disclosure;

[0065] Fig. 14a to Fig.14g show sampling health trajectories according to an embodiment of the present disclosure;

[0066] Fig. 15a to Fig. 15d show that predictions of an embodiment of the present disclosure remain accurate in a more distant future;

[0067] Fig. 16a to Fig. 16d show that explainable Al according to an embodiment of the present disclosure offers insights into disease progression;

[0068] Fig. 17a to Fig. 17f show attention maps according to an embodiment of the present disclosure; and

[0069] Fig. 18a to Fig. 18d show an external validation using Danish registries according to an embodiment of the present disclosure.

[0070] Detailed description

[0071] Model Architecture

[0072] The model according to the present disclosure is a generative model configured for a single data stream. In one embodiment of the disclosure, the model is based on a transformer model, preferably based on the GPT-2 architecture. That is, the model according to the present disclosure is based on a decoder-type only architecture.

[0073] Fig. 1 shows a block diagram of an embodiment of the model according to the present disclosure. The model according to the present disclosure comprises a modified decoder part 1012 of a GPT model.

[0074] The model according to the present disclosure has been extended to accept an age, i.e. time information, and a health information token sequence as input.

[0075] The extensions may comprise: a) an age encoding layer for embedding of time information; b) temporal causal attention masks; and c) an exponential waiting time head. It is noted that the exponential waiting time head may comprise, in an embodiment, approximations of the exponential waiting and may still be referred to an exponential waiting time head. That is, the term is a functional definition.

[0076] The first extension according to the present disclosure is replacing the usual positional encoding with an age-dependent embedding layer 1012, preferably using sine and cosine basis functions. That is, the agedependent embedding 1012 is configured to encode the input times 1001 from a continuous time axis, while a positional encoding normally encodes discrete positions, e.g., in a sequence of words.

[0077] In embodiments of the present disclosure, the age-dependent embedding is based on the age values. Following the logic for positional embeddings, in a preferred embodiment, age is represented by sine and cosine functions of different frequencies, where the lowest frequency is given by 1 / (100*365), corresponding to a linear and constant basis function each.

[0078] These functions are subsequently linearly combined by a trainable linear transformation, which enables the model to share the same basis function across multiple embedding dimensions. Another advantage of using an age embedding is that Delphi can handle token inputs of arbitrary length as no parameters are associated with token positions.

[0079] In this disclosure the terms age and time are used. Age refers to the age of the individual, while time refers to the date of the diagnosis. It is noted that both terms refer to the same information and a respective conversion is implied, even if not explicitly mentioned for reasons of improved intelligibility.

[0080] In a preferred embodiment, the age encoding layer 1012 uses a set of sine and cosine basis functions with different frequency coefficients C to transform a scalar age in years into a vector representation, for example:

[0081] C = [C1 ,C2,...]

[0082] Age_encoding(age) = [sin(C1*age), sin(C2*age), ...

[0083] Cos(C1*age), cos(C2*age), ...]

[0084] The number of coefficients C may depend on desired output dimensionality. The coefficients are non- trainable and can be selected beforehand depending on the problem at hand.

[0085] The age encoding, i.e., the output of the age encoding layer 1012, is combined with a token embedding 1011 , preferably via summation 1013 of both data structures 1012 and 1011. This sum forms the input to the decoder 1020.

[0086] The token embedding 1011 is obtained from a sequence of tokens 1002a and 1002b. That is, according to the present disclosure, the sequence of tokens comprises disease tokens 1002b and other tokens 1002a. The tokens are numerical values. The other tokens 1002a are of preferably three categories: sex tokens 101 , lifestyle tokens 103, and no event 104, i.e., padding tokens.

[0087] For tokenization of health information, in a preferred embodiment of the present disclosure, diagnosis codes, e.g. ICD10 codes are assigned integer values and represented as long sequences across an individual’s health care record including baseline measures (sex, height, weight, BMI, etc.), padding tokens, and “no event” (or healthy) tokens placed as specific points to allow better calibration of the time domain.

[0088] The disease tokens 1002b are preferably a numerical representation of a diagnoses represented by level- 3 ICD10 codes. The sex tokens 101 are preferably assigned at birth. The lifestyle tokens 103 are a numerical representation of one or more of BMI, as well as smoking and alcohol consumption indicators.

[0089] Together with “no event” padding tokens, which may be randomly added at a predetermined rate, preferably of 1 / 5 years, that is an average of 1 token in every 5 years, basic temporal continuity of inputs is ensured.

[0090] The term “randomly adding” refers to randomly placing tokens across the sequence at a specific predetermined rate. According to the present disclosure each token is paired with the corresponding age, measured from birth, the aforementioned times 1001.

[0091] In a preferred embodiment of the present disclosure, the token vocabulary consists of n=1257 different ICD-10 level 3 disease tokens, plus n=9 tokens for alcohol, smoking and BMI, each represented by 3 different levels, as well as n=2 tokens for sex and n=1 no-event padding token as well as n=1 additional, non-informative padding token at the beginning or end of the input sequences.

[0092] That is, the token vocabulary consists of a predetermined number of disease tokens, e.g., based on a coding scheme for diseases, plus a predetermined number of lifestyle tokens, preferably at different levels, a predetermined number of sex token, no-event padding tokens as well as n=1 additional, non-informative padding token at the beginning or end of the input sequences.

[0093] In a preferred embodiment of the present disclosure, non-informative padding tokens are added to Delphi token sequences at the beginning or end of the input sequences.

[0094] For embedding the token sequence, the embedding layer uses a weight matrix, which projects from the input vocabulary dimension to the dimension of the embedding dimension of the model. It is noted that the same weight matrix used for projecting the output embedding to the vocabulary dimension.

[0095] The second extension of the model according to the present disclosure, is the integration of causal continuous time masks 1021 . The effect of said causal continuous time masks 1021 to resolve ties of tokens recorded at the same time. That is, inside the transformer only information looking back not forward is processed, which allows for one or more of the tokens to be recorded at the same time. That is, according to a preferred embodiment of the present disclosure, attention masks are updated by setting attention values to 0 for padding tokens and co-occuring tokens. Padding, co-occuring and non predicted tokens are masked from both loss functions, see below. Attention masks are updated to mask padding and co-occruing tokens such that the model does not pay attention to them. Non predicted tokens are attended to but not included in the loss functions.

[0096] Standard causal self-attention enables a GPT model to attend to all preceding tokens, that is, for sequential language data, these are found to the left of the token sequence.

[0097] Yet in the case of time-dependent data, as in the present disclosure, tokens can be recorded at the same time with no specified order.

[0098] Thus, for the embodiments of the present disclosure, the attention masks were amended to mask positions that occurred at the same time as the predicted token. Furthermore, Non-informative padding tokens were masked for predictions of other tokens.

[0099] The third extension of the architecture according to the present disclosure, is the addition of another output head 1044 that predicts time to the next event, i.e., next disease token. This, along with a corresponding loss term, cf. below, ensures that predicted probabilities can be directly interpreted as incidences, that is the probability of a token to occur in a specified period.

[0100] That is the additional output head in a transformer as defined in the present disclosure refers to a computational entity configured to receive input from the attention layers (i.e. the output embedding) and recognizing the previous history of the tokens, i.e., the health events, and is configured to predict the time to the next token, i.e. health event.

[0101] One further additional extension according to preferred embodiments of the present disclosure, is the inclusion of non predicted baseline and lifestyle tokens which are also masked and the model is deliberately not trained to predict them,

[0102] Multiple non predictions tokens may be included to aid predictions but are not of interest to be predicted themselves. Some examples of these measurements, include sex, height, weight, BMI, smoking status and alcohol consumption in a tokenized fashion. The inclusion of more non predicted inputs make the model extremely flexible. Additional non-tokenized inputs may also include medications, imaging derived features, genetics, extended baseline measurements and exposures; numerical representations thereof (scalar or vector valued) are mapped by a linear or non-linear transformation to the embedding space, analogous to the token embedding 1011.

[0103] The input data to the embodiments of the present disclosure are bivariate pairs (j, t) of the next token class and the time to the next token. Delphi is motivated by the theory of competing exponentials. Let T be the waiting times from the current event to one of / = 1 ,... ,n competing events, where n is the number of predictable tokens.

[0104] Assuming the 71 are each exponentially distributed waiting times with rates exp(A), The next event being j is equivalent to Tj being the first of the competing waiting times, i.e. Tj= min 71, or equivalently j = argmin 71.

[0105] It can be shown that the corresponding probability is P(j = argmin 71) = exp(Aj) I Ziexp(A), which is the softmax function over the vector A. Conveniently, this definition corresponds to the classical cross-entropy model for classification with A = logits.

[0106] Hence Delphi preferably uses a conventional loss term for token classification: loss_j = - log p ( j ) = -c ross_ent ropy( logits , tokens )

[0107] Furthermore, in the competing exponential model the time to the next event T* = min{7 } is also exponentially distributed with logarithmic rate A* = log(Zi exp(A)), which is the logsumexp function of the logarithmic rates A. The loss function of exponential waiting times T between tokens is simply given by: loss_T = - log p (T) = - ( logsumexp ( logits ) - sum( exp ( logits ) ) * T) .

[0108] These approximations hold as long as the log rates Ai are constant in time, which is a reasonable assumption over short periods. For this reason, according to the present disclosure, padding tokens were introduced to ensure that waiting times are modelled over a relatively short period, which does not exceed 5 years in expectation.

[0109] In line with the tie-breaking logic used for causal self-attention, co-occurring events are preferably predicted from the last non-co-occurring token each.

[0110] The total loss of the model is then given by: loss = loss_j + loss_T .

[0111] In a preferred embodiment of the present disclosure, non-informative padding, as well as sex, alcohol, smoking, and BMI were considered as mere input tokens and thus removed from the loss terms above.

[0112] This is preferably achieved by setting their logits to -Inf and by evaluating the loss terms only on disease and healthy padding tokens.

[0113] The primary output of the decoder 1020 according to the present disclosure is an output embedding in the embedding space, obtained from the layer normalization 1041. The linear layer 1042 maps the primary output embedding to the space that contains all possible token (same dimension as the input). This step uses the same weight matrix as in the token embedding layer 1011. That is, from the output embedding the logits for each token of the input token vocabulary may be obtained by multiplication with the weight matrix.

[0114] The sharing of the weight matrix is also referred to as weight tying, which is a known approach for regularization that is used in deep learning models. It implies sharing the weight matrix between the input- to-token-embedding and final-embedding-to-logits layers. This, among other effects, improves the model stability.

[0115] Since the interpretation of the embedding does not change throughout the model, at all times, the weight matrix may be used to obtain a comprehensible output from an embedding.

[0116] From the logits the probability of the next event 1051 and time to the next event 1052 may be obtained. The probability of the next event 1051 is preferably obtained bases on a Softmax layer with cross entropy loss. The probability of time to the next event 1052 is preferably obtained bases on a competing exponential model 1044.

[0117] Training Phase

[0118] The training according to the present disclosure is performed based on at least one health database, preferably the UK Biobank database, because it offers a systematic and curated data collection.

[0119] Fig. 2 shows a bock diagram of the training according to the present disclosure.

[0120] The database 2000 comprises a plurality of datasets 2001 each corresponding to one individual. Each dataset comprises health information 2011 , corresponding time information 2012 and lifestyle information 2013.

[0121] To form one training dataset from a dataset 2001 a series of health information 2011 , corresponding time information 2012 and lifestyle information is retrieved. As discussed above the health information is tokenized together with the other information (other tokens).

[0122] According to an embodiment of the present disclosure only a subset of the token sequence is provided to the decoder 1020. According also just a corresponding subset of the time information is encoded. The resulting output embedding obtained from the decoder 1020 is then processed as discussed above to obtain a next event token 1051 and a corresponding and time to event 1052.

[0123] This prediction for a next token and time is compared 3060 to the actual next token in the input sequence 1001 and corresponding time information 1002. Based on the comparison an optimization 3070 of the decoder 1020 is performed. There are different methods to obtain suitable subsets of tokens and different methods to obtain an optimization for the model.

[0124] In a preferred embodiment of the present disclosure, models use two different loss function, i) the next disease event prediction loss function using softmax and ii) the time to next event loss function using an exponential loss function. These two loss function are summed to give the overall loss of the model.

[0125] In a preferred embodiment of the present disclosure, models are trained by stochastic gradient optimization using the Adam optimizer for 200k iterations.

[0126] The batch size, the size of the input sequences fed into the model during training, preferably is 128. The learning rate, i.e., a measurement of how quickly the cross entropy loss decreases overtime during training, is decayed preferably using a cosine function from 6e-4 to 6e-5.

[0127] Training data according to the present disclosure comprises 400,000 (80%) participants of the UK Biobank recorded before the 1st of July 2020. Validation data sets, i.e. data by which the trained model is evaluated and which is completely unseen during model training, were (i) the remaining 102,485 (20%) participants and (ii) data for all participants still alive on the 1st of July 2020 for longitudinal validation until the 1st of July 2022.

[0128] Additional external validation was carried out on a test set based on Danish disease registry data covering 1.93M Danish nationals aged 50-80 years, with data available between 1978 and 2018.

[0129] It is noted that for the used training data, i.e. the UK Biobank, sex tokens were presented at birth. Smoking, alcohol and lifestyle were recorded at the enrolment into the UK Biobank.

[0130] This specific time also coincides with the end of immortal time bias (probands had to be alive when they were recruited). Therefore, in a preferred embodiment, smoking alcohol and BMI tokens times were randomized by -20 to +40 years from this point in time to break an otherwise confounding correlation leading to a sudden jump in mortality rates (and possibly other diseases with high mortality such as cancers) associated with the recording of these tokens. This is likely also to diminish the true effect of these tokens.

[0131] In other population settings (for example, a random subset of a population) such manipulations may not be needed.

[0132] After the training is completed a trained model 1020a is obtained.

[0133] Inference Phase The inference phase of the trained model 1020a is used to infer a prediction from a health information dataset. The inference may be performed in different modes, each relating to a different embodiment of the present disclosure.

[0134] Fig. 3 shows the inference based on one individuum according to an embodiment of the present disclosure. For the individuum a previous health data 4001 is obtained. The health data 4001 comprises health information 4011 , corresponding time information 4012 and lifestyle information 4013.

[0135] The health data may be obtained from at least one health database, preferably the Biobank database. Additionally or alternatively, personal health data records and / or manual data input may be used to obtain a suitable sequence of heath events and corresponding time information.

[0136] The health data is tokenized 3000 and the corresponding embedding 1013 is input to the trained model 1020a. The trained model outputs an output embedding 1050 for the individuum.

[0137] Based on the output embedding, the next event token 1051 and the time to event 1052 is obtained, as discussed above. This pair of token and time is appended to the input sequence and the process is iterated to generate a health trajectory for the individuum.

[0138] In a preferred embodiment, the output embedding 1050 provides a vector of size 1 ,258 for each of the relevant tokens that is predicted per individual. Based on the above discussed model, with the two corresponding loss functions, the output embedding may be used in two ways (i) to predict the next token and (ii) to predict the time until the next token.

[0139] For (i) the output embedding corresponds to logits and can be transformed to a probability via an inverse mapping (logistic), the logistic being the inverse of the logit function and both are standard functions.

[0140] This transformation then provides the probabilities for each of the 1 ,258 tokens to occur next. Based on these probabilities the next token, e.g, from a multinomial distribution, may be sampled.

[0141] For (ii) the output embedding may be transformed to correspond to the rate of an exponential distribution via taking the exponential (this can also be understood as the hazard of an exponential survival model).

[0142] If one wants to now compute the probability of this token to appear within the next year for example, one computes: P(T < 365.25) = 1 - exp(exp(lambda) * 365.25), wherein lambda corresponds to the output embedding for a specific token here.

[0143] This probability can then be understood as the risk of this token in the next year. As the model is based on exponentials, the time to the next token - irrespectively of which of the 1 ,258 tokens - is simply: sum(exp(lambda)) - the sum over all the rates. Trajectories are formed through an iterative sampling process. Given an input sequence a pass through the model is done and then the most likely next token is computed with the corresponding time / age (see above).

[0144] Then this new token is appended to the initial input sequence and repeat the whole process, therefore, sequentially assembling the trajectory.

[0145] Results of a preferred Embodiment

[0146] The present disclosure is now discussed by a detailed discussion of a preferred embodiment, which is trained on health trajectory data available in the UK Biobank. All features of this embodiment, or a subset of features may be used in other embodiments. Furthermore, some features may be omitted, for better intelligibility.

[0147] As discussed above, a person’s health trajectory can be represented by a sequence of diagnoses using by top level ICD10 codes recorded at the age of first diagnosis as well as death.

[0148] According to an embodiment of the present disclosure, “no event” padding indicators are further included, which were randomly added at an average rate of 1 / 5 years to ensure basic temporal continuity of inputs.

[0149] Together, these data comprise 1 ,258 distinct states (or tokens in LLM terminology), each paired with the corresponding age, measured from birth, the sequence and timing of which are auto-regressively predicted.

[0150] Further health-related information includes sex, BMI, as well as smoking and alcohol consumption indicators, which are added at the age when the corresponding information becomes available, but are not themselves part of predictions, see Fig. 4.

[0151] Fig. 4 shoes an illustration of a token sequence according to an embodiment of the present disclosure. The token sequence shown as part of the tokenization 3000 as discussed above. The sequence preferably comprises sex tokens (horizontal pattern), lifestyle tokens (vertical pattern), disease tokens (left pointed pattern), and no event, padding tokens, (right pointed pattern).

[0152] The model according to the present embodiment, was trained as discussed above. Fig. 5 shows an illustration of the training data according to an embodiment of the present disclosure.

[0153] Training data 2000 comprised 400,000 (80%) participants of the UK Biobank recorded before the 1st of July 2020. Test data sets were (i) the remaining 102,485 (20%) participants and (ii) data for 471 ,057 (94%) participants still alive on the 1 st of July 2020, reference sign 2100, for longitudinal validation up until the 1 st of July 2022. Additional external validation 2200 was carried out on a test set based on Danish disease registry data covering 1.93M Danish nationals aged 50-80 years old, with data available between 1978 and 2018.

[0154] To model disease history data, which, unlike text, occurs on a continuous time axis, as discussed above, in the present disclosure the GPT-2 architecture, see [Ref 25] has been extended. Reference is made to Fig. 1 and the description thereof in the above.

[0155] Transformer models map their inputs into a lower dimensional embedding space, in which information is successively aggregated to facilitate auto-regressive predictions. The first change thus replaces GPT’s positional encoding, a mapping which identifies the discrete position of each text token, with an encoding of continuous age using sine and cosine basis functions, see [Ref 12],

[0156] Standard GPT models only classify the next token using a multinomial probability model. Therefore, the second extension is the addition of another output head to also predict the time to the next token using an exponential waiting time model. As the rates of competing exponentially distributed waiting times define the logit probabilities of the next event, this parametrization extends GPT’s existing classification step, see Fig. 6.

[0157] Fig. 6 shows the accuracy of predicted time-to-event based on an embodiment of the present disclosure. In the drawing the observed days to next token are shown over the expected days to next token, and the average is shown as solid line.

[0158] Third, GPT’s so-called causal attention masks, which ensure that the model only accesses information from past events, are amended to resolve ties of tokens recorded at the same time (It is noted that such temporal ordering does not imply the model of embodiments the present disclosure is capable of identifying causal mechanisms in the sense of modifiable features. Padding, lifestyle and sex tokens use a similar encoding, but do not enter the likelihoods, because in an embodiment of the present disclosure the model of embodiments the present disclosure is deliberately not trained to predict them).

[0159] The above discussed model is termed Delphi (Delphi Large Predictive Health Inference). This architecture allows to provide the model of embodiments the present disclosure with a partial health trajectory (prompt in LLM terminology) in order to calculate the subsequent rate (per day) for each of the 1 ,256 disease tokens plus death. Furthermore, the next token and the time to this event can be sampled based on these rates. Iteratively, this procedure samples entire health trajectories.

[0160] An example Input may be:

[0161] Age Token

[0162] 0.0: Male

[0163] 2.0: B01 Varicella (chickenpox)

[0164] 3.0: L20 Atopic dermatitis

[0165] 5.0: No event 10.0 No event

[0166] 15.0 No event

[0167] 20.0 No event

[0168] 20.0 G43 Migraine

[0169] 21 .0 E73 Lactose intolerance

[0170] 22.0 B27 Infectious mononucleosis

[0171] 25.0 No event

[0172] 28.0 J11 Influenza, virus not identified

[0173] 30.0 No event

[0174] 35.0 No event

[0175] 40.0 No event

[0176] 41.0 Smoking low

[0177] 41.0 BMI mid

[0178] 41.0 Alcohol low

[0179] 42.0 No event

[0180] The corresponding Output may be:

[0181] 43 2 No event

[0182] 43 5 M54 Dorsalgia

[0183] 44.6 IS6 Varicose veins of other sites

[0184] 50.4 K52 Other non-infective gastro-enteritis and colitis

[0185] 52.2 H83 Other diseases of inner ear

[0186] 53.9 J22 Unspecified acute lower respiratory infection

[0187] 54.5 L30 Other dermatitis

[0188] 55.3 No event

[0189] 57.5 L50 Urticaria

[0190] 59.4 K62 Other diseases at anus and rectum

[0191] [...]

[0192] 68.8 J90 Pleural effusion not elsewhere classified

[0193] 70.0 K21 Gastro-esophageal reflux disease

[0194] 70.1 K76 Other diseases of liver

[0195] 70.3. 110 Essential primary hypertension

[0196] 70 4 M85 Other disorders of bone density and structure

[0197] 70 7 M81 Osteoporosis without pathological fracture

[0198] 71.2 J98 Other respiratory disorders

[0199] 72.1 J80 Adult respiratory distress syndrome

[0200] 72.2 No event

[0201] 72.7 Death

[0202] A systematic screen of architecture hyperparameters (embedding dimensionality, number of layers, heads) confirms the reported empirical scaling laws, see [Ref 26], which state that model performance increases with the number of data points and, up to a limit defined by the available data, as the number of parameters increase, see Fig. 7 and Fig. 8.

[0203] Fig. 7 shows the scaling laws according to an embodiment of the present disclosure. In Fig.7 the minimum validation loss is shown over the model size (M parameters).

[0204] Fig. 8a to Fig. 8c show the results of a parameter screen according to an embodiment of the present disclosure. Figs. 8a shows validation cross-entropy (leftmost axis) for models trained with different architectural hyperparameter values (other axes). Fig. 8b, shows the same as Fig. 8a, taking a closer look at each parameter individually. The boxplots feature median as the center line, the box from the first to the third quartile and the whiskers for 1 ,5x IQR. Fig. 8c shows the importance of different hyperparameters (via Weights & Biases Sweeps). Fig. 8a is shows with heat-map type color-mapping. A colored version is provided at the time of filing.

[0205] The parameter screen indicates that for the UK Biobank dataset, optimal Delphi models have around 2M parameters. One of the models of embodiments the present disclosure within the optimal range, contains 120 internal embedding dimensions, 12 layers and 12 heads amounting to a total of 2.2M parameters, this model is a preferred embodiment. Unless indicated otherwise, results based on this model parameterization, termed Delphi-2M, are discussed throughout the rest of the disclosure.

[0206] It is noted that that qualitatively similar results may be obtained from other parameter choices.

[0207] That is, the number of internal (embedding) dimensions may be selected with restrictions only arising from available GPU resources preferably between 80 and 156, and most preferably 120 for the given training data set. The number of layers may be selected with restrictions only arising from available GPU resources, preferably between 8 and 14, and most preferably 12 for the given training data set. The number of heads may be with restrictions only arising from available GPU resources, preferably between 8 and 14, and most preferably 12 for the given training data set. The number of input tokens may be selected with restrictions only arising from available GPU resources preferably between 48 and 60, and most preferably 48 for the given training data set

[0208] It is noted the discussed parameters are effectively unlimited in the model implementation. It is only the hardware used, that sets bounds. Also, as the investigation have shown that a finite training data set imposes thresholds beyond which the model performance stops increasing. The preferred values are all based on UK Biobank training data with n=400,000 persons.

[0209] The selection is based on an optimization and the total size of the training data, the hyper parameter scan may be performed to find the optimal parameters. It has been noticed that the model size follows known scaling laws and although a particular layout is preferred (as above), it is the total number of parameters that matter most - relative to the size of the training data. A sensible range of parameter layouts has been tested and an increase in performance has been observed in relation to the number of parameters - up to a point where a performance plateau is indicating bigger models do not add further improvement in performance but come with an expense of resources (and time) for training, cf. Fig. 7 and Fig, 8a.

[0210] An ablation analysis shows how Delphi-2M's architectural modifications contribute to a better age-and-sex- stratified cross-entropy compared to a standard GPT model, see Fig. 9 and Fig. 10 as well as Table 1 below.

[0211] + healthy + age enc. &

[0212] Loss Age and Sex GPT tokens masks

[0213] Next token class 0 (baseline) -0.206 -0.478 -0.567 (cross-entropy)

[0214] Time to next token 0 (baseline) N / A N / A N / A (exponential)

[0215] Table 1

[0216] Table 1 shows ablation results for Delphi-2M. Top: Change in cross-entropy of disease token predictions relative to a baseline derived from age (in years) and sex incidence. Bottom: Change in exponential loss for the waiting time until the next token.

[0217] Fig. 9 shows the ablation results measured by the cross-entropy differences relative to an age and sex based baseline for an embodiment of the present disclosure. Fig. 9 shows the results color coded. A colored version of Fig. 9 is provided at the time of filing.

[0218] Fig. 10a to Fig. 10e show results of an ablation study for an embodiment of the present disclosure. Fig. 10a shows the average cross entropy loss for disease tokens. Fig. 10b shows cross-entropy loss for disease tokens relative to sex- and age-incidence baseline. Fig. 10c shows the average cross entropy loss for disease and healthy tokens. Fig. 10d shows the average cross-entropy loss for disease and healthy tokens relative to sex- and age-incidence baseline. Fig. 10e shows time to next token for sex- and age-incidence. Fig. 10a to Fig. 10d show the results color coded. A colored version of Fig. 10a to Fig. 10d is provided at the time of filing.

[0219] A good, albeit slightly inferior, classification performance at different ages may already be achieved by adding regular “no event” padding tokens to the input data with GPT models alone.

[0220] However, a key distinguishing feature of Delphi compared to basic GPT models is its ability to calculate the absolute rates of tokens, which in a statistically accurate sense represent inter-event times, see Fig. 6. This property also implies that the rates may be interpreted as the incidences of tokens, as recorded in the training data. In Fig. 11 a to Fig. 11d Delphi-2M’s accuracy in predicting diverse disease outcomes in the validation cohort is compared to the sex and age-stratified incidence as an epidemiological baseline. Delphi surpasses age & sex baseline on a wide range of diseases.

[0221] Fig. 11 a shows that predicted disease rates follow observed age-rate curves. Fig. 11 b shows that when forecasting one token ahead, Delphi-2M predicts a higher than the baseline model rate for a token that actually is going to be observed. Fig. 11 c, shows the same as Fig. 11 b, the results are aggregated by ICD- 10 chapter. The boxplots feature median as the center line, the box from the first to the third quartile and the whiskers for 0.025 and 0.975 quantiles. Fig. 11 d shows a comparison of AUC-ROC curves for Delphi and established individual disease risk models.

[0222] As can be seen in the ten examples shown in Figure11 a, the incidence curves are very varied, with some diseases such as chickenpox peaking in infancy, whereas others such as asthma or depression are relatively flat and with most rising exponentially in old age. Moreover, there are noticeable differences between the sexes, which are obvious for breast cancer, but also noticeable for diabetes, depression, acute myocardial infarction and death.

[0223] Delphi-2M’s predictions are updated for each individual when new inputs are recorded. The predictions largely follow the sex- and age-stratified incidence curves but also indicate events or periods when the individual risk remains below or rises above the population average.

[0224] For some diseases such as asthma or arthrosis, the spread is narrow, indicating a limited ability to predict beyond the sex- and age-incidence trend. Yet for other diseases including septicaemia and also death, the spread is wide, indicating measurable inter-individual differences of disease rates. Importantly, calibration analyses in 5-year age brackets show that the predicted rates (grouped in increments of Iog10) closely match the observed number of cases, showing that the models’ predictions of the imminent next tokens are statistically accurate, see Fig. 11 b.

[0225] Often the predicted rate rises sharply above the baseline towards the timestep before a disease is recorded, indicating that acute rather than chronic events influence the predictions (Fig. 11 a). Delphi-2M’s ability to predict the next diagnosis token across the spectrum of human disease is confirmed by the average age- stratified AUCs (area under the receiver operating characteristic), which averages at values of approximately 0.8 in the internal validation split (Fig. 11 b).

[0226] For more than 99% of diagnosis the AUC was greater than 0.5 indicating that the vast majority of diagnoses in the data followed patterns with at least partially predictability. These patterns were found to be true across the different chapters of the ICD-10 spectrum, which define broad groups of disease. Among the most confidently predicted events in the short term is death with an age-stratified AUC of 0.98. Fig. 12a to Fig. 12c shows longitudinal validation results on UK Biobank data for an embodiment of the present disclosure. Fig. 12a to Fig. 12c show the results color coded. A colored version of Fig. 12a to Fig. 12c is provided at the time of filing.

[0227] Fig. 12a sows ROC-AUC results of Delphi-2M for each token with at least 25 occurrences colored by the respective ICD-10 chapter. Predictions are based on UKB adults (50-80 yrs of age at cutoff) with data up until 30. June. 2020 and evaluated on incidence between 1. July. 2021 - 1. July. 2022. Below are the respective differences of AUC values between Delphi and a UK Biobank age-sex baseline.

[0228] Fig. 12b shows calibration plots split by each ICD-10 chapter. Each line represents an ICD10 disease evaluated for each decile of the Delphi rate and compared against the observed rate in the population.

[0229] Fig. 12c shows a comparison of Delphi-2M against established risk models (rows) for cardiovascular disease and death with a varying data gap between inputs and outcomes (columns). Shown are the ROC curves for Delphi-2M and the corresponding alternative risk prediction model.

[0230] When enforcing a 1 year gap between inputs and outcomes to exclude short-term effects that may be due to the Delphi-2M’s predictions were still found to exceed the age and sex baseline by approximately 0.1 across most chapters indicating that its predictions are also relevant for prognostication (see Fig. 12a and Fig. 12c). The same analysis, a longitudinal validation using UK Biobank data from 2020-2022 that was not part of model training, also shows that there was no substantial shift in diagnostic patterns throughout Biobank’s follow-up.

[0231] Delphi-2M’s disease predictions on the longitudinal validation data in the UK Biobank are comparable to clinical risk models like the Framingham, see [Ref 7] risk score for cardiovascular diseases, the UKBDRS, see [Ref 27] risk score for dementia and the Charlson, see [Ref 28] comorbidity index for mortality (see Fig. 10d).

[0232] Delphi-2M’s performance also appears stable when evaluated over varying prediction horizons ranging from 1-24 months, albeit with slightly decreasing performance over longer time periods (Supplementary Fig. 12c). This shows that Delphi-2Ms’ multi-disease predictions match current risk models for individual disease outcomes, but offer the great advantage of being enabling the simultaneous assessment of more than 1 ,000 diseases at any given time.

[0233] One of the most promising features of generative models is to sample disease trajectories, conditional on data recorded up to a certain point. This is a property which few conventional epidemiological models possess. Reassuringly, trajectories sampled from birth, only provided with the sex token, closely mirror the empirical age incidences and other population-level statistics for both males and females. At the same time, samples appear unique and do not copy any trajectories from the training dataset, see Fig. 13 and Fig. 14. Fig. 13a to Fig. 13e show sampling health trajectories according to an embodiment of the present disclosure.

[0234] Fig 13a shows Delphi-modelled disease rates match observed rates. Fig.13b shows that breaking the trajectory structure by shuffling causes performance drop. Fig. 13c shows that Delphi time to the next event matches the observed time across all ages. Fig. 13d shows Delphi disease rate (number of simulated tokens / 15yr follow up from 60-75) versus the observed rate for each participant in the same period. Fig. 13e shows the distribution of number of currently predicted tokens.

[0235] Fig. 13a to Fig. 13e show the results color coded. A colored version of Fig. 13a to Fig. 13e is provided at the time of filing.

[0236] Fig. 14a to Fig.14g show sampling health trajectories according to an embodiment of the present disclosure. Fig. 14a to Fig. 14g show color coded results. A colored version of Fig. 14a to Fig. 14g is provided at the time of filing.

[0237] Fig. Ma to Fig. 14g show that Delphi-sampled trajectories are unique. Fig. 14a shows that when completing the partial trajectories from the train and validation subsets of the UKB dataset, the fractions of correctly guessed tokens are the same, indicating that there is no train set overfit. The boxplots feature median as the center line, the box from the first to the third quartile, the whiskers for 1 ,5x IQR and the outliers.

[0238] Fig. 14b shows that when sampling from scratch, even the trajectories that are similar to real ones tokenwise have major shifts time-wise.

[0239] Fig. 14c to Fig. 15f each show that, in terms of fraction of matching tokens are normalized Levensthein distance, Delphi-sampled trajectories are closer to real ones compared to age-sex sampled.

[0240] Fig. 14g shows that Delphi-2M-synthetic, a model that is trained on trajectories sampled from Delphi-2M, still performs better than age-sex epidemiological baseline.

[0241] Fig. 15a to Fig. 15d show that predictions of an embodiment of the present disclosure remain accurate in a more distant future.

[0242] Fig. 15a shows a schematic representation of the experiment design. Delphi is used to continue real trajectories, the proposed continuations are compared to ground truth.

[0243] Fig. 15b shows that Delphi-modelled disease rates match observed rates.

[0244] Fig. 15c shows that Delphi predictions are more accurate compared to the age & sex baseline.

[0245] Fig. 15d shows that Delphi accurately stratifies trajectories into low-, mid- and high-risk groups. In order to systematically assess the influence of medical histories on future health, health trajectories are sampled for each participant from the UK Biobank validation cohort based on data available until the age of 60 (Fig. 15a). This provides the opportunity to compare 100,000 sampled and observed trajectories. When evaluated at the population level, the overall incidences at age 70 are well recapitulated, showing that the overall distributions are well preserved by iterative sampling (Fig. 15b).

[0246] This is further confirmed by the cross-entropy loss of sampled trajectories, which is, on average, indistinguishable from the observed data, but drops when the preceding disease histories are shuffled between participants (Fig. 13a to Fig.13e).

[0247] In the first year of sampling, there are, on average 17% disease tokens that are correctly predicted, which drops to less than 14% 20 years later. These figures compare to values of 12-13% of correctly predicted disease tokens using sex and age alone, confirming that the conditional generation helps make more accurate predictions of future events (Fig. 15c).

[0248] Assessing the cumulative incidences of selected diseases among high and low risk groups reveals that Delphi-2M can predict the disease burden across a period of 20 years (Fig. 15d). For the selected diseases the highest and lowest risk groups differ approximately 20-fold, which is largely reflected by Delphi-2M’s sampled trajectories. While the precision is lower than that of short-term predictions these analyses show that Delphi-2M’s conditional samples provide meaningful extrapolations for future health courses. This is not trivial as many of the imminent risks can change over time as more diseases accrue.

[0249] The use of synthetic data has been proposed to help overcome issues with privacy in biomedical modeling if such data sets do not reveal characteristics specific to any one person. Turning back to data sampled from birth, i.e., without any condition except for sex, our assessment shows that the generated trajectories do not exhibit any greater similarity to the training data than those from the validation cohort (Fig, 14a to Fig. 14g).

[0250] While partially overlapping disease trajectories may be found in terms of absolute disease tokens, the extent of overlap would be expected based on the observed incidences and co-morbidity patterns. Also, matching tokens occur on average 10 years apart underscoring that Delphi-2M does not memorize specific trajectories. This is also in keeping with its statistical rates, which at values between 10-4to 10-2per year are usually well below levels that would imply determinism (Fig. 11 a).

[0251] Another piece of evidence that speaks to the quality of the generated synthetic trajectories is that they can be used to train Delphi-2M from scratch. Remarkably, such a recursively trained model retains many features of the original Delphi-2M model, for instance, it surpasses the age-sex baseline in terms of quality of next-token prediction (Fig. 14g). Insights into how Delphi-2M utilizes past information to predict future disease rates can be obtained by assessing the structure of the disease embeddings. GPT models linearly map inputs into a lower dimensional embedding space, in which the temporal sequence of events is iteratively aggregated to produce a state from which predictions of each next token are derived (Figure 1).

[0252] Delphi-2M’s specific implementation uses the same mapping from this final embedding state to the token risk, which guides the interpretation as the embedding matrix reflects the observed structure of co-morbidity risks.

[0253] Fig. 16a to Fig. 16d show that explainable Al according to an embodiment of the present disclosure offers insights into disease progression. Fig. 16a to Fig. 16d show color coded results. A colored version of Fig. 16a to Fig. 16d is provided at the time of filing.

[0254] Fig. 16a shows an UMAP projection of token embeddings. Selected diseases are shown in the zoomed-in areas. Colours define disease chapters.

[0255] Fig. 16b shows SHAP-explained token risk contributions for individual trajectories. The top shows the risk of pancreatic cancer immediately prior to diagnosis at age 68.2, which was found to be 19x increased. The bottom panel shows the SHAP-based estimates of contributions to estimated mortality at age 63.5, which was greatly increased, in large parts due to the preceding diagnosis of pancreatic cancer.

[0256] Fig. 16c shows a heatmap of average rate increase that is by each of n=778 disease tokens with more than 5 occurrences and grouped by chapter (y-axis) on the same set of tokens plus death (x-axis). Red colours indicate a risk increase, whereas blue is a decrease.

[0257] Fig. 16d shows a relative effect (SHAP value) on mortality (y-axis) as a function of time after diagnosis (x- axis) for selected diseases.

[0258] As shown by the UMAP representation of Delphi-2M’s embedding matrix in Fig. 16a, disease codes cluster closely by the underlying chapter, a property the model has no direct knowledge of, and which purely reflects co-occurrence patterns in the data.

[0259] Yet there are also noticeable exceptions, for example, cancers and precancers of the female reproductive tract. Another noteworthy cluster involves the two types of diabetes, retinal disorders and the neuropathies caused by them. Diseases with high acute mortality such as myocardial infarction or septicaemia are clustering with death.

[0260] As participants age they progress through the embedding space, albeit often in a discontinuous fashion marked by jumps caused by acquired disease, and tend to attract certain health trajectories causing momentary shifts in the future risk of surrounding diseases. To gain further insights into how individual tokens influence future risks SHAP (SHapley Additive exPlanations), see [Ref29], values measure the influence of each token from the input trajectory on model predictions by a systematic assessment of subsampled data for individual prediction.

[0261] As illustrated with the example of the proband’s partial trajectory shown in Fig. 16b, this analysis reveals that a series of disease diagnoses of the digestive tract (Chapter XI) elevated their pancreatic cancer risk 19-fold. The subsequent pancreatic cancer diagnosis, in turn, increased the rate of mortality almost ten thousand fold.

[0262] SHAP analysis of data from 100,000 individuals of the validation cohort reveals the mutual dependencies by which each disease, sex and lifestyle tokens influences the rate of disease tokens within the next 5 years (Fig. 16c, left).

[0263] Effects mostly increase the rates of other diseases and are often found among diseases of the same chapters, underscoring that the recorded patterns of co-morbidities often cluster within specific ICD-10 disease chapters. Of note such patterns often appeared symmetrical, indicating similar predicted effect sizes of one disease token influencing another as vice versa.

[0264] Particular clusters spanning entire disease chapters are visible for chapters V. Mental Disorders and XV. Pregnancy & Childbirth. Moreover, mortality is highly associated with diagnostic tokens from a broad range of chapters, including neoplasms.

[0265] Similar patterns are observed for dependencies after 10 years, even though the strength of associations is greatly attenuated (Fig. 16c, right). The cluster relating diseases of chapter XV. Pregnancy & Childbirth within 5 years is entirely absent after 10 years, which is expected given that pregnancy-associated diseases occur within a finite period.

[0266] Dependencies among mental disorders, however, remain apparent, similar to the effects of neoplasms on mortality. These observations are noteworthy as the quantification of temporal dependencies on past events poses a particular challenge for conventional epidemiological models whereas Delphi-2M’s GPT model uses attention based weights which are updated with every new input, including the ‘no other event’ paddings.

[0267] To further illustrate Delphi-2M’s capabilities of modeling temporal dependencies it is noted that for some diseases such as cancers, the influence on mortality decays with a half-life of several years reflecting the sustained risks of recurrence or impacts of treatment (Fig. 16d).

[0268] For septicaemia, however, the influence on mortality is much more short-lived and drops sharply, effectively recovering to values close to the population average. This inference agrees with traditional Nelson-Aalen analyses of the hazard rates. This behaviour is also reflected by Delphi-2M’s attention maps, which show that cancer tokens are attended to for long periods, while that of septicaemia, myocardial infarction and many other diseases tends to be short-lived (Fig. 17a to 17f).

[0269] Fig. 17a to Fig. 17f show attention maps according to an embodiment of the present disclosure. The darker the color, the higher the value.

[0270] Fig. 17a to Fig. 17f show the attention for 6 selected trajectories representative of the token highlighted on the y-axis. Shown is the maximum attention across the 12 layers and 12 heads utilised by Delphi-2M. Darker values indicate higher attention (range 0-1). Typically, the last (token, age) pair is most strongly attended to after which attention drops. Sex and selected other inputs are attended for longer periods. Rows without labels are “No other event” tokens.

[0271] Fig. 18a to 18d show an external validation using Danish registries according to an embodiment of the present disclosure. Fig. 18a to Fig. 18d show color coded results. A colored version of Fig. 18a to Fig. 18d is provided at the time of filing.

[0272] Fig. 18a shows ROC-AUC results of Delphi for each token with at least 25 occurrences coloured by the respective ICD-10 chapter. Predictions are based on Danish adults (50-80 yrs. of age) with data up until 1. Jan. 2016 and evaluated on incidence between 1. Jan. 2017 - 1. Jan. 2018. Below are the respective differences of AUC values between Delphi and an age-sex baseline based on Danish data from 2010-2016.

[0273] Fig. 18b shows a comparison between AUC values in the UKB longitudinal validation and the Danish validation. Correlation is measured as Pearson’s rho.

[0274] Fig. 18c shows a comparison between delta AUC values in the UKB longitudinal validation and the Danish validation with their respective age-sex baselines. Correlation is measured as pearson’s roh.

[0275] Fig. 18d shows calibration plots on the Danish data (similar to Fig. 18a) split by each ICD10 chapter. Each line represents an ICD10 disease evaluated for each decentile of the Delphi rate and compared against the observed rate in the population.

[0276] In the above it has been shown that Delphi-2M accurately reproduces health trajectories and the underlying complex dependencies of health and disease tokens as recorded in the UK Biobank. However, there is a risk that such patterns are specific to the particular training data and do not generalize to data sets from other populations.

[0277] While the UK Biobank cohort has been recruited according to epidemiological best practices, due to the age of recruitment and the volunteer nature, the demographics are not entirely representative of the general population of the UK30. Furthermore, there may be differences in the availability of health-related data and how it is recorded between different healthcare systems. To assess whether Delphi-2M’s inference generalizes to data from another country, an external validation using Danish registry data has been performed. For this purpose, Delphi-2M has been transferred with the learned weights from the UK Biobank training and evaluated predictions on the Danish data, hence, no retraining or any adjustments have been made.

[0278] A cross-section of the entire Danish population aged 50-80 years on 1. Jan 2016 is used and Delphi-2M’s predictions on the incidence of the corresponding population are evaluated between 1. Jan 2017 and 1. Jan 2018, therefore, enforcing a 1-year data gap to also assess prognostication.

[0279] Data are collected for 1.93M individuals (51 % females and 49% males) with 11.51 M disease tokens, recorded between 1978 and 2016. Predictions are evaluated on 0.96M disease tokens across 771 ICD10 codes (each with at least 25 cases).

[0280] The average ROC-AUC when Delphi-2M is applied to Danish data was 0.674 (s.d. =0.087) with similar performance across most chapters. The AUC difference to age-sex baselines were on average 0.068 (s.d. =0.084) points higher for Delphi-2M with 83% of all diseases showing a signal above the baseline (Figure 5a).

[0281] When compared to the longitudinal validation in the UK Biobank, AUC values are similar with a correlation of 0.78 (95% Cl: 0.710-0.792) (Fig. 18b, and Fig. 12a to 12c) Further, the AUC differences over an agesex baseline remain comparable across the two cohorts with a correlation of 0.546 (95% Cl: 0.476-0.608) (Fig. 18c).

[0282] Comparing the observed rate of individual diseases with the rate predicted by Delphi-2M across each decile shows good overall calibration (Fig. 18d). However, there is a noticeable underestimation, especially for mortality, which will be discussed in the below. Overall, Delphi-2M’s predictions transfer well across two healthcare systems, without any reconfiguration, and show considerable prognostic performance across most diseases.

[0283] While Al models can learn diverse and complex patterns, they may also replicate biases that exist in the underlying training data. Delphi-2M is no exception to this and its reduced accuracy on Danish registries suggests a range of biases in the UK Biobank.

[0284] Epidemiological cohorts often only approximately reflect the general population due to differential response rates in different subpopulations. The UK Biobank comprises more white British citizens than other ethnicities and participants tend to be, on average, more affluent and educated than the general population30.

[0285] While Delphi-2M accurately replicates the observed rates of disease tokens in UK Biobank training and validation data, there are more pronounced differences to those recorded in Danish registries, even though a correlation of 0.7 clearly remains. On average incidence differed by a factor of 3 with a slight tendency of underreporting in the UK Biobank in line with a healthy volunteer bias.

[0286] Other biases arise from the use of retrospective data in a cohort. In the context of the UK Biobank, most individuals have been recruited between 40-50 years of age. This creates a type of immortality bias as no deaths are recorded prior to recruitment.

[0287] This has direct implications on estimated mortality, and indirectly also on the incidences of diseases associated with high mortality, such as cancers, as only survivors are included in the UK Biobank. For timedependent analyses, the jump of mortality to non-zero values at recruitment can also lead to false attribution of the apparent increase to unrelated variables recorded at the time of recruitment.

[0288] Furthermore, the heterogeneous sources of UK Biobank’s first occurrences data and their missingness patterns introduce artificial correlations. The first occurrence data has been collated from self-reports, primary care, hospital admissions, cancer and death registries where each source contributes characteristic disease tokens, but may be missing in a subset of individuals or cover only specific periods in time. Selfreporting and GP records contain mostly common diseases, with self-reporting covering 78% of participants, while GP records were only available for just 45% of participants.

[0289] Data from hospital records covered 86% participants and contributed more aggressive disease tokens, such as myocardial infarction or septicaemia, similar to the death registry. Lastly, cancers were provided by the national cancer registry.

[0290] Delphi-2M can learn patterns of missingness and use them to predict future rates accordingly - even though this concerns data availability rather than true incidence. Accordingly, the predicted rates of diseases which are specific to hospital records are on average 10x higher in individuals which have a history including other hospital records.

[0291] Septicaemia, for example, is diagnosed in 93% of cases in a hospital setting and is predicted to occur at 8x greater rates in individuals with any other hospital data. Similarly, the predicted rates for primary care linked diseases are on average 3x higher in participants with other primary care data.

[0292] For example, acute sinusitis is almost exclusively (99%) found in primary care data and is predicted to occur at 3.5x greater rate in individuals with any linked primary care data. While some of these associations may reflect true diagnostic pathways linked to a particular setting, it nevertheless appears that many of these associations are artefacts stemming from the incomplete aggregated nature of UK Biobank’s data.

[0293] Another consideration of healthcare models is whether predictions are made with similar accuracy in different population groups. The predicted rates of observed diseases are largely comparable between the two sexes, however it is lower for individuals self-reporting a “White” ethnic background compared to any other ethnic group. This is likely a consequence of participants of “White” ethnic background having lower numbers of disease tokens across time, which reduces the ability to predict future disease burden. Similarly, Delphi-2M’s predicted rate of disease is highest in the top quintile of Townsend deprivation, which mirrors the average number of tokens per individual in this group. Together these observations indicate that Delphi-2M’s predictions reflect the differential burden of disease seen across different population subgroups.

[0294] An embodiment of the present disclosure, as discussed above, is Delphi-2M, a GPT-based model of multidisease progression. Delphi-2M extends the GPT large language model to account for the temporal nature of health trajectory data. Analogous to LLMs, which learn the grammar and contextual logic of language from large bodies of text, Delphi-2M inferred the patterns of multi-disease progression when trained on data of more than 1 ,000 diseases and baseline health information recorded for 400k UK Biobank participants.

[0295] For this training corpus a, model size of approximately 2M parameters appeared optimal with bigger data sets poised to yield even greater accuracy with larger models.

[0296] A detailed assessment of Delphi-2M’s predictions shows that they faithfully recapitulate the patterns of disease occurrence at the population scale as recorded in UK Biobank.

[0297] Such patterns are largely dominated by age and sex, but also prior health data. For the majority of diseases Delphi-2M’s multi-disease state model predicted future rates at comparable accuracy as established singledisease risk models.

[0298] Furthermore, Delphi-2M achieves very good performance when applied to data from Danish disease registries without additional finetuning, demonstrating that models are largely applicable across national healthcare systems.

[0299] Delphi-2M is uniquely capable of sampling future disease trajectories, which enables estimating cumulative disease burdens over periods of up to 20 years, conditional on prior health information. It is worth noting that Delphi’s predictions are generally strongly influenced by statistical chance and compatible with a range of outcomes for a given individual.

[0300] The ability to generate synthetic data may also help create data sets that preserve the statistical cooccurrence patterns without revealing any specific data, which could facilitate the development of further Al models with a decreased risk of revealing personal information.

[0301] Delphi-2M’s underlying GPT-2 model offers insights into the modes of disease progression. The ability to cluster disease risks may be useful for genomic association studies that focus on comorbidities or are stratified by the risks derived from health trajectories. Delphi-2M’s capability to quantify the temporal influence of previous health data revealed that cancers increase mortality in a sustained fashion, while the effects of myocardial infarction or septicaemia regressed within 5 years. Similar analyses also revealed clusters of persisting co-morbidities such as mental health conditions, which informs healthcare planning.

[0302] A further promising feature of Delphi-2M’s implementation is the relative simplicity with which additional data layers may be incorporated in future extensions. Examples of this are the simple additive nature of age encoding or how lifestyle and sex tokens are superimposed on the embeddings of disease history information.

[0303] Immediate refinements of Delphi-2M may incorporate additional lifestyle data, prescription records and blood tests, both usually available in a general healthcare setting. Further multi-modal extensions could include genomic data, richer metabolomic information, or data from wearables can be added to Delphi-2M’s embedding layer similar to how lifestyle tokens are currently incorporated.

[0304] Such multi-modal extensions will shed light on how different layers of biology interface in shaping health outcomes and thus help address long-standing questions in biomedicine. Lastly, while ICD-10 provides a predefined tokenization of diseases, LLMs have been shown to also conceptualize natural language making it plausible to expect that future models may derive similar meaning directly from free text records.

[0305] These considerations illustrate the wide range of applications of generative models for biomedical research and, ultimately, also for healthcare. The ability to model instantaneous and cumulative risks harbour large potential for health economics and healthcare itself.

[0306] Risk models are important to efficient cancer screening and to enable preventative medicine. The ability to generate health trajectories may be used to project future health burdens, which will help allocate resources that can be scarce in ageing populations.

[0307] Future model extensions that also incorporate treatments and outcomes may be used to directly assess treatment effects with respect to prior health data and other information including genomics with utility for the individual and society.

[0308] Methods

[0309] In the above discussion, a number of methods have been discussed. For better intelligibility of the disclosure the methods are summarized and discussed hereinafter. All of the features disclosed hereinafter should be considered optional features of all embodiments of the present disclosure, unless indicated otherwise.

[0310] Data: UK Biobank Cohort: The UK Biobank is a cohort-based prospective study comprising approximately 500,000 individuals from various demographic backgrounds recruited across the UK between 2006 and 2010. At the time of recruitment, individuals were between the ages of 37 and 73, see [Ref 31],

[0311] Disease first occurrence data. The main data source for health-related outcomes is built on the first occurrence data assembled in category 1712 from the UK Biobank. These data include ICD10-level 3 codes (e.g., E11 : Type 2 diabetes mellitus) for diseases in chapters l-XVIl excluding chapter II - neoplasms (1256 in total) plus death.

[0312] The data is pre-assembled by UK Biobank and includes the first reported occurrence of a disease in either the linked primary care data (cat. 3000), inpatient hospital admissions (cat. 2000), death registry (fields: 40001 , 40002) or self-reported data through questionnaires (field: 20002).

[0313] Information on neoplasms was not included in category 1712 by the UK Biobank, hence said was included the data through addition of the linked cancer registry data in fields 40005 and 40006 (subset to first occurrence and mapped to ICD10 level 3 codes).

[0314] Lifestyle and demographics. Information is extracted on the self-reported sex of participants as recorded in field 31 (indicators for female and male), a physical assessment of body mass index at recruitment from field 21001 which is split into 3 indicators encoding bmi < 22 kg / m2, bmi > 28 kg / m2, and otherwise, as well as smoking behaviour from field 1239 with indicators for smoker (UKB coding: 1), occasionally smoking (2) and never smoker (0) and alcohol intake frequency from field 1558 with indicators for daily (1), moderately (2, 3) and limited (4, 5, 6).

[0315] Further, information that is extracted and used for stratification to assess model performance in subgroups but were not part of the data for model training include self-reported ethnic background as available in field 21000 participants grouped into 5 level groups (White, Mixed, Asian or Asian British, Black or Black British and Chinese) and an index of multiple deprivation as available in field 26410.

[0316] The index combines information across seven domains including Income, Employment Derivation, Health and Disability, Education Skills and Training, Barriers to Housing and Services, Living Environment, and Crime.

[0317] Additionally, information is extracted on diastolic blood pressure (field: 4079), systolic blood pressure (field: 4080), high density lipoprotein - hdl (field: 30760), total cholesterol (field: 30690) and hypertensive medication use (field: 20003 - any mentioned of: enalapril, lisinopril, perindopril, ramipril, candesartan, irbesartan, losartan, valsartan, olmesartan, amlodipine, felodipine, nifedipine, diltiazem, verapamil, indapamide, bendroflumethiazide, atenolol, bisoprolol, doxazosin, amiloride, spironolactone) which are used for the Framingham score evaluation. Further, information on years of education (field: 845), dementia of parents (fields: 20107, 20110) as well as living alone (field: 709) were extracted for the UKBDRS evaluation.

[0318] Data: Danish registries

[0319] Cohort. Exploring comorbidities and health-related factors is uniquely facilitated by Denmark's comprehensive registries, which gather up to 40 years of interconnected data from across the entire population.

[0320] All of the used registries are linkable through a unique personal identification number provided in the Central Person Registry along with information on sex and date of birth. Further, the Danish National Patient Registry, see [Ref32], was utilized, a nationwide longitudinal register with data on hospital admissions across all of Denmark since 1977 along with the Danish Register of Causes of Death, see [Ref 33] since 1970, to extract information on an individual’s acquired diagnoses throughout their lifetime.

[0321] The current data extract covers information up until around 2019 when reporting to the LPR was updated to LPR3. Further, the cohort is restricted to individuals aged 50-80 on 1st of January 2016, to obtain a similar age range as in the UK Biobank. The 1 st of January 2016 was chosen as the cutoff point as it is the latest time point for which reliable coverage may be guaranteed across the entire population over the entire prediction horizon.

[0322] Feature adjustments. In order to obtain a dataset that resembles the UK Biobank data only the first occurrence of an individual’s diagnosis is retained and all codes are transformed to ICD10-level 3 codes. Diagnoses before 1995 are reported in ICD8 and have been converted to ICD10 codes using published mappings, see [Ref34],

[0323] Codes that may be present in the Danish register but were not in the UK Biobank are removed. Information on lifestyle is not available, hence indicators for BMI, smoking and alcohol intake have been treated as absent.

[0324] Data splits in UK Biobank. The models were trained on UK Biobank data for 402,786 (80%) individuals using data from birth until 30th of June 2020. For horizontal validation, data contains the remaining 100,636 (20%) individuals for the same period.

[0325] Internal longitudinal validation was carried out using data for all individuals still alive by the cutoff date (471 ,057) and evaluated on incidence from 1st of July 2021 to 1st of July 2022, therefore enforcing a 1- year data gap between predictions and evaluation.

[0326] Horizontal validation assesses how well the model generalized to different individuals from the same cohort. Longitudinal validation investigates whether the model’s performance changes over time and if it can be used for prognostic purposes. Data splits in Denmark. External longitudinal validation was conducted on the Danish registries. All individuals residing in Denmark aged 50-80 years on the 1st of January 2016 were included. Predictions are based on the available data up to this point and subsequently evaluated on incidence from 1st of January 2017 - 1st of January 2018 similar to the internal longitudinal validation.

[0327] GPT model. Delphi’s architecture is based on GPT-2, see [Ref 25], as implemented in github.com / nanoGPT. The basic GPT model uses standard transformer blocks with causal self-attention. A standard lookup table embedding layer with positional encoding was used to obtain the embeddings.

[0328] The embedding and casual self-attention layers are followed by layer normalisation and a fully connected feedforward network. Transformer layers, consisting of causal self-attention and feedforward blocks, are repeated multiple times before the final linear projection that yields the logits of the token predictions.

[0329] The residual connections within a transformer layer are identical to those in the original GPT implementation. A weight tying of the token embeddings and final layer weights is used , which has the advantage of reducing the number of parameters and that input and output embeddings can be similarly interpreted.

[0330] Data representation and padding tokens. Each data point consists of pairs (token, age) recording the token value and the proband’s age, measured in days from birth, at which the token was recorded. The token vocabulary consists of n=1257 different ICD-10 level 3 disease tokens, plus n=9 tokens for alcohol, smoking and BMI, each represented by 3 different levels, as well as n=2 tokens for sex and n=1 no-event padding token as well as n=1 additional, non-informative padding token at the beginning or end of the input sequences.

[0331] No-event padding tokens were added to the data with a constant rate of 1 / 5 years by uniformly sampling 20 tokens from the range of (0, 36525) and interleaving those with the data tokens after intersecting with the data range for each person.

[0332] Sex tokens were presented at birth. Smoking, alcohol and lifestyle were recorded at the enrollment into the UK Biobank. As this specific time also coincided with the end of immortal time bias (probands had to be alive when they were recruited), smoking alcohol and BMI tokens times were randomized by -20 to +40 years from this point in time to break an otherwise confounding correlation leading to a sudden jump in mortality rates (and possibly other diseases with high mortality such as cancers) associated with the recording of these tokens.

[0333] Age encoding. Delphi replaces GPT’s positional encoding with an encoding based on the age values. Following the logic frequently used for positional encodings, age is represented by sine and cosine functions of different frequencies, where the lowest frequency is given by 1 / (100*365), corresponding to a linear and constant basis function each. These functions are subsequently linearly combined by a trainable linear transformation, which enables the model to share the same basis function across multiple encoding dimensions. Another advantage of using age encoding is that Delphi can handle token inputs of arbitrary length, as no parameters are associated with token positions.

[0334] Causal self-attention. Standard causal self-attention enables the GPT model to attend to all preceding tokens. For sequential data, these are found to the left of the token sequence. Yet, in the case of timedependent data, tokens can be recorded at the same time with no specified order. Thus, attention masks were amended to mask positions that occurred at the same time as the predicted token. Non-informative padding tokens were masked for predictions of other tokens.

[0335] Exponential waiting time model. The input data to Delphi are bivariate pairs (j, t) of the next token class and the time to the next token. Delphi is motivated by the theory of competing exponentials. Let T be the waiting times from the current event to one of / = 1 ,...,n competing events, where n is the number of predictable tokens. Assuming the T are each exponentially distributed waiting times with rates exp(Ai), The next event being j is equivalent to Tj being the first of the competing waiting times, ie Tj = min 71, or equivalently j = argmin 71 It can be shown that the corresponding probability is P(j = argmin T) = exp(Aj) I Ziexp(Ai), which is the softmax function over the vector A. Conveniently, this definition corresponds to the classical crossentropy model for classification with A = logits. Hence Delphi uses a conventional loss term for token classification: loss_j = - log p( j ) = -cross_entropy( logits, tokens)

[0336] Furthermore, in the competing exponential model the time to the next event T* = min{7j} is also exponentially distributed with logarithmic rate A* = log(Zi exp(Ai)), which is the logsumexp function of the logarithmic rates A. The loss function of exponential waiting times T between tokens is simply given by: loss_T = - log p(T) = - ( logsumexp( logits) - sum(exp( logits) ) * T) .

[0337] These approximations hold as long as the log rates Ai are constant in time, which is a reasonable assumption over short periods. For this reason, padding tokens were introduced to ensure that waiting times are modelled over a relatively short period, which does not exceed 5 years in expectation. In line with the tie-breaking logic used for causal self-attention, co-occurring events were predicted from the last non- co-occurring token each.

[0338] Loss function. The total loss of the model is then given by: loss = loss_j + loss_T . Non-informative padding, as well as sex, alcohol, smoking, and BMI were considered mere input tokens and thus removed from the loss terms above. This was achieved by setting their logits to -Inf and by evaluating the loss terms only on disease and “no event” padding tokens.

[0339] Sampling procedure. The next disease event is sampled by sampling the disease token and the time until the next event. The disease token is sampled from the distribution that originates from applying softmax to logits. For sampling the time, samples from all exponential distributions with rates exp(Ai) are taken, and the minimum is taken. Logits of non-disease tokens (sex, lifestyle) are discarded from the procedure to sample disease events only.

[0340] Models were trained by stochastic gradient optimization using the Adam optimizer with standard parameters for 200k iterations. The batch size was 128. After 1000 iterations of warmup, the learning rate was decayed using a cosine scheduler from 6e-4 to 6e-5. 32-bit float precision was used.

[0341] Modelled incidence. In the exponential waiting time definition above, the logits of the model correspond to the log probabilities of daily occurrences of each token. The probability of an event occurring within a year is given by P(T < 365.25) = 1 - exp(exp(lambda) * 365.25).

[0342] Age- and sex-stratified incidence. For the training set age- and sex-stratified incidences were calculated in annual age brackets. The observed counts were divided by the number of individuals at risk in each age and sex bracket, which was given by the number of probands for each sex minus the cumulative number of deaths to account for censoring.

[0343] Model calibration. Calibration curves were calculated based on predicted incidences. To this end, all cases of a given token accruing in five year age bins were identified. Subsequently for all other probands a control data point was randomly selected in the same age band.

[0344] Predictions were evaluated at the preceding token given that the time difference was less than a year. The predicted incidences were then further grouped log-linearly into risk bins from 10-6to 1 with multiplicative increments of logio(5) . The observed annual incidence was then calculated as the average of cases and control in age bins, divided by 5yr. The procedure was separately executed for each sex.

[0345] Relative risks. The relative risk of each prediction was calculated as the ratio of the modelled and yearly incidence to the age- and sex stratified incidence of the same token in the year in which the prediction was made. A summary was calculated as the geometric mean of each relative risk.

[0346] Incidence cross-entropy. To compare the distribution of annual incidences of model and observed data a cross-entropy metric was used. Let pi be the annual occurrence of token i in each year. Hence the age- and sex-based entropy across tokens is is given H(p,q) = - p * log(q) - (1-p) * log(1-q). For low incidences p,q the latter term is usually small. The cross-entropy is evaluated across all age groups and sexes. Generated trajectories. To evaluate the potential of generating disease trajectories two experiments were conducted using data from the validation cohort.

[0347] First, trajectories were generated from birth using only sex tokens. This was used to assess whether Delphi- 2M recapitulates the overall sex-specific incidence patterns.

[0348] Second, all available data until the age of 60 was used to simulate subsequent trajectories conditional on the previous health information. A single trajectory was evaluated per proband. Trajectories were truncated after the age of 80 as currently little training data was available. Incidence patterns were evaluated as described above.

[0349] Study design. To validate the predictions of the model a longitudinal validation is performed, internally for the UK Biobank data and externally on the Danish health registries. This has two advantages (i) one may enforce an explicit cutoff and separate data to avoid any potential time-leakage (ii) one may obtain insights into Delphi-2M prognostic capabilities.

[0350] As mentioned in the data splits two different cutoff dates are used between the two data sources, mainly due to differential data availability, however, the principle set-up applies to both in the exact same way.

[0351] Data up to a specific cutoff data is collated for each individual and use Delphi-2M to predict an individual’s future rate across all disease tokens. Building on the exponential waiting time representation rates are obtained over a 1 year time frame. The preceding year after the cutoff date is discarded to introduce a data gap. Subsequently the incidence in the next year is used for evaluation.

[0352] Algorithms for comparison. As the task of predicting all disease outcomes simultaneously is novel there is no other method to compare against. Therefore, a standard epidemiological baseline is build based on the sex- and age-stratified population rates.

[0353] These are based on the Nelson-Aalen estimator, see [Ref35] and [Ref 36], a nonparametric estimator of the cumulative hazard rate, across all diseases. For the UK Biobank the estimators are based on the same training data as Delphi-2M and for the Danish registries the entire Danish population is used in the time period from 2010-2016.

[0354] As the UK Biobank contains a wide range of phenotypic measures clinically established models were estimated for comparison. These include the Framingham score as implemented in R-CVrisk for cardiovascular disease, the Charlson comorbidity index for mortality implemented in R-comorbidity and the UKBDRS risk score for dementia based on our own implementation as reported in the original paper.

[0355] The Charlson comorbidity index is based on the same data as Delphi, however, the Charlson comorbidity index is originally based on ICD10-level 4 codes. Therefore, two versions are estimated , (i) that uses the level 3 codes and (ii) a version that maps the level 3 codes to all possible level 4 codes and the version is retained that shows the best performance.

[0356] Additionally, different versions of the comorbidity index are evaluated, (i) original Charlson, see [Ref 28], (ii) Charlson with weights by Quan et al., see [Ref 6], (iii) Elixhauser, see [Ref 37], and (iv) the Swiss Elixhauser, see [Ref 38], and again retain the best performing one.

[0357] The Framingham score is based on the 2008 version, see [Ref 7], with lab measurements. If covariates are missing, a simple mean imputation within sex has been used. If multiple measurements are available, the most recent one is retained. For individuals older than 74, their Framingham score at age 74 is used as the current implementation does not allow for older age groups.

[0358] Diabetes is classified as a diagnostic code in ICD10: E10-14. The assessed outcomes was cardiovascular disease (CVD) containing a diagnosis in either ischaemic heart disease (ICD10: I20-25), Heart failure or cardiac arrest (ICD10: I46, I50) or stroke (ICD10: I63, I64).

[0359] Last, the UKBDRS score is used to evaluate dementia as a diagnosis in (ICD10: FOO, F01 , F03, G30, G31). The risk score is build following the instructions in the paper as closely as possible. For years of education values based on the reported mean value in the original paper (13.54) are input. Again as only 3rd level ICD10 codes are retained minor adjustments have been made, precisely, depression (ICD10: F32, F33), stroke (ICD10: I60, 161 , I63, I64, G45), high cholesterol (ICD10: E77-78) and hypertension (ICD10: 110-15 or hypertensive drug usage as constructed for the Framingham score).

[0360] Performance measures. To assess the discriminatory power of the predicted rates, the area under the receiver operating curve (ROC-AUC) as implemented in python-scikit-learn is used. All diseases with at least 25 cases were assessed.

[0361] Calibration of predicted rates. Further, the predicted rates from Delphi-2M are compared to the observed incidence to determine calibration of the predicted rates using python-scikit-learn. Delphi-2M predicted rates are split into deciles and for each bin Delphi-2M’s average rate is compared against the observed rate within the bin. All diseases with at least 25 cases are included.

[0362] Token embedding UMAP. The low-dimensional representation of token space was constructed by applying the UMAP, see Ref

[0039] , dimensionality reduction algorithm to the learned token embeddings for Delphi-2M (1270x120 matrix). The cosine metric was used.

[0363] SHAP. To evaluate the influence of each token in a trajectory on the next predicted token, the SHAP, see [Ref 29] methodology has been adopted. Each trajectory from the validation cohort was augmented by masking one or several tokens and then used for prediction. The change of logits after many such augmentation were aggregated by a PartitionExplainer from the SHAP Python package. Masking procedure. The number of augmentations for each trajectory was determined by the PartitionExplainer masking algorithm. When masked, tokens were replaced by a “no event” placeholder that was also used during training. Sex tokens when masked were replaced with the corresponding token of the opposite sex.

[0364] SHAP values evaluation. The described procedure was applied to each of 100,636 trajectories in the validation cohort. The predicted token was always the last available token in the trajectory.

[0365] Nonparametric hazard ratios. To complement the SHAP analysis and the assessment of Delphi-2M’s modelling of time-dependent effects are performed and evaluation based on the Nelson-Aalen estimator. For a given token, individuals are identified with the token and estimate their corresponding cumulative hazard from the occurrence of the token onwards.

[0366] Additionally, 5 age-sex matched individuals are randomly selected for each case and estimate the cumulative hazard in this comparison group. One may then obtain an estimate of the hazard rate by taking the derivative of the cumulative hazard. A gaussian kernel is applied to acquire a smooth estimate. Subsequently, one may take the ratio of the two hazards and obtain a crude nonparametric estimate for the hazard ratio of the token over time.

[0367] While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not by way of limitation. Likewise, the various diagrams may depict an example architectural or configuration, which are provided to enable persons of ordinary skill in the art to understand exemplary features and functions of the present disclosure. Such persons would understand, however, that the present disclosure is not restricted to the illustrated example architectures or configurations, but can be implemented using a variety of alternative architectures and configurations. Additionally, as would be understood by persons of ordinary skill in the art, one or more features of one embodiment can be combined with one or more features of another embodiment described herein. Thus, the breadth and scope of the present disclosure should not be limited by any one of the above-described exemplary embodiments.

[0368] It is also understood that any reference to an element herein using a designation such as "first," "second," and so forth does not generally limit the quantity or order of those elements. Rather, these designations can be used herein as a convenient means of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not mean that only two elements can be employed, or that the first element must precede the second element in some manner.

[0369] Additionally, a person having ordinary skill in the art would understand that information and signals can be represented using any one of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits and symbols, for example, which may be referenced in the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. A skilled person would further appreciate that any one of the various illustrative logical blocks, units, processors, means, circuits, methods and functions described in connection with the aspects disclosed herein can be implemented by electronic hardware (e.g., a digital implementation, an analog implementation, or a combination of the two), firmware, various forms of program or design code incorporating instructions (which can be referred to herein, for convenience, as "software" or a "software unit”), or any combination of these techniques.

[0370] To clearly illustrate this interchangeability of hardware, firmware and software, various illustrative components, blocks, units, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware, firmware or software, or a combination of these techniques, depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in various ways for each particular application, but such implementation decisions do not cause a departure from the scope of the present disclosure. In accordance with various embodiments, a processor, device, component, circuit, structure, machine, unit, etc. can be configured to perform one or more of the functions described herein. The term “configured to” or “configured for” as used herein with respect to a specified operation or function refers to a processor, device, component, circuit, structure, machine, unit, etc. that is physically constructed, programmed and / or arranged to perform the specified operation or function.

[0371] In this document, the term "unit" as used herein, refers to software, firmware, hardware, and any combination of these elements for performing the associated functions described herein. Additionally, for purpose of discussion, the various units are described as discrete units; however, as would be apparent to one of ordinary skill in the art, two or more units may be combined to form a single unit that performs the associated functions according to embodiments of the present disclosure.

[0372] Additionally, memory or other storage, as well as communication components, may be employed in embodiments of the present disclosure. It will be appreciated that, for clarity purposes, the above description has described embodiments of the present disclosure with reference to different functional units and processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processing logic elements or domains may be used without detracting from the present disclosure. For example, functionality illustrated to be performed by separate processing logic elements, or controllers, may be performed by the same processing logic element, or controller. Hence, references to specific functional units are only references to a suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization.

[0373] All methods described herein may relate to computer implemented methods. Their respective steps may be implemented as mentioned above, e.g. by a processor executing the respective steps.

[0374] Various modifications to the implementations described in this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the scope of this disclosure. Thus, the disclosure is not intended to be limited to the implementations shown herein, but is to be accorded the widest scope consistent with the novel features and principles disclosed herein, as recited in the claims below.

[0375] The present disclosure is further defined by the following items:

[0376] Item 1. Method for determining of a next health event data from a health information dataset based on a machine learning model, wherein the machine learning model is based on a transformer and comprises an additional output head for predicting a time to a next token, wherein the transformer comprises causal attention masks, wherein the health information dataset comprises a sequence of health event data and corresponding time data, and wherein the method comprises: a) tokenizing and embedding the sequence of health event data; b) embedding the corresponding time data based on an age-dependent encoding; c) combining the embedding of the corresponding time data and the embedding of the tokenized health event data into an input embedding; d) providing the input embedding to a trained transformer; wherein inside the transformer only information looking back not forward is processed based on the causal attention masks; e) receiving an output embedding from the trained transformer; f) determining a next token from the output embedding; g) determining a respective time-to-the-next-token from the output embedding based on the additional output head; and h) determining the next health event data from the next token and the time-to-the-next- token.

[0377] Item 2. Method according to item 1 , wherein the transformer is a GPT-based transformer.

[0378] Item 3. Method according to item 1 or 2, wherein a weight tying between the token embeddings and logit weights is used.

[0379] Item 4. Method according to any one of items 1 to 3, wherein the tokenizing is based on a token vocabulary consisting of one or more of: i) a diagnosis code token set, preferably n=1257 different ICD-10 level 3 disease tokens; ii) a lifestyle token set, preferably n=9 tokens for alcohol, smoking and BMI, each represented by 3 different levels; iii) a sex token set, preferably n=2 tokens for sex; iv) a healthy padding token, preferably n=1 no-event padding token; v) a non-informative padding token at the beginning or end of the sequence of health data, preferably n=1 additional non-informative padding token.

[0380] Item 5. Method according to any one of items 1 to 4, wherein during the tokenizing step healthy padding tokens are added to the sequence of health data, and wherein preferably the healthy padding tokens are randomly added at a predetermined rate, preferably at a rate of 1 in 5 years.

[0381] Item 6. Method according to item 5, wherein the no-event padding tokens are randomly added at a predetermined rate, preferably at a rate of 1 / 5 years.

[0382] Item 7. Method according to any one of items 1 to 6, wherein the age-dependent encoding is based on sine and cosine base functions.

[0383] Item 8. A method for training a transformer model for prediction of a next health event data from a health information dataset, preferably for a method according to any one of items 1 to 7, based on a health information database, comprising the steps of: a) retrieving a plurality of health data sets and corresponding time information data from the health information database; c) splitting the data points into training data points, preferably 80% thereof, and validation data points, preferably 20% thereof; d) training the transformer model by stochastic gradient optimization based on the training data points and validation data points, wherein during the training the total loss of the transformer model is the sum of a next disease event prediction loss function based on softmax and a time to next event loss function based on an exponential loss function defined by: loss = loss_j + loss_T .

[0384] Item 9. Method according to item 8, wherein loss_ j is defined as loss_j = - log p( j ) = -cross_entropy( logits, tokens) , and wherein loss_T is defined as loss_T = - log p(T) = - ( logsumexp( logits) - sum(exp( logits) ) * T) .

[0385] Item 10. Method according to item 8 or item 9, wherein the transformer model is trained using an Adam optimizer, preferably for 200k iterations, preferably for a batch size of 128, preferably decaying the learning rate using a cosine function from 6e-4 to 6e-5. Item 11 . A method for health trajectory prediction, the method comprising iteratively performing the steps of the method according to any one of items 1 to 7 for determining a plurality of next health event data as the health trajectory prediction; and outputting the health trajectory prediction.

[0386] Item 12. A data processing system comprising means for carrying out the steps of the method according to any one of items 1 to 7 and / or items 8 to 10 and / or item 11 .

[0387] Item 13. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any one of items 1 to 7 and / or items 8 to 10 and / or item 11.

[0388] Item 14. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of items 1 to 7 and / or items 8 to 10 and / or item 11 .

[0389] List of References

[0390] The above disclosure refers to the following documents. The content of said reference is incorporated by reference. In particular the definitions of terms, which are used in the above disclosure, and which are found in the referenced documents are incorporated by reference.

[0391] [Ref 1] Nyberg, S. T. et al. Association of healthy lifestyle with years lived without major chronic diseases. JAMA Intern. Med. 180, 760-768 (2020).

[0392] [Ref 2] Link, B. G. & Phelan, J. Social conditions as fundamental causes of disease. J. Health Soc. Behav. Spec No, 80-94 (1995).

[0393] [Ref 3] Zhu, Z. et al. Causal associations between risk factors and common diseases inferred from GWAS summary data. Nat. Commun. 9, 224 (2018).

[0394] [Ref 4] Mukama, T. et al. Risk-Adapted Starting Age of Screening for Relatives of Patients With Breast Cancer. JAMA Oncol 6, 68-74 (2020).

[0395] [Ref 5] Tian, Y. et al. Calculating the Starting Age for Screening in Relatives of Patients With Colorectal Cancer Based on Data From Large Nationwide Data Sets. Gastroenterology 159, 159-168. e3 (2020).

[0396] [Ref 6] Quan, H. et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am. J. Epidemiol. 173, 676-682 (2011).

[0397] [Ref 7] D’Agostino, R. B., Sr et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study: The Framingham heart study. Circulation 117, 743-753 (2008). [Ref 8] Hippisley-Cox, J. & Coupland, C. Development and validation of risk prediction algorithms to estimate future risk of common cancers in men and women: prospective cohort study. BMJ Open 5, e007825 (2015).

[0398] [Ref 9] Bray, F. et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. (2024) doi:10.3322 / caac.21834.

[0399] [Ref 10] Watt, T. et al. Health in 2040: projected patterns of illness in England. The Health Foundation (2023).

[0400] [Ref 11] Jung, A. W. & Gerstung, M. Bayesian Cox regression for large-scale inference with applications to electronic health records, aoas 17, 1064-1085 (2023).

[0401] [Ref 12] Vaswani, A. et al. Attention Is All You Need. arXiv [cs.CL] Preprint at http: / / arxiv.org / absZ1706.03762 (2017).

[0402] [Ref 13] Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877-1901 (2020).

[0403] [Ref 14] Gemini Team et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv [cs.CL] (2023).

[0404] [Ref 16] Touvron, H. et al. LLaMA: Open and Efficient Foundation Language Models. arXiv [cs.CL] (2023).

[0405] [Ref 16] Ouyang, L. et al. Training language models to follow instructions with human feedback. arXiv [cs.CL] (2022).

[0406] [Ref 17] OpenAI et al. GPT-4 Technical Report. arXiv [cs.CL] (2023).

[0407] [Ref 18] Rasmy, L., Xiang, Y., Xie, Z., Tao, C. & Zhi, D. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digit Med 4, 86 (2021).

[0408] [Ref 19] Li, Y. et al. BEHRT: Transformer for electronic health records. Sci. Rep. 10, 7155 (2020).

[0409] [Ref 20] Li, Y. et al. Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records. IEEE J. Biomed. Health Inform. 27, 1106-1117 (2023).

[0410] [Ref 21] Yang, Z., Mitra, A., Liu, W., Berlowitz, D. & Yu, H. TransformEHR: transformer-based encoderdecoder generative model to enhance prediction of disease outcomes using electronic health records. Nat. Commun. 14, 7857 (2023).

[0411] [Ref 22] Placido, D. et al. A deep learning algorithm to predict risk of pancreatic cancer from disease trajectories. Nat. Med. 29, 1113-1122 (2023).

[0412] [Ref 23] Savcisens, G. et al. Using sequences of life-events to predict human lives. Nat. Comput. Sci. 1- 14 (2023).

[0413] [Ref 24] Kraljevic, Z. et al. Foresight — a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study. Lancet Digit. Health 6, e281-e290 (2024).

[0414] [Ref 25] Radford, A. et al. Language Models are Unsupervised Multitask Learners. Preprint at https: / / d4mucfpksywv.cloudfront.net / better-language-models / language-models.pdf.

[0415] [Ref 26] Hoffmann, J. et al. Training Compute-Optimal Large Language Models. arXiv [cs.CL] (2022). [Ref 27] Anaturk, M. et al. Development and validation of a dementia risk score in the UK Biobank and Whitehall II cohorts. BMJ Ment Health 26, (2023).

[0416] [Ref 28] Charlson, M. E., Pompei, P., Ales, K. L. & MacKenzie, C. R. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J. Chronic Dis. 40, 373-383 (1987).

[0417] [Ref 29] Lundberg, S. M. & Lee, S.-l. A Unified Approach to Interpreting Model Predictions, in Advances in Neural Information Processing Systems (eds. Guyon, I. et al.) vol. 30 (Curran Associates, Inc., 2017).

[0418] [Ref 30] Fry, A. et al. Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants With Those of the General Population. Am. J. Epidemiol. 186, 1026-1034 (2017).

[0419] [Ref 31 ] Sudlow, C. et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 12, e1001779 (2015).

[0420] [Ref 32] Schmidt, M. et al. The Danish National Patient Registry: a review of content, data quality, and research potential. Clin. Epidemiol. 7, 449-490 (2015).

[0421] [Ref 33] Helweg-Larsen, K. The Danish Register of Causes of Death. Scand. J. Public Health 39, 26-29 (2011).

[0422] [Ref 34] Pedersen, M. K. et al. A unidirectional mapping of ICD-8 to ICD-10 codes, for harmonized longitudinal analysis of diseases. Eur. J. Epidemiol. 38, 1043-1052 (2023).

[0423] [Ref 35] Aalen, O. Nonparametric Inference for a Family of Counting Processes. Ann. Stat. 6, 701-726 (1978).

[0424] [Ref 36] Nelson, W. Theory and applications of hazard plotting for censored failure data. Technometrics 14, 945 (1972).

[0425] [Ref 37] van Walraven, C., Austin, P. C., Jennings, A., Quan, H. & Forster, A. J. A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data. Med. Care 47, 626-633 (2009).

[0426] [Ref 38] Sharma, N., Schwendimann, R., Endrich, O., Ausserhofer, D. & Simon, M. Comparing Charlson and Elixhauser comorbidity indices with different weightings to predict in-hospital mortality: an analysis of national inpatient data. BMC Health Serv. Res. 21 , 13 (2021).

[0427] [Ref 39] Mclnnes, L., Healy, J. & Melville, J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv [stat. ML] (2018).

Claims

CLAIMS1 . Method for determining of a next health event data on a health trajectory from a health information dataset based on a machine learning model, wherein the machine learning model is based on a transformer and comprises an additional output head configured to predict a time to a next token, wherein the transformer comprises causal attention masks, wherein the health information dataset comprises a sequence of health event data and corresponding time data, and wherein the method comprises: a) tokenizing and embedding the sequence of health event data; b) embedding the corresponding time data based on an age-dependent encoding; c) combining the embedding of the corresponding time data and the embedding of the tokenized health event data into a single data structure forming an input embedding; d) providing the input embedding to the transformer; wherein the causal attention masks comprised in the transformer are configured such that only information looking back not forward in time is processed ; ; e) receiving an output embedding, based on processing the input embedding with the transformer; f) determining a next token from the output embedding; g) determining a respective time-to-the-next-token from the output embedding based on the additional output head; and h) determining the next health event data from the next token and the time-to-the-next- token.

2. Method according to claim 1 , wherein the additional output head is configured to predict a time to a next token using an exponential waiting time model.

3. Method according to claim 1 or 2, wherein the age-dependent encoding is based on an agedependent embedding layer and configured to encode the input times from a continuous time axis, preferably using sine and cosine basis functions.

4. Method according to any one of claims 1 to 3, wherein the causal attention masks are configured to resolve ties of tokens recorded at the same time.

5. Method according to any one of claims 1 to 4, wherein the transformer is a GPT-based transformer.

6. Method according to any one of claims 1 to 5,wherein a weight tying between the token embeddings and logit weights is used.

7. Method according to any one of claims 1 to 6, wherein the tokenizing is based on a token vocabulary consisting of one or more of: i) a diagnosis code token set, preferably n=1257 different ICD-10 level 3 disease tokens; ii) a lifestyle token set, preferably n=9 tokens for alcohol, smoking and BMI, each represented by 3 different levels; iii) a sex token set, preferably n=2 tokens for sex; iv) a healthy padding token, preferably n=1 no-event padding token; v) a non-informative padding token at the beginning or end of the sequence of health data, preferably n=1 additional non-informative padding token.

8. Method according to any one of claims 1 to 7, wherein during the tokenizing step healthy padding tokens are added to the sequence of health data, and wherein preferably the healthy padding tokens are randomly added at a predetermined rate, preferably at a rate of 1 in 5 years.

9. Method according to claim 8, wherein the no-event padding tokens are randomly added at a predetermined rate, preferably at a rate of 1 / 5 years.

10. Method according to any one of claims 1 to 9, wherein the age-dependent encoding is based on sine and cosine base functions.

11. A method for training a transformer model for prediction of a next health event data from a health information dataset, preferably for a method according to any one of claims 1 to 10, based on a health information database, comprising the steps of: a) retrieving a plurality of health data sets and corresponding time information data from the health information database; c) splitting the data points into training data points, preferably 80% thereof, and validation data points, preferably 20% thereof; d) training the transformer model by stochastic gradient optimization based on the training data points and validation data points, wherein during the training the total loss of the transformer model is the sum of a next disease event prediction loss function based on softmax and a time to next event loss function based on an exponential loss function defined by: loss = loss_j + loss_T .

12. Method according to claim 11 , wherein loss_ j is defined asloss_j = - log p( j ) = -cross_entropy( logits, tokens) , and wherein loss_T is defined as loss_T = - log p(T) = - ( logsumexp( logits) - sum(exp( logits) ) * T) .

13. Method according to claim 11 or claim 12, wherein the transformer model is trained using an Adam optimizer, preferably for 200k iterations, preferably for a batch size of 128, preferably decaying the learning rate using a cosine function from 6e-4 to 6e-5.

14. A method for health trajectory prediction, the method comprising iteratively performing the steps of the method according to any one of claims 1 to 10 for determining a plurality of next health event data as the health trajectory prediction; and outputting the health trajectory prediction.

15. A data processing system comprising means for carrying out the steps of the method according to any one of claims 1 to 10 and / or claims 11 to 13 and / or claim 14.

16. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 10 and / or claims 11 to 13 and / or claim 14.

17. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 10 and / or claims 11 to 13 and / or claim 14.

Images