Space-time attention for clinical outcome prediction

Abstract

A disease prognosis model is provided that can be trained to determine a clinical outcome of a disease based on longitudinal data including health records for each point in time in a sequence of points in time. The disease prognosis model may include a recurrent neural network trained to extract, from each health record, a set of features representing local dependencies present within the health record. The disease prognosis model may include spatio-temporal attention trained to determine an importance of each feature in the set of features at each point in time in the sequence of points in time. The disease prognosis model may include a feed-forward neural network trained to determine the clinical outcome of the disease based on the importance of each feature in the set of features at each point in time in the sequence of points in time. A trained disease prognosis model may be applied to determine the clinical outcome of the disease for one or more patients.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 376,957, entitled “Spatial-Temporal Attention for Clinical Outcome Prediction,” filed on September 23, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter described herein relates generally to machine learning, and more particularly to machine learning-based prognostic models for predicting clinical outcomes of diseases. Background Art

[0004] Many diseases can cause serious long-term complications. For example, infection with the novel coronavirus SARS-CoV-2 can lead to coronavirus disease (COVID-19) with a clinical syndrome including cough, headache, fever, etc. Among patients infected with SARS-CoV-2, a considerable number will experience persistent post-infection sequelae. Patients with COVID-19 may present with prolonged COVID-19 symptoms, such as fatigue, dyspnea, and memory problems, lasting for at least two months after the initial acute infection. In some cases, symptoms associated with COVID-19 may appear, recur, and linger for months or even years after the initial acute infection. In the most severe cases, chronic COVID-19 symptoms may even be life-threatening. Therefore, identifying groups at high risk for serious long-term disease complications can facilitate treatment planning and allocation of medical resources. Summary of the invention

[0005] Systems, methods, and articles of manufacture (including computer program products) for machine learning-enabled prediction of clinical outcomes are provided. In one aspect, a system for machine learning-enabled prediction of clinical outcomes is provided. The system may include at least one processor and at least one memory. The at least one memory may include program code that, when executed by at least one processor, provides operations. The operation may include: training a disease prognosis model to determine a clinical outcome of a disease based at least on longitudinal data, the longitudinal data including a health record for each time point in a sequence of time points, the training of the disease prognosis model including training a recurrent neural network, spatiotemporal attention, and a feedforward neural network, the recurrent neural network being trained to extract a feature set representing one or more local dependencies present within the health record from each health record, the spatiotemporal attention being trained to determine the importance of each feature in the feature set at each time point in the sequence of time points, and the feedforward neural network being trained to determine the clinical outcome of the disease based at least on the importance of each feature in the feature set at each time point in the sequence of time points; and applying the trained disease prognosis model to determine the clinical outcome of the disease for a patient associated with the first health record and the second health record based at least on a first health record from a first time point and a second health record from a second time point.

[0006] In another aspect, a method for machine learning-enabled prediction of clinical outcomes is provided. The method may include: training a disease prognosis model to determine a clinical outcome of a disease based at least on longitudinal data, the longitudinal data including a health record for each time point in a sequence of time points, the training of the disease prognosis model including training a recurrent neural network trained to extract a feature set representing one or more local dependencies present in the health record from each health record, the spatiotemporal attention trained to determine the importance of each feature in the feature set at each time point in the sequence of time points, and the feedforward neural network trained to determine the clinical outcome of the disease based at least on the importance of each feature in the feature set at each time point in the sequence of time points; and applying the trained disease prognosis model to determine the clinical outcome of the disease for a patient associated with the first health record and the second health record based at least on a first health record from a first time point and a second health record from a second time point.

[0007] In another aspect, a computer program product for machine learning-enabled prediction of clinical outcomes is provided. The computer program product may include a non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, cause operations. The operations may include: training a disease prognosis model to determine a clinical outcome of a disease based at least on longitudinal data, the longitudinal data including a health record for each time point in a sequence of time points, the training of the disease prognosis model including training a recurrent neural network, a spatiotemporal attention, and a feedforward neural network, the recurrent neural network being trained to extract a feature set representing one or more local dependencies present within the health record from each health record, the spatiotemporal attention being trained to determine the importance of each feature in the feature set at each time point in the sequence of time points, and the feedforward neural network being trained to determine the clinical outcome of the disease based at least on the importance of each feature in the feature set at each time point in the sequence of time points; and applying the trained disease prognosis model to determine the clinical outcome of the disease for a patient associated with the first health record and the second health record based at least on a first health record from a first time point and a second health record from a second time point.

[0008] In some variations of the methods, systems, and non-transitory computer-readable media, one or more of the following features may optionally be included in any feasible combination.

[0009] In some embodiments, the recurrent neural network can be a bidirectional recurrent neural network (RNN), a long short-term memory (LSTM) network, a local long short-term memory (LSTM) network with a given window size for a time point, or a gated recurrent unit (GRU) network.

[0010] In some variations, the feedforward neural network may be a multilayer perceptron model.

[0011] In some embodiments, the trained disease prognosis model can determine the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least applying the trained recurrent neural network to extract from the first health record a first set of feature values ​​for a hidden feature set representing a first set of local dependencies present within the first health record, and applying the trained recurrent neural network to extract from the second health record a second set of feature values ​​for the hidden feature set representing a second set of local dependencies present within the second health record.

[0012] In some embodiments, the trained recurrent neural network can output a feature map for trained spatiotemporal attention ingestion, the feature map comprising the first set of feature values ​​from the first time point and the second set of feature values ​​from the second time point.

[0013] In some embodiments, the trained disease prognosis model can further determine the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least applying the trained spatiotemporal attention to determine the importance of each feature in the hidden feature set at each of the first time point and the second time point based at least on a feature map including the first set of feature values ​​and the second set of feature values.

[0014] In some variations, the trained spatiotemporal attention may include one or more two-dimensional convolutional layers trained to determine the importance of each feature in the set of hidden features across a temporal dimension and a feature dimension.

[0015] In some variations, the one or more two-dimensional convolutional layers may include a 1×1 convolutional filter configured to jointly weight the importance of each feature in the hidden feature set across the time dimension and the feature dimension.

[0016] In some variations, the trained spatiotemporal attention may determine, for a first feature from the set of hidden features, a first importance of the first feature at the first point in time and a second importance of the first feature at the second point in time.

[0017] In some variations, the trained spatiotemporal attention may further determine, for a second feature from the set of hidden features, a third importance of the second feature at the first time point and a fourth importance of the second feature at the second time point.

[0018] In some embodiments, the trained disease prognosis model can further determine the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least applying the trained feed-forward neural network to determine the clinical outcome of the disease for the patient based at least on the importance of each feature in the hidden feature set at each of the first time point and the second time point.

[0019] In some variations, the disease prognosis model can be further trained to determine the clinical outcome of the disease based on non-longitudinal data. The non-longitudinal data can be static across the sequence of time points. The non-longitudinal data can be concatenated with the health record associated with each time point in the sequence of time points.

[0020] In some variations, one or more missing values ​​for a non-longitudinal variable constituting the non-longitudinal data may be identified. The one or more missing values ​​may be replaced with the mean value of the non-longitudinal variable observed in the available data set.

[0021] In some variations, the non-longitudinal data may include medical image data and / or electrogram data corresponding to metrics quantifying the severity of the disease depicted in one or more medical images and / or electrograms.

[0022] In some variations, the health record associated with each time point in the sequence of time points may include a value for each of a plurality of vital sign statistics.

[0023] In some variations, the health record associated with each time point in the sequence of time points includes values ​​for each of a plurality of laboratory test variables.

[0024] In some variations, the health record associated with each time point in the sequence of time points may include medical image data and / or electrogram data. The medical image data includes metrics quantifying the severity of the disease depicted in one or more medical images and / or electrograms.

[0025] In some variations, the longitudinal data may be determined to include a missing value at a first time point for a longitudinal variable. The missing value may be replaced with (i) a first value of the longitudinal variable from a second time point before the first time point, (ii) a second value of the longitudinal variable from a third time point after the first time point, or (iii) a third value determined based on the first value and the second value.

[0026] In some variations, the disease can be coronavirus disease (COVID-19), Alzheimer's disease, or age-related macular degeneration.

[0027] In some variations, the clinical outcome of the disease may include a probability associated with one or more of cure, progression, and death.

[0028] Specific implementations of the current subject matter may include, but are not limited to, methods consistent with the description provided herein and articles including tangibly embodied machine-readable media that are operable to cause one or more machines (e.g., computers, etc.) to cause operations that implement one or more of the features described. Similarly, a computer system that may include one or more processors and one or more memories coupled to the one or more processors is also described. A memory that may include a non-transitory computer-readable or machine-readable storage medium may include, encode, store, etc., one or more programs that cause one or more processors to perform one or more of the operations described herein. A computer-implemented method consistent with one or more implementations of the current subject matter may be implemented by one or more data processors present in a single computing system or multiple computing systems. Such multiple computing systems may be connected and may exchange data and / or commands or other instructions, etc., via one or more connections, including, for example, via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.) via a direct connection between one or more computing systems in the multiple computing systems, etc.

[0029] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will become apparent by reference to the description and drawings, and to the claims. Although certain features of the presently disclosed subject matter are described in the context of post-acute sequelae of COVID-19 for illustrative purposes related to the prediction of clinical outcomes, it should be readily understood that such features are not intended to be limiting. The claims following this disclosure are intended to define the scope of the protected subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed implementations.

[0031] Figure 1 depicts a system diagram showing an example of a prognostic system according to some exemplary embodiments;

[0032] Figure 2 depicts a schematic diagram showing an example of computing spatiotemporal attention across a time dimension and a feature dimension according to some exemplary embodiments;

[0033] Figure 3A depicts a schematic diagram showing an example of a disease prognosis model according to some exemplary embodiments;

[0034] Figure 3Bdepicts a schematic diagram showing an example of a disease prognosis model according to some exemplary embodiments;

[0035] Figure 4A depicts a flow chart showing an example of a process for training a disease prognosis model for clinical outcome prediction according to some exemplary embodiments;

[0036] Figure 4B depicts a flow chart showing an example of a process for machine learning enabled clinical outcome prediction according to some exemplary embodiments;

[0037] Figure 4C depicts a flow chart showing an example of a process for machine learning enabled clinical outcome prediction according to some exemplary embodiments;

[0038] Figure 5A depicts a chart showing an example of classifying disease severity of a patient based on Acute Physiology and Chronic Health Evaluation II (APACHE II) scores over time according to some exemplary embodiments;

[0039] Figure 5B depicts a chart showing an example of breaking down a patient's Acute Physiology and Chronic Health Evaluation II (APACHE II) score by different physiological variables according to some exemplary embodiments;

[0040] Figure 5C depicts a table showing exemplary outputs of spatiotemporal attention according to some exemplary embodiments; and

[0041] Figure 6 Depicted is a block diagram illustrating an example of a computing system in accordance with some example embodiments.

[0042] When applicable, like reference numerals refer to like structures, features, or elements. DETAILED DESCRIPTION

[0043] Identifying groups at high risk for severe long-term morbidity can benefit treatment planning and allocation of medical resources. In the case of COVID-19, identifying patients at high risk for severe long-term complications may be crucial for timely medical intervention. Accurate prognosis of long-term disease outcomes (such as prolonged morbidity and mortality) requires comprehensive assessment of multimodal data, including static non-longitudinal data (such as the initial condition of the patient at the onset of the disease) and longitudinal data that track disease progression. However, due to the heterogeneous phenotypes and chronic conditions presented by patients, it may be difficult to predict patient outcomes from longitudinal data (such as a series of electronic health records obtained at different time points). In particular, longitudinal data exhibit a combination of short-term and long-term dependencies that circumvent conventional methods of prognostication of disease outcomes. For example, in patients hospitalized for COVID-19 pneumonia, fibrotic-like abnormalities that are common during the first three months will mostly disappear over a year, while consolidation will usually dissipate within six months. Therefore, the characteristic importance of fibrosis should not only change over time, but also be different from the characteristic importance of consolidation in terms of the possibility of COVID-19 and prolonged and / or recurrent symptoms. However, conventional methods for disease outcome prognosis may either identify time-dependent feature importance in the absence of feature diversity or provide spatial feature importance that is static over time.

[0044] Thus, in some exemplary embodiments, a disease prognosis model for determining the clinical outcome of a disease may include a spatiotemporal attention mechanism that is capable of jointly weighting feature importance from the time dimension and feature space of longitudinal data. Examples of longitudinal data include medical data in the form of a health record (e.g., an electronic health record (EHR)) for each time point in a sequence of continuous time points. As used herein, the term "health record" may refer to various types of data across multiple modalities. For example, each health record may include values ​​for each of a plurality of vital statistics data (such as systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, etc.). Alternatively and / or in addition, each health record may include values ​​for each of a plurality of laboratory test variables. Examples of laboratory test variables may include fibrinogen, C-reactive protein, prothrombin international normalized ratio, prothrombin time, lactate dehydrogenase, D-dimer, albumin, ferritin, alanine aminotransferase, aspartate aminotransferase, chloride, protein, alkaline phosphatase, bilirubin, calcium, creatinine, glucose, hematocrit, hemoglobin, potassium, platelets, red blood cells, sodium, white blood cells, etc. In some cases, the health record associated with each time point in the sequence of time points may include medical image data associated with: X-ray, magnetic resonance imaging (MRI) scan, computed tomography (CT) scan, positron emission tomography (PET) scan, optical coherence tomography (OCT) scan, etc. In addition, in some cases, each health record may also include one or more electrogram data associated with: electroencephalogram (EEG), electrocorticogram (ECoG or iEEG), electrooculogram (EOG), electroretinogram (ERG), electrocardiogram (ECG), electromyogram (EMG), etc.

[0045] In some exemplary embodiments, the disease prognosis model may include a recurrent neural network and a feedforward neural network coupled with a spatiotemporal attention mechanism. For example, a recurrent neural network may be trained to extract a feature set representing one or more local dependencies present in the health record from each health record included in the longitudinal data. Spatiotemporal attention may be trained to determine the importance of each feature in the feature set at each time point in the time point sequence. In addition, a feedforward neural network may be trained to determine the clinical outcome of the disease based on the importance of each feature in the feature set at each time point in the time point sequence. The combination of a recurrent neural network and spatiotemporal attention enables the disease prognosis model to effectively capture short-term and long-term dependencies present in longitudinal data such as a series of health records (e.g., electronic health records) from different time points. In particular, the trained disease prognosis model can be applied to identify key time points and feature patterns to determine the clinical outcomes of the disease, such as, for example, the probability of cure, persistence, recurrence, deterioration and / or death for COVID-19.

[0046] Figure 1 A system diagram illustrating an example of a prognostic system 100 according to some exemplary embodiments is depicted. Figure 1 , the prognosis system 100 may include a prognosis engine 110, a client device 120, and a data storage area 130. Figure 1 As shown, the prognosis engine 110, the client device 120, and the data storage area 130 can be communicatively coupled via a network 140. The client device 120 can be a processor-based device, including, for example, a workstation, a desktop computer, a laptop computer, a smart phone, a tablet computer, a wearable device, etc. The data storage area 130 can be a relational database, a non-structured query language (NoSQL) database, an in-memory database, a graph database, a key-value store, a document store, etc. The network 140 can be a wired network and / or a wireless network, including, for example, a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), a public land mobile network (PLMN), the Internet, etc.

[0047] In some exemplary embodiments, the prognostic engine 110 may apply the disease prognostic model 115 to the longitudinal data 133 from the data store 130 to determine a clinical outcome of a disease for a patient associated with the longitudinal data 133. For example, in some cases, the longitudinal data 133 may include a sequence {x i |x1,x2,…,x T}, where T represents the length of the sequence (or the number of time points), and x i In some cases, x i The longitudinal data 133 may be a health record (e.g., an electronic health record (EHR)) represented as a vector, a matrix, or a tensor. Thus, the longitudinal data 133 may include, for example, a first health record x1 of a patient from a first time point t1 and a second health record x2 of the patient from a second time point t2. The disease prognosis model 115 may be trained based on the sequence {x i |x1,x2,…,x T}Determine a clinical outcome of a disease for a patient associated with the longitudinal data 133, the clinical outcome including, for example, cure, persistence, relapse, deterioration, and / or death. In some cases, the clinical outcome prediction performed by the disease prognosis model 115 can be formulated as a sequence to a single problem. For example, the disease prognosis model 115 can extract a feature set (e.g., hidden features) from the longitudinal data 133 that represents short-term dependencies present in the longitudinal data 133. In addition, the disease prognosis model 115 can determine the importance of each feature in the feature set for each time point in the longitudinal data 133. The clinical outcome of the disease can be determined based on the importance of each feature in the feature set for each time point in the longitudinal data 133.

[0048] Since various features (e.g., disease symptoms) exist at different time points, the above sequence to a single problem for disease outcome prediction may be challenging. In some cases, features (e.g., disease symptoms) present at an earlier time point may be related to features present at a later time point. Although accurate prognosis requires an overall analysis of feature aspect information and temporal information, existing outcome prediction methods can consider either time-dependent feature importance or spatial feature importance, but not both at the same time. In contrast, various implementations of the disease prognosis model 115 disclosed herein include spatiotemporal attention 200 that is trained to jointly weight feature importance on the time axis and feature space.

[0049] Figure 2 A schematic diagram illustrating an example of a spatiotemporal attention 200 for computing feature importance across a time dimension and a feature dimension according to some exemplary embodiments is depicted. Figure 2 As shown, the feature importance of feature f at any specific time point can be based on the value of each feature at each time point f ij To calculate. In some cases, the spatiotemporal attention 200 can be implemented as one or more two-dimensional convolutional layers that are trained to use convolutional filters (e.g., 1×1 convolutional filters) to calculate the key, query, and value for each feature extracted from the longitudinal data 133 to calculate the comparison score as the feature importance. The benefit of using 1×1 convolutional filters for attention calculation is that the feature importance is jointly weighted from two dimensions including the time dimension and the feature dimension. Equation (1) below shows that the features extracted from the longitudinal data 133 (e.g., hidden features) can be adjusted by the spatiotemporal attention 200 using the weighting factor γ.

[0050] f′=f+γ·a(f) (1)

[0051] where f represents the hidden features extracted from the longitudinal data 133 (in a manner described in more detail below), and f′ represents the adjusted value of each feature determined by the spatiotemporal attention a(·).

[0052] The following equation (2) shows the calculation of the spatiotemporal attention a(·).

[0053]

[0054] Where H represents the size of the feature space extracted from the longitudinal data, and T represents the number of time points in the longitudinal data 133. As shown in equation (2), the feature importance of a given feature f is determined based on other features {f ij |i=1,2,…,H;j=1,2,…,T}. In some cases, feature importance may be measured by computing the alignment score via a key k(), value v(), and query q() operation implemented by a convolutional filter (eg, a 1×1 convolutional filter).

[0055] As noted, the disease prognosis model 115 can extract a feature set (e.g., hidden features) from the longitudinal data 133 that represents short-term dependencies present within the longitudinal data 133. In some cases, the disease prognosis model 115 can include a recurrent neural network (RNN) that is trained to extract a feature set from the longitudinal data 133 before applying the spatiotemporal attention 200 to determine the importance of each feature for each time point in the longitudinal data 133. Figures 3A to 3B A schematic diagram is depicted showing an example of a disease prognosis model 115 in which spatiotemporal attention 200 is integrated into a recurrent neural network 300. In some cases, the recurrent neural network 300 can be a bidirectional recurrent neural network (RNN), a long short-term memory (LSTM) network, a local long short-term memory (LSTM) network with a given window size for a time point, a gated recurrent unit (GRU) network, and the like.

[0056] refer to Figure 3A In some cases, the recurrent neural network 310 can be a long short-term memory (LSTM) network trained to extract short-term and long-term dependencies from the longitudinal data 133 to form a feature map 325. The size of the feature map 325 can be N×H, where N corresponds to the number of stacked recurrent layers and H corresponds to the number of features in the hidden state of the LSTM network. The spatiotemporal attention 200 can operate on the feature map 325 to learn the correlation between the H number of features across the T number of time points. Figure 3AAs shown, the spatiotemporal attention 200 can output an adjusted feature map for ingestion by the feedforward neural network 350, which indicates the importance of each of the H number of features at each of the T number of time points. For example, in some cases, the adjusted feature map can indicate the first importance of the first feature at the first time point and the second importance of the first feature at the second time point. In addition, in some cases, the adjusted feature map can indicate the third importance of the second feature at the first time point and the fourth importance of the second feature at the second time point. The feedforward neural network 350 (which can be a multilayer perceptron model in some cases) can operate on the adjusted feature map to determine the clinical outcome of the disease.

[0057] While spatiotemporal attention 200 may be proficient in learning long-term dependencies, spatiotemporal attention may lack the ability to model short-term (or local) dependencies in order. Some attention-based models, such as transformers, rely on positional embeddings to encode the order of short-term dependencies. However, unlike images or sentences in which the order of elements has contextual meaning, health records (e.g., electronic health records (EHRs)) included in the longitudinal data 133 are not strictly ordered. For example, a patient may undergo a laboratory test before undergoing a medical imaging examination, and vice versa. Therefore, positional embeddings are not suitable for encoding the order of short-term dependencies present in the longitudinal data 133. Nevertheless, long short-term memory (LSTM) networks do not enforce strict ordering, at least because when the local order undergoes changes, the signals stored in the memory cells can still propagate. In order to sort out the learning of short-term and long-term dependencies, the learning of short-term dependencies can be restricted to a set of local long short-term memory (LSTM) networks, and the learning of long-term dependencies can be restricted to spatiotemporal attention 200. Figure 3B An example of a disease prognosis model 115 is shown in FIG, where the recurrent neural network 115 is implemented as a set of local long short-term memory (LSTM) networks. The local long short-term memory (LSTM) network is limited to learning sequence patterns existing within a specific window size and extracting corresponding local patterns as hidden features.

[0058] Reference again Figure 3B, after concatenating the hidden states from each local LSTM network, the stacked hidden states can become a matrix containing T number of vectors of length H. The spatiotemporal attention 200 then refines the hidden states by mining feature-wise and long-term dependencies before: outputting a set of adjusted hidden states for ingestion by a feedforward neural network 350 (e.g., a multilayer perceptron model) to determine clinical outcomes. In some cases, the set of local LSTM networks and the spatiotemporal attention 200 can form an R-transformer, which is different from the transformer model used in computer vision and natural language processing, in which position information is encoded using position embedding.

[0059] In some exemplary embodiments, the prognostic engine 110 may apply the disease prognostic model 115 to the combination of the longitudinal data 133 and the non-longitudinal data in order to determine a clinical outcome of a disease for a patient associated with the longitudinal data 133 and the non-longitudinal data 135. In this context, the non-longitudinal data 135 may include data whose values ​​remain fixed across different time points. Examples of the non-longitudinal data 135 may include demographic information, medical history (e.g., pre-existing conditions such as hypertension, obesity, hyperlipidemia, diabetes, etc.), medical image data (e.g., a grading of disease severity depicted in one or more medical images), and / or electrogram data (e.g., a grading of disease severity indicated by one or more electrograms). The prognostic engine 110 may combine the longitudinal data 133 with the non-longitudinal data 135 by at least concatenating the non-longitudinal data 135 with the longitudinal data 133 from each time point. For example, the non-longitudinal data 135 associated with a patient may be concatenated with a first health record x1 from a first time point t1 and a second health record x2 from a second time point t2.

[0060] In some exemplary embodiments, the prognostic engine 110 may preprocess the longitudinal data 133 and / or the non-longitudinal data 135 before applying the disease prognostic model 115. For example, preprocessing the longitudinal data 133 and / or the non-longitudinal data 135 may include normalizing one or more of the values ​​present therein. For at least some medical images, the preprocessing performed by the prognostic engine 110 may include determining a metric that quantifies the severity of the disease depicted in the medical image. For example, in the case of a chest X-ray, the prognostic engine 110 may calculate a radiological assessment of pulmonary edema (RALE) score to characterize the severity of the disease of acute respiratory distress syndrome (ARDS) in patients who are positive for COVID-19. For numerical variables, the prognostic engine 110 may apply minimum-maximum normalization to display the value of each variable in the same range (e.g., a range from zero to one). The value of a binary variable present in the longitudinal data 133 and / or the non-longitudinal data 135 may be presented as one of two values ​​(e.g., 0 and 1).

[0061] In some cases, the longitudinal data 133 and the non-longitudinal data 135 may exhibit sparsity, where values ​​for one or more variables are missing. Therefore, in some exemplary embodiments, preprocessing of the longitudinal data 133 and / or the non-longitudinal data 135 may include excluding from further analysis one or more variables that are available for less than a threshold number of patients (e.g., 95% of the patients). In addition, in the case where the prognostic engine 110 detects that the patient's non-longitudinal data 135 includes one or more missing values ​​for a non-longitudinal variable (e.g., a variable with a static value across time points), the prognostic engine 110 may perform mean filling, in which the one or more missing values ​​are replaced with the average value of the variable observed in the available data set (e.g., the average value of the variable associated with other patients). In the case where the prognostic engine 110 detects that the patient's longitudinal data 133 includes missing values ​​for a longitudinal variable (e.g., a variable with a fluctuating value across time points) at a first time point, the prognostic engine 110 may perform forward filling, in which the missing value is replaced with the first value of the longitudinal variable from a second time point prior to the first time point. Alternatively, the prognostic engine 110 can perform backward filling, where the missing value is replaced with a second value of the longitudinal variable from a third time point after the first time point. In some cases, the missing value for the longitudinal variable at the first time point can be interpolated based on the first value of the longitudinal variable from the second time point before the first time point and the second value of the longitudinal variable from the third time point after the first time point.

[0062] Figure 4A A flow chart illustrating an example of a process 400 for training a disease prognosis model 115 for clinical outcome prediction according to some exemplary embodiments is depicted. Figures 1 to 2, 3A to 3B and 4A, the process 400 can be performed, for example, by the prognosis engine 110 to train the disease prognosis model 115.

[0063] At 402, the prognosis engine 110 may train the disease prognosis model 115 by at least training the recurrent neural network 300, the spatiotemporal attention 200, and the feedforward neural network 350 included in the disease prognosis model 115. For example, in some exemplary embodiments, the disease prognosis model 115 may be trained to determine a clinical outcome for a disease of a patient based at least on the longitudinal data 133 of the patient. Figures 3A to 3B As shown, the disease prognosis model 115 may include a recurrent neural network 330, a spatiotemporal attention 200, and a feedforward neural network 350. In addition, the longitudinal data 133 may include a corresponding health record for each time point in the time point sequence. Therefore, training the disease prognosis model 115 may include training the recurrent neural network 330, the spatiotemporal attention 200, and the feedforward neural network 350. For example, the recurrent neural network 330 may be trained to extract a feature set representing one or more local dependencies present in the health record from each health record included in the longitudinal data 133. The spatiotemporal attention 200 may be trained to determine the importance of each feature in the feature set at each time point in the time point sequence. The feedforward neural network 350 may be trained to determine the clinical outcome of the disease based on the importance of each feature in the feature set at each time point in the time point sequence.

[0064] At 404, the prognostic engine 110 may apply the trained disease prognostic model 115 to determine a clinical outcome of a disease for one or more patients. In some exemplary embodiments, the trained disease prognostic model 115 may be applied to determine clinical outcomes of various diseases including, for example, coronavirus disease (COVID-19), Alzheimer's disease, age-related macular degeneration, etc. The output of the trained disease prognostic model 115 may include a probability associated with one or more of cure, persistence, relapse, deterioration, and death as a clinical outcome of the disease.

[0065] Figure 4B A flow chart illustrating an example of a process 430 for enabling machine learning clinical outcome prediction according to some exemplary embodiments is depicted. Figures 1 to 2 , 3A to 3B and 4A to 4B, process 430 may be performed by the prognosis engine 110 and may be implemented Figure 4A Operation 404 of process 400 is shown.

[0066] At 432, the prognosis engine 110 may receive the patient's longitudinal data 133. For example, in some exemplary embodiments, the prognosis engine 110 may receive a first health record of the patient at a first time point and a second health record of the patient at a second time point as part of the patient's longitudinal data 133. In some cases, the patient's longitudinal data 133 may be combined with the patient's non-longitudinal data 135. The patient's non-longitudinal data 135 may include one or more non-longitudinal variables whose values ​​remain static at consecutive time points, while the patient's longitudinal data 133 may include one or more longitudinal variables whose values ​​fluctuate at consecutive time points. Therefore, the combination of the patient's longitudinal data 133 and the non-longitudinal data 135 may include concatenating the value of the longitudinal variable at each time point (e.g., included in the health record associated with each time point) with the value of the non-longitudinal variable.

[0067] At 434, the prognosis engine 110 may apply the trained disease prognosis model 115 to determine a clinical outcome for the patient's disease based at least on the patient's longitudinal data 133. For example, in some exemplary embodiments, the trained disease prognosis model 115 may be applied to determine a clinical outcome for the patient's disease based at least on a first health record of the patient at a first time point and a second health record of the patient at a second time point. The disease prognosis model 115 including the spatiotemporal attention 200 integrated into the recurrent neural network 300 may be able to identify short-term as well as long-term dependencies present in the longitudinal data 133. In particular, the recurrent neural network 300 may be trained to extract various features that represent short-term dependencies present in the longitudinal data 133, while the spatiotemporal attention 200 may be trained to recognize changes in the feature importance of each feature across different time points. Doing so may enable the disease prognosis model 115 to generate an accurate prognosis for the clinical outcome of the patient's disease.

[0068] Figure 4C A flow chart illustrating an example of a process 450 for enabling machine learning-based clinical outcome prediction according to some exemplary embodiments is depicted. Figures 1 to 2 , 3A to 3B and 4A to 4C, process 450 can be performed by the disease prognosis model 115 and can be implemented Figure 4B Operation 434 of operation 430 is shown.

[0069] At 452, the disease prognosis model 115 can apply the recurrent neural network 300 to generate a feature map, which includes a first set of feature values ​​for the hidden feature set (representing a first set of local dependencies existing in a first health record of a patient from a first time point) and a second set of feature values ​​for the hidden feature set (representing a second set of local dependencies existing in a second health record of a patient from a second time point). In some exemplary embodiments, the recurrent neural network 300 can be trained to extract local (or short-term) dependencies existing in, for example, the following: i |x1,x2,…,x T} from the first time point t1 and the second health record x2 from the second time point t2. Figures 3A to 3B As shown, the recurrent neural network 300 can extract H features (e.g., hidden features) from the first health record x1 and the second health record x2. Figures 3A to 3B As further shown, the output of the recurrent neural network 300 may be a feature map 325 of size N×H. For example, the feature map 325 may include a corresponding vector of length H for each health record from a number of time points in T. In the case where the recurrent neural network 300 is implemented as a local long short-term memory (LSTM) network, the recurrent neural network 300 may be limited to learning sequential patterns that exist within a specific window size and extracting corresponding local patterns as hidden features.

[0070] At 454, the disease prognosis model 115 may apply the spatiotemporal attention 200 to determine the importance of each feature at each of the first time point and the second time point based at least on the feature map. In some exemplary embodiments, the spatiotemporal attention 200 may operate on the feature map 325 to determine the correlation between H number of features across T number of time points and output a corresponding adjusted feature map indicating the importance of each of the H number of features at each of the T number of time points. For example, the spatiotemporal attention 200 may determine the first importance of the first feature at the first time point t1 and the second importance of the first feature at the second time point t2. In addition, the spatiotemporal attention 200 may determine the third importance of the second feature at the first time point t1 and the fourth importance of the second feature at the second time point t2.

[0071] As noted, the importance of different features can change over time, which is consistent with clinical observations such as in COVID-19 patients, in which fibrotic abnormalities that are common during the first three months will mostly disappear after a year, while consolidation will typically dissipate within six months. Therefore, the adjusted feature graph can include adjusted feature values ​​for each of the first feature and the second feature. For example, the value of the first feature at the first time point t1 and the second time point t2 can be adjusted based on the first importance of the first feature at the first time point t1 and the second importance of the first feature at the second time point t2, respectively. The values ​​of the second feature at the first time point t1 and the second time point t2 can be adjusted based on the third importance of the second feature at the first time point t1 and the fourth importance of the second feature at the second time point t2, respectively.

[0072] At 456, the disease prognosis model 115 can apply the feedforward neural network 350 to determine the clinical outcome of the disease for the patient based on at least the importance of each feature at each of the first time point and the second time point. In some exemplary embodiments, the disease prognosis model 115 can apply the feedforward neural network 350, which can operate on the adjusted feature map to determine the clinical outcome of the disease for the patient. For example, the feedforward neural network 350 can be implemented as a multilayer perceptron model. In addition, the feedforward neural network 350 operating on the adjusted feature map can determine the clinical outcome of the disease for the patient based on at least the importance of each feature at different time points in the longitudinal data 133.

[0073] In some exemplary embodiments, the performance of the disease prognosis model 115 in predicting the clinical outcomes of COVID-19 is evaluated based on data associated with a group of 365 patients hospitalized for severe COVID-19 pneumonia. The non-longitudinal data 135 collected for each patient at the time of initial admission include demographic information, medical history, and medical image data (e.g., quantifying the radiological assessment of pulmonary edema (RALE) reflected by chest X-rays to determine the severity of the disease). Longitudinal data 133 associated with each patient, including laboratory test results and vital signs, are collected during follow-up. The average number of time points for each patient is 10, with a standard deviation of 6. Clinical outcomes for each patient, such as survival status, are collected on the 60th day after initial hospitalization. Table 1 below generates patient characteristics at initial admission.

[0074] Table 1

[0075]

[0076]

[0077] Table 2 below shows laboratory test variables from an exemplary patient on day 1. As shown in Table 2, laboratory test variables may include fibrinogen, C-reactive protein, prothrombin international normalized ratio, prothrombin time, lactate dehydrogenase, D-dimer, albumin, ferritin, alanine aminotransferase, aspartate aminotransferase, chloride, protein, alkaline phosphatase, bilirubin, calcium, creatinine, glucose, hematocrit, hemoglobin, potassium, platelets, red blood cells, sodium, and white blood cells.

[0078] Table 2

[0079]

[0080]

[0081] In some exemplary embodiments, the performance of the disease prognosis model 115 can be evaluated by: using the training set, validation set and test set generated by splitting the above-mentioned data set at a ratio of 7:1:2 to train, validate and test the disease prognosis model 115. In some cases, the training of the disease prognosis model 115 may include stochastic gradient optimization, and the validation of the disease prognosis model 115 may include tuning the hyperparameters of the disease prognosis model 115 on the validation set. For example, in the case where the disease prognosis model 115 includes an R-converter formed by a set of local long short-term memory (LSTM) networks and spatiotemporal attention 200, a window size of 6 can be used to limit the local long short-term memory (LSTM) network to learning short-term dependencies. In addition, the size of the hidden feature set (e.g., the value of H) can be set to 32 in some cases. Using the tuned hyperparameters, the disease prognosis model 115 is trained for 50 batches of size 2. The training of the disease prognosis model 115 can also follow a learning rate schedule, such as an annealed learning rate that gradually decreases from 1e-3 to 1e-5. The disease prognosis model 115 with the best performance on the validation set was evaluated on the test set.

[0082] In some exemplary embodiments, the evaluation of the performance of the disease prognosis model 115 may include assessing the prognostic value of different data modalities by gradually incorporating different data modalities into the disease prognosis model 115. For example, the spatiotemporal attention 200 for clinical outcome prediction can be interpreted by first visualizing the spatiotemporal feature map (e.g., the importance of changes in different features across consecutive time points). The key time points identified by the spatiotemporal attention 200 (e.g., the time points with the highest feature importance) can be compared with the key time points identified by the Acute Physiology and Chronic Health Evaluation II (APACHE II) system (e.g., the time points with the highest increase in Apache II scores). The latter is a clinical nomogram that quantifies the severity of the disease. By measuring physiological variables, age, and previous health status, Apache II gives a score between 0 and 71, where a higher score indicates a higher risk of death. A Mann-Whitney U test is performed to compare the key time points identified by the spatiotemporal attention 200 and by the Apache II system.

[0083] Table 3 shows the area under the curve (AUC) when the disease prognosis model 115 (e.g., implemented using a long short-term memory (LSTM) network) is used to determine the clinical outcomes of COVID-19 patients. As shown in Table 3, when the disease prognosis model 115 operates alone on laboratory test data (as a type of longitudinal data 133), the disease prognosis model 115 is able to achieve an area under the curve (AUC) of 0.63 on the test set. When vital signs (as another type of longitudinal data 133) are incorporated, the performance of the disease prognosis model 115 increases to an area under the curve (AUC) of 0.70. These two results demonstrate the effectiveness of the disease prognosis model 115 in modeling longitudinal data. In addition, the incorporation of the same type of non-longitudinal data 135 (e.g., static data, including demographic data, medical history data, and medical data) further improves the performance of the disease prognosis model 115 to areas under the curve (AUC) of 0.73, 0.75, and 0.76, respectively.

[0084] Table 3

[0085]

[0086]

[0087] In some exemplary embodiments, Figures 3A to 3BThe performance of the disease prognosis model 115 is further evaluated by using different network architectures shown (e.g., a long short-term memory (LSTM) network with spatiotemporal attention 200 and an R-transformer with spatiotemporal attention 200). The performance of the disease prognosis model 115 is also compared with the performance of two conventional models (including a long short-term memory (LSTM) network with embedded temporal attention and a transformer with embedded spatial attention). Table 4 shows the performance of different models.

[0088] Table 4

[0089]

[0090] The clinical model in Table 4 (considering only non-longitudinal data collected at the time of initial admission) achieves an area under the curve (AUC) of 0.61. The poor performance of the clinical model may be attributed to its inability to consider nonlinear interactions between variables and the exclusion of longitudinal data. A conventional long short-term memory (LSTM) network is able to achieve an area under the curve (AUC) of 0.76 alone and an area under the curve (AUC) of 0.77 with the assistance of temporal attention. By adding spatiotemporal attention 200, the disease prognosis model 115 has an area under the curve of 0.80 when implemented using a long short-term memory (LSTM) network, and an area under the curve of 0.94 when implemented as an R-converter. These results indicate that separating the learning of short-term dependencies and long-term dependencies (as is the case when the disease prognosis model 115 is implemented using an R-converter) can enhance the accuracy of clinical outcome predictions.

[0091] The adjusted feature map output by the spatiotemporal attention 200 may include various nonlinear interactions between features. Interpreting the spatiotemporal attention 200 with hidden features may not be as direct and intuitive as interpreting variables with physical meanings (such as heart rate, body temperature, etc.). Therefore, in some exemplary embodiments, the Apache II system can be used as a bridge to interpret where the spatiotemporal attention 200 is focusing. Figure 5A Depicted is a graph showing an example of one patient's Apache II score over time. Figure 5B Depicted is a graph showing the increase in Apache II score broken down by different physiological variables for this patient. Figure 5C A table providing a visualization of an exemplary output of spatiotemporal attention 200 is depicted. Figure 5C As shown, the importance of each individual feature may not only change across consecutive time points, but may also differ from the importance of other features.

[0092] Based on statistical testing of key time points identified by the Spatiotemporal Attention 200 and Apache II systems, values ​​of features present at the initial time point (e.g., corresponding to disease onset) and the final time point (e.g., corresponding to the most recent condition) tended to have greater importance than values ​​of features from other time points in determining clinical outcomes for COVID-19 for patients, including the likelihood of persistent and / or recurrent symptoms. The significance of the initial time point is consistent with the finding that experiencing more than five symptoms during the patient's first week of illness is associated with COVID-19. Significant (p<0.05) correlations between the Spatiotemporal Attention 200 and Apache II systems for clinical outcomes of COVID-19 included i) when respiratory rate was abnormal at the initial time point and the 60-day time point and ii) when both heart rate and creatinine level were within abnormal ranges at any time point.

[0093] In view of the above specific implementation of the subject matter, the present application discloses the following list of examples, wherein one feature of a single example or a combination of more than one feature of the example, and optionally a combination with one or more features of one or more other examples, are other examples that also fall within the disclosure scope of the present application:

[0094] Item 1: A computer-implemented method comprising: training a disease prognosis model to determine a clinical outcome of a disease based at least on longitudinal data, the longitudinal data comprising a health record for each time point in a sequence of time points, the training of the disease prognosis model comprising training a recurrent neural network, spatiotemporal attention, and a feedforward neural network, the recurrent neural network being trained to extract a feature set representing one or more local dependencies present within the health record from each health record, the spatiotemporal attention being trained to determine the importance of each feature in the feature set at each time point in the sequence of time points, and the feedforward neural network being trained to determine the clinical outcome of the disease based at least on the importance of each feature in the feature set at each time point in the sequence of time points; and applying the trained disease prognosis model to determine the clinical outcome of the disease for a patient associated with the first health record and the second health record based at least on a first health record from a first time point and a second health record from a second time point.

[0095] Item 2: The method according to Item 1, wherein the recurrent neural network is a bidirectional recurrent neural network (RNN), a long short-term memory (LSTM) network, a local long short-term memory (LSTM) network with a given window size for a time point, or a gated recurrent unit (GRU) network.

[0096] Item 3: A method according to any one of Items 1 to 2, wherein the feedforward neural network is a multilayer perceptron model.

[0097] Item 4: A method according to any one of Items 1 to 3, wherein the trained disease prognosis model determines the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least: applying the trained recurrent neural network to extract from the first health record a first set of feature values ​​for a hidden feature set representing a first set of local dependencies present within the first health record, and applying the trained recurrent neural network to extract from the second health record a second set of feature values ​​for the hidden feature set representing a second set of local dependencies present within the second health record.

[0098] Item 5: According to the method described in Item 4, the trained recurrent neural network outputs a feature map for trained spatiotemporal attention ingestion, the feature map comprising the first set of feature values ​​from the first time point and the second set of feature values ​​from the second time point.

[0099] Item 6: A method according to any one of Items 4 to 5, wherein the trained disease prognosis model further determines the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least: applying the trained spatiotemporal attention to determine the importance of each feature in the hidden feature set at each of the first time point and the second time point based at least on a feature map comprising the first set of feature values ​​and the second set of feature values.

[0100] Item 7: A method according to Item 6, wherein the trained spatiotemporal attention comprises one or more two-dimensional convolutional layers, which are trained to determine the importance of each feature in the hidden feature set across the time dimension and the feature dimension.

[0101] Item 8: A method according to Item 7, wherein the one or more two-dimensional convolutional layers include a 1×1 convolutional filter configured to jointly weight the importance of each feature in the hidden feature set across the time dimension and the feature dimension.

[0102] Item 9: A method according to any one of Items 6 to 8, wherein the trained spatiotemporal attention determines, for a first feature from the hidden feature set, a first importance of the first feature at the first time point and a second importance of the first feature at the second time point.

[0103] Item 10: The method according to Item 9, wherein the trained spatiotemporal attention further determines, for a second feature from the hidden feature set, a third importance of the second feature at the first time point and a fourth importance of the second feature at the second time point.

[0104] Item 11: A method according to any one of Items 6 to 10, wherein the trained disease prognosis model further determines the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least: applying the trained feedforward neural network to determine the clinical outcome of the disease for the patient based at least on the importance of each feature in the hidden feature set at each of the first time point and the second time point.

[0105] Item 12: A method according to any one of Items 1 to 11, wherein the disease prognosis model is further trained to determine the clinical outcome of the disease based on non-longitudinal data, wherein the non-longitudinal data is static across the sequence of time points, and wherein the non-longitudinal data is concatenated with the health record associated with each time point in the sequence of time points.

[0106] Item 13: The method according to Item 12 further comprises: identifying one or more missing values ​​for a non-longitudinal variable constituting the non-longitudinal data; and replacing the one or more missing values ​​with the observed mean value of the non-longitudinal variable in the available data set.

[0107] Item 14: A method according to any one of Items 12 to 13, wherein the non-longitudinal data includes at least one of demographic information and medical history.

[0108] Item 15: A method according to any one of Items 12 to 14, wherein the non-longitudinal data includes medical image data and / or electrogram data corresponding to a metric quantifying the severity of the disease depicted in one or more medical images and / or electrograms.

[0109] Item 16: A method according to any one of Items 1 to 15, wherein the health record associated with each time point in the sequence of time points includes a value for each of a plurality of vital sign statistics.

[0110] Item 17: The method of Item 16, wherein the plurality of vital sign statistics include one or more of systolic blood pressure, diastolic blood pressure, pulse rate, and respiratory rate.

[0111] Item 18: A method according to any one of Items 1 to 17, wherein the health record associated with each time point in the sequence of time points includes values ​​for each of a plurality of laboratory test variables.

[0112] Item 19: The method of Item 18, wherein the plurality of laboratory variables comprises one or more of the following: fibrinogen, C-reactive protein, prothrombin international normalized ratio, prothrombin time, lactate dehydrogenase, D-dimer, albumin, ferritin, alanine aminotransferase, aspartate aminotransferase, chloride, protein, alkaline phosphatase, bilirubin, calcium, creatinine, glucose, hematocrit, hemoglobin, potassium, platelets, red blood cells, sodium, and white blood cells.

[0113] Item 20: A method according to any one of Items 1 to 19, wherein the health record associated with each time point in the sequence of time points includes medical image data and / or electrogram data.

[0114] Item 21: A method according to Item 20, wherein the medical image data includes metrics quantifying the severity of the disease as depicted in one or more medical images and / or electrograms.

[0115] Item 22: A method according to any one of Items 1 to 21, further comprising: determining that the longitudinal data includes missing values ​​for a longitudinal variable at a first time point; and replacing the missing value with (i) a first value of the longitudinal variable from a second time point before the first time point, (ii) a second value of the longitudinal variable from a third time point after the first time point, or (iii) a third value determined based on the first value and the second value.

[0116] Item 23: A method according to any one of Items 1 to 22, wherein the disease is coronavirus disease (COVID-19), Alzheimer's disease or age-related macular degeneration.

[0117] Item 24: A method according to any one of Items 1 to 23, wherein the clinical outcome of the disease comprises a probability associated with one or more of cure, progression and death.

[0118] Item 25: A method according to any one of Items 1 to 24, wherein the health record associated with each time point in the sequence of time points is an electronic health record (EHR).

[0119] Item 26: A system comprising: at least one data processor; and at least one memory storing instructions that, when executed by the at least one data processor, result in operations including those of the method described in any one of Items 1 to 25.

[0120] Item 27: A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, result in operations including those of the method described in any one of Items 1 to 25.

[0121] Figure 6 A block diagram illustrating an example of a computing system 600 consistent with implementations of the current subject matter is depicted. Figures 1 to 6 , computing system 600 may be used to implement database management system 110 and / or any components thereof.

[0122] like Figure 6 As shown, the computing system 600 may include a processor 610, a memory 620, a storage device 630, and an input / output device 640. The processor 610, the memory 620, the storage device 630, and the input / output device 640 may be interconnected via a system bus 650. The processor 610 is capable of processing instructions for execution within the computing system 600. Such executed instructions may implement, for example, one or more components of the database management system 110. In some exemplary embodiments, the processor 610 may be a single-threaded processor. Alternatively, the processor 610 may be a multi-threaded processor. The processor 610 is capable of processing instructions stored on the memory 620 and / or the storage device 630 to display graphical information for a user interface provided via the input / output device 640.

[0123] Memory 620 is a computer-readable medium, such as a volatile or non-volatile computer-readable medium, that stores information within computing system 600. For example, memory 620 may store a data structure representing a configuration object database. Storage device 630 is capable of providing persistent storage for computing system 600. Storage device 630 may be a solid-state drive, a floppy disk device, a hard disk device, an optical disk device, or a tape device or other suitable persistent storage device. Input / output device 640 provides input / output operations for computing system 600. In some exemplary embodiments, input / output device 640 includes a keyboard and / or a pointing device. In various specific implementations, input / output device 640 includes a display unit for displaying a graphical user interface.

[0124] According to some exemplary embodiments, the input / output device 640 can provide input / output operations for network devices. For example, the input / output device 640 can include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0125] In some exemplary embodiments, the computing system 600 can be used to execute various interactive computer software applications that can be used to organize, analyze and / or store data in various formats. Alternatively, the computing system 600 can be used to execute any type of software application. These applications can be used to perform various functionalities, such as planning functionality (e.g., generating, managing, editing electronic spreadsheet documents, word processing documents and / or any other objects, etc.), computing functionality, communication functionality, etc. The application can include various additional functionality or can be an independent computing product and / or functionality. After activation within the application, the function can be used to generate a user interface provided via the input / output device 640. The user interface can be generated by the computing system 600 and presented to the user (e.g., on a computer screen monitor, etc.).

[0126] One or more aspects or features of the subject matter described herein can be implemented with digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software and / or combinations thereof. These various aspects or features can be included in the implementation in one or more computer programs, which are executable and / or interpretable on a programmable system, which includes at least one programmable processor (which can be dedicated or general, coupled to receive data and instructions from a storage system, at least one input device and at least one output device and to transmit data and instructions thereto). A programmable system or computing system can include a client and a server. Typically, the client and the server are remotely arranged from each other, and generally interact through a communication network. The relationship between the client and the server is generated by means of a computer program running on each computer and a client-server relationship between each other.

[0127] These computer programs (which may also be referred to as programs, software, software applications, applications, components or codes) include machine instructions for programmable processors and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer product, apparatus and / or device (such as, for example, a disk, an optical disk, a memory and a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily (such as, for example, a non-temporary solid-state memory or a magnetic hard drive or any equivalent storage medium). A machine-readable medium may store such machine instructions alternatively or additionally in a transient manner (such as, for example, a processor cache or other random access memory associated with one or more physical processor cores).

[0128] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (such as, for example, a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user) and a keyboard and a pointing device (such as, for example, a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide interaction with a user. For example, the loop provided to the user may be any form of sensory loop, such as, for example, a visual loop, an auditory loop, or a tactile loop; input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include a touch screen or other touch-sensitive device, such as a single-point or multi-point resistive or capacitive tracking pad, voice recognition hardware and software, an optical scanner, an optical pointer, a digital image capture device and associated interpretation software, and the like.

[0129] In the above description and claims, phrases such as "at least one" or "one or more" may appear, followed by a list of combinations of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, the phrase is intended to represent any element or feature listed alone, or any other described element or feature combined with any other described element or feature. For example, the phrase "at least one of A and B"; "one or more of A and B"; "A and / or B" are each intended to represent "single A, single B, or A and B together". A similar interpretation also applies to lists including three or more items. For example, the phrase "at least one of A and B and C"; "one or more of A, B, and C" and "A, B, and / or C" are each intended to represent "single A, single B, single C, A and B together, A and C together, B and C together, or A and B and C together". The use of the term "based on" above and in the claims is intended to represent "based at least in part", so that undescribed features or elements are also permissible.

[0130] Depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The embodiments described in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Instead, they are only some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to those features and / or variations described herein, other features and / or variations may also be provided. For example, the above-mentioned specific implementations may be directed to various combinations and sub-combinations of the disclosed features and / or to combinations and sub-combinations of several further features disclosed above. In addition, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other specific implementations may be within the scope of the following claims.

Claims

1. A computer-implemented method comprising: training a disease prognosis model to determine a clinical outcome of a disease based at least on longitudinal data, the longitudinal data comprising a health record for each time point in a sequence of time points, The training of the disease prognosis model includes training a recurrent neural network, a spatiotemporal attention, and a feedforward neural network, the recurrent neural network being trained to extract a feature set representing one or more local dependencies existing in the health record from each health record, the spatiotemporal attention being trained to determine the importance of each feature in the feature set at each time point in the sequence of time points, and the feedforward neural network being trained to determine the clinical outcome of the disease based at least on the importance of each feature in the feature set at each time point in the sequence of time points; as well as The trained disease prognosis model is applied to determine the clinical outcome of the disease for a patient associated with the first health record and the second health record based at least on a first health record from a first time point and a second health record from a second time point.

2. The method according to claim 1, wherein the recurrent neural network is a bidirectional recurrent neural network (RNN), a long short-term memory (LSTM) network, a local long short-term memory (LSTM) network with a given window size for a time point, or a gated recurrent unit (GRU) network.

3. The method according to any one of claims 1 to 2, wherein the feedforward neural network is a multilayer perceptron model.

4. The method according to any one of claims 1 to 3, wherein the trained disease prognosis model determines the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least: applying the trained recurrent neural network to extract from the first health record a first set of feature values ​​for a hidden feature set representing a first set of local dependencies present within the first health record, and The trained recurrent neural network is applied to extract from the second health record a second set of feature values ​​for the hidden feature set representing a second set of local dependencies present within the second health record.

5. The method of claim 4, wherein the trained recurrent neural network outputs a feature map for trained spatiotemporal attention ingestion, the feature map comprising the first set of feature values ​​from the first time point and the second set of feature values ​​from the second time point.

6. A method according to any one of claims 4 to 5, wherein the trained disease prognosis model further determines the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least applying the trained spatiotemporal attention to determine the importance of each feature in the hidden feature set at each of the first time point and the second time point based at least on a feature map including the first set of feature values ​​and the second set of feature values.

7. The method of claim 6, wherein the trained spatiotemporal attention comprises one or more two-dimensional convolutional layers, the one or more two-dimensional convolutional layers being trained to determine the importance of each feature in the hidden feature set across a temporal dimension and a feature dimension.

8. The method of claim 7, wherein the one or more two-dimensional convolutional layers comprise 1×1 convolutional filters configured to jointly weight the importance of each feature in the hidden feature set across the time dimension and the feature dimension.

9. The method according to any one of claims 6 to 8, wherein the trained spatiotemporal attention determines, for a first feature from the hidden feature set, a first importance of the first feature at the first time point and a second importance of the first feature at the second time point.

10. The method of claim 9, wherein the trained spatiotemporal attention further determines, for a second feature from the set of hidden features, a third importance of the second feature at the first time point and a fourth importance of the second feature at the second time point.

11. A method according to any one of claims 6 to 10, wherein the trained disease prognosis model further determines the clinical outcome of the disease for the patient associated with the first health record and the second health record by at least applying a trained feedforward neural network to determine the clinical outcome of the disease for the patient based at least on the importance of each feature in the hidden feature set at each of the first time point and the second time point.

12. A method according to any one of claims 1 to 11, wherein the disease prognosis model is further trained to determine the clinical outcome of the disease based on non-longitudinal data, wherein the non-longitudinal data is static across the sequence of time points, and wherein the non-longitudinal data is concatenated with the health record associated with each time point in the sequence of time points.

13. The method according to claim 12, further comprising: identifying one or more missing values ​​for a non-longitudinal variable constituting the non-longitudinal data; as well as The one or more missing values ​​are replaced with the mean value of the non-longitudinal variable observed in the available data set.

14. A method according to any one of claims 12 to 13, wherein the non-longitudinal data includes medical image data and / or electrogram data corresponding to metrics quantifying the severity of the disease depicted in one or more medical images and / or electrograms.

15. The method of any one of claims 1 to 14, wherein the health record associated with each time point in the sequence of time points includes a value for each of a plurality of vital sign statistics.

16. The method of any one of claims 1 to 15, wherein the health record associated with each time point in the sequence of time points includes values ​​for each of a plurality of laboratory test variables.

17. A method according to any one of claims 1 to 16, wherein the health record associated with each time point in the sequence of time points includes medical image data and / or electrogram data, and wherein the medical image data includes metrics quantifying the severity of the disease depicted in one or more medical images and / or electrograms.

18. The method according to any one of claims 1 to 17, further comprising: determining that the longitudinal data includes missing values ​​for a longitudinal variable at a first time point; as well as The missing value is replaced with (i) a first value of the longitudinal variable from a second time point before the first time point, (ii) a second value of the longitudinal variable from a third time point after the first time point, or (iii) a third value determined based on the first value and the second value.

19. The method according to any one of claims 1 to 18, wherein the disease is coronavirus disease (COVID-19), Alzheimer's disease, or age-related macular degeneration.

20. The method of any one of claims 1 to 19, wherein the clinical outcome of the disease comprises a probability associated with one or more of cure, progression, and death.

21. A system comprising: at least one data processor; as well as At least one memory storing instructions which, when executed by said at least one data processor, result in operations including the method according to any one of claims 1 to 21.

22. A non-transitory computer readable medium storing instructions which, when executed by at least one data processor, result in operations including the method of any one of claims 1 to 21.

Images