Determination of cardiac function indexes

Abstract

A computer-implemented method for determining a cardiac performance index of a patient includes receiving an image of a fundus of the patient, encoding the received image into a joint latent space, decoding a representation of the patient's heart from the joint latent space, providing the decoded representation from the joint latent space to a neural network configured to generate a cardiac performance index, and outputting the cardiac performance index generated by the neural network in response to receiving the decoded representation of the patient's heart.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for estimating a cardiac performance index of a patient. [Background technology]

[0002] Cardiovascular diseases (CVD) are a leading cause of death and socioeconomic burden worldwide. According to the World Health Organization, there are an estimated 17.9 million CVD-related deaths worldwide annually. Identification and timely treatment of CVD risk factors are important strategies for reducing CVD incidence in populations and for risk modification in individuals.

[0003] Traditionally, CVD risk is determined using demographic and clinical parameters, such as age, sex, ethnicity, smoking status, family history, and medical history of hyperlipidemia, diabetes mellitus, or hypertension. Imaging tests, such as coronary CT imaging, echocardiography, and cardiovascular magnetic resonance (CMR), help stratify patient risk by assessing coronary calcium burden, myocardial scar burden, ischemia, and cardiac chamber size and function. However, cardiovascular imaging is typically only performed in secondary care and is relatively expensive, limiting its availability in underdeveloped and developing countries. In developed countries, then, access prioritization to proceed with cardiovascular imaging in high-risk patients can avoid the collapse of health care services and cost-effective resource use. Summary of the Invention [Problem to be solved by the invention]

[0004] It is an object of the present invention to obviate or mitigate one or more of the problems set forth above. [Means for solving the problem]

[0005] In an example described herein, there is a computer-implemented method (100) for determining a cardiac function index (111) of a patient, comprising receiving an image (101) of a fundus of the patient, encoding the received image (101) into a joint latent space, decoding a representation (106) of the patient's heart from the joint latent space, providing the decoded representation from the joint latent space to a neural network (110) configured to generate a cardiac function index, and outputting the cardiac function index generated by the neural network in response to receiving the decoded representation of the patient's heart.

[0006] Advantageously, the method may enable a patient's cardiac performance indicators to be determined by using a captured fundus image rather than a captured image of the heart (such as a CMR or CT image).

[0007] The representation decoded from the joint latent space may be, for example, an image of the patient's heart, such as a CMR image or a CT image. Alternatively, the representation decoded from the joint latent space may be an abstract representation of the patient's heart. The abstract representation may be a low-dimensional abstract representation (compared to the decoded image).

[0008] The method may further include the use of a neural network (109) configured to process the input and provide an output. The input to the neural network (109) may be a first characteristic (108) of the patient. The output from the neural network (109) may be provided to a neural network (110) configured to determine the cardiac function index. Beneficially, including the patient characteristic as input data may provide a sensitive method for determining the cardiac function index. The first characteristic may be one of a plurality of first characteristics.

[0009] The method may further include using a neural network (107) configured to process the representations decoded from the combined latent space. An input to the neural network (107) configured to process the representations decoded from the combined latent space may include the representations decoded from the combined latent space (106). An output of the neural network (107) configured to process the representations decoded from the combined latent space may be provided as an input for a neural network (110) configured to determine a cardiac function index.

[0010] The neural network (107) configured to process the representations decoded from the joint latent space may be a convolutional neural network (CNN).

[0011] The determined cardiac function indices (111) may include, among others, left ventricular mass (LVM), left ventricular end-diastolic volume (LVEDV), ejection fraction (EF), cardiac output (CO), LV end-systolic volume (LVESV), regional wall thickening (WT), regional wall motion (WM), and / or myocardial strain.

[0012] The method may further determine the patient's risk of an adverse cardiovascular characteristic or event (114). The method may further include a neural network (112) for determining the patient's risk of an adverse cardiovascular characteristic / event. An input to the neural network (112) for predicting the patient's risk of an adverse cardiovascular characteristic / event may include the determined cardiac performance index (111). Advantageously, this may provide a method for predicting the patient's risk of an adverse cardiovascular characteristic / event without the need for imaging of the patient's heart, which provides greater accuracy than prior art methods.

[0013] The input to the neural network (112) for determining the patient's risk of an adverse cardiovascular trait / event may further include a second patient characteristic (113). The second patient characteristic (113) may include characteristics that are the same as or different from the first patient characteristic (108). Advantageously, this may further increase the accuracy of the method in determining the patient's risk of an adverse cardiovascular trait / event.

[0014] The computer-readable medium may include instructions that, when executed by one or more computers, cause the one or more computers to perform any one or more of the methods disclosed herein. The one or more computers may be configured to perform the methods disclosed herein. In some exemplary systems, the one or more computers may comprise one or more wearable devices, such as a digital retinal implant or biosensing glasses.

[0015] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying schematic drawings, in which: [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic diagram of an example of a system for determining cardiac function indices and / or predicting the risk of adverse cardiovascular events. [Diagram 2] 1 is a flow chart depicting an example of a method for predicting risk of an adverse cardiovascular event. [Diagram 3] 2 is a flowchart depicting an exemplary method of training the system of FIG. 1 to determine cardiac function indices and / or predict risk of adverse cardiovascular events. [Figure 4] FIG. 2 is a schematic diagram of an exemplary method for training the multi-channel variational autoencoder (mcVAE) depicted in FIG. 1. [Figure 5a] FIG. 2 shows a plot of the performance of the system of FIG. 1 in determining cardiac function indices in comparison with manual annotations on CMR images. [Figure 5b]FIG. 2 shows a plot of the performance of the system of FIG. 1 for determining cardiac function indices compared with automatic annotations on CMR images. [Figure 6] FIG. 2 shows a plot comparing the performance of the system of FIG. 1 for estimating cardiac function indices with different resolutions of captured fundus images. [Figure 7] FIG. 2 shows a plot of the performance of the system of FIG. 1 for predicting risk of adverse cardiovascular events. [Figure 8] FIG. 2 compares the performance of the system of FIG. 1 for predicting risk of adverse cardiovascular events when using different patient characteristics. [Figure 9] FIG. 2 illustrates an exemplary distribution of latent variables after encoding using the joint latent space depicted in FIG. 1. [Figure 10] FIG. 2 illustrates an exemplary distribution of Frechet inception distance scores for a representation of a patient's heart decoded from the joint latent space depicted in FIG. 1. [Figure 11] FIG. 3 illustrates exemplary values ​​of regressor coefficients in the method of FIG. 2. [Figure 12] FIG. 1 depicts an exemplary computer system that can be used to execute the methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The devices and methods disclosed herein can utilize retinal images (e.g., fundus photographs or optical coherence cross-sectional scans, optionally together with patient characteristics and / or demographic data) to determine (or predict / estimate) one or more cardiac function indices by jointly learning the latent space of the retinal and CMR images. The cardiac function indices may include, among others, left ventricular (LV) mass (LVM), LV end-diastolic volume (LVEDV), ejection fraction (EF), cardiac output (CO), LV end-systolic volume (LVESV), regional wall thickening (WT), regional wall motion (WM), and myocardial strain. A review of at least some of the relevant indices is provided in Frangi AF, Niessen WJ, Viergever MA, Three-dimensional modeling for functional analysis of cardiac images: a review IEEE Trans Med Imaging, January 2001;20(1):2-25.

[0018] Cardiac performance indices (optionally together with patient characteristics and / or demographic data) may be used to determine (or predict / estimate) the risk of adverse cardiovascular characteristics / events. Such adverse cardiovascular characteristics may include, for example, arrhythmias, valvular heart disease, cardiomyopathies (enlarged hearts), carotid or coronary artery disease, among others, each of which may lead to adverse events such as myocardial infarction.

[0019] By using the devices and methods described herein to determine cardiac performance indices, patients may be evaluated for risk of adverse cardiovascular events during routine ophthalmology or optician visits using readily available equipment, rather than requiring the presence of a specialist cardiologist or specialized equipment such as magnetic resonance imaging (MRI) or computed tomography (CT) scanners. Assessment of risk of adverse cardiovascular events during routine ophthalmology visits may further allow timely referral of patients for further testing. Additionally or alternatively, cardiac performance indices may also be used to identify, diagnose, and / or monitor possible pathological cardiac remodeling and / or hypertension signs, incidents, or indicators. By recording or monitoring a patient's cardiac performance indices over time, problematic changes may be observed more quickly and easily, allowing the patient to be referred to a cardiologist for further evaluation (e.g., when a significant change in the cardiac performance indices is detected). An ophthalmologist, optician, or automated risk detection system may record a patient's cardiac performance indices over time and / or determine the risk of adverse events. Patient monitoring may also be performed during clinical trials to observe response to treatment, which can reduce the cost and complexity of clinical trials. Additionally, the use of more readily available equipment can allow clinical trial participants to be more geographically distributed, which can allow for the inclusion of representative cohorts within the study.

[0020] The cardiac index or risk of adverse cardiovascular events may be used for further purposes. Examples of other purposes include more personalized calculation of health insurance premiums (compared to calculations that do not use the cardiac index or risk of adverse cardiovascular events) and automatic generation and output of diet and lifestyle modifications. Further intelligent personalization can be achieved by integration into retinal implants or specialized head-mounted devices.

[0021] The system may be implemented on a computer system, which may comprise one or more computers. For example, the computer system may include or be part of a wearable device configured to capture retinal images. The system may be used to screen patients at regular optician examinations or as an indication for secondary referral at an eye clinic. All patients (e.g., at an optician or eye clinic) may be screened, or alternatively, only high-risk patients (i.e., patients identified as at high risk of suffering an adverse cardiovascular event based on one or more patient characteristics) may be screened. The computer system may comprise a retinal implant and / or a head-mounted component, such as smart glasses or an AR / VR headset.

[0022] For example, the smart glasses (i.e., a pair of smart glasses) may include a camera. For example, the smart glasses may include a camera for eye tracking. The camera of the smart glasses may be configured to record an image of one or both of the user's eyes while the user is wearing the smart glasses in accordance with the present technique. The image may be a still image or a moving image (i.e., a digital film) from which one or more still images may be extracted. The still image may capture a portion of the fundus of the wearer's eye. The still image may then be analyzed by the smart glasses to determine a cardiac performance index and / or determine a risk of an adverse cardiovascular characteristic or event. Additionally or alternatively, the still image (or a moving image from which one or more still images may be extracted) may be transmitted (e.g., wirelessly, e.g., via a WLAN or 4G network, or wired) to a computer (e.g., a server), which may analyze (and optionally extract) the still image to determine a cardiac performance index and / or determine a risk of an adverse cardiovascular characteristic or event.

[0023] Output from the system (i.e., the determined cardiac performance index and / or the determined risk of an adverse cardiovascular characteristic or event) may be provided to the user, for example, via a user (I / O) interface of the smart glasses (e.g., on a head-up display) or a computer of the computer system (e.g., on a display of a smartphone device paired with the smart glasses). The output may include a number indicating the value of the determined cardiac performance index and / or the determined risk of an adverse cardiovascular characteristic or event. Additionally or alternatively, the output may prompt the user, for example, to consult a medical professional. Further additionally or alternatively, the output may be provided to a medical professional or may trigger a communication with a medical professional (e.g., an email sent by the computer) prompting the medical professional to contact the user.

[0024] FIG. 1 illustrates an example of a computer-implemented system 100 for predicting cardiac performance indices based on retinal images.

[0025] As depicted in FIG. 1 , a captured fundus image 101 (i.e., an image of a patient's fundus captured by an image sensor) is received. The captured fundus image 101 may be received from a camera or any other suitable image sensor, as will be readily apparent to those skilled in the art. For example, the camera may include a camera of a portable user device, such as a smartphone. In a first step, the captured fundus image 101 may be pre-processed to generate a pre-processed fundus image 102. Examples of pre-processing operations that may be performed are described in more detail below. However, it will be understood that in general, pre-processing is not an essential feature of the techniques described herein. For example, the captured fundus image 101 may be received in a format that does not require pre-processing. Thus, any reference below to further operations performed on the captured fundus image 101 should be understood to be operations performed on the captured fundus image 101 or the pre-processed captured fundus image 102, as appropriate.

[0026] The captured fundus image 101 is encoded into a latent space 104, for example, using a multi-channel variational autoencoder (mcVAE). The latent space 104 is a joint latent space that provides an incorporation of both the fundus image and the cardiac magnetic resonance (CMR) image, as described below. The joint latent space 104 may be obtained by other techniques, for example, Bayesian mixture of experts models or disentangled representation learning techniques.

[0027] As known to those skilled in the art, a cardiac magnetic resonance (CMR) image may be one or more 3D images or one or more 2D images. The multiple 3D or 2D images may provide a time series of (2D or 3D) images (e.g., over one or more complete cardiac cycles). The multiple 2D images may correspond to a trajectory of 2D slices in 3D space. Although CMR images are described in the following examples, it will be apparent to those skilled in the art that other imaging techniques, such as CT scans, may be used to obtain images of the patient's heart.

[0028] The patient's heart representation 106 may be decoded from the joint latent space 104 using a decoder (or generative model) 105. The representation may be, for example, a representation of a CMR image (or another type of image used to train the joint latent space 104). Alternatively, the representation may be an abstract representation of the patient's heart (e.g., an output may be obtained from the joint latent space 104 without generating a complete image representation, but it nevertheless represents the patient's heart). For the purposes of the following example, it is assumed that the CMR image is decoded from the joint latent space 104. An exemplary decoder architecture that is particularly effective is shown in Table 1 below, although it will be understood that other decoder architectures may be used. In Table 1, each row corresponds to a layer of the respective model. The patient's heart representation 106 is provided as an input to a neural network 107 configured to process the patient's heart representation 106. The output from the neural network 107 may be a low-dimensional representation of the patient's heart representation 106. The patient's characteristics 108 (of the patient for whom the fundus image 101 was obtained) are provided as input to the neural network 109. The output from the neural network 109 may be a low-dimensional representation of the patient characteristics 108.

[0029] The outputs from the neural networks 107 and 109 (i.e., the neural network 107 configured to process the representation 106 of the patient's heart, and the neural network configured to process the patient's characteristics 108) are provided as one or more inputs to the neural network 110 configured to determine a cardiac performance index. The outputs from the neural networks 107 and 109 may be provided as separate inputs (i.e., two separate inputs) to the neural network 110 configured to determine a cardiac performance index, or alternatively, may be concatenated (or otherwise fused) together before being provided as a single input to the neural network 110 configured to determine a cardiac performance index. The neural network 110 configured to determine a cardiac performance index output outputs a determined cardiac performance index 111. The determined cardiac performance index 111 may be provided as an input, optionally together with the patient's characteristics 113, to the neural network 112 configured to predict the patient's risk of an adverse cardiovascular event. The neural network 112 configured to predict a patient's risk of an adverse cardiovascular event outputs an indicated risk 114 of an adverse cardiovascular event. [Table 1] Table 1: Exemplary Encoder and Decoder Architectures. In the exemplary retina encoder architecture, the third dimension in the input layer corresponds to the three channels of each pixel in the image: red, green, and blue channels.

[0030] The captured fundus image 101 may have been previously captured and stored in a computer readable memory. The captured fundus image 101 may be received by the system of FIG. 1 from another entity over a network such as the Internet. Alternatively, the captured fundus image 101 may be used as an input for the system of FIG. 1 upon its capture. The captured fundus image 101 may be a digital image captured using a digital camera. Alternatively, the captured fundus image 101 may be a digital image formed by scanning an analog film or may be an analog photographic image. As a further alternative, the captured fundus image 101 may be formed using any direct fundus scanning methodology that uses any physical or physiological contrast from the health of the eye.

[0031] As described above, pre-processing may be applied to the captured fundus image 101. Pre-processing may include one or more pre-processing steps. For example, the captured fundus image 101 may be cropped, filtered, enhanced, restored, interpolated, super-resolved, or otherwise processed. For example, if the captured fundus image 101 includes unnecessary (or useless) information (e.g., pixels that do not depict the fundus of the patent), the captured fundus image 101 may be cropped to remove the unnecessary information. The radius of the field of view may be determined, for example, using a suitable thresholding technique to identify foreground pixels. Such thresholding techniques are well known to those skilled in the art and will not be described in detail herein. Another example of pre-processing that may be performed is that the captured fundus image 101 may be resampled from a first image resolution (e.g., 2048×1536 pixels) to a second image resolution. The first image resolution may depend on the image sensor used to capture the captured fundus image 101. The second image resolution may be preferably 128×128 pixels, or alternatively 256×256 pixels, 512×512 pixels, etc. The second image resolution of 128×128 pixels may be preferred as experimental data shown in FIG. 6 and described below suggests that such image resolution provides better discrimination than other image resolutions. Image resolution and resampling may alternatively be referred to as image size and resizing, respectively. Resampling the captured fundus image 101 may reduce the computational resources required for subsequent processing or best fit the embedded hardware constraints. As another example of pre-processing that may be performed, a process called "unsharp masking" may be applied to the captured fundus image 101. Unsharp masking subtracts a smoothed image from the original image (which may be the captured fundus image with or without other pre-processing applied). For example, a smoothed image may be generated by applying a Gaussian filter (e.g., of size 4 x 4 pixels) to the captured fundus image, and the smoothed image is then subtracted from the captured fundus image.Any one or more of the pre-processing steps described above may be applied to the captured fundus image 101. Alternatively or additionally, other applicable pre-processing steps may be used, as known to those skilled in the art.

[0032] The mcVAE uses a multivariate technique to jointly analyze heterogeneous data by projecting observations from different sources into a joint latent space (also called a common latent space, hidden space, or embedded space). As described above, a captured fundus image 101 may be encoded into the joint latent space 104 using the mcVAE, which includes one or more encoders 103 and one or more decoders 105. The mcVAE may be considered an extension of a variational autoencoder (VAE), which allows for the generation or reconstruction of observations by sampling from a learned latent representation. The mcVAE is described in more detail in Antelmi et al., "Sparse Multi-Channel Variational Auto-encoder for the Joint Analysis of Heterogeneous Data," Proceedings of the 36th International Conference on Machine Learning-PMLR, 2019, pp. 302-311.

[0033] X={x c} c=1,...,C Given C data channels of different size and dimensionality for each observation, denoted as z, mcVAE estimates a joint latent space common to all channels. This latent space is typically represented by a low-dimensional vector, denoted as z, with a multivariate Gaussian prior distribution p(z). Assuming that each data channel is conditionally independent from all others, the joint log-likelihood of an observation is

number

[0034] Compared to VAE, mcVAE is based on channel-specific encoding functions that map each channel in a joint latent space, and cross-channel decoding functions that simultaneously decode all other channels from the latent representation. The variational posterior distribution of the mcVAE's latent space may be optimized to maximize data reconstruction. Since the latent representations that mcVAE encode are shared, multiple channels may be decoded using information encoded from a single channel. This feature allows for imputation, imputation, or decoding of missing channels from those available. Thus, by first training the mcVAE with pairs of images from multiple training subjects (each pair including a captured fundus image 101 and a captured CMR image), a decoded CMR image may be decoded from the latent representation of the captured fundus image 101 using the decoder 105. The training of the mcVAE (i.e., the joint latent space 104 and the architecture of the encoder 103 and decoder 105) is described in more detail below. The mcVAE may be sparse. mcVAEs can be sparse in the sense that each feature (i.e., each dimension) of the dataset depends on a small subset of the latent factors. More specifically, dropout, a technique known in the art for regularizing neural networks, can be naturally incorporated into VAEs to result in a sparse representation of the variational parameters. Using sparse mcVAEs can ensure that the lower bound on evidence generally reaches a maximum upon convergence when the number of latent dimensions matches the true dimensions used to generate the data. The lower bound on evidence is a lower bound on the probability of observing some data under the model and can be used as an optimization criterion to approximate the posterior distribution p(z|x,Θ) with a simpler (parametric) distribution.

[0035] The representation 106 of the patient's heart may be decoded from the joint latent space 104 105. The format of the representation 106 of the patient's heart may depend on the format of the CMR images in the training data used to train the joint latent space 104. For example, the format of the representation 106 of the patient's heart may be the same as the format of the CMR images used to train the joint latent space 104.

[0036] As depicted in FIG. 1, the patient's heart representation 106 may be provided as an input to a neural network 107 configured to process the patient's heart representation 106. The neural network 107 configured to process the patient's heart representation 106 may be a convolutional neural network (CNN). The CNN may use a rectifier such as ReLu, a data augmentation technique, or a regularization technique such as "dropout" (e.g., as described in Srinivastava et al., "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," Journal of Machine Learning Research 15 (2014) 1929-1958). The softmax layer of the CNN may be replaced with a fully connected recurrent layer with a linear, sigmoid, or other activation function. For example, the CNN may be a ResNet50 or a VGG16. ResNet50 may be preferred over VGG16 because it uses fewer parameters. More generally, the neural network 107 may include other attention-based networks, such as a self-attention network or an image transformer network.

[0037] The instrument may provide a way to measure a patient characteristic. For example, a set of scales may provide a way to measure a patient's weight. As depicted in FIG. 1, one or more patient characteristics 108, 113 may be received. The patient characteristics 108, 113 may be limited to such characteristics (e.g., age and sex) that should be available to an optician, ophthalmologist, or eye clinic, thereby allowing cardiac function indicators to be determined in clinical (or non-clinical) situations where the system is less sophisticated. Alternatively, broader patient characteristics 108, 113 may be used, for example, as characteristics that should be available to a cardiologist or cardiology department. More generally, the patient characteristics 108, 113 may include one or more of sex, age, sex, body mass index (BMI), weight, lean body mass, diastolic BP, systolic BP, HbA1c score, blood glucose level, cholesterol, smoking habits, and alcohol intake. Other additional patient characteristics may be used, such as data derived from the patient's medical history, blood samples, genetic profiling, microbiome analysis, etc.

[0038] 1, one or more of the patient characteristics 108 are optionally processed by a neural network 109. The neural network 109 may be a fully connected neural network. Again, it will be appreciated that other suitable network architectures may process the patient characteristics 108 instead of or in addition to the network 109, as known to those skilled in the art. The output of the neural network 109 may be used for further processing, as described below.

[0039] As described above, the patient's cardiac performance index 111 may be determined by a neural network 110 configured to estimate the cardiac performance index using an output from a neural network 107 configured to process the representation 106 of the patient's heart and, optionally (e.g., if present), an output from a neural network 109. For example, the neural network 110 configured to estimate the cardiac performance index may be referred to as a decision network. The determined cardiac performance index 111 may be one or more of a left ventricular mass (LVM) or a left ventricular end-diastolic volume (LVEDV).

[0040] As described above, the determined cardiac function index 111 may be provided as an input to a neural network 112 configured to predict a patient's risk of an adverse cardiovascular event. Alternatively, the neural network 112 may be configured to use other techniques or architectures (other than CNN) known to those skilled in the art. For example, the neural network 112 configured to predict a patient's risk of an adverse cardiovascular event may be a CNN, in particular a CNN configured to calculate an output using logistic regression. Beneficially, the use of logistic regression facilitates interpretability and allows comparison between coefficients of variables used to predict the risk of an adverse cardiovascular event.

[0041] Patient characteristics 113 may also be provided as inputs to a neural network 112 configured to predict a patient's risk of an adverse cardiovascular event. Patient characteristics 113 may include characteristics described above with respect to patient characteristics 108. Patient characteristics 113 may include (or may be a subset of) patient characteristics different from patient characteristics 108 provided as inputs to neural network 109.

[0042] One or more of the patient's determined cardiac function index 111, the patient's predicted risk of an adverse cardiovascular event 114, or another output from one of the neural networks 107, 110, 112 may be used for further activities. For example, the output of the system 100 (at any stage) may be used to calculate the insurance premium charged by a health insurance provider. The health insurance provider may receive the captured fundus image 101 and the patient's characteristics 108, 113 provided as part of the enrollment or renewal process. The health insurance provider may then use the method or system disclosed herein. Alternatively, the health insurance provider may receive the determined cardiac function index 111, the patient's predicted risk of an adverse cardiovascular event 114, or another output from one of the neural networks 107, 110, 112 provided as part of the enrollment or renewal process. For example, a lower predicted risk of the patient suffering from an adverse cardiovascular event may indicate that the patient is less likely to require expensive medical care. As such, the health insurance provider may lower the premium charged accordingly.

[0043] One or more of the patient's determined cardiac performance indicators 111, the patient's predicted risk of an adverse cardiovascular event 114, or another output from one of the neural networks 107, 110, 112 may be used to provide a notification or warning to the patient, another individual (e.g., a family member or caregiver), or a service provider. For example, if the predicted risk of an adverse cardiovascular event crosses a threshold, the patient (or the patient's clinician) may be notified of the increased predicted risk.

[0044] One or more of the patient's determined cardiac performance indicators 111, the patient's predicted risk of an adverse cardiovascular event 114, or another output from one of the neural networks 107, 110, 112 can trigger the automatic generation and output of dietary and / or lifestyle modifications. For example, based on a predicted risk of an adverse cardiovascular event exceeding a threshold, dietary or lifestyle modifications may be generated and communicated to the patient or the patient's clinician.

[0045] As will be appreciated by those skilled in the art, system 100 may be modified. For example, neural networks 107, 110, and 112 may be replaced by a single neural network for determining risk of adverse cardiovascular events 114 from the representation 106 of the patient's heart, the output from neural network 109, and patient characteristics 113.

[0046] It will be appreciated that the architecture of any of the neural networks 107, 109, 110 may be of various types of architectures, such as a fully connected neural network (multilayer perceptron), a deep recurrent network, a transformer network, etc. By way of example only, the neural network 109 may be a fully connected network, and / or one or more layers in the neural network 109 may be replaced with one or more convolutional layers. However, recent research has shown that other factors such as data preprocessing dominate performance, obscuring nuances in the architecture, loss function, or activation function [https: / / arxiv.org / abs / 1803.08450].

[0047] 2 depicts an exemplary method 200 in which the computer-implemented system 100 of FIG. 1 may be used to determine a risk of an adverse cardiovascular event. It will be appreciated that although the method 200 determines a risk of an adverse cardiovascular event, the method may end after outputting a determined cardiac function index without determining the risk of a cardiovascular event. In step 201, the computer-implemented system 100 receives a captured image of a patient's eye, such as the captured fundus image 101 of FIG. 1. In step 202, recorded patient characteristics, such as the patient characteristics 108, 113 of FIG. 1, are received. In step 203, the method estimates a cardiac function index of the patient using the system 100 and as described above with respect to FIG. 1.

[0048] In step 204, following determination of the cardiac performance index, the computer-implemented system may predict the patient's risk 114 of an adverse cardiovascular event using the neural network 112 and as described above. Also as described above, although FIG. 2 illustrates receiving patient characteristics in step 202, it will be understood that the receipt and use of patient characteristics is not considered an essential feature of the method of FIG.

[0049] 3 illustrates an exemplary method 300 for training system 100. The mcVAE (i.e., encoder 103, joint latent space 104, and decoder 105), neural network 107 configured to process a representation 106 of the patient's heart, neural network 109, neural network 110 configured to estimate cardiac performance index 111, and neural network 112 configured to predict the patient's risk of an adverse cardiovascular event are all trained. Method 300 is exemplary and system 100 may be trained using other methods.

[0050] Since the mcVAE provides input to neural network 107 configured to process a representation of the patient's heart, the mcVAE may be trained prior to training neural network 107. Neural network 107 is configured to process representation 106 of the patient's heart, and neural network 109 may provide input to neural network 110 configured to estimate cardiac performance indices, and neural network 107 and neural network 109 may be trained prior to training neural network 110. Since configured neural network 110 may provide input to neural network 112 configured to predict the patient's risk of an adverse cardiovascular event, neural network 110 may be trained prior to training neural network 112.

[0051] In step 301, a first group of participants is selected from available training data. As described above, the training data includes retinal images and CMR images. The retinal and CMR images can take any suitable form. By way of example only, in an exemplary implementation, a suitable training dataset was obtained from the UK Biobank (UKB), referred to herein as the UKB dataset. The UKB dataset includes CMR images of participants who underwent CMR imaging (e.g., using a clinical wide-bore 1.5 Tesla MRI system (such as MAG-NETOM Aera, Syngo Platform VD13A, Siemens Healthcare, Erlangen, Germany)) as well as retinal images (45° field of view centered on and / or including both the optic nerve head and the macula) of participants who underwent retinal imaging using a Top-con 3D OCT 1000 Mark 2. The UKB dataset includes data corresponding to 84,760 participants. The retinal images in the UKB dataset have an image resolution of 2048×1536 pixels (before any resampling).

[0052] A first group of participants (i.e., images of those participants) may be selected. The first group of participants may be selected by starting with a dataset (e.g., the complete UKB dataset) and excluding participants. Exemplary criteria for excluding participants are shown in Table 2 with further details regarding the specific criteria below. Table 2 also shows the number of participants rejected from the UKB dataset according to each criterion when the criteria are applied in the order shown in the table. The first group of participants includes 5,663 participants according to the criteria shown in Table 2. The first group of participants may be used to train the mcVAE, the neural network 107 configured to process a representation of the patient's heart, the neural network 109, and the neural network 110 configured to estimate cardiac function indices, [Table 2] Table 2: Exemplary criteria for selecting the first group of participants

[0053] A participant may be excluded from the first group of participants due to a medical history known to affect LV mass. For example, a participant may be excluded from the first group of participants due to a medical history of diabetes, previous myocardial infarction, cardiomyopathy, or frequent strenuous exercise habits. A participant may be excluded from the first group of participants due to the participant data (e.g., captured fundus images) being of insufficient quality upon evaluation of data quality. For example, data quality may be determined by a deep learning method for quality assessment (QA) (e.g., as described in Fu et al., "Evaluation of Retinal Image Quality Assessment Networks in Different Color-spaces," MICCAI 2019). Specific criteria may be specified prior to QA. Training and performance validation of the QA method may use additional datasets, e.g., EyePACS. EyePACS is a public dataset presented within the Kaggle platform for automated diabetic retinopathy detection. Participants whose corresponding participant data fail the QA evaluation may be excluded from the first group of participants. Participants that are not excluded may be identified as suitable for training and having corresponding participant data that includes good quality captured fundus images. Training can use all participants in the first group of participants or can use a portion of the participants in the first group of participants (e.g., 25%, 50%, or 75%).

[0054] In step 302, a system 400 for training the mcVAE of Figure 1 is trained as depicted in Figure 4. After training of the system 400, the encoder 403 and the decoder 411 may be used as the encoder 103 and the decoder 105 of Figure 1.

[0055] With reference to Figure 4, a captured fundus image 401 is received. Pre-processing may optionally be applied to generate a pre-processed fundus image 402. Pre-processing may include, for example, one or more of the pre-processing steps described above with reference to Figure 1. Alternatively, the captured fundus image 401 may be suitable for use without pre-processing.

[0056] A captured CMR image 407 is also received. Pre-processing may optionally be applied to generate a pre-processed CMR image 408. Pre-processing may include, for example, detection of a region of interest (ROI) 413 around the heart depicted in the captured CMR image 407. Pre-processing may include cropping around the ROI 413. Advantageously, cropping may reduce computational time and resources required to train the mcVAE because the ROI 413 is smaller and the system 400 processes only the region of interest. Pre-processing of the captured CMR image 407 may include the use of deep learning techniques, such as a CNN. For example, detection of the ROI 413 may include providing the captured CMR image 407 as an input to a CNN. In one example, the CNN may be a U-Net or a variant of U-Net (e.g., as described in Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas (2015) "U-Net: Convolutional Networks for Biomedical Image Segmentation") that iteratively detects the heart in a short-axis MRI stack from the basal to the apical slices.

[0057] As a further example of pre-processing that may be performed on the captured CMR images 407, each captured CMR image 407 may be resampled to a normalized volume. For example, each CMR image 407 may be resampled to a normalized volume, where the normalized volume may include the LV of the heart. In a training example using data from the UKB dataset, each CMR image 407 may be resampled to 15 slices. Additionally, each captured CMR image 407 may be resampled to a particular resolution. For example, each captured CMR image 407 may be resampled to a 1 mm 3 Each captured CMR image 407 may be resampled to an isotropic resolution of 100 kb / s. For example, each captured CMR image 407 may be resampled using cubic B-spline interpolation. Each captured CMR image 407 may be normalized, for example, by normalizing the pixel intensity value range to 0 to 1.

[0058] The mcVAE may be trained using participant data, particularly from the first group of participants. In particular, two pairs of encoders 403, 409 and decoders 405, 411 (i.e., a first pair of encoders 403 and decoders 405 corresponding to a first data channel, and a second pair of encoders 409 and decoders 411 corresponding to a second data channel) may be trained to encode into a joint latent space 404. It will be understood that after training, the joint latent space 404 may be the same as the joint latent space 104 of FIG. 1. The first data channel may include a fundus image data channel, and the second data channel may include a CMR image data channel, or vice versa. The first pair of encoders 403 and decoders 405 may be trained on the first data channel (fundus image). The second pair of encoders 409 and decoders 411 may be trained on the second data channel (CMR image). Exemplary architectures for the encoders 403, 409 and decoders 405, 411 are described in Table 1. The mcVAE may be trained using any suitable method, such as backpropagation, known to those skilled in the art. An exemplary training strategy is described in Antelmi, Ayache, Robert, and Lorenzi (Sparse Multi-Channel Variational Auto-encoder for the Joint Analysis of Heterogeneous Data (2019) Proc Int Conf Mach Learning, PMLR97:302-311).

[0059] Returning to FIG. 3, in step 303, a neural network 107 configured to process the patient's heart representation 106 may be trained to output an abstract representation 106 of the patient's heart. The neural network 107 may be trained using any suitable method, such as backpropagation. The neural network 107 may be trained on the participant's heart representation. Training data for training the neural network 107 may be obtained from a first group of participants. The training data may use the participant's heart representation (decoded from the joint latent space representation). Alternatively, the participant's heart representation forming the training data for the mcVAE may be a captured CMR image or a preprocessed captured CMR image. Alternatively or additionally, the training data may include a representation of the CMR image (captured or preprocessed and captured) or a representation of the participant's heart representation. The representation may provide a measure of the cardiac function index as ground truth. The representation may be automatic or manual. For example, manual delineation may be provided through the use of post-processing software (such as cvi42 from Circle Cardiovascular Imaging Inc.) and / or expert analysis. Alternatively, automated delineation may be provided by methods known in the art (e.g., as described in Attar R, Pereaenez M, Gooya A, Alba X, Zhang L, de Vila MH, Lee AM, Aung N, Lukaschuk E, Sanghvi MM, Fung K, Paiva JM, Piechnik SK, Neubauer S, Petersen SE, Frangi AF. Quantitative CMR population imaging on 20,000 subjects of the UK Biobank imaging study: LV / RV quantification pipeline and its evaluation. Med Image Anal. 2019 Aug;56:26-42. DOI:10.1016 / j.media.2019.05.006).

[0060] In step 304, the neural network 109 may be trained to output an abstract representation of the patient characteristics 108. The neural network 109 may be trained using any suitable method, such as backpropagation. Training data for training the neural network 109 may include patient characteristics from the first group of participants. The patient characteristics may include the same characteristics described above with respect to the patient characteristics 108, 113 of FIG. 1. Levels of Hb1A1c have been shown to be positively correlated with cardiovascular mortality, even in subjects with no history of diabetes. Thus, the training data may include an indication of the participants' HbA1c levels, despite excluding participants from the first group of participants with diabetes.

[0061] In step 305, the neural network 110 configured to output an estimate of the cardiac function index may be trained using any suitable method. For example, the neural network 110 may be trained using backpropagation. The training data for training the neural network 110 may include training data from the first group of participants. The neural network 110 may be trained separately from the neural networks 107, 109. Alternatively, the neural network 110 may be trained together with the neural networks 107, 109 (i.e., end-to-end). In either case, the neural network 110 may be trained to output an indication of the cardiac function index in response to an input including an output from the neural network 107 configured to process a representation 106 of the patient's heart (optionally together with an input including an output from the neural network 109). Thus, the ground truth for training the neural network 110 is provided from the same data used to train the neural networks 107, 109 (such as the first group of participants). For example, taking a participant in the first group of participants who have fundus images and CMR images and whose cardiac function indices are known, the neural network 110 may be trained to determine the correct cardiac function indices from the output of the neural network 107 (which receives as input a representation of the patient's heart generated by the decoder 411 in response to receipt by the encoder 409 of that participant's fundus image).

[0062] In step 306, a second set of training data (e.g., a second group of participants from the group) may be selected. The second group of participants may be selected by excluding participants from the dataset (e.g., the UKB dataset). Exemplary criteria for excluding participants are shown in Table 3. Table 3 also shows the number of participants excluded from the UKB dataset (in the exemplary implementation) according to each criterion when the criteria are applied in the order shown in the table. The second group of participants includes 71,515 participants according to the criteria shown in Table 3. For example, other criteria may be considered by one skilled in the art to optimize the data available for training the neural network 112. [Table 3] Table 3: Exemplary criteria for selecting the second group of participants

[0063] The second group of participants may include two classes, the first class corresponding to participants who suffered an adverse cardiovascular event after the fundus image was captured, and the second class corresponding to participants who did not suffer an adverse cardiovascular event after the fundus image was captured. The number of participants in each of the two classes may not be equal or similar, in other words, the data may be unbalanced. In response to the unbalanced data, the data may be resampled, and a resampled subset of data for each class is created to improve the time efficiency of training and prevent overtraining. For example, the majority class may be resampled and / or the minority class may be resampled. Beneficially, resampling the majority class is a robust solution when the minority class includes hundreds of cases, e.g., less than a thousand cases.

[0064] In step 307, a neural network 112 configured to predict a patient's risk of an adverse cardiovascular event may be trained. The training data used to train the neural network 112 may include data from a second group of participants. The neural network 112 may be trained using cardiac function indices (optionally together with patient characteristics) and known characteristics / incidents of adverse cardiac events. The cardiac function indices may be determined cardiac function indices 111, or alternatively measured cardiac function indices. The performance of the neural network 112 in predicting a patient's risk of an adverse cardiovascular event may be evaluated using known statistical techniques. Cross-validation may be used to train and evaluate the performance of the neural network 112. For example, K-fold cross-validation, such as 10-fold cross-validation, may be used.

[0065] As described above, cross-validation may be used to evaluate the performance of any component of the system 100. Following training, the performance of the system 100 may be evaluated. The performance of the system 100 may be evaluated using additional data sets. For example, an alternative data set (e.g., the AREDS database) may be used to provide a validation plot.

[0066] 5a shows a set of exemplary Bland-Altman plots 501, 503 and correlation plots 502, 504 for cardiac function indices determined using system 100 compared to cardiac function indices calculated using manual delineation (from captured CMR images) using methods known in the art. The correlation coefficients (r) for LVM and LVEDV are 0.65 and 0.45, respectively. A subsample (approximately 10%) of the first group of participants was used to generate the plots 501, 502, 503, 504.

[0067] FIG. 5b shows a first set 510 and a second set 520 of exemplary Bland-Altman plots 511, 513, 521, 521 and correlation plots 512, 514, 522, 524 for determined cardiac function indices (determined from input captured fundus images) compared with cardiac function indices calculated from captured CMR images. In the first set of plots 510, the calculated cardiac function indices are determined manually by one or more clinicians by inspecting the captured CMR images. In the second set of plots 520, the calculated cardiac function indices are calculated using an automated delineation using methods known in the art (e.g., using the method described by Attar et al.). The correlation coefficients (r) for LVM and LVEDV are 0.56 and 0.34, respectively.

[0068] Table 2 compares the bias and limits of agreement of the determined cardiac performance indices 111 of the system 101 compared to the prior art methods. [Table 4] Table 2: Comparison of methods for estimating LVEDV and LVM. LoA relates to the 95% limits of agreement.

[0069] FIG. 6 shows three sets of plots 600. Each set of plots 610, 620, 630 shows the performance of the system of FIG. 1 for estimating cardiac function indexes using different sizes of captured fundus images (128×128 pixels, 256×256 pixels, and 512×512 pixels, respectively). Each set of plots 610, 620, 630 includes a Bland-Altman plot for the determined LVM 611, 621, 631, a Bland-Altman plot for the determined LVEDV 612, 622, 633, a correlation plot for the determined LVM 613, 623, 633, a correlation plot for the determined LVEDV 614, 624, 634, and a ROC curve 615, 625, 635 for the predicted risk of adverse cardiovascular events. Each ROC curve 615, 625, 635 was calculated using the same method shown in FIG. 7 described below.

[0070] FIG. 7 shows a set of exemplary ROC curves 700 evaluating the performance of the system when predicting the risk of myocardial infarction. Each ROC curve in the set of ROC curves 700 was generated using 10-fold cross-validation. ROC curve 701 shows the performance of the system when using only patient characteristics, which has an accuracy of 0.66±0.03, a sensitivity of 0.7±0.04, a specificity of 0.64±0.03, a precision of 0.64±0.03, and an F1 score of 0.66±0.03. ROC curve 702 shows the performance when using the determined cardiac function index and demographic data, which has an accuracy of 0.74±0.03, a sensitivity of 0.76±0.02, a specificity of 0.73±0.05, a precision of 0.73±0.05, and an F1 score of 0.74±0.03. ROC curve 702 shows an improvement compared to ROC curve 701, demonstrating the improvement in the performance of the system when the determined cardiac function index is used, and when the determined cardiac function index is used together with other patient characteristics.

[0071] Figure 8 shows an exemplary set of ROC curves 800 evaluating the performance of a system in predicting risk of myocardial infarction, created according to the same method used to create the set of ROC curves 700 shown in Figure 7. ROC curve 801 shows the performance of system 100 when using captured fundus images 101 and patient characteristics 108, 113 available at the optician or eye clinic. ROC curve 802 shows the performance of system 100 when using captured fundus images 101 and patient characteristics 108, 113 available at the cardiology department.

[0072] FIG. 9 shows a plot 900 illustrating an example distribution of latent variables encoded in the joint latent space 104. The first line 901 shows the distribution of latent variables when only the captured fundus image is used for training and then encoded in the joint latent space 104. The second line 902 shows the distribution of latent variables when both the captured fundus image and the captured CMR image are used for training and then encoded in the joint latent space 104. The difference in variance between the first line 901 and the second line 902 demonstrates that both the captured retinal image and the captured CMR image are encoded in the joint latent space 104. In other words, the variance is larger when (both) the retinal image and the CMR image are encoded (compared to when only the retinal image is encoded) due to the latent variables encoding richer information coming from both the retinal image and the CMR image.

[0073] FIG. 10 shows a plot 1000 illustrating an example distribution of Fréchet initiation distance (FID) scores 1001 calculated for the representation of the patient's heart decoded from the joint latent space 104. The FID score may be used to capture the similarity of the generated image to the actual image, for example, as described by Heusel et al. ("Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", NIPS 2017). The FID score may be calculated using an initiation architecture pre-trained on ImageNet. The distribution of FID scores 1001 indicates that the representation of the patient's heart is similar to the captured CMR image.

[0074] 11 illustrates an exemplary set of coefficient values ​​1101 obtained from a neural network 112 configured to predict a patient's risk of an adverse cardiovascular event. The set of values ​​1101 indicates that LVM 1102, sex 1103, age 1104, and LVEDV 1105 contribute most to predicting an adverse cardiovascular event.

[0075] 12 illustrates an exemplary computer system 1200. The computer-implemented system 100 may be implemented on a computer system, such as computer system 1200. Computer system 1200 may include a central processing unit 1210, a memory 1220, one or more storage devices 1230, an input / output processor 1240, and circuitry 1250 connecting components to one or more input / output devices 1260.

[0076] This specification uses the term "configured" in connection with systems and computer program components. In the case of a system of one or more computers configured to perform a particular operation or activity, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or activity. In the case of one or more computer programs configured to perform a particular operation or activity, the one or more programs contain instructions that, when executed by a data processor, cause a device to perform the operation or activity.

[0077] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus.

[0078] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus can optionally include code that creates an execution environment for a computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. An apparatus can also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0079] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or separate unit suitable for use in a computing environment. A program may correspond to a file in a file system, but this need not be the case. A program can be stored as part of a file that holds other programs or data, for example one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files, for example files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0080] The processes and logic flows described herein may be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0081] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The central processing unit and memory can be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Moreover, a computer can be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0082] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM disks and DVD-ROM disks.

[0083] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone, running a messaging application, and receiving a response message from the user in return.

[0084] A data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling common computationally intensive parts of machine learning training or production, i.e. inference, workloads.

[0085] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0086] An embodiment of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0087] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship. In some embodiments, the server transmits data, e.g., HTML pages, to a user device, e.g., to display data to a user interacting with a device functioning as a client and to receive user input from the user. Data generated at the user device, e.g., results of user interaction, can be received from the device at the server.

[0088] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as functioning in a particular combination and even initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0089] Similarly, although operations are depicted in the figures and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0090] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the following claims. For example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. For example, the activities recited in the claims can be performed in a different order and still achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method (100) for determining a cardiac performance index (111) of a patient in a first computing system having one or more processors, comprising: receiving an image (101) of the fundus of the patient; encoding the received image (101) into a joint latent space; decoding a representation (106) of the patient's heart from the combined latent space; providing the representations decoded from the combined latent space to a neural network (110) configured to generate a cardiac function index; outputting the cardiac performance index generated by the neural network in response to receiving the decoded representation of the patient's heart; A method comprising:

2. The method of claim 1 , wherein the representation decoded from the joint latent space is an image of the patient's heart.

3. The method of claim 1 , wherein the representation decoded from the combined latent space is an abstract representation of the patient's heart.

4. The method of claim 3 , wherein the abstract representation is a lower dimensional abstract representation than an image that provides a visual representation of the patient's heart.

5. 2. The method of claim 1, further comprising processing the first characteristic of the patient in a neural network configured to process an input including a first characteristic and provide an output from the neural network configured to process the first characteristic as an input to the neural network configured to determine a cardiac function index.

6. 2. The method of claim 1, further comprising using a neural network (107) configured to process the representations decoded from the joint latent space, wherein an input to the neural network (107) configured to process the representations decoded from the joint latent space includes the representations (106) decoded from the joint latent space, and an output of the neural network (107) configured to process the representations decoded from the joint latent space is provided as an input to the neural network (110) configured to determine a cardiac function index.

7. 7. The method of claim 6, wherein the neural network (107) configured to process the representations decoded from the joint latent space is a convolutional neural network (CNN).

8. 2. The method of claim 1, wherein the determined cardiac function indices (111) include left ventricular mass (LVM), left ventricular end diastolic volume (LVEDV), ejection fraction (EF), cardiac output (CO), LV end systolic volume (LVESV), regional wall thickening (WT), regional wall motion (WM), and / or myocardial strain.

9. The method of claim 1 , further comprising determining the patient's risk (114) of an adverse cardiovascular characteristic or event.

10. 2. The method of claim 1, further comprising a neural network (112) configured to determine the patient's risk of an adverse cardiovascular characteristic / event, and an input to the neural network for predicting the patient's risk of an adverse cardiovascular characteristic / event comprises the determined cardiac performance index (111).

11. The method of claim 10 , wherein the input to the neural network (112) for determining the patient's risk of an adverse cardiovascular characteristic / event includes a second patient characteristic (113).

12. When executed by one or more computers, the one or more computers Receiving an image (101) of a fundus of a patient; encoding the received image (101) into a joint latent space; decoding a representation (106) of the patient's heart from the combined latent space; providing the representations decoded from the combined latent space to a neural network (110) configured to generate a cardiac function index; outputting the cardiac performance index generated by the neural network in response to receiving the decoded representation of the patient's heart; 23. A non-transitory computer-readable medium comprising instructions for causing the one or more computers to determine a cardiac function index (111) of the patient, the non-transitory computer-readable medium comprising:

13. one or more processors; one or more non-transitory computer readable media; A computer system comprising: The non-transitory computer readable medium comprises: Receiving an image (101) of the fundus of a patient; encoding the received image (101) into a joint latent space; decoding a representation (106) of the patient's heart from the combined latent space; providing the representations decoded from the combined latent space to a neural network (110) configured to generate a cardiac function index; and outputting the cardiac performance index generated by the neural network in response to receiving the decoded representation of the patient's heart. a computer system storing instructions configured to cause the one or more processors to determine a cardiac function index (111) of the patient by

14. The computer system of claim 13 , further comprising one or more wearable devices configured to capture the images of the fundus of the patient.

Images