SYSTEM AND METHOD FOR DEEP NEURAL NETWORKS TO ENHANCE PREDICTION OF PATIENT ENDPOINTS USING CARDIAC VIDEOS

Abstract

The present disclosure provides a method for determining a predicted risk level of a clinical endpoint for a patient, the method including receiving cardiac video frames, the video frames associated with the patient, receiving electronic health record data including several variables related to the patient, providing the video frames and the electronic health record data to a trained neural network, receiving a risk score from the trained neural network, and outputting a report based on the risk score to at least one of a display or a memory.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 62 / 936,377, filed November 5, 2019. [Background technology]

[0002] The present invention relates to a system and method for the analysis of cardiac anatomy. Imaging is essential for treatment decisions in most modern medical specialties and has become one of the most data-intensive components of electronic health records (EHRs). For example, during a single routine ultrasound examination of the heart (echocardiogram), approximately 10-50 videos (approximately 3000 images) are acquired to evaluate the cardiac anatomy and function. In clinical practice, the cardiologist realistically has 10-20 minutes to interpret these 3000 images in the context of numerous other data streams such as laboratory values, vital signs, additional imaging tests (radiography, magnetic resonance imaging, nuclear imaging, computed tomography), and other diagnostics (e.g., electrocardiograms). These numerous data sources offer the potential for more precise and accurate clinical predictions, but humans are limited in their ability to integrate data in decision making. Thus, there is both a need and a substantial opportunity to leverage technologies such as artificial intelligence and machine learning to manage this volume of data and ultimately provide intelligent computer assistance to the physician.

[0003] Therefore, what is needed is a system for efficiently and accurately analyzing cardiac videos, e.g., those acquired during echocardiography, cardiac magnetic resonance imaging (MRI), or cardiac computed tomography (CT), to assist physicians in assessing cardiac anatomy and function and to provide accurate predictions of future clinical events. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure includes systems and methods for efficiently and accurately analyzing cardiac videos to assist physicians in assessing cardiac anatomy and function by providing prognostic assistance. More specifically, the present disclosure provides a neural network capable of receiving echocardiography videos as well as electronic health records (EHRs) and outputting a mortality risk score or level associated with a patient's predicted mortality. [Means for solving the problem]

[0005] Some embodiments of the present disclosure provide a method including receiving an echocardiographic video of a heart associated with a patient; providing the echocardiographic video to a trained neural network, the trained neural network being trained to generate a mortality risk score based on the input echocardiographic video; receiving a mortality risk score associated with the patient from the trained neural network; and outputting the mortality risk score associated with the patient to at least one of a memory or a display for viewing by a physician or healthcare administrator.

[0006] In the method, a mortality risk score associated with the patient can be indicative of the likelihood that the patient will die within a predetermined time period from when the echocardiographic video was generated. A mortality risk score associated with the patient can be indicative of the likelihood that the patient will die from all causes within a predetermined time period.

[0007] In the method, the trained neural network may be trained based on a training dataset including a plurality of videos, each video of the plurality of videos being associated with an echocardiographic view selected from among a number of echocardiographic views, and at least a portion of the training dataset being: eMERGE GuidelinesThe number of echocardiographic views can include at least one echocardiographic view. Some echocardiographic views can include at least one of an apical two-chamber view, an apical three-chamber view, an apical four-chamber view, an apical four-chamber view focused on the right ventricle, an apical five-chamber view, a parasternal long axis view, a parasternal long descending aorta view, a parasternal long mitral valve view, a parasternal long pulmonary valve view, a parasternal long right ventricular inflow view, a parasternal long zoom aortic valve view, a parasternal short aortic valve view, a parasternal short pulmonary valve and pulmonary artery view, a parasternal short tricuspid valve view, a short axis apical view, a short axis base view, a short axis central papillary view, a subcostal four-chamber view, a subcostal hepatic vein view, a subcostal atrial septum view, a subcostal inferior vena cava view, a subcostal right ventricular view, a suprasternal fossa view, a short axis central papillary view, a short axis apical view, an apical three-chamber zoom view, an apical two-chamber zoom view, or a short axis base view. The training dataset may further include a plurality of survival outcomes, and each video of the plurality of videos may be associated with a survival outcome included within the plurality of survival outcomes. The training dataset may further include an electronic health record dataset, and each video of the plurality of videos may be associated with a portion of the electronic health record dataset. The electronic health record dataset may be associated with a number of patients and may include values ​​of a number of parameters including age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function. The electronic health record dataset may include values ​​of a number of parameters including demographic parameters, vital parameters, laboratory measured parameters, echocardiogram-based parameters, and diagnostic parameters. The demographic parameters may include age, sex, and smoking status. The vital parameters may include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure. The laboratory measured parameters may include low density lipoprotein level and high density lipoprotein level.Echocardiogram-based parameters were physician-reported left ventricular ejection fraction, aortic regurgitation deceleration gradient, aortic regurgitation peak velocity, velocity-time integral of flow distal to the aortic valve, peak velocity of flow distal to the aortic valve, mean velocity of flow distal to the aortic valve, aortic root diameter, ascending aorta diameter, left ventricular end-diastolic volume: apical two-chamber; modified ellipsoid, left ventricular end-diastolic volume: apical four-chamber; modified ellipsoid, left ventricular end-diastolic volume: apical two-chamber; single plane, left ventricular end-diastolic volume: apical four-chamber; single plane, left ventricular systole. End systolic volume: apical two chambers; modified ellipsoid, Left ventricular end systolic volume: apical four chambers; modified ellipsoid, Left ventricular end systolic volume: apical two chambers; single plane, Left ventricular end systolic volume: apical four chambers; single plane, Left ventricular septum diameter at end diastole, Left atrial diameter, Left atrial volume derived from apical two chambers; modified ellipsoid, Left atrial volume derived from apical four chambers; modified ellipsoid, Velocity-time integral proximal to occlusion, Maximum left ventricular velocity proximal to occlusion, Mean left ventricular velocity proximal to occlusion, Left ventricular area at end diastole derived from apical two chambers, Left atrial area derived from apical four chambers left ventricular area at end diastole derived from apical two chambers, left ventricular area at end systole derived from apical four chambers, left ventricular intraventricular size at end diastole, left ventricular intraventricular size at end systole, left ventricular long axis length at end diastole derived from apical two chambers, left ventricular long axis length at end diastole derived from apical four chambers, left ventricular long axis length at end systole derived from apical two chambers, left ventricular long axis length at end systole derived from apical four chambers, left ventricular outflow tract area, left ventricular outflow tract diameter, left ventricular These may include left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C.The diagnostic parameters may include a diagnosis of acute rheumatic fever, a diagnosis of chronic rheumatic heart disease, a diagnosis of hypertensive disease, a diagnosis of ischemic heart disease, a diagnosis of pulmonary heart disease and pulmonary circulation diseases, a diagnosis of acute pericarditis, a diagnosis of other forms of heart disease, a diagnosis of acute myocarditis, a diagnosis of cardiomyopathy, a diagnosis of cardiac arrest, a diagnosis of paroxysmal tachycardia, a diagnosis of atrial fibrillation, a diagnosis of heart failure, a diagnosis of cerebrovascular disease, a diagnosis of diseases of the arteries, arterioles, and capillaries, a diagnosis of diseases of the veins, lymphatic vessels, and lymph nodes, a diagnosis of hypotension, a diagnosis of other and unspecified diseases of the circulatory system, a diagnosis of diabetes, a diagnosis of congenital heart disease, a diagnosis of dyslipidemia, and a diagnosis of chronic kidney disease.

[0008] The method may further include receiving electronic health record information associated with the patient and providing the electronic health record information to a trained neural network, the trained neural network being further trained to generate a mortality risk score based on the input electronic health record information. The electronic health record information includes values ​​of several parameters including age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function, the values ​​being associated with the patient. The electronic health record information may include values ​​of several parameters including demographic parameters, vital parameters, laboratory measured parameters, echocardiogram-based parameters, and diagnostic parameters. The demographic parameters may include age, sex, and smoking status. The vital parameters may include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure. The laboratory measured parameters may include low density lipoprotein level and high density lipoprotein level.Echocardiogram-based parameters were physician-reported left ventricular ejection fraction, aortic regurgitation deceleration gradient, aortic regurgitation peak velocity, velocity-time integral of flow distal to the aortic valve, peak velocity of flow distal to the aortic valve, mean velocity of flow distal to the aortic valve, aortic root diameter, ascending aorta diameter, left ventricular end-diastolic volume: apical two-chamber; modified ellipsoid, left ventricular end-diastolic volume: apical four-chamber; modified ellipsoid, left ventricular end-diastolic volume: apical two-chamber; single plane, left ventricular end-diastolic volume: apical four-chamber; single plane, left ventricular systole. End systolic volume: apical two chambers; modified ellipsoid, Left ventricular end systolic volume: apical four chambers; modified ellipsoid, Left ventricular end systolic volume: apical two chambers; single plane, Left ventricular end systolic volume: apical four chambers; single plane, Left ventricular septum diameter at end diastole, Left atrial diameter, Left atrial volume derived from apical two chambers; modified ellipsoid, Left atrial volume derived from apical four chambers; modified ellipsoid, Velocity-time integral proximal to occlusion, Maximum left ventricular velocity proximal to occlusion, Mean left ventricular velocity proximal to occlusion, Left ventricular area at end diastole derived from apical two chambers, Left atrial area derived from apical four chambers left ventricular area at end diastole derived from apical two chambers, left ventricular area at end systole derived from apical four chambers, left ventricular intraventricular size at end diastole, left ventricular intraventricular size at end systole, left ventricular long axis length at end diastole derived from apical two chambers, left ventricular long axis length at end diastole derived from apical four chambers, left ventricular long axis length at end systole derived from apical two chambers, left ventricular long axis length at end systole derived from apical four chambers, left ventricular outflow tract area, left ventricular outflow tract diameter, left ventricular These may include left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C., left ventricular posterior wall thickness at 120° C. to 180° C.The diagnostic parameters may include a diagnosis of acute rheumatic fever, a diagnosis of chronic rheumatic heart disease, a diagnosis of hypertensive disease, a diagnosis of ischemic heart disease, a diagnosis of pulmonary heart disease and pulmonary circulation diseases, a diagnosis of acute pericarditis, a diagnosis of other forms of heart disease, a diagnosis of acute myocarditis, a diagnosis of cardiomyopathy, a diagnosis of cardiac arrest, a diagnosis of paroxysmal tachycardia, a diagnosis of atrial fibrillation, a diagnosis of heart failure, a diagnosis of cerebrovascular disease, a diagnosis of diseases of the arteries, arterioles, and capillaries, a diagnosis of diseases of the veins, lymphatic vessels, and lymph nodes, a diagnosis of hypotension, a diagnosis of other and unspecified diseases of the circulatory system, a diagnosis of diabetes, a diagnosis of congenital heart disease, a diagnosis of dyslipidemia, and a diagnosis of chronic kidney disease.

[0009] In the method, the echocardiographic video of the heart associated with the patient can be associated with a single echocardiographic view, which can be a parasternal long axis view.

[0010] The method may further include generating a report based on the mortality risk score, wherein the report includes at least one of information regarding potential treatments for the patient or links to information regarding potential treatments for the patient, wherein the potential treatments for the patient include at least one of a heart transplant, implantation of a mechanical assist device, placement of a defibrillator, palliative care, or hospice.

[0011] In the method, the mortality risk score associated with the patient can provide a physician or health care administrator with sufficient information to make a decision regarding potential treatment for the patient, the potential treatment including at least one of a heart transplant, implantation of a mechanical assist device, placement of a defibrillator, palliative care, or hospice. The decision can be eligibility for the treatment.

[0012] Some embodiments of the present disclosure provide a system including at least one processor coupled to at least one memory including instructions, the at least one processor executing the instructions to receive echocardiographic video of a heart associated with a patient, provide the echocardiographic video to a trained neural network, where the trained neural network is trained to generate a mortality risk score based on the input echocardiographic video, receive a mortality risk score associated with the patient from the trained neural network, and output the mortality risk score associated with the patient to at least one of the memory or a display for viewing by a physician or health care administrator.

[0013] In the system, the at least one processor can further execute instructions to receive electronic health record information associated with the patient and provide the electronic health record information to a trained neural network, the trained neural network being further trained to generate a mortality risk score based on the input electronic health record information, the electronic health record information including values ​​for several parameters including age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function.

[0014] To the accomplishment of the foregoing and related ends, the invention comprises the features hereinafter fully described. The following description and the annexed drawings set forth in detail certain illustrative aspects of the invention. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention may be employed. Other aspects, advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the drawings. [Brief description of the drawings]

[0015] [Figure 1]FIG. 1 is a diagram of a neural network architecture for mortality prediction from echocardiographic video and electronic health record (EHR) data. [Diagram 2] FIG. 1-year mortality prediction performance ranking for echocardiographic views alone. [Diagram 3] FIG. 13 is a graph of model performance quantified as the area under the curve (AUC) of 1-year mortality prediction across all views using four different neural network architectures. [Figure 4] Graph of the AUC of 1-year mortality prediction across all views using various levels of reduced resolution from native (×1) to 4-fold (×4). [Diagram 5] Graph of mortality prediction performance for echocardiographic videos alone at 3, 6, 9, and 12 months for all views. [Figure 6A] This is a graph of the accuracy rate for two cardiologists compared to the machine. [Figure 6B] FIG. 13 is a diagram of receiver operating characteristic curves for two cardiologists and the machine demonstrating predictive performance. [Figure 7] Graph of the learning curve for the full (158) EHR variable model compared to the full EHR variable plus video model. [Figure 8A] FIG. 2 is an example live video diagram. [Figure 8B] FIG. 2 is an example annotated video. [Figure 9] 1 is a plot of the number of patients for the study who required 3, 6, 9, and 12 months of follow-up. [Figure 10] Graph of 1-year mortality prediction performance for all echocardiographic views using only raw video (blue) and using raw video together with optical flow features (grey). [Figure 11]FIG. 1 is a diagram of the interface of a web application developed for cardiologists to predict one-year survival after echocardiography. [Figure 12] FIG. 1 is a diagram of an exemplary process for predicting all-cause mortality in a patient for a given period (i.e., one year) based on echocardiography data as well as EHR data. [Figure 13] FIG. 1 is a diagram of an exemplary system for implementing the foregoing disclosure. [Figure 14] FIG. 13 is a diagram of an example interface for a first survey. [Figure 15] FIG. 1 is a diagram of an interface for paired investigations. [Figure 16] FIG. 11 is a diagram of the interface for the model-assisted portion of the third study. [Figure 17] FIG. 2 is a diagram of an exemplary first model and an exemplary second model. [Figure 18] FIG. 11 is a diagram of an exemplary third model and an exemplary fourth model. [Figure 19] FIG. 1 is a diagram of an exemplary trained model. [Figure 20] FIG. 2 is a diagram of another exemplary process for predicting a relevant clinical endpoint, such as all-cause mortality, in a patient for a given period of time. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the description herein of specific embodiments is not intended to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims.

[0017] Various aspects of the present invention will now be described with reference to the accompanying drawings. It should be understood, however, that the drawings and the following detailed description relating thereto are not intended to limit the claimed subject matter to the particular forms disclosed. Rather, the intent is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claimed subject matter.

[0018] As used herein, terms such as "component," "system," and the like are intended to refer to a computer-related entity that is either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computer and a computer may be a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed among two or more computers or processors.

[0019] The term "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0020] Furthermore, the disclosed subject matter may be implemented as a system, method, apparatus, or article of manufacture using programming and / or engineering techniques to generate software, firmware, hardware, or any combination thereof for controlling a computer or processor-based device to implement the aspects detailed herein. The term "article of manufacture" (or alternatively, "computer program product") as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to, magnetic storage devices (hard disks, floppy disks, magnetic strips, etc.), optical disks (compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (cards, sticks, etc.). Alternatively, it should be understood that carrier waves may be used to carry computer-readable electronic data, such as those used in sending and receiving e-mail or in accessing a network, such as the Internet or a local area network (LAN). Transient computer-readable media (carrier wave and signal-based) should be considered separately from non-transient computer-readable media, such as those described above. Of course, those skilled in the art will recognize that many modifications can be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0021] Hereinafter, unless otherwise indicated, the following terms and phrases are used in this disclosure as described: The term "provider" is used to refer to the entity that operates the entire system disclosed herein, and includes a company or other entity that runs the servers, maintains the databases, and employs people with the many different knowledge and skills required to configure, maintain, and adapt the disclosed system to accommodate new data types, new medical and treatment insights, and other needs. Exemplary provider employees may include researchers, clinical trial designers, oncologists, neurologists, psychiatrists, data scientists, and many others with specialized knowledge and skills.

[0022] The term "physician" is used generally to refer to health care providers including, but not limited to, primary care physicians, specialists, oncologists, neurologists, nurses, and medical assistants.

[0023] The term "researcher" is used generally to refer to anyone who performs research, including but not limited to a radiologist, data scientist, or other health care provider. Some may be both physicians and researchers, while others may only function in one of those capacities.

[0024] Recent advances in "deep" learning (deep neural network (DNN)) techniques such as convolutional neural networks (CNN), recurrent neural networks (RNN), dropout regularization, and adaptive gradient descent algorithms, combined with massively parallel computing hardware (graphics processing units), have enabled state-of-the-art predictive models for image data, time series data, and video-based data. For example, DNNs have shown promise in diagnostic applications such as the pathogenesis of diabetic retinopathy, skin cancer, pulmonary nodules, cerebral microbleeds, and cardiac hypertrophy. However, machine learning opportunities are not limited to such diagnostic tasks.

[0025] For example, prediction of future clinical events is a natural but relatively unexplored extension of machine learning in medicine. Nearly all medical decisions depend on accurate predictions. Diagnosis is offered to patients as it helps establish the typical future clinical course of patients with similar symptoms, and treatment is offered as a prediction of how to positively impact the predicted future clinical course. Thus, using computer-based methods to directly predict future clinical events is an important task where computers are likely to be able to assist with human interpretation due to the inherent complexity of the problem. For example, a recent paper on 216,221 patients demonstrated how a random forest model can predict in-hospital mortality with high accuracy. Deep learning models have also recently been used to predict mortality risk among hospitalized patients to support palliative care referrals. In cardiology, variables derived from electronic health records have been used to predict 2-5 year all-cause mortality in patients undergoing coronary computed tomography, 5 year cardiovascular mortality in the general clinical population, and up to 5 year all-cause mortality in patients undergoing echocardiography.

[0026] Notably, these early outcome prediction studies in cardiology used only human-derived features from the images, i.e., “hand-crafted” features, rather than automatically analyzing raw imaging data. While this use of hand-crafted features is important, techniques that are not biased by human opinion and are not limited by human perception, human ability in pattern recognition, and labor may be more robust. That is, strong potential exists for automated analysis that leverages all available data in the image, rather than a select few clinical or clinically inspired measurements. Furthermore, the potential benefits of this technique for echocardiography may be enhanced by the additional availability of abundant temporal (video) data. DNNs make this unique technique possible. However, using video data also increases technical complexity, and thus early efforts to apply deep learning to echocardiography focused on incorporating individual images, rather than entire videos.

[0027] In this disclosure, it is shown that DNNs can predict one-year mortality directly from echocardiographic videos with good accuracy, and that this accuracy can be improved by incorporating additional clinical variables from electronic health records. This is done through technological advances that leverage the entire echocardiographic video to make predictions using a 3D DNN. In addition to this technological advancement, direct clinical relevance is demonstrated by showing that DNNs are more accurate in predicting one-year mortality compared to two expert physician cardiologists.

[0028] result In this work, a full 3D convolutional neural network (CNN) design is utilized (Figure 1). CNNs are neural networks that exploit spatial coherence in images to drastically reduce the number of parameters that a fully connected network needs to learn. CNNs have shown promise in image classification tasks, even surpassing human capabilities. Details of the additional model architecture attempted (time-sliced ​​2D CNN + long short-term memory network [LSTM]) are described in the Methods section.

[0029] Figure 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. The convolutional layers (Conv) are shown in solid outline in the top box, and the tabular EHR data layer (Tab) is shown in dashed outline in the bottom box. The convolutional layers consist of a Convolutional Neural Network (CNN), a Batch Normalization (Batch Norm.), a Rectified Linear Unit (ReLU), and a 3D Max Pool layer. The tabular layers consist of a Fully Connected Layer (Dense) with Sigmoid Activation and a Dropout Layer. The input video dimensions are 150x109x60 pixels, and the output dimensions of all layers are shown. The mortality prediction is output as a risk score associated with the patient's predicted mortality.

[0030] 723754 clinically acquired echocardiographic videos were obtained (approximately 45 million images) from 27028 patients who were linked to longitudinal follow-up data for at least 1 year to know if the patient survived or died within that time frame. Overall, 16% of patients in this cohort died within 1 year of the echocardiogram being obtained. Based on the power calculations detailed in the "Methods" section, data were separated from 600 patients for validation and comparison against two independent cardiologists, and the remaining data were used for a 5-fold cross-validation scheme.

[0031] During an echocardiogram acquisition (or other medical video acquisition of the heart, including but not limited to videos generated using cardiac MRI or CT), images of the heart and great vessels are acquired in different two-dimensional planes, or "views", standardized according to clinical guidelines. A separate model was generated for each of the 21 standard echocardiographic views, and we showed that the proposed models were able to accurately predict 1-year survival using only raw video data as input (Figure 2). The selected 3D CNN architecture (AUC range: 0.695-0.784) outperformed the 2D CNN+LSTM architecture (AUC range: 0.703-0.752) for most views. In both cases, the parasternal long axis ("PL DEEP") view had the best performance. This result was consistent with clinical intuition, as the PL DEEP view is typically reported by cardiologists as the most informative "summary" view of overall cardiac health. This is because the PL DEEP view includes elements of the left ventricle, left atrium, right ventricle, aortic valve, and mitral valve, as well as whether or not pericardial or left pleural effusion is present, all within a single view.

[0032] Figure 2 shows the 1-year mortality prediction performance ranking for each echocardiographic view only (no EHR data) using the 2D CNN+LSTM architecture (grey) and the 3D CNN (blue) model. Error bars indicate one standard deviation above and below the mean across five folds. See Table 1 below for abbreviations of all view labels. [Table 1]

[0033] These results were relatively insensitive to image resolution (no significant differences were observed between models using full native resolution images (400 × 600 pixels) and models using reduced resolution images (100 × 150 pixels); Figure 4). Figure 4 is a graph of the AUC of 1-year mortality prediction across all views using various levels of reduced resolution ranging from native (×1) to 4-fold (×4). Similarly, adding derived optical flow velocity maps to the model along with pixel-level data did not improve prediction accuracy (Figure 10). Figure 10 is a graph of 1-year mortality prediction performance ranking for all echocardiographic views using only raw video (blue) and all echocardiographic views using raw video with optical flow features (gray). Note that full native resolution training was only performed for selected views due to the computational time required to complete the experiments at this resolution.

[0034] The predictive accuracy of the model was then investigated in additional survival intervals, including 3, 6, 9, and 12 months after echocardiography. The model generally performed better in longer intervals, but the AUC for all cases was greater than 0.64 (Figure 5).

[0035] Figure 5 is a graph of mortality prediction performance for echocardiographic videos only at 3, 6, 9, and 12 months for all views. Error bars indicate one standard deviation above and below the mean across five folds.

[0036] Clinical ("EHR") variables were then selected from each patient, including age, tricuspid regurgitation peak velocity, heart rate, low-density lipoprotein [LDL], left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function. These 10 variables have previously been shown to contain greater than 95% power to predict 1-year survival in 171,510 patients, and their addition improved the accuracy of predicting 1-year survival for all echocardiographic views, with AUCs ranging from 0.79 to 0.82 (compared to 0.70 to 0.78 without these 10 EHR variables). A software platform was then developed (see Methods below) that was used to display echocardiographic videos of interest along with the 10 selected EHR variables to two independent cardiologists who were blinded to the clinical outcomes. Based on the data presented, cardiologists assessed whether each of the 600 patients (a separate test set randomly drawn from the original dataset of parasternal long axis views and not used to train the machine) would survive for one year. The final trained model (trained on all but these 600) was also applied to the same separate test set.

[0037] The overall accuracy of the model (75%) was significantly higher than that of the cardiologists (56% and 61%, p = 4.2 × 10 by Bonferroni-adjusted post-hoc analysis). -11 and 6.9 × 10 -7 , Fig. 6A). Cardiologists were found to tend to overestimate the likelihood of survival, resulting in high specificity (97% and 91%, respectively), but low sensitivity (16% and 31%, respectively), whereas the model, by design, was balanced between sensitivity and specificity (both 75%). Furthermore, as shown in Fig. 6B, the operating points of the individual cardiologists fell below the receiver operating characteristic curve of the model (as opposed to falling at a different point on this curve), suggesting poorer predictive performance in this task.

[0038] Figures 6A and 6B are graphs of cardiologist vs. machine performance for 1-year mortality prediction for 600 samples with balanced prevalence. Figure 6A shows the accuracy rate for each cardiologist and machine, and Figure 6B shows the sensitivity vs. specificity for each cardiologist and machine. Figure 6A shows accuracy in bars, and sensitivity (red) and specificity (green) as triangles. Figure 6B shows the cardiologist operating points as orange dots, the receiver operating characteristic curve for the machine performance in blue, and the machine operating points as blue dots.

[0039] Beyond the limited inputs selected for clinical expert comparison, we sought to further characterize model performance unconstrained by data input limitations. That is, additional experiments were completed that varied the input combinations of structured data (none, limited set [top 10 EHR variables], full set [158 EHR variables as described in Methods below]) and echocardiographic videos (none, single view, all 21 views). Models without video were trained using all available data in the Structured Echocardiography Measurements Database (501449 valid studies), and models with video were trained with all available videos for each view, ranging from 11020 to 22407 for single videos and 26428 for combinations. In all cases, the test set was 600 patients that were excluded for comparison with clinical experts.

[0040] Table 2 below shows that all videos combined with the full EHR variable set had the highest AUC in the test set of 600 studies that were excluded, indicating the potential to further enhance the performance of an already clinically excellent model. Several general trends were also noted. First, a single video view outperformed a model that included 10 EHR variables as input. Second, multiple videos had higher performance than a single video. Third, the learning curve for multi-video prediction (Figure 7) showed that, despite having access to a large dataset (26428 echocardiographic videos), more samples could potentially yield better performance for multi-video prediction. In contrast, the performance of the model with only the full EHR data, which was consistently inferior to the model with the full EHR plus videos, was beginning to plateau. Thus, the novel multimodal DNN approach including echocardiographic videos provides improved performance for this clinical prediction task compared to what could be achieved using only EHR data (including handcrafted features derived by humans from the videos). [Table 2]

[0041] Table 2 shows the AUC scores for each data modality of EHR and echo video combinations in the 600 excluded studies used for comparison with cardiologists. The "No Video" model was trained on all available studies, while "Single Video" and "All Videos" were trained on the subset where video data was available. The "No EHR Variables" and "No Video" cells show random guesses.

[0042] Figure 7 is a graph of the learning curve for the full (158) EHR variables model compared to the full EHR variables plus video. The AUC is reported for a set of 600 patients as a function of training set size, ranging from 10 to the maximum number of datasets available for the given data input, which was 501449 for EHR variables and 26428 for the full EHR+video.

[0043] Using both raw video data and relevant clinical data extracted from electronic health records, we show that DNNs may help cardiologists predict clinically relevant endpoints, e.g., mortality after echocardiography. A dataset of 723,754 clinically acquired videos of the heart, consisting of approximately 45 million images, was utilized to train the DNN. The ability of the DNN to discriminate 1-year survival was shown to exceed that of trained cardiologists, even with limited model input, suggesting that these models can add value beyond standard clinical interpretation. To our knowledge, no prior work has demonstrated the ability to train deep neural networks to directly predict future clinically relevant events from image pixel-level data. Additional experiments demonstrated the opportunity to incorporate more EHR variables, use all echocardiography views simultaneously, and leverage more data for model training to achieve further significant performance improvements.

[0044] To demonstrate feasibility for this initial study, one-year all-cause mortality was chosen as a highly significant and easily measured clinical outcome. Importantly, all-cause mortality is a well-defined endpoint free of bias that may be introduced in endpoints such as cardiovascular-specific mortality, and can be easily extracted from EHRs that have been validated against national death index databases. Furthermore, mortality prediction is highly relevant for many applications in cardiology, as evidenced by the large number of clinical risk scores currently in clinical use (e.g., Framingham score, TIMI score, and GRACE score). It is understood that DNNs can be trained to predict all-cause mortality over periods other than one year, e.g., six months, two years, three years, five years. Furthermore, DNNs can be trained to predict many other clinically relevant endpoints, such as hospitalizations, future disease onset, response to treatment, and health care utilization.

[0045] method Image acquisition and pre-processing Echocardiographic studies consisted of several videos containing multiple views of the heart. Two clinical databases, Philips iSite and Xcelera, contained all echocardiograms collected at Geisinger. DCM4CHEE (version 2.0.29) and AcuoMed (version 6.0) software were used to obtain DICOM files for each echocardiographic video.

[0046] The acquired DICOM files included annotated videos (e.g., marked with view names) and raw videos as the device was configured to store them. Without loss of generality, the raw videos were used for all analyses. FIG. 8A is an exemplary raw video, and FIG. 8B is an exemplary annotated video. The raw videos included only the beamformed ultrasound images stored in a stream of bytes format, and the annotated videos included the raw videos plus artificial annotations. All raw videos were linearly interpolated to 30 frames per second.

[0047] Along with the video data, the DICOM files contained tags labeling the view as to which particular image orientation was acquired. These view tags had slight variations between studies for the same type of view. For example, an apical 4-chamber view may be tagged as “a4”, “a4 2d”, or “ap4”. Samples of each unique tag were visually inspected and grouped into 30 common views (Table 1). As each video from a view group could potentially have different dimensions, all videos were normalized to the most common row and column dimensions from the view. Each frame was cropped / padded with zeros to match the most common dimensions within the view group. Finally, a Philips-generated DICOM file with the raw video and view labels was obtained, and videos less than 1 second long were filtered out.

[0048] Preprocessing of Electronic Health Record Data The EHR contained 594,862 echocardiogram studies from 272,280 unique patients performed over a 19-year period (February 1998 to September 2018). For each study, automated or physician-reported echocardiogram measurements (n=480) were extracted along with patient demographics (n=3), vitals (n=5), laboratory (n=2), and problem list diagnosis codes (n=90; International Classification of Diseases, 10th Revision (ICD-10)). For measurements taken outside of the echocardiogram study, such as fasting LDL, HDL, blood pressure, heart rate, and weight and height measurements, the closest (pre or post) within 6 months was obtained.

[0049] All continuous variables were removed for values ​​outside physiological limits that may have been caused by input errors. When limits could not be defined for measurements, extreme outliers that met two rules were removed: 1) values ​​exceeding the mean plus or minus three standard deviations and 2) values ​​below the 25th percentile minus three interquartile ranges or above the 75th percentile plus three interquartile ranges. Removed outliers were set as missing.

[0050] Missing data were imputed from continuous variables in two steps. First, time interpolation was performed to fill in missing measurements using all available studies of an individual patient; missing values ​​between echocardiographic sessions were linearly interpolated if complete values ​​were found in adjacent echocardiograms. Then, 115 of the 480 echocardiographic measurement variables with more than 10% non-missing measurements were retained for multivariate imputation by chained equations (MICE) to complete the full dataset.

[0051] Reported diastolic function was coded in an ordinal fashion, with −1 for normal, 0 for impaired function (no grade reported), and 1, 2, and 3 for diastolic function impairment grades I, II, and III, respectively. After imputation of continuous measures, missing diastolic function assessments were imputed by training a logistic regression classifier to predict impairment grade (−1, 1, 2, or 3) in a one-versus-others classifier framework, using the 278,160 studies in which diastolic function was known.

[0052] Following imputation, physicians reported left ventricular ejection fraction (LVEF) plus 57 other independent, nonredundant echocardiographic measures (i.e., variables derived from other measures were retained; total n = 58 echocardiographic measures).

[0053] Patient age and survival time were calculated from the date of echocardiogram. Patient status (dead / alive) was based on the last known live encounter or confirmed date of death and was cross-referenced monthly against the national death index database in the EHR from which it was obtained.

[0054] A list and description of all 158 EHR variables used in the proposed model are presented below in Table 3. [Table 3A] [Table 3B] [Table 3C] [Table 3D] [Table 3E]

[0055] Data pruning Image collection and preprocessing resulted in 723754 videos (average 22.7 videos per study) from 31874 studies performed on 27028 patients. Imaging and EHR data were linked and imaging without EHR data was discarded. Studies without sufficient follow-up for a given survival experiment (3, 6, 9, and 12 months) were also removed. A single study per patient was then retained by randomly sampling one study per patient. This ensured that images from a single patient did not appear multiple times across training, validation, and testing groups.

[0056] To estimate and compare the prognostic accuracy between the model and the two cardiologists, at least 600 patients (300 surviving and 300 deceased) were required, as indicated by a sample size calculation using Pearson's chi-square test. A 10% difference in accuracy between the machine and the cardiologists (80% vs. 70%), a power of 80%, a significance level of 5%, and a discrepancy of about 40% were assumed. This was calculated using power analysis software (PASS v15). Thus, 300 studies of patients who survived and 300 studies of patients who died within the set experimental thresholds were randomly sampled per view and set aside from the valid sample to later compare the machine's performance against the two independent cardiologists. Only the parasternal long axis view (representing the best performing model, and the cardiologists' preference for the most comprehensive single view) was finally used for the cardiologist comparison. The total number of valid samples per experiment and view is shown in Table 4 below and Figure 9. FIG. 9 is a plot of the number of patients for the studies that required 3, 6, 9, and 12 months of follow-up, and the percentage of patients who died (hatched bars) (shown in Table 3 above). [Table 4]

[0057] The parasternal long mitral valve view, parasternal long pulmonary valve view, short axis apical zoom view, short axis midpapillary zoom view, parasternal long flaccid view, apical 3 zoom, and apical 2 zoom were excluded because there were not enough available samples to perform the experiments.

[0058] Model Selection Four different architectures were investigated for echocardiography video classification: 1) time-distributed 2D convolutional neural network (2D CNN) with long short-term memory (LSTM), 2) time-distributed 2D CNN with Global Average Pooling (GAP), 3) 3D CNN, and 4) 3D CNN with GAP. For simplicity, the four candidate architectures are abbreviated as 2D CNN+LSTM, 2D CNN+GAP, 3D CNN, and 3D CNN+GAP.

[0059] 2D CNN+LSTM consists of 2D CNN branches distributed across all frames of a video. This architecture was used for video description problems where all frames from a video belong to the same scene or action. Since all frames of an echocardiography video belong to the same scene or view, it is correct to assume that static features are commonly found by the same 2D kernel across the entire video. This assumption was put into practice for echocardiography view classification. The LSTM layer aggregates the CNN features over time to output a vector that represents the entire sequence.

[0060] The 2D CNN+GAP approach replaced the LSTM layer with average CNN features as a temporal aggregation of frames. The GAP layer offers two advantages: it does not require trainable parameters, saving 1008 parameters from the LSTM layer, and it enables feature interpretation. The final fully connected layer after GAP provides a weighted average of the CNN features, which can indicate which sections of the video were weighted more in the final decision.

[0061] 3D CNN methods aggregate temporal and spatial features as the input data flows through the network. 3D CNN has also shown successful application to video classification. In contrast to 2D CNN methods, 3D CNN incorporates information from neighboring frames at every layer to extract time-space dependent features.

[0062] The 3D CNN approach replaces the flattening operation for a GAP layer. Similar to the 2D CNN+GAP approach, the GAP layer reduces the number of input features to the final dense layer, thus reducing the number of parameters from 641 to 17, while at the same time enabling traceback of the contributions of the video features.

[0063] The convolutional units for 2D and 3D CNN were defined as a sequence of seven layers in the following configuration: CNN layer, batch normalization, ReLU, CNN layer, batch normalization, ReLU, and max pooling (see Figure 1). The dimensions of all kernels were set to 3, and max pooling was applied in a 3x3 window for 2D kernels and 3x3x3 for 3D kernels.

[0064] A detailed description of the parameter numbers for the 2D CNN+LSTM architecture is shown in Table 5 below, the 2D CNN+GAP is shown in Table 6 below, the 3D CNN is shown in Table 7 below, and the 3D CNN+GAP is shown in Table 8 below. [Table 5] [Table 6] [Table 7] [Table 8]

[0065] All four candidate architectures were applied to all identified echocardiographic views with 1-year mortality labels, and 3D CNN consistently performed the best (Figure 3). Figure 3 shows a plot of the AUC for 1-year mortality prediction across all views by four different neural network architectures, namely, 2D CNN+Global Average Pooling (GAP; dark grey), 2D CNN+Long Short-Term Memory (LSTM; light grey), 3D CNN+GAP (light blue), and 3D CNN (dark blue).

[0066] Similarly, the performance improvement was evaluated at different image resolutions. The video resolution was reduced by a factor of 2, 3, and 4. No significant performance degradation was observed that was consistent across all views (Figure 4). Therefore, it was decided to perform all experiments at a factor of 4 reduced resolution to reduce the computational cost.

[0067] To incorporate EHR data into the prediction, a three-layer multi-layer perceptron (MLP) with 10 hidden units in each layer was trained. The last 10 hidden units were then concatenated with a CNN branch (see Figure 1).

[0068] Training Algorithm The RMSProp algorithm was used to train the network with LSTM coupling, and AdaGrad was used for the 3D CNN architecture. Each iteration of the 5-fold cross-validation included training, validation, and test sets. The training and test sets were sampled to have the same prevalence of surviving patients, while the validation set was sampled in a balanced ratio. The validation set consisted of 10% of the training set.

[0069] As the DNN was trained, the loss (binary cross entropy) on the validation set was evaluated at each epoch. If the validation loss did not decrease for more than 10 epochs, the training was stopped and the performance in AUC on the test set was reported. The maximum number of epochs was set to 1000, keeping the default training parameters defined by the software Keras (version 2.2). Training was always terminated before the maximum number of epochs was reached.

[0070] Since the prevalence of each patient class was disproportionate (approximately 16% of deceased patients), the weights for each class were set as follows:

number

[0071] All training was performed on an NVIDIA DGX1 platform. Each fold was trained separately on each of the eight available GPUs. The main experiment, shown in Figure 2, took a total of six days to complete.

[0072] The effect of adding optical flow input Optical flow velocity maps have been shown to be informative along with the original video for classification tasks. Therefore, dense optical flow vectors of echocardiography raw videos were calculated using Gunnar Farneback's algorithm implemented in the OpenCV (version 2.4.13.7) software library. The pyramid scale was set to 0.5, the number of levels was set to 3, and the window size was set to 5 pixels. The vectors were then converted to a color video where color indicates direction and brightness indicates amplitude (as in the HSV color space). This resulted in an image video that was fed into the neural network model through a separate 3D CNN branch along with the raw video. As can be seen in Figure 10, this combination of optical flow video and raw video did not consistently result in improved model performance compared to models using only raw video. Therefore, optical flow was not used for the final study analysis.

[0073] The use of balanced outcomes in cardiologist survey datasets As explained in the data pruning section, the study of 600 patients used to compare cardiologists and model accuracy was purposely balanced with respect to mortality outcomes (300 deaths and 300 survival at 1 year) to ensure sufficient power to detect differences in performance. Cardiologists were blinded to this distribution at the time of review.

[0074] Cardiologist Survey Software A web application was deployed with the interface shown in Figure 11, which is a web application developed for cardiologists to predict survival one year after an echocardiogram. The application required the cardiologist to enter their institution's credentials for access. Ten EHR variables and two versions of the video (raw and annotated) are shown. The application then recorded the cardiologist's prediction upon clicking either the "survive" or "death" button.

[0075] Statistical analysis of machine vs. cardiologist comparison The cardiologist responses were binary and the machine responses were continuous. Before running the final comparison experiment, a threshold of 0.5 was set for the machine response. As all responses were recorded for the same sample, a Cochran's Q test was performed to assess whether the three responses significantly differed in the proportion of correctly classified samples. This test showed that there was sufficient evidence that at least one of the responses was significantly different with a p-value of 1.8e-15. Post-hoc analysis of pairwise comparisons between the three responses showed a p-value of 0.003, 4.2e for the pairs Cardiologist 1 vs Cardiologist 2, Cardiologist 1 vs Machine, and Cardiologist 2 vs Machine, respectively. -11 , and 6.9e -7 This resulted in a Bonferroni adjusted p-value of .

[0076] Turning now to FIG. 12 as well as FIG. 1, an exemplary process 100 for predicting relevant clinical endpoints, such as all-cause mortality in a patient for a predetermined period (i.e., one year), based on cardiac videos (in this case, echocardiography data) as well as any additional available EHR data is shown. The process 100 predicts a risk score for a patient based on a neural network, which may be a deep neural network, such as a convolutional neural network, trained using cardiac videos, such as echocardiographic videos, and the EHR variables described above. The process 100 may be employed in a health analytics module used by a care team including a physician, for example, a physician deploying resources across a population of 10,000 patients with heart failure, to treat a patient or for population-level management of patients. In some embodiments, the process 100 may be implemented as instructions (e.g., computer-readable instructions) on at least one memory and executed by one or more processors coupled to the at least one memory.

[0077] At 102, the process 100 may receive an echocardiographic video of a heart associated with a patient. The echocardiographic video may include echocardiographic video frames. The video frames may include video frames taken from one or more views of the patient's heart. For example, the video frames may include video frames taken at 21 different views of the heart. In some embodiments, the echocardiographic video may be associated with one or more echocardiographic views including an apical two-chamber view, an apical three-chamber view, an apical four-chamber view, an apical four-chamber view focused on the right ventricle, an apical five-chamber view, a parasternal long axis view, a parasternal long descending aorta view, a parasternal long mitral valve view, a parasternal long pulmonary valve view, a parasternal long right ventricular inflow view, a parasternal long zoom aortic valve view, a parasternal short aortic valve view, a parasternal short pulmonary valve and pulmonary artery view, a parasternal short tricuspid valve view, a short axis apical view, a short axis base view, a short axis central papillary view, a subcostal four-chamber view, a subcostal hepatic vein view, a subcostal atrial septum view, a subcostal inferior vena cava view, a subcostal right ventricular view, a suprasternal fossa view, a short axis central papillary view, a short axis apical view, an apical three-chamber zoom view, an apical two-chamber zoom view, and / or a short axis base view. In some embodiments, the echocardiographic video may be associated with a single view of the heart. In some embodiments, the single view may be a parasternal long axis view. The parasternal long axis view has been shown to be superior to other single views, as explained above. Process 100 may then proceed to 104.

[0078] At 104, the process 100 may receive EHR data including values ​​of several parameters associated with the patient. The EHR data is optional and in some embodiments, the process 100 may receive only the echocardiogram video at 102. Using the EHR data may improve the performance of the risk score generation. In some embodiments, the parameters may include age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein [LDL], left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function. In some embodiments, the variables may include at least some of the variables listed in Table 3 above. In some embodiments, the EHR data may include values ​​of several parameters including demographic parameters, vital parameters, laboratory measurement parameters, echocardiogram-based parameters, and diagnostic parameters. The values ​​may be associated with the patient. In some embodiments, the demographic parameters may include age, sex, and / or smoking status. In some embodiments, the vital parameters may include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure. In some embodiments, the laboratory measured parameters can include low density lipoprotein levels and / or high density lipoprotein levels. In some embodiments, the echocardiogram-based parameters can include physician-reported left ventricular ejection fraction, aortic regurgitation deceleration gradient, aortic regurgitation peak velocity, velocity-time integral of flow distal to the aortic valve, peak velocity of flow distal to the aortic valve, mean velocity of flow distal to the aortic valve, aortic root diameter, ascending aorta diameter, left ventricular end-diastolic volume: apical two chamber, left ventricular end-diastolic volume: apical four ... Volume: apical two chambers, Left ventricular end diastolic volume: apical four chambers, Left ventricular end systolic volume: apical two chambers, Left ventricular end systolic volume: apical four chambers, Left ventricular end systolic volume: apical two chambers, Left ventricular end systolic volume: apical four chambers, Left ventricular septal diameter at end diastole, Left atrial diameter, Left atrial volume derived from apical two chambers, Left atrial volume derived from apical four chambers, Velocity-time integral proximal to occlusion, Maximum left ventricular velocity proximal to occlusion, Mean left ventricular velocity proximal to occlusion degree, left ventricular area at end diastole derived from apical two chambers, left ventricular area at end diastole derived from apical four chambers, left ventricular area at end systole derived from apical two chambers, left ventricular area at end systole derived from apical four chambers, left ventricular intraventricular measurement at end diastole, left ventricular intraventricular measurement at end systole, left ventricular long axis length at end diastole derived from apical two chambers, left ventricular long axis length at end diastole derived from apical four chambers, left ventricular long axis length at end systole derived from apical two chambers, left ventricular long axis length at end systole derived from apical four chambers, left ventricular outflow tract area, These may include left ventricular outflow tract diameter, left ventricular posterior wall thickness at end diastole, peak mitral regurgitation velocity, peak mitral flow velocity at point a, peak mitral flow velocity at point e, peak mitral flow velocity, mitral deceleration gradient, mitral deceleration time, peak flow velocity distal to the pulmonary valve, pulmonary acceleration gradient, pulmonary acceleration time, pulmonary rr-time interval, right atrial mean end systolic pressure, right ventricular diameter at end diastole, peak tricuspid regurgitation velocity, aortic regurgitation, mitral regurgitation, tricuspid regurgitation, pulmonary regurgitation, aortic stenosis, mitral stenosis, tricuspid stenosis, pulmonary stenosis, and / or physician-reported diastolic function. In some embodiments, the diagnostic parameters may include a diagnosis of acute rheumatic fever, a diagnosis of chronic rheumatic heart disease, a diagnosis of hypertensive disease, a diagnosis of ischemic heart disease, a diagnosis of pulmonary heart disease and pulmonary circulation disease, a diagnosis of acute pericarditis, a diagnosis of other forms of heart disease, a diagnosis of acute myocarditis, a diagnosis of cardiomyopathy, a diagnosis of cardiac arrest, a diagnosis of paroxysmal tachycardia, a diagnosis of atrial fibrillation, a diagnosis of heart failure, a diagnosis of cerebrovascular disease, a diagnosis of diseases of arteries, arterioles, and capillaries, a diagnosis of diseases of veins, lymphatic vessels, and lymph nodes, a diagnosis of hypotension, a diagnosis of other and unspecified diseases of the circulatory system, a diagnosis of diabetes, a diagnosis of congenital heart disease, a diagnosis of dyslipidemia, and / or a diagnosis of chronic kidney disease. eMERGE Guidelines The parameters may be those that fall within predetermined guidelines, such as: Then, the process 100 may proceed to 106.

[0079] At 106, the process 100 can provide the video frames to a trained neural network. In some embodiments, the process 100 can provide the video frames and the EHR data to the trained neural network. As described above, the trained neural network can be a convolutional neural network. In some embodiments, the trained neural network can be the neural network architecture 10 shown in FIG. 1. In some embodiments, the trained neural network can be trained based on a training dataset that includes a plurality of videos, where each video included in the plurality of videos is associated with an echocardiographic view included within a number of echocardiographic views. The number of echocardiographic views can include one or more views. In some embodiments, some echocardiographic views can include at least one of an apical two-chamber view, an apical three-chamber view, an apical four-chamber view, an apical four-chamber view focused on the right ventricle, an apical five-chamber view, a parasternal long axis view, a parasternal long descending aorta view, a parasternal long mitral valve view, a parasternal long pulmonary valve view, a parasternal long right ventricular inflow view, a parasternal long zoom aortic valve view, a parasternal short aortic valve view, a parasternal short pulmonary valve and pulmonary artery view, a parasternal short tricuspid valve view, a short axis apical view, a short axis base view, a short axis central papillary view, a subcostal four-chamber view, a subcostal hepatic vein view, a subcostal atrial septum view, a subcostal inferior vena cava view, a subcostal right ventricular view, a suprasternal fossa view, a short axis central papillary view, a short axis apical view, an apical three-chamber zoom view, an apical two-chamber zoom view, or a short axis base view. The training dataset may further include a plurality of survival outcomes, and each video included in the plurality of videos is associated with a survival outcome included in the plurality of survival outcomes. In this manner, the neural network may be trained based on actual patient outcomes. In some embodiments, the training dataset may further include an electronic health record dataset, and each video included in the plurality of videos is associated with a portion of the electronic health record dataset.In some embodiments, the electronic health record dataset may be associated with a number of patients and may include values ​​for several parameters including age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function. As explained above, these parameters have been shown to perform well, and these ten parameters have been shown to provide good predictive power.

[0080] In some embodiments, the electronic health record dataset may include values ​​for several parameters, including demographic parameters, vital parameters, laboratory measured parameters, echocardiogram-based parameters, and diagnostic parameters. In some embodiments, the demographic parameters may include age, sex, and / or smoking status. The vital parameters may include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure. In some embodiments, the laboratory measured parameters may include low density lipoprotein levels and / or high density lipoprotein levels.In some embodiments, the echocardiogram-based parameters include physician-reported left ventricular ejection fraction, aortic regurgitation deceleration gradient, aortic regurgitation peak velocity, velocity-time integral of flow distal to the aortic valve, peak velocity of flow distal to the aortic valve, mean velocity of flow distal to the aortic valve, aortic root diameter, ascending aorta diameter, left ventricular end diastolic volume: apical two chamber, left ventricular end diastolic volume: apical four chamber, left ventricular end diastolic volume: apical two chamber, left ventricular end diastolic volume: apical four chamber, left ventricular collapse End-systolic volume: apical two chambers, left ventricular end-systolic volume: apical four chambers, left ventricular end-systolic volume: apical two chambers, left ventricular end-systolic volume: apical four chambers, left ventricular septum diameter at end diastole, left atrial diameter, left atrial volume derived from apical two chambers, left atrial volume derived from apical four chambers, velocity-time integral proximal to occlusion, maximum left ventricular velocity proximal to occlusion, mean left ventricular velocity proximal to occlusion, left ventricular area at end diastole derived from apical two chambers, left ventricular area at end diastole derived from apical four chambers, velocity-time integral proximal to occlusion, maximum left ventricular velocity proximal to occlusion, left ventricular area at end diastole derived from apical two chambers, left ventricular area at end diastole derived from apical four chambers, left ventricular area at end systole derived from apical four chambers, left ventricular intraventricular size at end diastole, left ventricular intraventricular size at end systole, left ventricular long axis length at end diastole derived from apical two chambers, left ventricular long axis length at end diastole derived from apical four chambers, left ventricular long axis length at end systole derived from apical two chambers, left ventricular long axis length at end systole derived from apical four chambers, left ventricular outflow tract area, left ventricular outflow tract diameter, left ventricular posterior wall thickness at end diastole, mitral valve regurgitation These may include maximum velocity, mitral flow point a maximum velocity, mitral flow point e maximum velocity, mitral flow maximum velocity, mitral deceleration gradient, mitral deceleration time, maximum velocity of flow distal to the pulmonary valve, pulmonary artery acceleration gradient, pulmonary artery acceleration time, pulmonary rr-time interval, right atrial mean end systolic pressure, right ventricular diameter at end diastolic pressure, tricuspid regurgitation maximum velocity, aortic regurgitation, mitral regurgitation, tricuspid regurgitation, pulmonary regurgitation, aortic stenosis, mitral stenosis, tricuspid stenosis, pulmonary stenosis, and / or physician-reported diastolic function.In some embodiments, the diagnostic parameters can include a diagnosis of acute rheumatic fever, a diagnosis of chronic rheumatic heart disease, a diagnosis of hypertensive disease, a diagnosis of ischemic heart disease, a diagnosis of pulmonary heart disease and pulmonary circulation disease, a diagnosis of acute pericarditis, a diagnosis of other forms of heart disease, a diagnosis of acute myocarditis, a diagnosis of cardiomyopathy, a diagnosis of cardiac arrest, a diagnosis of paroxysmal tachycardia, a diagnosis of atrial fibrillation, a diagnosis of heart failure, a diagnosis of cerebrovascular disease, a diagnosis of diseases of arteries, arterioles, and capillaries, a diagnosis of diseases of veins, lymphatic vessels, and lymph nodes, a diagnosis of hypotension, a diagnosis of other and unspecified diseases of the circulatory system, a diagnosis of diabetes, a diagnosis of congenital heart disease, a diagnosis of dyslipidemia, and / or a diagnosis of chronic kidney disease. eMERGE Guidelines The parameters may be parameters that fall within predetermined guidelines, such as: Then, the process 100 may proceed to 108.

[0081] At 108, the process 100 may receive a risk score from the trained neural network. The risk score may be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score may be a mortality risk score. In some embodiments, the mortality risk score may be an all-cause mortality risk score. In some embodiments, the mortality risk score associated with the patient may provide a physician or healthcare administrator with sufficient information to make a decision about a potential treatment for the patient. In some embodiments, the potential treatment includes at least one of a heart transplant, implantation of a mechanical assist device, placement of a defibrillator, palliative care, or hospice. In some embodiments, the decision may be eligibility for the potential treatment. The process 100 may then proceed to 110.

[0082] At 110, the process 100 may output the raw risk scores to at least one of a memory or a display for viewing by a physician or health care administrator. In some embodiments, the process 100 may generate and output a report based on the risk scores. The report may include the raw risk scores. The report may include any suitable graphs and / or charts generated based on the risk scores. The report may be displayed to the physician using a display such as a computer monitor or a screen integrated into a tablet computer, smartphone, laptop computer, or the like. In some embodiments, the report may be output to a storage device including a memory. In some embodiments, the report may include information regarding potential treatments for the patient and / or links to information regarding potential treatments for the patient. In some embodiments, the links may be hyperlinks. In some embodiments, the potential treatments for the patient may include heart transplant, implantation of a mechanical assist device, placement of a defibrillator, palliative care, and / or hospice. In some embodiments, the physician may make a decision (e.g., an eligibility decision) for the patient based on the report.

[0083] Turning now to FIG. 13, an exemplary system for implementing the foregoing disclosure is shown. The system 210 may include one or more computing devices 212a, 212b in communication with each other, as well as with a server 214 and one or more databases or other data repositories 216, for example, via the Internet, an intranet, Ethernet, LAN, WAN, etc. The computing devices may also communicate with additional computing devices 212c, 212d via a separate network 218. Although particular attention is given to the computing device 212a, each computing device may include a processor 220, one or more computer-readable media drives 222, a network interface 224, and one or more I / O interfaces 226. The device 212a may also include a memory 228 including instructions configured to cause the processor to execute an operating system 230, and a health analysis module 232 for predicting important clinical endpoints, such as all-cause mortality in a patient for a predetermined period (i.e., one year) based on video data, such as echocardiography data, as well as EHR data as described herein. The health analysis module 232 may be used to perform at least a portion of the process 100 described above in connection with FIG.

[0084] Additional examples of techniques for generating risk scores (e.g., mortality risk scores), progression-free survival (PGS) scores, etc. based on echocardiographic videos are now described. First, experiments used to design a network architecture for the trained model are described. Further below, the performance of the trained model is compared to other techniques.

[0085] Model Design Electronic Health Records Data Processing The institution's echocardiography archive used contained a total of 683,662 echocardiography studies from 305,282 unique patients collected over the past 22 years as of January 2020. All structured physician-reported echocardiography-derived measurements (n=58) were extracted from these studies. Additionally, through the institution's Phenomics Initiative database, echocardiography-derived measurements were linked to patient demographic (3), vital (5), laboratory (2), and problem list data (90; International Classification of Diseases, 10th Revision (ICD-10) codes) from the institution's EHR (Epic Systems; 1996-present). Table 3 above provides a list and description of all 158 "EHR variables" used in the study.

[0086] All continuous variables were removed for values ​​outside physiological limits (manually defined by a cardiologist); these values ​​were presumed to reflect input errors and were set as missing. Eight categorical variables were identified for echocardiographic measurements, each reporting five severity levels of valvular regurgitation and stenosis (including no rating) and transforming them into 40 one-hot coded binary variables. An ordinal variable reporting diastolic function was also identified, coded as -1 for normal, 0 for dysfunction (no grade reported), and 1, 2, and 3 for diastolic dysfunction grades I, II, and III, respectively. For measurements not derived from echocardiography, such as LDL, HDL, blood pressure, heart rate (if not taken in the study), weight, and height measurements, the most recent historical measurements were obtained within 1 year of the echocardiogram acquisition date.

[0087] Patient age and survival time were calculated as of the echocardiogram date. Patient status (dead or alive) was based on the last known live encounter or confirmed date of death, which was cross-referenced monthly against the national death index database. To label 1-year mortality, a positive sample was defined as an echocardiography study within 1 year of the patient's date of death. A negative 1-year mortality label was defined as an echocardiography study performed more than 1 year prior to the date of death (if deceased) or the last known physical encounter in the system (if alive). Studies without a date of death or at least 1 year of follow-up (physical encounter) were excluded.

[0088] Image acquisition and pre-processing An echocardiography study consists of several videos containing multiple views of the heart. The acquired DICOM files included the annotated videos and the raw videos when the device was configured to store them. As shown in Figures 8A and 8B, the raw videos included only the beamformed ultrasound images stored in the format of a stream of bytes, while the annotated videos included annotations (such as view names) in addition to the raw videos. For all analyses, the raw videos were used. As the videos in raw format had different frame rates depending on the study, all videos were linearly interpolated to 30 frames per second.

[0089] Along with the video data, the DICOM files contained tags, sometimes called "views," that labeled each image with the specific image orientation in which it was acquired. These view tags had slight variations between studies for the same type of view. For example, an apical 4-chamber view may be tagged as "a4," "a4 2d," or "ap4." Samples for each unique tag were visually inspected and grouped into common views, as shown in Table 9. For the entire cross-validation cohort, the mean number of views available for negative samples was 19.4, with quartiles of 19 and 22. For positive samples, the mean was 18.3 videos, with quartiles of 18 and 22 videos per sample. The median number of videos was 20 for both positive and negative samples. If a study had multiple videos from the same view, the video with the longest duration was selected. [Table 9]

[0090] Since each video from a view group could potentially have different dimensions, every video was normalized to the most common row and column dimension pair of its corresponding view. Each frame was cropped / padded with zeros to match the most common dimension within the view group while keeping the beamformed image centered. Note that since standard echocardiography views are centered on the anatomical region of interest, image size normalization (cropping and padding) had minimal impact on the videos. For example, less than 3% of PL DEEP videos were cropped or padded beyond 6 rows, and of those, only 17 cases were cropped and the rest were zero padded. In general, the border regions did not contain features of interest.

[0091] Data Selection Echocardiography studies were extracted from the clinical imaging archive (acquired after February 2011) onto a research server for this analysis, and only raw video data from these studies were retained, when available. This extracted subset of the total clinical archive was split into three different groups (characteristics of each are described below in Table 10) for the experiments described above. In all cases, follow-up >1 year or date of death within 1 year was known. [Table 10]

[0092] 1. Cross-validation experiment: The experiment includes 42,095 studies with 812,278 videos collected from 34,362 patients drawn without predefined patient selection criteria from the clinical echocardiography archive.

[0093] 2. Cardiologist survey: These surveys included 600 studies with 11,357 videos collected from 600 unique patients, again taken from unselected clinical data extraction, but excluded from cross-validation experiments set or pre-specified to have balanced outcome labels (300 deaths and 300 survival at 1 year).

[0094] 3. Heart failure experiment: The experiment included 3384 studies with 58561 videos collected from 2404 patients specifically selected from clinical archives based on the presence of heart failure according to "sure" eMERGE algorithm criteria at the time of the echocardiogram. The 42095 studies in the cross-validation set are a subset of the previously published cohort.

[0095] Cardiologist survey As explained above, the ten most predictive clinical ("EHR") variables for one-year mortality after an echocardiogram are age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function. To efficiently assess cardiologist performance, the top ten variables were presented as a summary of the patient's condition on the date of the echocardiogram. Along with these ten measurements, the parasternal long axis video was also presented. This view is typically reported by cardiologists as the most informative "summary" view of overall cardiac health, as it includes elements of the left ventricle, left atrium, right ventricle, aortic valve, and mitral valve, and whether or not there is a pericardial or left pleural effusion, all within a single view.

[0096] Following sample size calculations (Pearson's chi-square test) to estimate and compare the prognostic accuracy between the cardiologists and the model, the cardiologists completed a study set of 600 samples. A 10% difference in accuracy between the machine and the cardiologists (80% vs. 70%), a power of 80%, a significance level of 5%, and a discrepancy of approximately 40% were assumed. Calculations (performed using the power analysis software PASS v15) indicated that at least 600 patients (300 alive and 300 deceased) were required. Thus, 300 positive studies and 300 negative studies including parasternal long axis views were randomly sampled to ensure that these patients were not left within the cross-validation set.

[0097] The first study was designed to present one patient sample at a time and score the aggregated discrimination ability of the cardiologists. Figure 14 shows the interface for the first study. Ten EHR variables were displayed in a table with two versions of the video (raw and annotated). The application then recorded the cardiologist's prediction when he or she clicked either the "survive" or "death" button.

[0098] The second study was designed to present a paired sample and evaluate the discrimination ability of each cardiologist while controlling for the prevalence of mortality. 300 pairs were prepared based on gender, age (within 5 years), and left ventricular EF (within 10%). All 300 positive cases were paired with a negative case, 213 negatives were unique, and the remaining 87 pairs had to include a negative already used to maintain the matching criteria. Thus, all positive cases were unique. Figure 15 shows the interface for the paired study where the video and 10 EHR variables for two patients matched for age, gender, and EF are shown.

[0099] The third and final study presented individual samples back to back with the same samples with additional information extracted from the DNN model. The machine scores and occlusion maps were presented to evaluate whether the inclusion of machine information could improve the cardiologists' aggregated score performance. The same 600 patients were presented twice. First, the individual samples were displayed as shown in Figure 14, and immediately afterwards, the same samples were shown with the calibrated risk scores from the model and the occlusion map. The cardiologists then revised or repeated their predictions.

[0100] To avoid incremental performance changes while the cardiologists were progressing through the study, the cardiologists were presented with 80 examples with machine predictions, occlusion maps, and true outcomes from the cross-validation set before conducting the study. The 80 examples were divided into four groups of 20 examples, grouped by history of heart failure only, history of myocardial infarction only, history of both, and neither. Each of the four groups was further divided into five examples that fell into each of the four quadrants of the confusion matrix. Figure 16 shows the interface for the model-assisted portion of the third study, with the "machine prediction" row and occlusion map video added.

[0101] Note that individual patient-level response feedback was not presented to cardiologists between any surveys (to avoid conflating the results of subsequent surveys with knowledge gained through previous surveys) and a minimum of 15 days elapsed between surveys for a given cardiologist.

[0102] Neural Network Architecture Here, four different architectures are presented: 1) time-distributed 2D convolutional neural network (2D CNN) with LSTM, 2) time-distributed 2D CNN with global average pooling (GAP), 3) 3D CNN, and 4) 3D CNN with GAP. For simplicity, the four candidate architectures are abbreviated as 2D CNN+LSTM, 2D CNN+GAP, 3D CNN, and 3D CNN+GAP.

[0103] FIG. 17 illustrates a first model 300 and a second model 304. The first model 300 may be a 2D CNN architecture with LSTM. The second model 304 may be a 2D CNN with GAP. Both the first model 300 and the second model 304 may include several convolutional layers 308A-F. An exemplary convolutional layer 312 is shown that may be used as any of the convolutional layers 308A-F. In some embodiments, the convolutional layer 312 may include a first 2D CNN layer 316, a first batch normalization 320, a first ReLU 324, a second 2D CNN layer 328, a second batch normalization 332, a second ReLU 336, and a 2D max pooling layer 340. In some embodiments, the kernel dimension of the first model 300 and the second model 304 may be 3 and the 2D max pooling layer 340 may be applied in a 3×3 window. In some embodiments, the kernel dimension of the first model 300 and the second model 304 may be 5 and the 2D max pooling layer 340 may be applied in a 5×5 window.

[0104] The first model 300 can include a flattening layer 344, an LSTM layer 348, and a dense layer 352 that outputs a risk score 356. In some embodiments, the LSTM layer 348 can include two LSTM units. The risk score 356 output by the first model 300 can be referred to as a video risk score. The second model 304 can include a GAP layer 360 and a dense layer 364 that outputs a risk score 368. The risk score 368 output by the second model 304 can be referred to as a video risk score. Exemplary video dimensions are shown for an input video 372 that the first model 300 and / or the second model 304 can receive. It is understood that some layer sizes and / or video sizes can be adjusted depending on the application in which the first model 300 and / or the second model 304 are used.

[0105] FIG. 18 illustrates a third model 400 and a fourth model 404. The third model 400 may be a 3D CNN. The fourth model 404 may be a 3D CNN with GAP. Both the third model 400 and the fourth model 404 may include several convolutional layers 408A-F. An exemplary convolutional layer 412 is shown that may be used as any of the convolutional layers 408A-F. In some embodiments, the convolutional layer 412 may include a first 3D CNN layer 416, a first batch normalization 420, a first ReLU 424, a second 3D CNN layer 428, a second batch normalization 432, a second ReLU 436, and a 3D max pooling layer 440. In some embodiments, the kernel dimension of the third model 400 and the fourth model 404 may be 3, and the 3D max pooling layer 440 may be applied in a 3×3 window. In some embodiments, the kernel dimension of the third model 400 and the fourth model 404 may be 5, and the 3D max pooling layer 440 may be applied in a 5×5 window.

[0106] The third model 400 may include a flattened layer 444 and a densified layer 448 that outputs a risk score 452. The risk score 452 output by the third model 400 may be referred to as a video risk score.

[0107] The fourth model 404 may include a GAP layer 456 and a high density layer 460 that outputs a risk score 464. The risk score 464 output by the fourth model 404 may be referred to as a video risk score. Exemplary video dimensions are shown for an input video 468 that may be received by the third model 400 and / or the fourth model 404. It is understood that some layer sizes and / or video sizes may be adjusted depending on the application in which the third model 400 and / or the fourth model 404 are used.

[0108] In some embodiments, the first model 300, the second model 304, the third model 400, and / or the fourth model 404 may be neural networks (e.g., convolutional neural networks and / or deep neural networks). Exemplary parameters and layer information for the first model 300, the second model 304, the third model 400, and the fourth model 404 are shown in Tables 11, 12, 13, and 14 below, respectively. [Table 11] [Table 12] [Table 13] [Table 14]

[0109] In testing, the convolutional units of 2D and 3D CNN were defined as a sequence of 7 layers in the following configuration: CNN layer, batch normalization, ReLU, CNN layer, batch normalization, ReLU, and max pooling. The dimensions of all kernels were set to 3, and max pooling was applied in a 3x3 window for 2D kernels and 3x3x3 for 3D kernels. Four additional versions were added by increasing the kernel size from 3 to 5 pixels in all dimensions, resulting in a total of 8 candidate video models per echocardiography view.

[0110] In general, the models 300, 304, 400, 404 are low-parameter designs. The low-parameter designs were chosen due to the high computational cost of the presented experiments and to reduce the possibility of overfitting. The 2D CNN+LSTM (e.g., the first model 300) consists of 2D CNN branches distributed across all frames of a video. This architecture was used for video description problems where all frames from a video belong to the same scene or action. Thus, it is assumed that static features are commonly found by the same 2D kernel across the video. This assumption was put into practice for echocardiography view classification. The LSTM layer aggregates the CNN features over time to output a vector representing the entire sequence.

[0111] The 2D CNN+GAP approach (e.g., second model 304) replaced the LSTM layer with average CNN features as a temporal aggregation of frames. The GAP layer provided two advantages: it did not require trainable parameters, saving 10736 parameters from the LSTM layer, and it enabled feature interpretation. The final fully connected layer after GAP provided a weighted average of the CNN features, which can indicate which sections of the video were weighted more in the final decision. The 3D CNN approach aggregated temporal and spatial features as the input data flowed through the network.

[0112] In contrast to 2D CNN methods, 3D CNN incorporates information from neighboring frames at every layer to extract spatio-temporal dependent features that have also proven useful for video classification. In the 3D CNN method, the GAP layer reduced the fully connected layer input from the feature map size to the number of filters. Thus, the GAP layer also reduced the number of parameters from 641 to 17.

[0113] As explained above, the low-parameter design was chosen due to the high computational cost of the presented experiments and to reduce the possibility of overfitting. A total of 1152 neural network models (24 view x 5 fold x 8 models for the cross-validation experiments plus 24 view x 8 models for the final version) were fitted to complete all experiments, which fully occupied all 16 GPUs in the NVIDIA DGX2 for about 40 days. Deep learning models typically consist of millions of parameters. For example, the Inception model has 25 million parameters and ResNet has over 40 million parameters, making the computational cost of training such large networks prohibitive and potentially unnecessary given the performance demonstrated in the disclosed models. Even the relatively large disclosed models contained over 20,000 parameters.

[0114] In some embodiments, the models may be implemented using the docker container tensorflow:19.08py3 (available at nvcr.io / nvidia / ) with Python version 3.6.8, Tensorflow module version 1.14, and Keras module version 2.2.4tf.

[0115] Cross-validation procedure

[0116] Using the cross-validation set described in Table 10 above, the echocardiography studies were divided into 5 folds, where in each of the 5 iterations, one fold was used for testing and the rest were used for training. Two constraints were imposed on the content of the folds: 1) studies from the same patient could not be present in more than one fold, and 2) each fold contained a similar positive prevalence (of 1-year all-cause mortality) as the entire dataset. For each training set, one tenth of studies with balanced prevalence was reserved as a surrogate for the test set for internal validation. Once the DNN was trained, the loss (binary cross-entropy) was evaluated at the internal validation set at each epoch. If the internal validation loss did not decrease for more than 10 epochs, the training was stopped and the model weights were restored at the minimum validation loss.

[0117] All video architectures were trained on all available views in the training set. For each view, the architecture with the highest AUC was selected in the internal validation set, and that model was used to report the performance for that view in all subsequent experiments. A summary of the architectures selected for each view is shown in Table 15, and examples of PL DEEP views are shown in Table 16. EHR-derived features and video risk scores were concatenated for each view to fit a classification pipeline consisting of an interquartile range scaler, multivariate imputation with chained equations, and an XGboost classifier. This pipeline was fit on each training fold and applied to its corresponding test set to generate an output risk score.

[0118] Because the prevalence of mortality in the entire dataset was disproportionate (14.6% of patients died within 1 year of the echocardiography study), the weights for each class were set as follows:

number

[0119] In the tests, all training was performed on an NVIDIA DGX2 platform by fitting each model independently on each of the 16 available GPUs. [Table 15] [Table 16]

[0120] statistical analysis In all survival analyses, time from echocardiography study to death or last known alive encounter (censoring) and predicted labels were used to stratify survival probabilities for Kaplan-Meier plots and Cox proportional hazards ratio analyses. Analyses were performed using the lifelines python package version 0.25.4. The threshold for both the DNN and SHF models was chosen as the midpoint in the score range.

[0121] For cross-validation experiments where fold-wise AUC estimates were obtained, the mean across the five folds and the 95% CI calculated at ±1.96σ / √5 were reported.

[0122] For the remaining experiments where only a single AUC was available (heart failure and survey cohorts), AUC estimates were bootstrapped with 10,000 iterations, and the 2.5th and 97.5th percentiles were reported as 95% CIs.

[0123] To report significant differences when comparing prediction performance with paired survey data, paired proportion tests were performed on the number of correct answers out of 300 samples. A total of four tests were performed comparing each of the four cardiologists with the DNN model, therefore the p-value correction threshold was 0.05 / 4. For statistical calculations, the statistical models package for Python version 0.11.1 was used.

[0124] Seattle Heart Failure Implementation The SHF score was calculated using typical parameters, except that systolic blood pressure, hemoglobin, percentage of white blood cells in lymphocyte morphology, uric acid, total cholesterol, and sodium were defined as the most recent available measurements on or before (within 1 year) the date of the echocardiogram, instead of using potentially closer measurements in the future. To predict future events, both the DNN and SHF models were blinded to data collected after the date the echocardiogram was obtained.

[0125] Defining Heart Failure Subtypes The type of heart failure (i.e., reduced vs. preserved ejection fraction) was determined for each sample using all available historical ejection fraction measurements up to 6 months prior to heart failure diagnosis as follows: 1) heart failure with reduced ejection fraction (HFrEF) if any LVEF ≤ 50%, 2) heart failure with preserved ejection fraction (HFpEF) if all LVEF ≥ 50%, and 3) no subtype assigned if LVEF was not reported.

[0126] Performance evaluation First, the DNN models were cross-validated on a clinically acquired echocardiography video database (812278 videos). Independent models were trained for each individual view (parasternal long axis, apical four-chamber, etc.) and aggregated to form a feature vector consisting of the outputs from the individual view models. On average, using all echocardiography video views combined yielded higher performance (area under the receiver operating characteristic curve (AUC): 0.83, 95% CI) for predicting 1-year mortality than using either the 58 EDM (AUC: 0.75, 95% CI), or a combination of 100 additional clinical variables including the 58 EDM, relevant cardiovascular-related diagnoses, laboratory values, demographics, and vital signs (AUC: 0.81, 95% CI). The largest model combining all views and 158 EHR-derived measurements yielded an AUC of 0.84, 95% CI. The individual view models ranged in performance from 0.70 to 0.80 AUC, with the parasternal long axis view yielding the best individual performance. Finally, the PCE score, a clinical gold standard measure of future cardiovascular disease, was calculated for the same sample. The PCE score yielded an AUC (95% CI) of 0.64 for 1-year mortality prediction, which was inferior to all DNN models tested.

[0127] Given this proof of concept from the cross-validation experiment for predicting 1-year mortality from echocardiography videos using DNNs, the DNN model was retrained using 812278 videos from the cross-validation experiment to evaluate its performance in two new separate groups of patients. The first group was an independent set (study set) of 600 patients balanced for 1-year mortality outcomes (i.e., 300 patients who survived 1 year after echocardiography and 300 patients who died within 1 year). The second group was a cohort of 2404 patients with heart failure (defined as "definite" heart failure by the eMERGE guidelines) who underwent 3384 echocardiograms.

[0128] The survey set was used to evaluate the performance of four experienced cardiologists, three Core Cardiovascular Training Statement (COCATS) level 3, and one level 2, in echocardiography. The cardiologists were asked separately and blindly to determine whether each patient was alive or dead 1 year after the echocardiogram. To efficiently evaluate the cardiologists' performance, a limited input set of a single video from the parasternal long axis view (the best performing individual view), and 10 EHR variables were presented to compare their performance to a model trained on the same input set. A risk score from the cardiologists' responses was created by aggregating the number of positive predictions (died within 1 year) per patient. The DNN model yielded an AUC, 95% CI of 0.84, and the aggregated cardiologist score yielded an inferior AUC, 95% CI of 0.68.

[0129] To further evaluate the performance of the DNN model compared to cardiologists, the study set was rearranged to show matched pairs. No individual-level feedback was provided to the cardiologists between experiments. In the second study, the cardiologists and the model were presented with two studies at a time, one study from a patient who died within one year of the echocardiogram, and the other study from a patient who survived more than one year from the echocardiogram. Both the cardiologists and the model were asked to select the patient from each pair who was more likely to die at one year. The 300 pairs were matched by sex, age (within 5 years), and left ventricular ejection fraction (EF) (within 10% absolute difference). This study was designed to control for outcome prevalence and directly measure discrimination performance. The DNN model yielded an accuracy of 82%, while the four cardiologists scored 66, 70, 73, and 76%. Note that simple heuristics such as selecting older patients or lower EF as positive samples resulted in accuracies of 43% (131 samples) and 36% (108 samples), respectively. Using paired proportions tests, the model performed significantly better than three of four cardiologists after correcting for multiple comparisons (p<0.05 / 4).

[0130] Next, it was evaluated whether cardiologists improved their performance when assisted by the model. As in the first study, a single study was shown at a time, collecting cardiologist predictions, and then immediately presenting the same study along with the machine prediction score. The aggregated cardiologist score AUC improved from 0.72, 95% CI to 0.78, 95% CI with assistance from the model predictions, which overlaps slightly with the DNN performance. On average, cardiologists in the study correctly changed 10.3% of their predictions and incorrectly changed 3.8% of their predictions. Sensitivity increased by 13% and specificity decreased by an average of less than 1%.

[0131] The second group of patients to which the fully trained DNN model (all views + EHR) was applied was a cohort of 2404 patients with heart failure (defined as "definite" heart failure by the eMERGE guidelines) who underwent 3384 echocardiographic examinations. This group of patients was selected as an important additional clinical validation, since heart failure is prevalent and costly, and heart failure management is highly dependent on survival prediction models such as the SHF risk score. Within this cohort, the SHF score yielded an AUC, 95% CI of 0.70, and the DNN model yielded an AUC, 95% CI of 0.76. Notably, this superior performance of the DNN was observed for both patients with reduced EF (HFrEF) and those with preserved EF (HFpEF), see Table 16.

[0132] Predictions were calculated based on mid-range thresholds for the DNN model (0.5) and the SHF score (1.5) to distinguish between high and low risk. Scores ranged from 0 to 1 for the DNN model and 1 to 4 for the SHF model.

[0133] Finally, it was investigated what features the DNN model had learned from the echocardiography videos. To do this, a sample video was occluded with 10x10x10 three-dimensional voxels, and the difference in likelihood score resulting from occluding that particular region was calculated. As the result of occlusion is a video, the first frame and overlaid red regions were displayed to indicate significant changes (>2.5 standard deviations) in risk score for at least 10 frames. These occlusion experiments included the four patients with the highest predictive scores who died within one year, and the four patients with the lowest predictive scores who survived beyond one year. These patients were selected from the test set of the first cross-validation experiment fold. It is noted that for high-risk patients, occlusion reduces the risk score, and for low-risk patients, occlusion increases the risk score. In general, it was observed that the most influential regions coincided with the anatomically relevant regions of the heart, in particular the left atrium, left ventricle, and the mitral and aortic valve planes. These regions appeared to be more restricted and localized in the lower-risk videos, whereas the higher-risk videos appeared to take advantage of the surrounding anatomy more; however, when presented with several examples of the obstruction maps, the cardiologists reported that, based on their personal opinion, they were unable to identify any patterns that would help them better identify patient survival outcomes.

[0134] In summary, we demonstrated the ability of neural networks to assist physicians in the important clinical task of predicting 1-year all-cause mortality. The ability of the DNN model to identify 1-year mortality exceeded that of models leveraging only image-derived standard clinical measures from the EHR as well as multiple existing clinical risk scores. Furthermore, the DNN model improved the predictive performance of four trained cardiologists. Thus, this echocardiography video-based DNN model can add value beyond standard clinical interpretation.

[0135] Survival was selected as a highly significant and unequivocal clinical outcome, although other outcomes such as PFS may be used. Even if observer variability in echocardiography may be present for predicting human-defined outcomes, the use of mortality labels can help minimize, if not eliminate, this challenge. Improving predictive performance may directly improve patient risk assessment prior to elective surgery or may impact treatment guidance for both primary and secondary prevention of cardiovascular disease in the outpatient setting. Also, at the population level, improved mortality risk models may enable health systems and insurers to better understand their own patient populations and optimally deploy resources, as previously demonstrated using only EHR variables in patients with heart failure. For heart failure specifically, methods for determining patient candidacy for advanced therapies such as cardiac transplantation and implantation of durable mechanical assist devices have historically relied on mortality risk assessments based in part on peak oxygen consumption and invasive hemodynamics. Considerations for defibrillator placement in heart failure patients are also based on a reasonable expectation of meaningful survival beyond 1 year. Implementation of more accurate mortality-based risk tools may have additional benefits. Finally, estimates of 1-year mortality are particularly important for planning transition to palliative care and hospice. Further studies are needed to evaluate the performance of neural network models to predict additional clinically relevant outcomes in cardiology, such as future hospitalization or the need for major procedures such as valve replacement.

[0136] 19, an exemplary trained model 500 is shown. In some embodiments, the model 500 can receive at least one echocardiographic video 504 and / or EHR data 508 associated with a patient. In some embodiments, the EHR information can include values ​​of several patient parameters, such as the parameters in Table 3. In some embodiments, the model 500 can include several trained video models 512. Each video model included in the several trained video models 512 can be associated with an echocardiographic view. In some embodiments, each video model included in the several trained video models 512 can be associated with a unique echocardiographic view included in Table 9 and / or Table 4. In some embodiments, the several trained video models 512 can include the first model 300, the second model 304, the third model 400, and / or the fourth model 404. Each trained model included in the trained video models 512 can be selected by determining the best performing (e.g., highest AUC) model architecture for each echocardiographic view. For example, for a first echocardiographic view, the first model 300 may be best, and for a second echocardiographic view, the fourth model 400 may be best.

[0137] In some embodiments, the trained model 500 can receive at least one echocardiographic video 504, provide each echocardiographic video in the at least one echocardiographic video 504 to a trained video model in a number of trained video models 512, and receive a risk score (e.g., a video risk score) from each of the number of trained video models 512. In some embodiments, the trained model 500 can provide each risk score and / or EHR data 508 to a trained sub-model 516. In some embodiments, the trained sub-model 516 can receive each risk score and / or EHR data 508 and generate an output value, such as a mortality risk score 520 (e.g., an all-cause mortality risk score) and / or a progression-free survival score.

[0138] Referring now to FIG. 20, another exemplary process 600 for predicting relevant clinical endpoints such as all-cause mortality in a patient for a predetermined period (e.g., one year) based on cardiac videos (in this case, echocardiography data) as well as any additional available EHR data is shown. In some embodiments, the process 600 can predict a risk score for a patient based on a neural network, which may be a deep neural network, such as a convolutional neural network, trained using cardiac videos, such as echocardiography videos, and EHR variables as described above. In some embodiments, the process 600 can be employed in a health analytics module used by a care team including a physician, for example, a physician deploying resources across a population of 10,000 patients with heart failure, to treat a patient or for population-level management of patients. In some embodiments, the process 600 can be implemented as instructions (e.g., computer-readable instructions) on at least one memory and executed by one or more processors coupled to the at least one memory.

[0139] At 602, the process 600 can receive a number of echocardiographic videos of a heart associated with a patient. Each echocardiographic video can include echocardiographic video frames. The video frames can include video frames taken from one or more views of the patient's heart. For example, the video frames can include video frames taken at 21 different views of the heart. In some embodiments, the echocardiographic video may be associated with one or more echocardiographic views including an apical two-chamber view, an apical three-chamber view, an apical four-chamber view, an apical four-chamber view focused on the right ventricle, an apical five-chamber view, a parasternal long axis view, a parasternal long descending aorta view, a parasternal long mitral valve view, a parasternal long pulmonary valve view, a parasternal long right ventricular inflow view, a parasternal long zoom aortic valve view, a parasternal short aortic valve view, a parasternal short pulmonary valve and pulmonary artery view, a parasternal short tricuspid valve view, a short axis apical view, a short axis base view, a short axis central papillary view, a subcostal four-chamber view, a subcostal hepatic vein view, a subcostal atrial septum view, a subcostal inferior vena cava view, a subcostal right ventricular view, a suprasternal fossa view, a short axis central papillary view, a short axis apical view, an apical three-chamber zoom view, an apical two-chamber zoom view, and / or a short axis base view. The process 600 may then proceed to 604.

[0140] At 604, process 600 may receive EHR data including values ​​of several parameters associated with the patient. In some embodiments, 604 may be substantially similar to 104 in FIG. 12. Process 600 may then proceed to 606.

[0141] At 606, the process 600 can provide several echocardiographic videos and EHR data to a trained model. In some embodiments, the trained model can be model 500 in FIG. 19. In some embodiments, the process 600 can provide each echocardiographic video to an associated trained video model (e.g., a trained video model associated with the same echocardiographic view as the video) contained within the trained model. In some embodiments, the process 600 can provide the EHR data to a trained sub-model (e.g., sub-model 516 in model 500) along with a risk score (video risk score) generated by the trained video model based on the echocardiographic video. At 606, the process 600 can analyze one or more regions of the heart using the trained model. In some embodiments, the process 600 can analyze one or more regions of the heart including the left ventricle, the left ventricle, the mitral valve, and / or the aortic valve. The process 600 can then proceed to process 608.

[0142] At 608, process 600 can receive a risk score from the trained model. In some embodiments, the risk score can be risk score 520 in FIG. 19. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the risk score can be a progression-free survival score. Process 600 can then proceed to 610.

[0143] At 610, the process 600 can output the raw risk score to at least one of a memory or a display for viewing by a physician or health care administrator. In some embodiments, the process 600 can generate and output a report based on the risk score. The report can include the raw risk score. The report can include any suitable graphs and / or charts generated based on the risk score. The report can be displayed to the physician using a display such as a computer monitor or a screen integrated into a tablet computer, smartphone, laptop computer, etc. In some embodiments, the report can be output to a storage device including a memory. In some embodiments, the report can include information regarding potential treatments for the patient and / or links to information regarding potential treatments for the patient. In some embodiments, the links can be hyperlinks. In some embodiments, the potential treatments for the patient can include heart transplant, implantation of a mechanical assist device, placement of a defibrillator, palliative care, and / or hospice. In some embodiments, the physician can make a decision (e.g., an eligibility decision) for the patient based on the report.

[0144] In conclusion, a methodology and architecture for extracting clinically relevant predictive information from medical videos using deep neural networks is disclosed.

[0145] Thus, as described herein, the present disclosure provides systems and methods for efficiently and accurately analyzing cardiac videos, such as videos acquired during echocardiography, to assess cardiac anatomy and function and to predict future clinical events.

[0146] While the present disclosure may be susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail herein. It should be understood, however, that the present disclosure is not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the following appended claims.

[0147] This written description uses examples to disclose the disclosure, including the best mode, and also to enable those skilled in the art to practice the disclosure, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the disclosure is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the language of the claims, or if they include equivalent structural elements that are not substantially different from the language of the claims.

[0148] Finally, it is expressly contemplated that any of the processes or steps described herein may be combined, eliminated, or rearranged. Accordingly, this description is intended to be construed as illustrative only and is not intended to otherwise limit the scope of the present disclosure. [Explanation of symbols]

[0149] 10 Neural Network Architecture 210 System 212a Computing devices, devices 212b Computing Devices 212c Computing Device 212d Computing Devices 214 Server 216 Data Repository 218 Network 220 Processor 222 Computer Readable Media Drive 224 Network Interface 226 I / O Interface 228 Memory 230 Operating Systems 232 Health Analysis Module 300 First Model 304 Second Model 308A~F Convolutional layer 312 Convolutional Layer 316 First 2D CNN Layer 320 First Batch Normalization 324 First ReLU 328 Second 2D CNN Layer 332 Second Batch Normalization 336 Second ReLU 340 2D Max Pooling Layer 344 Planarization layer 348 LSTM layer 352 High density layer 356 Risk Score 360 GAP layer 364 dense layer 368 Risk Score 372 Input Video 400 3rd Model 404 The Fourth Model 408A~F Convolutional layer 412 Convolutional Layer 416 First 3D CNN Layer 420 First Batch Normalization 424 First ReLU 428 Second 3D CNN Layer 432 Second Batch Normalization 436 Second ReLU 440 3D Max Pooling Layer 444 Planarization layer 448 High density layer 452 Risk Score 456 GAP layer 460 high density layer 464 Risk Score 468 Input Video 500 pre-trained models,models 504 Echocardiogram Video 508 EHR data 512 pre-trained video models 516 trained sub-models 520 Mortality Risk Score

Claims

1. A method for detecting a heart condition comprising the steps of: receiving at least one echocardiographic video of a heart associated with an individual; providing the at least one echocardiographic video to at least one trained model in a trained neural network, the trained neural network being trained to generate at least one risk score based on the input echocardiographic video; receiving electronic health record information relating to the individual; linking the at least one risk score with the electronic health record information for each view associated with the at least one echocardiographic video; providing the concatenated at least one risk score and electronic health record information as inputs to a sub-model of the trained neural network, the sub-model configured to generate a clinical outcome risk score associated with the individual; and receiving the clinical outcome risk score associated with the individual from the sub-model of the trained neural network; outputting the clinical outcome risk score associated with the individual to at least one of a memory or a display for viewing by a physician or health care administrator; generating a report based on the clinical outcome risk score; The method includes:

2. 2. The method of claim 1, wherein the clinical outcome risk score associated with the individual indicates the likelihood that the individual will die within a predetermined time period from when the at least one echocardiographic video was generated.

3. 3. The method of claim 2, wherein the clinical outcome risk score associated with the individual indicates the likelihood that the individual will die from any cause within the given time period.

4. 2. The method of claim 1, wherein the trained neural network is trained based on a training dataset including a plurality of videos, each video of the plurality of videos being associated with an echocardiographic view selected from among a number of echocardiographic views, and at least a portion of the training dataset being associated with definite heart failure according to eMERGE guidelines.

5. The method of claim 4 , wherein the number of echocardiographic views includes at least one echocardiographic view.

6. 5. The method of claim 4, wherein the several echocardiographic views include at least one of an apical two-chamber view, an apical three-chamber view, an apical four-chamber view, an apical four-chamber view focused on the right ventricle, an apical five-chamber view, a parasternal long axis view, a parasternal long descending aorta view, a parasternal long mitral valve view, a parasternal long pulmonary valve view, a parasternal long right ventricular inflow view, a parasternal long zoom aortic valve view, a parasternal short aortic valve view, a parasternal short pulmonary valve and pulmonary artery view, a parasternal short tricuspid valve view, a short axis apical view, a short axis base view, a short axis central papillary view, a subcostal four-chamber view, a subcostal hepatic vein view, a subcostal atrial septum view, a subcostal inferior vena cava view, a subcostal right ventricular view, a suprasternal fossa view, a short axis central papillary view, a short axis apical view, an apical three-chamber zoom view, an apical two-chamber zoom view, or a short axis base view.

7. The method of claim 4 , wherein the training dataset further comprises a plurality of survival outcomes, and each video of the plurality of videos is associated with a survival outcome included within the plurality of survival outcomes.

8. The method of claim 4 , wherein the training dataset further comprises an electronic health record dataset, and each video of the plurality of videos is associated with a portion of the electronic health record dataset.

9. 9. The method of claim 8, wherein the electronic health record dataset is associated with a number of individuals and includes values ​​for a number of parameters including age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function.

10. 9. The method of claim 8, wherein the electronic health record dataset includes values ​​for a number of parameters including demographic parameters, vital parameters, laboratory measurement parameters, echocardiogram-based parameters, and diagnostic parameters.

11. The method of claim 10, wherein the demographic parameters include age, sex, and smoking status.

12. The method of claim 10, wherein the vital parameters include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure.

13. The method of claim 10, wherein the laboratory measured parameters include low density lipoprotein levels and high density lipoprotein levels.

14. The echocardiogram-based parameter is physician-reported left ventricular ejection fraction; Aortic regurgitation deceleration gradient, Aortic regurgitation maximum velocity, velocity-time integral of flow distal to the aortic valve, The peak velocity of flow distal to the aortic valve, The mean velocity of the flow distal to the aortic valve, Aortic root diameter, Diameter of the ascending aorta, Left ventricular end-diastolic volume: apical two-chamber; modified ellipsoid, Left ventricular end-diastolic volume: apical four-chamber; modified ellipsoid, Left ventricular end-diastolic volume: apical two-chamber; single plane, Left ventricular end-diastolic volume: apical four-chamber; single plane, Left ventricular end-systolic volume: apical two-chamber; modified ellipsoid, Left ventricular end-systolic volume: apical four-chamber; modified ellipsoid, Left ventricular end-systolic volume: apical two-chamber; single plane, Left ventricular end-systolic volume: apical four-chamber; single plane; Left ventricular septal diameter at end diastole, left atrial diameter, Left atrial volume derived from the apical two-chamber; modified ellipsoid, Left atrial volume derived from the apical four chambers; modified ellipsoid, velocity-time integral proximal to the occlusion, Maximum left ventricular velocity proximal to the occlusion, Mean left ventricular velocity proximal to the occlusion, Left ventricular area at end diastole derived from the apical two-chamber Left ventricular area at end diastole derived from the apical four chambers; Left ventricular area at end systole derived from the apical two chambers, Left ventricular area at end systole derived from the apical four chambers; Left ventricular intraventricular method at end diastole, Left ventricular intraventricular method at end systole, Left ventricular long axis length at end diastole derived from the apical two-chamber Left ventricular long axis length at end diastole derived from the apical four chambers, Left ventricular long axis length at end systole derived from the apical two-chamber Left ventricular long axis length at end systole derived from the apical four chambers, left ventricular outflow tract area, left ventricular outflow tract diameter, Left ventricular posterior wall thickness at end diastole, Maximum mitral regurgitation velocity, Maximum velocity of mitral valve blood flow at point a, Maximum velocity of mitral valve blood flow at point e, The peak velocity of mitral blood flow, Mitral deceleration gradient, mitral valve deceleration time, The peak velocity of flow distal to the pulmonary valve, pulmonary artery acceleration gradient, pulmonary artery acceleration time, Pulmonary rr-time interval, Right atrial mean end-systolic pressure, Right ventricular diameter at end diastole, Tricuspid regurgitation maximum velocity, Aortic regurgitation, Mitral regurgitation, Tricuspid regurgitation, Pulmonary regurgitation, Aortic stenosis, Mitral stenosis, Tricuspid stenosis, Pulmonary artery stenosis, and / or Physician-reported diastolic function The method of claim 10, comprising:

15. 11. The method of claim 10, wherein the diagnostic parameters comprise a diagnosis of acute rheumatic fever, a diagnosis of chronic rheumatic heart disease, a diagnosis of hypertensive disease, a diagnosis of ischemic heart disease, a diagnosis of pulmonary heart disease and pulmonary circulation diseases, a diagnosis of acute pericarditis, a diagnosis of other forms of heart disease, a diagnosis of acute myocarditis, a diagnosis of cardiomyopathy, a diagnosis of cardiac arrest, a diagnosis of paroxysmal tachycardia, a diagnosis of atrial fibrillation, a diagnosis of heart failure, a diagnosis of cerebrovascular diseases, a diagnosis of diseases of arteries, arterioles and capillaries, a diagnosis of diseases of veins, lymphatic vessels and lymph nodes, a diagnosis of hypotension, a diagnosis of other and unspecified diseases of the circulatory system, a diagnosis of diabetes, a diagnosis of congenital heart disease, a diagnosis of dyslipidemia and / or a diagnosis of chronic kidney disease.

16. The method of claim 1, wherein the electronic health record information includes values ​​for several parameters including age, peak tricuspid regurgitation velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function, and the values ​​are associated with the individual.

17. The method of claim 1 , wherein the electronic health record information includes values ​​for several parameters, including demographic parameters, vital parameters, laboratory measurement parameters, echocardiogram-based parameters, and diagnostic parameters.

18. 18. The method of claim 17, wherein the demographic parameters include age, sex, and smoking status.

19. The method of claim 17, wherein the vital parameters include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure.

20. 18. The method of claim 17, wherein the laboratory measured parameters include low density lipoprotein levels and high density lipoprotein levels.

21. The echocardiogram-based parameters include physician-reported left ventricular ejection fraction, aortic regurgitation deceleration gradient, aortic regurgitation peak velocity, velocity-time integral of flow distal to the aortic valve, peak velocity of flow distal to the aortic valve, mean velocity of flow distal to the aortic valve, aortic root diameter, ascending aorta diameter, left ventricular end-diastolic volume: apical two chambers; modified ellipsoid, left ventricular end-diastolic volume: apical four chambers; modified ellipsoid, left ventricular end-diastolic volume: apical two chambers; single plane, left ventricular end-diastolic volume: apical four chambers; single plane, left ventricular end-systolic volume: apical four chambers; single plane, End systolic volume: apical 2 chambers; modified ellipsoid, Left ventricular end systolic volume: apical 4 chambers; modified ellipsoid, Left ventricular end systolic volume: apical 2 chambers; single plane, Left ventricular end systolic volume: apical 4 chambers; single plane, Left ventricular septum diameter at end diastole, Left atrial diameter, Left atrial volume derived from apical 2 chambers; modified ellipsoid, Left atrial volume derived from apical 4 chambers; modified ellipsoid, Velocity-time integral proximal to occlusion, Maximum left ventricular velocity proximal to occlusion, Mean left ventricular velocity proximal to occlusion, Left ventricular area at end diastole derived from apical 2 chambers, End diastolic volume derived from apical 4 chambers Left ventricular area at end systole, left ventricular area at end systole derived from apical two chambers, left ventricular area at end systole derived from apical four chambers, left ventricular intraventricular measurement at end diastole, left ventricular intraventricular measurement at end systole, left ventricular long axis length at end diastole derived from apical two chambers, left ventricular long axis length at end diastole derived from apical four chambers, left ventricular long axis length at end systole derived from apical two chambers, left ventricular long axis length at end systole derived from apical four chambers, left ventricular outflow tract area, left ventricular outflow tract diameter, left ventricular posterior wall at end diastole 18. The method of claim 17, comprising: thickness, mitral regurgitation peak velocity, mitral flow a-point peak velocity, mitral flow e-point peak velocity, mitral flow peak velocity, mitral deceleration gradient, mitral deceleration time, peak velocity of flow distal to the pulmonary valve, pulmonary acceleration gradient, pulmonary acceleration time, pulmonary rr-time interval, right atrial mean end systolic pressure, right ventricular diameter at end diastole, tricuspid regurgitation peak velocity, aortic regurgitation, mitral regurgitation, tricuspid regurgitation, pulmonary regurgitation, aortic stenosis, mitral stenosis, tricuspid stenosis, pulmonary stenosis, and / or physician-reported diastolic function.

22. 18. The method of claim 17, wherein the diagnostic parameters comprise a diagnosis of acute rheumatic fever, a diagnosis of chronic rheumatic heart disease, a diagnosis of hypertensive disease, a diagnosis of ischemic heart disease, a diagnosis of pulmonary heart disease and pulmonary circulation diseases, a diagnosis of acute pericarditis, a diagnosis of other forms of heart disease, a diagnosis of acute myocarditis, a diagnosis of cardiomyopathy, a diagnosis of cardiac arrest, a diagnosis of paroxysmal tachycardia, a diagnosis of atrial fibrillation, a diagnosis of heart failure, a diagnosis of cerebrovascular diseases, a diagnosis of diseases of arteries, arterioles and capillaries, a diagnosis of diseases of veins, lymphatic vessels and lymph nodes, a diagnosis of hypotension, a diagnosis of other and unspecified diseases of the circulatory system, a diagnosis of diabetes, a diagnosis of congenital heart disease, a diagnosis of dyslipidemia and / or a diagnosis of chronic kidney disease.

23. The method of claim 1 , wherein each of the at least one echocardiographic video of the heart associated with the individual is associated with a single echocardiographic view.

24. 24. The method of claim 23, wherein the single echocardiographic view is a parasternal long axis view.

25. 2. The method of claim 1, wherein the clinical outcome risk score associated with the individual can provide the physician or healthcare administrator with sufficient information to make a decision about potential treatment for the individual, the potential treatment comprising at least one of a heart transplant, implantation of a mechanical assist device, placement of a defibrillator, palliative care, or hospice.

26. 26. The method of claim 25, wherein the determination is eligibility for the treatment.

27. 1. A system comprising at least one processor coupled to at least one memory containing instructions, the at least one processor comprising: receiving at least one echocardiographic video of a heart associated with the individual; providing the at least one echocardiographic video to at least one trained model in a trained neural network, the trained neural network being trained to generate at least one risk score based on the input echocardiographic video; receiving electronic health record information relating to the individual; linking the at least one risk score with the electronic health record information for each view associated with the at least one echocardiographic video; providing the concatenated at least one risk score and electronic health record information as inputs to a sub-model of the trained neural network, the sub-model configured to generate a clinical outcome risk score associated with the individual; and receiving the clinical outcome risk score associated with the individual from the sub-model of the trained neural network; outputting the clinical outcome risk score associated with the individual to at least one of a memory or a display for viewing by a physician or health care administrator; generating a report based on said clinical outcome risk score; and executing said instructions to perform the steps of:

28. The electronic health record information comprising values ​​for several parameters including age, tricuspid regurgitation peak velocity, heart rate, low density lipoprotein, left ventricular ejection fraction, diastolic pressure, pulmonary artery acceleration time, systolic pressure, pulmonary artery acceleration gradient, and diastolic function.

28. The system of claim 27.

Images