Systems and methods for ai-assisted echocardiography

Abstract

A method for processing a sparsely populated data source, method for generating a training set for training a model to predict mitral regurgitation from echocardiograph data, and method of predicting heart failure from echocardiograph data including the steps of: retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports; analysing the echocardiograph data to determine unpopulated data fields; populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model; calculating a probability output from a trained model; analysing echocardiograph measurement data of individual patient records from the echocardiograph data to determine a prediction of the presence of a disease state in the patient on the basis of the calculated probability output; and associating the presence of the disease state to a prediction of heart failure in the patient.

Description

SYSTEMS AND METHODS FOR AI-ASSISTED ECHOCARDIOGRAPHYField of the Invention

[0001] The present invention relates to artificial intelligence and in particular to systems and methods for artificial intelligence integration to analysis of medical data and records.

[0002] The invention has been developed primarily for use in systems and methods for Al-assisted echocardiography for identification of various forms of left ventricular dysfunction that may manifest as clinical heart failure, and will be described hereinafter as “heart failure” with reference to this application. However, it will be appreciated that the invention is not limited to this particular field of use.Background

[0003] Any discussion of the background art throughout the specification should in no way be considered as an admission that such background art is prior art, nor that such background art is widely known or forms part of the common general knowledge in the field in Australia or worldwide.

[0004] Echocardiography (or simply “echo”) is a subspecialty of cardiology that makes use of specialised ultrasound equipment to take diagnostic images of the heart. It is particularly valuable as a first-line diagnostic tool due to its ability to noninvasively assess the internal structure and function of the heart in a cost-effective manner.

[0005] A comprehensive echo procedure can measure many features of the heart totalling approximately 150 unique variables and a total of approximately 400 variables if calculations from unique variables are included, but the full complement of characteristics is rarely measured due to time constraints and impracticality in the absence of disease.

[0006] Echocardiographic report and measurement data from a wide range of contributing sites Australia-wide are stored in the National Echocardiogram Database Australia (NEDA). In its current form, NEDA consists of measurements taken from echo procedures and report texts that are an analysis from that procedure for many patients. Currently, the database has echocardiography data relating to more than 1 million patients.

[0007] NEDA contains echocardiographic measurement and report data from participating real-world clinical echocardiography laboratories. The measurements are performed as part of standard clinical echocardiography, performed for clinical indications under standard echocardiography imaging protocols. Although there is some minor variation between laboratories, imaging techniques, protocols, image acquisition and structures to be measured,and the specific locations on each structure that measurements are performed, has been standardised. Guidelines for performing a comprehensive standard transthoracic echocardiogram have been published, see: https: / / www.asecho.org / wpcontent / uploads / 2018 / 10 / GuidelinesforPerformingaComprehensiveT r ansthoracicEchocardiographicExaminationinAdults.pdf.

[0008] As is standard in modern echocardiography, the images are stored in DICOM (Digital Imaging and Communications in Medicine) format with the measurements stored in Structured Reporting (SR) format along with the images. At the conclusion of the echocardiogram, the images and SR file are transferred to a Cardiology PACS (Picture Archive and Communications System). From there, the images are viewable with the accompanying measurements, for generation of the echocardiography report. Standard workflow involves a preliminary report by the echocardiographer who performed the study, and the finalised report by a Cardiologist which may involve additional changes to measurements and / or interpretation compared with the preliminary report. The final report is that which is sent to the medical record / referring medical practitioner as the definitive interpretation of the echocardiogram procedure. A final echocardiogram report typically contains the measurements that were transferred in the SR file along with the text interpretation of the echocardiogram, and a conclusions section. Recommendations for a standardised transthoracic echocardiogram report can be found at: https: / / www.asecho.org / wp-content / uploads / 2013 / 05 / Standardized_Echo_Report_Rev1 .pdf

[0009] For the purpose of this Application, we define “echocardiographic report data” to include all measurement and report information that is contained in the final echocardiogram report. NEDA has developed a proprietary system for capturing all retrospective echocardiographic report data from a participating echocardiography laboratory, allowing for all measured echocardiographic variables and all corresponding interpretive text information to be collated into a single database containing a unique record for each echocardiogram . Each database is then remotely transferred into the Master NEDA Database via a “vendor-agnostic”, automated data extraction process that transfers every measurement for each echocardiogram performed into a standardized NEDA data format (according to the NEDA Data Dictionary). Each individual contributing to NEDA is given a unique identifier along with their demographic profile (date of birth and sex) and all data recorded with their echocardiogram. Using this methodology, NEDA has collected over 1 ,000,000 echocardiographic reports and subsequently linked this data with the Australian National Deaths Index (NDI) through the data linkage unit at the Australian Institute of Health and Welfare (AIHW), Canberra, Australia. This is the Australian Government’s Asset of all Deaths in Australia, and NEDA has obtained multiple ethical approvals from Human Research Ethics Committees (HREC), covering both public and private echocardiography laboratoriesthroughout Australia, as well as the HREC at AIHW for mortality linkage. NEDA is registered by the Australasian Clinical Trials Registry:

[0010] This unique resource of echocardiographic measurement and report data and the accompanying deaths linkage data is held in a secure server environment, however it is important to note that no NDI data is used for training of Al systems. All NDI data is kept within the highly secure NEDA and Australian Government server network, and can only be used by NEDA authorised personnel for research purposes (and therefore not used for training of Al systems). Physical separation of patient identifier data and all content data is maintained to protect participant anonymity, and all data analyses are maintained and performed on a deidentified basis. This massive repository of echocardiographic report data can be used to train artificial intelligence (Al) systems, although by its nature is incomplete with typically about 1 / 3 of all possible measurements performed in a standard echo. Since the individual measurements performed vary on the clinical indication for the echocardiogram and the findings revealed as the echocardiogram is performed, there is no minimum dataset that is present in every echocardiogram. Thus, while NEDA contains a large amount of echocardiographic report data, it may be sparsely populated with certain measurements performed infrequently. Figure 1 shows a few real examples of different measurements present and missing in echo studies for a small randomly selected group of patients, being a typical example of sparse echo data where each row of the table is a record of the echo measurement available for a single patient.

[0011] The NEDA database contains the measurements required to diagnose most cardiac disease that can be identified by echocardiography, with the report data containing additional information obtained by visual inspection of the echocardiographic images. Since each cardiac disease identified by echocardiography has typical features (“phenotype”) that are contained within the measurement and text information, NEDA contains a rich tapestry of disease phenotypes, although each disease phenotype is not labelled (or identified) within the NEDA database. Therefore, NEDA does not include patient phenotype information to identify traits or groups of traits which are held by patients having common diseases.

[0012] The workflow for a typical prior art echocardiography study process is depicted in Figure 2 for existing echocardiography analysis methods. In summary, a typical workflow consists of the following steps:■ Images are acquired 101 by a sonographer using a special ultrasound machine.■ Specific physical features are measured 103 from the acquired echo images by the sonographer, for example, the diameter of the left ventricle is commonlymeasured. In other circumstances, the physical features are measured 103 from the acquired echo images automatically by an image-recognition Al system (Note: many of the measurements are typically taken during the image acquisition process with the patient present, while others may be measured from the images acquired during the procedure once the patient has left). In this workflow, the sonographer has complete and sole control over whether or not sufficient images of the patient have been acquired or whether additional images are required 104 for a meaningful diagnosis of the patient’s actual or suspected condition.■ The images and measurements are manually interpreted 105 by the sonographer.■ A preliminary report is prepared 107 either by an automated system or manually by the sonographer, using measurements either performed by the sonographer, or automatically acquired measurements using an image-recognition Al system. The automatically generated or manually populated report is then completed by the sonographer detailing their interpretation of the study.■ The cardiologist reads the preliminary report and inspects the manual analyses 109 and measurements.■ The cardiologist creates a final report 111 with their remarks and conclusions from the study.

[0013] A key point is that the set of images and measurements required to be taken to ensure that the sonographer or cardiologist has sufficient data to diagnose the patient’s condition is comprehensive, meaning an echo study is time-consuming for the sonographer and prone to error, such as particular data being missed during the scan by a sonographer who may be inexperienced or unfamiliar with the requirements for a particular study. In practice, however, not all possible measurements are taken, only a subset related to a suspected condition or disease are recorded by the sonographer. It is dependent upon the sonographer’s skill and experience to know which measurements are important for subsequent analysis and diagnosis.

[0014] Accordingly, there is a need for methods of obtaining complete echocardiography datasets for diagnosis of abnormal or disease conditions in patients within time and cost constraints of physical echocardiography procedures. Also, there is a need for methods and systems for determining meaningful data to provide a complete echocardiography record for past echocardiography patients.

[0015] Furthermore, there is a need for identifying patients at increased risk of suffering from, or increased risk of adverse outcomes from, a particular condition such as heart failure. Preferably, the likelihood of a particular condition in a patient should be identified automatically during the echocardiography investigation, which would assist the cardiologist (and potentially also assist the sonographer) in making a positive diagnosis when such a condition is found to be present. Alternatively, if the automated system identifies that the condition is unlikely, this information could be useful to assist the cardiologist (and potentially the sonographer) to identify that the disease is or is not present.

[0016] Heart Failure is a clinical syndrome that is typically associated with symptoms, signs of congestion (identified during clinical examination, biomarkers and / or echocardiography) along with echocardiographic abnormalities. Echocardiography studies are pivotal in characterising the Left Ventricular Dysfunction (LVD) that is characteristically present in the setting of heart failure. Since clinical diagnosis of heart failure can be technically challenging, with appropriate analysis, echo measurement Al may provide new and more granular insights into the various LVD phenotypes that are associated with heart failure. Thus, enhancing identification of these LVD phenotypes may assist physicians in their diagnosis of heart failure.Summary

[0017] It is an object of the present invention to overcome or ameliorate at least one or more of the disadvantages of the prior art, or to provide a useful alternative.

[0018] According to a first aspect of the present invention, there is provided a method for processing a sparsely populated data source. The method may comprise Step (a) retrieving data from a sparsely populated data source to form a base dataset comprising a plurality of patient echocardiography reports, the data source comprising a plurality of patient records, each patient record not requiring a full set of populated data fields corresponding to a medical measurement. The method may further comprise Step (b) dividing the base dataset into two portions. A first portion (i) may comprise a training dataset being a defined percentage, X%, of the base dataset. A second portion (ii) may comprise a holdout dataset being a defined percentage (100% - X%) of the base dataset. The method may further comprise Step (c) analysing the training data set to jointly model variable relationships using a non-linear function approximation algorithm applied iteratively to the records of the training dataset to obtain a trained imputation model and measurement prediction protocols for populating unpopulated fields in the training data set. The method may further comprise Step (d) imputing the predicted measurement values in the records of the holdout dataset. The method may further comprise Step (e) applying a primary filter to the imputed holdout dataset to remove patient echocardiography reports comprising an absence of a first indication of the presence of, absence of, or severity of a predetermined disease state. Themethod may further comprise Step (f) further dividing the imputed holdout dataset into two portions. A first portion (i) may comprise an interim disease training dataset being a defined percentage, Y%, of the test / validation dataset. A second portion (ii) may comprise an interim disease test dataset being a defined percentage (100% - Y%) of the base dataset. The method may further comprise Step (g) applying a secondary filter to each portion above to filter the interim datasets to remove patient echocardiography reports comprising a second indication of prior treatment for a predetermined disease state to obtain a disease training dataset and a disease test dataset. The method may further comprise Step (h) applying a tertiary filter to each of the disease training dataset and a disease test dataset to filter the datasets to select a valid echocardiography report for each patient, wherein validity refers to the presence of the first indicator and the absence of the second indicator. The method may further comprise Step (i) analysing the disease training dataset on the basis of predefined disease conditions in known patient records of the disease training dataset to form a classification model adapted to associate patient data to a probability output of a disease condition in patient records of the disease training data set. The method may further comprise Step (j) validating the model comprising analysing the disease test dataset using the classification model, wherein the records of the disease test dataset may comprise data associated with patient data, and determining a validation error which may comprise a probability of correctly predicting a patient associated with a disease state in the records of the disease test dataset . The method may further comprise Step (k) repeating Steps (i) to (j) to minimise the validation error and computing a probability output of a probable disease state for each patient record in the disease test dataset.

[0019] According to a particular arrangement of first aspect , there is provided a method for processing a sparsely populated data source comprising:(a) retrieving data from a sparsely populated data source to form a base dataset comprising a plurality of patient echocardiography reports, the data source comprising a plurality of patient records, each patient record not requiring a full set of populated data fields corresponding to a medical measurement;(b) dividing the base dataset into two portions:(i) a first portion comprising a training dataset being a defined percentage, X%, of the base dataset; and(ii) a second portion comprising a holdout dataset being a defined percentage (100% - X%) of the base dataset;(c) analysing the training data set to jointly model variable relationships using a non-linear function approximation algorithm applied iteratively to the records of the training dataset to obtain a trained imputation model and measurement prediction protocols for populating unpopulated fields in the training data set;(d) imputing the predicted measurement values in the records of the holdout dataset;(e) applying a primary filter to the imputed holdout dataset to remove patient echocardiography reports comprising an absence of a first indication of the presence of, absence of, or severity of a predetermined disease state;(f) further dividing the imputed holdout dataset into two portions:(i). a first portion comprising an interim disease training dataset being a defined percentage, Y%, of the test / validation dataset; and(ii). a second portion comprising an interim disease test dataset being a defined percentage (100% - Y%) of the base dataset;(g) applying a secondary filter to each portion above to filter the interim datasets to remove patient echocardiography reports comprising a second indication of prior treatment for a predetermined disease state to obtain a disease training dataset and a disease test dataset;(h) applying a tertiary filter to each of the disease training dataset and a disease test dataset to filter the datasets to select a valid echocardiography report for each patient, wherein validity refers to the presence of the first indicator and the absence of the second indicator;(i) analysing the disease training dataset on the basis of predefined disease conditions in known patient records of the disease training dataset to form a classification model adapted to associate patient data to a probability output of a disease condition in patient records of the disease training data set;(j) validating the model comprising analysing the disease test dataset using the classification model, wherein the records of the disease test dataset comprise data associated with patient data, and determining a validation error comprising a probability of correctly predicting a patient associated with a disease state in the records of the disease test dataset;(k) repeating Steps (i) to (j) to minimise the validation error and computing a probability output of a probable disease state for each patient record in the disease test dataset.

[0020] The method may further comprise recalibrating the probability output to correspond to the true probability of the disease state. The method may further comprise defining a series of threshold values to divide the probability output into bins that map to classes of disease severity e.g. low risk, medium risk, high risk, thereby to generate a predicted class that is indicative of degrees of LV Dysfunction that may be associated with a further disease state.

[0021] Analysing the training data set may be performed using a using a machine learning system.

[0022] The disease state may be mitral regurgitation, left ventricular systolic dysfunction, left ventricular diastolic dysfunction, and / or heart failure.

[0023] The method may further comprise: a first stage comprising an imputation model used to fill in missing values in the echocardiographic data, and a second stage comprising a classification model used to predict the disease state.

[0024] The imputation model may comprise a Mixture Density Network (MDM). The classification model may comprise an automated machine learning algorithm configured to: train a large number of candidate models: and build an ensemble classifier by choosing a weighted subset of the candidate models to optimize performance on a given classification metric.

[0025] The method may further comprise an F1 -score measuring a harmonic mean of precision and recall.

[0026] The candidate models may be selected from a list of standard machine techniques including: Categorical Boost Classifier, Light Gradient Boosting Model, Extra Trees Classifier, Random Forest Classifier, K-Nearest Neighbours Classifier and Support Vector Classifier.

[0027] The neural network pipelines may comprise one or more scaling techniques selected from the group including: Normalization, Minimum Maximum Scaling, Power Transformation, Quantile Transformation, Robust Scaling, Standard Scaling.

[0028] The neural network pipelines may comprise one or more scaling techniques selected from the group including: Fast Independent Component Analysis, Kernel Principal Component Analysis, Random Kitchen Sinks, Nystroem Kernel Approximation, Polynomial Features, Truncated Singular Value Decomposition, Extra Trees Preprocessor Classification, Feature Agglomeration, Random Trees Embedding, Select Percentile Classification, Lib Linear SVC Preprocessing,

[0029] The disease state may be mitral regurgitation. The disease state may be aortic stenosis. The further disease state may be clinical heart failure.

[0030] According to a second aspect of the present invention, there is provided a method for generating a training set for training a model to predict mitral regurgitation from echocardiograph data. The method may comprise the step of retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports. The method may further comprise the step of analysing the echocardiograph data to determine unpopulated data fields. The method may further comprise the step of populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model. The method may further comprise the step of filtering the patient records to remove patient records with no availablediagnosis of mitral regurgitation severity. The method may further comprise the step of filtering the patient records to remove patient records from patients with more than one patient record and to select a valid echocardiography report for each patient, wherein validity refers to the presence of a first indicator and the absence of a second indicator. The method may further comprise the step of generating a training set for training a machine learning or artificial intelligence system, the training set based on predefined disease conditions in the patient records.

[0031] According to a particular arrangement of the second aspect, there is provided a method for generating a training set for training a model to predict mitral regurgitation from echocardiograph data, comprising the steps of retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports; analysing the echocardiograph data to determine unpopulated data fields; populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model; filtering the patient records to remove patient records with no available diagnosis of mitral regurgitation severity; filtering the patient records to remove patient records from patients with more than one patient record and to select a valid echocardiography report for each patient, wherein validity refers to the presence of a first indicator and the absence of a second indicator; generating a training set for training a machine learning or artificial intelligence system, the training set based on predefined disease conditions in the patient records.

[0032] The first indicator may comprise the presence of, confirmed absence of, or severity of mitral regurgitation. The second indicator may comprise the presence of prior mitral valve replacement or repair.

[0033] According to a third aspect of the present invention, there is provided a method of predicting heart failure from echocardiograph data. The method may comprise the step of retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports. The method may further comprise the step of analysing the echocardiograph data to determine unpopulated data fields. The method may further comprise the step of populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model. The method may further comprise the step of calculating a probability output from a trained model. The method may further comprise the step of analysing echocardiograph measurement data of individual patient records from the echocardiograph data to determine a prediction of the presence of a disease state in the patient on the basis of the calculated probability output. The method may further comprise the step of associating the presence of the disease state to a prediction of heart failure in the patient.

[0034] According to a particular arrangement of the third aspect, there is provided a method of predicting heart failure from echocardiograph data, comprising the steps of: retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports; analysing the echocardiograph data to determine unpopulated data fields; populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model; calculating a probability output from a trained model; analysing echocardiograph measurement data of individual patient records from the echocardiograph data to determine a prediction of the presence of a disease state in the patient on the basis of the calculated probability output; and associating the presence of the disease state to a prediction of heart failure in the patient.

[0035] The disease state may be left ventricular dysfunction. A predetermined threshold may be applied to determine the clinical disease state and or likelihood of developing the clinical disease state of Heart Failure.

[0036] A method as claimed in claim 18 wherein a predetermined threshold can be applied to determine the clinical disease state and or likelihood of developing the clinical disease state of left ventricular dysfunction.Brief Description of the Drawings

[0037] Notwithstanding any other forms which may fall within the scope of the present invention, preferred embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:

[0038] Figure 1 shows an example of patient records in a sparsely populated dataset;

[0039] Figure 2 shows a typical workflow procedure for existing echocardiography analysis methods;

[0040] Figure 3 shows a workflow procedure for the filtering of training data for the Al models described herein; and

[0041] Figure 4 shows the frequency distribution as a function of disease severity for two variables closely associated with mitral regurgitation;

[0042] Figures 5A and 5B respectively show a heatmap of the correlation coefficients across all 1 19 echocardiograph variables, where Figure 5A highlights high correlation values with dark shading, and Figure 5B highlights low correlation values with dark shading.

[0043] Figure 6 shows a Precision-Recall (P-R) curve describing the relationship between evaluation metrics of the Al model, displaying precision (x-axis) vs recall (y-axis) for the AutoPyTorch (blue), XGBC (orange) and Logistic Regression (green) models with respect to diagnosis of moderate or greater MR.

[0044] Figure 7 shows a matrix of the Prediction ratio per class vs Precision.

[0045] Figures 8A and 8B show a haemodynamic analysis across the spectrum of MR severity wherein the Mean (and Standard Deviation) is shown across the severity classes for a number of relevant echocardiographic variables.

[0046] Figures 9A and 9B together shows a workflow procedure for the filtering of training data for the Al models described herein;

[0047] Figure 10 shows plots of the long-term cardiovascular-related mortality for each decile output of the LVD Al, adjusted for age and sex wherein the rising LVD Al decile was strongly associated with increasing mortality.

[0048] Figure 11 shows the partial dependence for a subset of model parameters with the global mean probability output marked on each plot.

[0049] Figure 12 shows the result of absolute gradient of the partial dependence against sampled variable value for each variable in the model feature set.

[0050] Figure 13 describes the averaged gradient across the normalised range of values for each variable in Figure 12.

[0051] Figure 14 shows Centred ICE Plots for demonstrative echocardiograph variables.

[0052] Figure 15 shows Normal ICE plots of the outlier gradients determined by the centred ICE plots of Figure 14.

[0053] Figures 16A, 16B, 16C show Contour Plots of the 2D Partial Dependence for demonstrative combinations of variables;

[0054] Figure 17 shows an example computer system for implementation of the Al model described herein.Definitions

[0055] The following definitions are provided as general definitions and should in no way limit the scope of the present invention to those terms alone, but are put forth for a better understanding of the following description.

[0056] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the invention belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. For the purposes of the present invention, additional terms are defined below. Furthermore, all definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms unless there is doubt as to the meaning of a particular term, in which case the common dictionary definition and / or common usage of the term will prevail.

[0057] For the purposes of the present invention, the following terms are defined below.

[0058] The articles “a” and “an” are used herein to refer to one or to more than one (i.e. to at least one) of the grammatical object of the article. By way of example, “an element” refers to one element or more than one element.

[0059] The term “about” is used herein to refer to quantities that vary by as much as 30%, preferably by as much as 20%, and more preferably by as much as 10% to a reference quantity. The use of the word ‘about’ to qualify a number is merely an express indication that the number is not to be construed as a precise value.

[0060] Throughout this specification, unless the context requires otherwise, the words “comprise”, “comprises” and “comprising” will be understood to imply the inclusion of a stated step or element or group of steps or elements but not the exclusion of any other step or element or group of steps or elements.

[0061] Any one of the terms: “including” or “which includes” or “that includes” as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, “including” is synonymous with and means “comprising”.

[0062] In the claims, as well as in the summary above and the description below, all transitional phrases such as “comprising”, “including”, “carrying”, “having”, “containing”, “involving”, “holding”, “composed of”, and the like are to be understood to be open-ended, i.e. to mean “including butnot limited to”. Only the transitional phrases “consisting of” and “consisting essentially of” alone shall be closed or semi-closed transitional phrases, respectively.

[0063] The term, “real-time”, for example “displaying real-time data”, refers to the display of the data without intentional delay, given the processing limitations of the system and the time required to accurately measure the data.

[0064] The term “near-real-time”, for example “obtaining real-time or near-real-time data” refers to the obtaining of data either without intentional delay (“real-time”) or as close to real-time as practically possible (i.e. with a small, but minimal, amount of delay whether intentional or not within the constraints and processing limitations of the of the system for obtaining and recording or transmitting the data.

[0065] Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, preferred methods and materials are described. It will be appreciated that the methods, apparatus and systems described herein may be implemented in a variety of ways and for a variety of purposes. The description here is by way of example only.

[0066] The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0067] In this respect, various inventive concepts may be embodied as a computer readable storage medium (or multiple computer readable storage media) (e.g. a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other non-transitory medium or tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the invention discussed above. The computer readable medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present invention as discussed above.

[0068] The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as discussed above.Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of the present invention need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present invention.

[0069] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0070] Also, data structures may be stored in computer-readable media in any suitable form. For simplicity of illustration, data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a computer-readable medium that convey relationship between the fields. However, any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements.

[0071] Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0072] The phrase “and / or”, as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e. elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e. ’’one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and / or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0073] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list,“or” or “and / or” shall be interpreted as being inclusive, i.e. the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of”, or, when used in the claims, “consisting of” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either”, “one of”, “only one of”, or “exactly one of”. “Consisting essentially of”, when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0074] As used herein in the specification and in the claims, the phrase “at least one”, in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B”, or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0075] For the purpose of this specification, where method steps are described in sequence, the sequence does not necessarily mean that the steps are to be carried out in chronological order in that sequence, unless there is no other logical manner of interpreting the sequence.

[0076] In addition, where features or aspects of the invention are described in terms of Markush groups, those skilled in the art will recognise that the invention is also thereby described in terms of any individual member or subgroup of members of the Markush group.Detailed Description

[0077] It should be noted in the following description that like or the same reference numerals in different embodiments denote the same or similar features.

[0078] Described herein include a number of echocardiography report data records sourced from the National Echo Database of Australia (NEDA) - a vendor agnostic source neutral database containing measurement and text outputs from multiple participating echocardiographiclaboratories across Australia. The database used is linked to the National Deaths Index (NDI) with mortality outcomes, however this deaths data is not available to Echo IQ and no mortality data formed part of these experiments. The mortality-linked database (NEDA V2.0) contains 1 ,077,145 studies on 631 ,824 individuals (Cohort 301), with the mortality data available only to NEDA investigators and reported.Study 1

[0079] Text from echo reports is sourced from NEDA. A total of 1 ,535,414 echo reports are included, each divided into one or more comments for a total of 14,271 ,548 individual text fields (Cohort 309). No patient data is included in this dataset, but the unique identifier StudylD can be cross-referenced with the original NEDA echocardiographic database (Cohort 301).

[0080] There are two goals of the present Study. Firstly, the goal is to extract and quantify reporting of the presence or absence of mitral regurgitation (MR), the presence or absence of comments about mitral regurgitation, or if a comment is present, to extract the reported MR severity. The second goal is to examine the potential for the Al to predict the types of echocardiographic measurements observed in various forms of left ventricular dysfunction, and that may also be present in the clinical syndrome of heart failure (HF).

[0081] Blocks of fully de-identified text from each echo report (Cohort 309) were parsed into a comma-separated-values (CSV) text file, containing a row for each report section indicating the study ID, report section title, and text for that section. The echo report text data was then processed by an NLP engine to fix misspellings and formatting issues and to determine a label for each of the echo studies indicating the physician reported severity of MR. The NLP engine processes data across a number of stages:(1 ) Pre-processing to fix misspellings and formatting issues.(2) Text extraction to identify echo-related content.(3) Quantitative extraction to extract measurements.(4) Qualitative extraction to extract specific reference to the presence of and disease grading reported in the body of the report. This includes the presence or absence of mitral regurgitation, accompanying comments on regurgitation severity, and the valve comments.

[0082] All relevant valve comment terms were included in the proprietary NLP system (for example, mitral valve thickening, calcification, prolapse etc). Chronicity was established using specific search terms.

[0083] Finally, the presence of any valve intervention, including mitral valve replacement (surgical or percutaneous), mitral valve repair, or transcatheter edge-to-edge repair, were extracted and these patients were excluded from further analysis as discussed below.

[0084] The NLP engine uses a range of techniques including regular expression parsing, open-source NLP libraries and heuristic rules, and were also combined with specialist-informed custom dictionaries for cardiovascular terms and misnomers (e.g., ‘regulation’ replaced with ‘regurgitation’ and similar common errors in the text of echo reports). Performance of the heuristic rules and identified dictionary values was tuned by processing text across the entire corpus of echo text in NEDA.

[0085] Performance and cross checking of the NLP engine was performed using Boolean and string extractions, and random manual comparisons of edge cases. Finally, to verify the performance of the NLP, manual checking of a random sample of >1000 cases was undertaken by a board-certified echo-cardiologist and no MR group re-allocation was required.

[0086] In an example, consider an Echo Report including the text:INDICATIONS: Shortness of breath on exertion. The echo demonstrates sever mitral regurgitation. The LV is moderately dilated with mild global LV systolic disfunction. The regurgitant jet originates centrally with regurgitant volume of 31 ml, a calculated distal jet are 40% of LAA, vena contract of 4 mm and EROA of 0.30 cm2, the mitral leaflets are moderately thickened, particularly the posterior leaflet. The left ventricle is moderately dilated with LVOT velocity of 0.8 m / sec. overall systolic function is mildly impaired with the akinetic septal apex ad inferior wall, severe hypokinesis inferoseptum. There is concentric LV hypertrophy. Trileaflet aortic valve. There is aortic root dilation. There is trivial aortic regurgitation.

[0087] After preprocessing and processing with the NLP engine described above, the NLP output, separated into the different processing stages, would include:Text Extraction-.Reason / s for echo: SOBOE; LV global systolic dysfunction: FOUND; Mitral thickening: FOUND; Hypertrophic cardiomyopathy: FOUND; Aortic valve morphology: Trileaflet; Aortic disease: FOUND.Regional Wall Motion Abnormality Detection:Akinetic: Septal; Akinetic: Inferior; Hypokinesis: Inferior.Quantitative Extraction:Regurgitant volume: 31 ml; Jet area: 40%; Vena contracta: 4 mm; EROA: 0.30 cm2; LVOT velocity: 0.8 m / secSeverity Extraction-.Mitral regurgitation: SEVERE; Aortic regurgitation; TRIVIAL.Study 1 - Model Development

[0088] The training and testing data for the model was sourced from the National Echo Database of Australia (NEDA) - a vendor agnostic source neutral database containing measurement and text outputs from multiple echocardiographic laboratories across Australia. A brief outline of the training data filtering for the model is shown in Figure 3. Only physical measurements with an acceptable fill rate were used, resulting in 119 variables. Missing values in the data were imputed. To do this we developed a Mixture Density Network (MDN) model.

[0089] The labelling for the Al model was sourced from the echo text data of the NEDA dataset 307. Multiple Natural Language Processing (NLP) methods were used to extract physician reported severity of mitral regurgitation (MR). The presence of any mitral valve replacement or mitral valve repair was also detected and these echo reports were removed.

[0090] The full pipeline from raw echo data to final probability output has two stages: The first stage is an imputation model used to fill in missing values in the echocardiographic data, the second stage is an Al classification model used to predict a disease state.

[0091] For the imputation model, we used a Mixture Density Network (MDN). An MDN model is a neural network multivariate regression model that learns the relationship between all parameters in a given dataset. The MDN is used to impute data for the downstream Al model to predict a disease state (MR or HF).

[0092] The imputation stage takes in 119 physical echo measurements. To improve model performance and facilitate explainability, highly correlated columns are removed from the output. For example, all measures of LV Diastolic ‘size’ such as LV_Diastolic_Diameter_PLAX and several methods of calculating the LV Diastolic Volume are removed except for LV_Diastolic_Volume_SIM, which acts as a proxy for the other variables. The final list of input columns to the Al classification model contains 47 physical variables, with Age and Gender also supplied to the Al classification model for a total of 49 variables.

[0093] For the classification model, a moderate or greater MR binary classifier was chosen as the labelling scheme to prioritise treatable forms of MR. To perform classification, an ensemble classifier Al model was trained. As an example, implementation of the Al model, an open-source Python package known as AutoPyTorch was used. AutoPyTorch is an automatedmachine-learning algorithm that trains a large number of candidate models and then builds an ensemble classifier by choosing a weighted subset of these models in order to optimize performance on a given classification metric (in our case, the F1 -score). The individual models consist of traditional machine-learning models (e.g. Support Vector Models, Gradient Boosting etc.) as well as neural network pipelines. Each pipeline consists of a traditional pre-processing algorithm (e.g. Random Trees Embedding, Principal Component Analysis etc.) followed by a neural network. Ensemble selection is performed via Bayesian optimization and hillclimbing algorithms.

[0094] Despite extensive training and re-training of the AutoPyTorch and other models, prediction of physician-reported MR severity was not robust enough for commercial application. However, clinical review of the outputs of the Al classification model showed very promising capacity to identify echocardiographic measurements typically associated with left ventricular dysfunction. The particular characteristics of these echocardiographic measurements corresponded to the phenotype of abnormalities typically seen in heart failure.

[0095] Echo - derived variables used in the Al model include an input of 49 different, filled variables, including:Hierarchy RV_Diastolic_Basal_Diameter MV_Deceleration_Time LV_Ejection_Fraction_Hierarchy LVOT Diameter PV Peak VelocityGenderfinal Pulmonary_Vein_A_Velocity RVSP_CombinedBody_Surface_Area Mitral_A_Duration Aortic_Arch_Diameter LVOT Peak Velocity IVC_Diameter_Expiration IVS_Diastolic_Thickness LVOT AV VTI Ratio Systolic_BP AVAVTIcalculated TR Peak Velocity LV_Stroke_Volume_SIM AV Peak Velocity Mitral_E_to_A_Ratio PI_Peak_Velocity RVOT_Velocity_Time_lntegralLV_Mass_2 D AS E l ndex Diastolic_BPLVOT_Stroke_Volume_lndex Right_Atrial_PressureHierarchy Pulmonary_Vein_Systolic -Velocity Pulmonary_Vein_S_D_RatioLV_Septal_E_Prime_Velocity

[0096] The following methodology is used to develop the presently disclosed Al model:1 . Choose labelling source for Mitral Regurgitation (MR) a. Investigate potential labelling sources for MR:i. NLP: extraction of physician -graded severity from echo report text data (Cohort 307). ii. Effective Regurgitant Orifice Area (EROA): direct MR measurement data from NEDA. iii. Mitral Regurgitant Stroke Volume: calculated proxy for MR Stroke Volume. b. Explore fill rate and potential accuracy rates of ground truth labels. Use MDN imputation model to fill in missing data within NEDA. a. A Mixture Density Network (MDN) imputation protocol, as discussed herein, was used for pre-processing and model training. b. Follow clinical guidance on appropriate variables for imputation. i. Include all physical echo measurements. ii. Exclude text outputs, NEDA disease labels, low fill rate columns. c. Note that mortality data is excluded from training Al databases and can only be used within the secure NEDA or Australian Government servers for testing the Al outputs. d. Split patients into train and test sets (Cohorts 303 + 305) with ratio 30:70 to allow large training set for downstream MR model. i. tested by data science team, no increase in error margins. Dataset filtering a. Select patients / studies from those with valid ground truth labels (Cohort 309). b. Split into train and test sets (Cohorts 315 + 317). Decide on labelling scheme (multiclass, binary (with different boundaries), regression) a. Investigate options for using grade of MR severity. i. E.g. Grade X / 4 MR. ii. No / trivial / mild / moderate / severe MR. b. Investigate label types: i. Binary (choose severity boundary) ii. Regression iii. Multiclass Train MR model a. Use open-source software, AutoPyTorch, to implement model training. b. T rain simple baseline models to validate the performance increase from using AutoPyTorchi. E.g. Logistic Regression, XGBC (extreme Gradient Boosting Classifier) c. Perform feature selection. i. Remove correlated features with clinical guidance.6. Validate model. a. Choice of evaluation metric to optimize: i. E.g. maximum F1 -score, F1 -score at specific precision / recall thresholds b. Class-based haemodynamics c. Performance metrics to be guided by clinical advice and feedback throughout the course of model development.Study 1 - ResultsStudy 1 - Ground truth labelNatural Language Processing

[0097] Data were pre-processed to fix misspellings and formatting issues on echo report text data (Cohort 307). Multiple NLP methods were used to extract physician graded severity of mitral regurgitation, which included:1 ) manual labelling of very common text comments,2) direct extraction of “[x] mitral regurgitation” where [x] is in a list of common severity labels (e.g. “no”, “trivial”, “minor”, “slight”, “mild”, “moderate”, “severe” etc.),3) heuristic text extraction (looking for common sentence structures, including negation and multiple clauses), and4) syntax parsing (automated detection of sentence structures and subsequent extraction of adjectives pertaining to mitral regurgitation).

[0098] Finally, the presence of any valve intervention, including mitral valve replacement (surgical or percutaneous) or mitral valve repair (including surgical or percutaneous) was extracted. The various grades and descriptions of MR were grouped into the following categories:• Severity = -1 : Unknown MR: mitral valve mentioned but no severity of MR extracted• Severity = 0: No MR: no / without mitral regurgitation• Severity = 1 : Trivial MR: trivial / trace / minor / slight mitral regurgitation• Severity = 2: Mild MR: includes trivial-to-mild MR• Severity = 3: Moderate MR: includes mild-to-moderate MR• Severity = 4: Severe MR: includes moderately severe MR, moderate-to-severe MR

[0099] There is an additional cohort with no valid NLP output for MR, corresponding to echoes where the mitral valve was not mentioned, or no report text was provided. These form the remainder of the database and will be excluded from all further analysis.

[0100] The final counts for each NLP extracted class are given below in Table 1 :Severity Grade # %Table 1: NLP labels from all provided text data

[0101] After filtering to StudylDs in NEDA v2.0 (i.e. with enough echo measurements to be potentially useful in creating an MR model) the following counts shown in Table 2 were obtained:Table 2: NLP labels from all NEDA v2.0 studies

[0102] Although the exact accuracy of the NLP labelling is hard to determine, manual sensitivity analysis of a random sample of >1000 cases was undertaken by a board -certified subspecialist echo-cardiologist to verify the performance of the NLP with No MR group re-allocation required. Other tests investigating random samples of difficult-to-classify comments against the NLP outputs approximates the accuracy at around 95% or higher. Having discrete NLP outputs of MR severity for more than 80% of echo reports in NEDA provides a suitable basis for a ground truth label in the context of machine learning models.NEDA Measurements

[0103] The Mitral Regurgitation Guidelines lists the following NEDA variables as directly relevant to diagnosis of the severity of MR: Mitral_Regurgitant_Volume, MV_Regurgitant_Volume, MV_Regurgitant_Fraction, MR_ERO_PISA, MR_Orifice_PISA. There are additional variables that measure properties of the mitral regurgitant flow. The low fill rates within the NEDA dataset limited the use of these variables as the ground truth (Table 3), below:NEDA Variable Fill rate (%)Mitral_Regurgitant_Volume 0.51 163MR_ERO_PISA 0.175464MR_Flow_Convergence_Radius 1 .145064MR_Flow_Rate_PISA 0.210464MR_Mean_Gradient 0.568447MR_Mean_Velocity 0.519336MR_Orifice_PISA 0.792465MR_Peak_Gradient 0.853088MR_Peak_Velocity 0.904335MR_Velocity_Time_lntegral 1 .078035MV_Regurgitant_Fraction 0.009748MV_Regurgitant_Volume 0.090981Table 3: Fill rates across MR variables in NEDA

[0104] The most direct quantitative measurement of MR severity is the Effective Regurgitant Orifice Area (EROA). There are two measures for EROA in NEDA: MR_ERO_PISA and MR_Orifice_PISA which are two terms for the same variable and are therefore never both measured on the same echo. After combining these two into a single variable “EROA”, the distribution of values was investigated across the severity spectrum (as determined by the NLP physician-graded severity). The EROA is almost never measured on echo reports with severity less than mild MR, as shown in Table 4 below, which is expected from a physiology perspective since no regurgitant orifice is present in the setting of minimal or absent regurgitation. In mild MR, although occasionally measured and can theoretically be measured frequently, in practice it is technically very difficult to accurately identify the correct border of the proximal isovelocity shell, the apex of the PISA radius and the vena contracta point. As a result, large errors are common when quantitation of mild MR is undertaken, particularly if the mild MR is an eccentric jet, or only present for part of systole (such as late-systolic MR, or early-systolic MR). Further compounding the problem, without having the capacity for independent image review to verify the accuracy of individual measurements, it is not possibleto identify the source of larger-than-anticipated errors in a very large data set. For example, in mild or greater MR, the maximum recorded EROA is at least an order of magnitude larger than regarded as humanly possible.

[0105] In order to highlight likely measurement errors, we defined (based on expert review) the humanly possible maximum EROA to be approximately the size of the largest possible mitral annulus (10 cm2). This suggests a high likelihood of significant errors (which may also be associated with unit conversion errors). Due to the low fill rate(especially on echo reports without significant MR) and the likelihood of inaccurate values, it was determined that the EROA is not a suitable candidate for the ground truth label.Table 4: Statistics for Effective Regurgitant Orifice Area (E OA) in NEDA

[0106] Note that it is not expected that quantitation would (or could) be performed in the absence of significant MR, since a proximal isovelocity shell is required to make these measurements. The predominance of measurements in moderate or severe MR is expected, although even in the setting of moderate or greater MR, the amount of quantitation is small, representing 4.19% (3338 of 92842) of moderate MR studies, and 16.5% (5301 of 32198) of severe MR studies.Mitral Regurgitant Stroke Volume

[0107] A ground truth label based on echo measurement data, the ‘Mitral Regurgitant Stroke Volume’ (MR Stroke Volume, also known as the MR Regurgitant Volume), was also utilised a potential basis for the model. The physiological basis for this variable is that blood pumped out of the left ventricle (LV) can only go out through the left ventricular outflow tract (LVOT), or through the mitral valve in the setting of mitral regurgitation (or potentially through another defect if present, such as a ventricular septal defect). Therefore, the difference between the LV Stroke Volume and the LVOT Stroke Volume provides a proxy for the Stroke Volume passing through the mitral valve:MR Stroke Volume= Left Ventricle Stroke Volume— Left Ventricular Outflow Tract Stroke Volume

[0108] Although theoretically this should be a robust method for estimation of MR severity, it assumes that the measurement of the LV stroke volume and LVOT stroke volume are both accurate. Unfortunately, this assumption does not hold true in standard clinical echocardiography because of variability of both measures, and since LV stroke volume and LVOT stroke volume are calculated using different modalities (2-D echo and pulsed-wave echocardiography), the errors may potentially compound each other.

[0109] To examine whether these limitations could be overcome, two methods for calculating the LV Stroke Volume were considered, the Teichholz formula and the Simpson’s biplane method. For both methods, there was a marginal increase in the MR Stroke Volume for echo reports with NLP-labelled severe MR, but no visible increase for mild or moderate MR as shown in Figure 4 where the frequency distribution for MR Stroke Volume and LA VolumeJndex as a function of severity class (see Table 1 ) are plotted. There is a distinctive leading edge in MR Stroke Volume for severity = 4 (severe MR) but less discriminating for severity classes -1 to 3 (unknown, no, trivial, mild, moderate MR). Most likely due to measurement error in the estimation of LV stroke volume, the overlap between severe MR and non-severe MR was considered too large, especially given that more standard echocardiographic variables like the LA VolumeJndex differentiate much more strongly between the different severity classes. Despite the relatively high fill rate and strong theoretical foundation for the MR Stroke Volume, the error rate was too high and it was deemed an unsuitable ground truth label for MR.Study 1 - MDN imputation of missing data

[0110] The model training methods used in the methods disclosed herein require a dataset with no missing values. Since the real word NEDA dataset is quite sparse in terms of fill rates, a Mixture Density Network (MDN) model was produced to impute the missing values (see Applicant’s prior publication WO / 2024 / 000041 for details of MDN imputation pre-processing and model training).• Model parameters were set as follows: o Number of training epochs N ~ 200 o Batch size B = 256 o Number of mixture components c = 3 o Minimum standard deviation eCT= 10-5o Holdout probability phoidout = 0-3 o Learning rate r| = 5 x 10-4o Network configuration 4 layers of 2048 nodes• Input features were chosen under clinical guidance to meet the following criteria: o Pass a minimum fill-rate threshold of 5% in the training set. o Include all physical echo measurements which meet the fill-rate condition. o Exclude text outputs, and NEDA-calculated disease labels.• Trim ranges are documented in and remain the same as were used in prior studies. No column collapsing was used in this model.• The dataset was split into training (30%) and test (70%) sets based on PatientID. The reason for using a smaller training set was to produce enough imputed data to allow for downstream MR model development. These variations in the train-test split were tested in prior studies and found to have no impact on imputation error rates.

[0111] The final list of features contained 1 19 variables suitable for imputation.Randomness in imputation sampling

[0112] The output of the MDN model applied to a dataset of input echo reports is a probability distribution for each feature of each echo. Whereas in previous model iterations, the probability distribution itself was used as the final output, in this study the distribution is sampled in order to produce a final value to be used in training the MR model.

[0113] In a computational context, sampling of a probability distribution is achieved by using a pseudo-random number generator, which outputs a very long (232) but predictable sequence of ‘random’ numbers. Each time a new sample is drawn, the result will differ from the previous sample as the next number in the sequence is used to draw that sample. In this way after many samples are taken, the distribution of sampled outputs matches the probability distribution from which they were drawn. The starting point of the sequence (i.e. how far along in the sequence to begin), is known as the ‘seed’ and is usually chosen arbitrarily to simulate randomness.

[0114] In a production context, the final output of the MR model applied to an input echo should be the same on successive runs of the algorithm. In order to achieve this, we can fix the seed so that the resulting sequence of random numbers is always the same. The sample itself will still be randomly drawn from the given probability distribution but the same sample will be produced each time.

[0115] The stability of this seeding method run across different computation environments has been tested. Preliminary experiments have shown that the computational environment does have an effect on the seeding process, but that successive runs in the same environmentproduce the same samples in a reproducible way. Future experimental work will determine the final code for the seeding process to ensure product stability.Study 1 - Dataset filtering

[0116] The MDN test set (Cohort 305) with missing data filled in through imputation forms the base dataset for the downstream MR model.

[0117] NEDA database data 301 is split into two groups:• an imputation training dataset 303;• and an imputation test dataset 305.

[0118] The Imputation Test dataset 305 was further filtered to remove those with no NLP output for mitral regurgitation (MR) to obtain an MR test dataset 307, and also to remove the “unknown MR” category. The overwhelming majority (-99%) of “unknown MR” text outputs originate from the text comment “Structurally normal mitral valve" with no mention of MR. However, it was clear that whatever label was provided, such ambiguous NLP outputs can be safely removed without affecting model development as there is enough data with more well-defined outputs.

[0119] MR test dataset is then separated into two groups: a training patient dataset 309 of 75% of MR dataset 307 and the remaining 25% designated the test patient dataset 311 .

[0120] A secondary filter was applied to remove echo reports with evidence of “mitral valve replacement’ or “mitral valve repair" (MVR). There is clear evidence in the data that these surgical or transcatheter interventions result in a rapid decrease in mitral regurgitation severity, but the echocardiographic measurements relied upon to predict MR may not return to “normal” values (i.e. the phenotype may remain abnormal even after resolution of the MR, at least for a period of time), and thus cannot be relied upon to predict mitral regurgitation in the setting of mitral valve replacement. Hence, echo reports with MVR tend to produce a large number of false positives.

[0121] A third filter was applied to select only one echo per patient, in particular the last valid echo per patient, where validity refers to the presence of an NLP output for MR and the absence of an MVR.

[0122] Once these filters were applied, the dataset was split 75%:25% into obtain the final training dataset 315 and test dataset 317 as shown in Table 5 below, describing the complete list of filtering steps applied to the data.

[0123] Due to the order in which these filters were applied, there were a very small number of echo reports in the final train and test sets that did not have direct evidence of an MVR but that previous echo reports for that patient did have evidence of an MVR. There were 23 in the test set (out of 84,946) and 44 in the train set (out of 254,735).Dataset Cohort # # Studies # PatientsMDN test set 305 758,283 505,688Exclude no NLP 574,372 391 ,569Exclude ‘unknown’ NLP 309 508,399 348,458Exclude MVR 494,421 344,568Last valid echo 339,681 339,681Final training set (75%) 315 254,735 254,735Final test set (25%) 317 84,946 84,946Table 5: Dataset filtering steps from the initial MDN output (Cohort 305) to the final training and test sets (Cohort 315 + 317)Inclusion Exclusion Criteria

[0124] The filtering steps described above were applied to the NEDA V2.0 data and reduced to the final data set by application of the inclusion / exclusion criteria as summarised in Table 6, below:Cohort ID Source No. # StudiesPatientsTable 6: Inclusion / Exclusion CriteriaStudy 1 - Labelling scheme

[0125] There is significant flexibility in the choice of labelling scheme to be used in model training. At a basic level, the NLP outputs of ‘no’ / ‘trivial’ / ‘mild’ / ‘moderate’ / ‘severe’ MR constitute a textbook example of an ‘ordinal regression’ framework in which the labels form discrete but ordered categories as shown in Table 7 below. However, we also have the freedom to interpret these labels in other, more simple frameworks such as binary classification, multiclass classification or as a regression problem.

[0126] More complex models can result in worse performance because the methods used to train the models are generally much less developed in both the literature and publicly available packages. In addition, they may be trying to solve problems we are not interested in (such as the boundary between trivial and mild MR, or the boundary between moderate and severe MR).

[0127] Additionally, the wider clinical and commercial context needs to be taken into account. In particular, mild MR is a very common condition that has a significant impact on mortality outcomes but is currently not recommended for treatment due to the absence of any evidence of a benefit, as well as potential danger and expense of intervention. Our choice of labelling scheme prioritises the highest mortality groups and the treatable forms of MR, namely severe and moderate MR.Label type Definition Advantages Disadvantages F1(approx.)corresponds to severityTable 7: Labelling schemes with their respective advantages, disadvantages and approximate performance. *Severity labels 0, 1,2, 3, 4 refer to no, trivial, mild, moderate, severe MR respectively

[0128] Preliminary tests showed that the labelling scheme which produced the best performance was the binary - mild+ classifier. The performance was similar for the labelling schemes binary - moderate+, regression (synthetic EROA and standard) and multiclass. The worst performance was obtained by the binary - severe+ classifier and the ordinal regression framework.

[0129] The mild+ classifier has the major disadvantage that mild MR is not currently recommended for any form of intervention. The improved performance largely comes from the fact that the haemodynamic boundary between mild and moderate MR is much less well-defined than between trivial and mild MR.

[0130] The labelling scheme which corresponds best to the ground truth label, the ordinal regression framework, did not produce good performance due to the high complexity of the model and the relative lack of well-developed techniques to solve problems of this kind.The moderate+ classifier is preferable to the other labelling schemes with similar performance in that it corresponds well to the clinical and commercial needs for diagnosing treatable MR, has an acceptable prevalence rate in the general population and there is a large range of techniques available to solve the problem of a binary classifier.Study 1 - Model training

[0131] The Python package, AutoPyTorch, was used to train the binary classifier model. This package trains a large number of individual models using different machine learning algorithms and neural network architectures along with a variety of preprocessing techniques and builds a final ensemble model to solve classification and regression problems. The relevant hyperparameters we used for training are:• Ensemble size: 50• Optimization metric: F1• Runtime: 40000 seconds (~11 hours)

[0132] Candidate traditional machine learning techniques included in AutoPyTorch and which may be used in the models disclosed herein according to requirements include: Categorical Boost Classifier (catboost), Light Gradient Boosting Model (lightgbm) as well as the scikit-learn modules ExtraTreesClassifier, RandomForestClassifier, K-Nearest Neighbours Classifier and Support Vector Classifier.

[0133] Candidate scaling techniques and which may be used in the models disclosed herein according to requirements include the scikit-learn modules Normalizer, MinMaxScaler, PowerTransformer, QuantileTransformer, RobustScaler, StandardScaler.

[0134] Candidate pre-processing techniques and which may be used in the models disclosed herein according to requirements include the scikit-learn modules Fast Independent Component Analysis, Kernel Principal Component Analysis, RandomKitchenSinks, NystroemKernel Approximation, PolynomialFeatures, Truncated Singular Value Decomposition, ExtraTreesPreprocessorClassification, Feature Agglomeration, RandomTreesEmbedding, SelectPercentileClassification, LibLinearSVCPreprocessor.

[0135] In addition to the AutoPyTorch model, two baseline models were also trained on the same training and testing datasets, namely an extreme Gradient Boosting Classifier (XGBC) and Logistic Regression classifier, in order to establish that the AutoPyTorch model performs better than a simple machine learning algorithm.Feature selection

[0136] In order to improve model performance and make model explainability easier, feature selection was performed in order to remove highly correlated columns. The correlation of both the real and imputed data was considered, given that the real data shows the true correlation between the variables, but the model also relies on imputed data to produce an output. The methodology followed is as follows:• Calculate correlation coefficient of both real and imputed data.• Choose an approximate threshold at which to group columns.■ Use clinical guidance to decide when correlated variables should not be grouped due to their high importance.• From each group, choose one variable to keep and drop the rest.■ Use clinical guidance to decide which variables are more reliable / appropriate.■ Consider fill rate of variables when deciding which variables to keep.

[0137] The final list of variables was reduced from 1 19 to 47 variables. Age and gender are also included as columns into the MR model, but since their fill rate in NEDA is 100%, they are not included in the MDN imputation model or in the correlated feature analysis. Figures 5A and 5B show a heatmap of the correlation coefficients across all 1 19 variables where Figure 5A highlights high correlation values with dark shading, and Figure 5B highlights low correlation values with dark shading.. . Empty cells (white) indicate no co-measurement of data to with which to calculate a correlation.Model calibration

[0138] The output of the binary classification algorithm is a continuous number between 0 and 1 , representing the probability of moderate+ MR. However, the raw output of any Al model is not guaranteed to be well-calibrated in the sense that the output can’t be directly interpreted as the likelihood of the positive class label. There exist simple methods tore-calibrate the output of the model to correspond to a true probability, for example fitting a sigmoid function which transforms the raw output into a calibrated output. In the following discussion, we will use the terminology “probability output for the raw output of the model.

[0139] In particular embodiments, the probability output may be further calibrated to correspond to the true probability of the disease state where the disease state of particular relevance to the current methods are mitral regurgitation (MR), left ventricular dysfunction (LVD) and / or heart failure (HF). However, it would be readily appreciated by the skilled addressee that the methods and systems disclosed herein would be well suited to procedure of using a machine learning system for analysing clinical data for any number of possible human ailments. The methods disclosed herein further comprise defining a series of threshold values to divide the probability output into bins that map to classes of disease severity e.g. low risk, medium risk, high risk, thereby to generate a predicted class that may be associated with a disease state. In the methods and models discussed herein, the disease states of particular relevance are mitral regurgitation (MR), left ventricular dysfunction (LVD) and / or heart failure (HF). However, it will be readily appreciated by the skilled addressee that the methods and models discussed herein are readily suitable for a large variety of possible disease states.

[0140] Stratification of the testing dataset into bins, e.g. quintiles or deciles, is completely independent of the process of calibration. Similarly, the assessment of individual features’ importance to the final output of the model is performed on a relative basis amongst variables and is largely unaffected by calibration.

[0141] For the purpose of simplicity, the rest of this document will use the raw probability output as the output of the model for all further analysis.Study 1 - Model validation

[0142] The principal metric for evaluating the performance of the algorithm is the F1 -score, which measures the harmonic mean of precision and recall. We have the freedom to choose a threshold for the probability output at which an echo should be classified as “MR” or “not MR”. The models we build can then be compared against each other in terms of their F1 -score once a threshold has been chosen. There are several options each with their own advantages and disadvantages.F1 -score at a threshold of 0.5:This is the most natural choice for a classifier. The AutoPyTorch ensemble method is designed to optimize the F1 -score at 0.5, though the actual maximum may not be at exactly 0.5.However, this choice of threshold might not suit our clinical and commercial needs (e.g. too many false positives, too many false negatives etc.).Maximum F1 -score:This choice is justified by trying to maximise one of the most commonly-accepted evaluation metrics.However, this choice of threshold might not suit our clinical and commercial needs (e.g. too many false positives, too many false negatives etc.).F1 -score at a specified precision or recall threshold:Based on clinical and commercial needs, we choose a suitable threshold for precision (reducing the rate of false positives returned) or recall (reducing the rate of false negatives missed). The F1 -score is calculated at this threshold and compared across models.Since recall and precision tend to have an inverse relationship, choosing a high threshold for one metric will tend to reduce the value of the other metric. Since the F1 -score is designed to balance the two metrics, it will also be reduced from its maximum value.

[0143] For each of these methods, the F1 score (Table 8) is higher for the AutoPyTorch model than XGBC or Logistic Regression.F1 method AutoPyTorch XGBC Logistic RegressionThreshold = 0.5 0.514 0.406 0.365Maximum F1 0.515 0.511 0.497Precision = 70% 0.283 0.2480.223Table 8: F1 scores for various methodologies and models at various thresholds

[0144] For the AutoPyTorch model, the performance at each of these thresholds in terms of Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) is seen in Table 9:F1 method Sensitivity Specificity PPV NPVTable 9: Performance metrics for the AutoPyTorch model at various thresholds

[0145] The relationship between these evaluation metrics can be visualised by plotting precision against recall at all values of the threshold (the Precision -Recall (P-R) curve of Figure 6) and by additionally overlaying lines of equal F1 (isocurves). Ideally, a better modelwould have all points of the P-R curve further to the right than a worse model. It can also be the case that a model has better performance when optimising for recall but worse performance when optimising for precision. The final decision on which model is best should be taken in the context of the clinical and commercial needs of the product. In Figure 6, Precision (x-axis) vs recall (y-axis) for the AutoPyTorch (blue plot 601 ), XGBC (orange plot 603) and Logistic Regression (green plot 605) models with respect to diagnosis of moderate or greater MR. Curves of equal F1 607 are overlayed in grey. Bold markers represent the performance at a fixed threshold value of 0.5 for each of the AutoPyTorch, XGBC and Logistic Regression) models (see Table 7).

[0146] Another clinical consideration is the performance of the algorithm across the different severity classes of MR. Ideally, the false negative rate for severe MR should be as low as possible so that the algorithm does not miss patients who are at the highest risk of mortality. We can visualise the performance across the severity spectrum at various levels of precision in order to make a value judgement of the usefulness of the model.

[0147] In Figure 7, the Prediction ratio per class vs Precision is shown where the numbers inside the cells [“X7”Y”] represent the number predicted positive (X) and the number predicted negative (Y) within each severity class at a given precision, where a positive prediction refers to the MR severity noted on the Y axis, and a negative prediction is the remainder of the patients in that severity group.. “No” and “trivial” MR are combined into class 1 due to their similarity in terms of haemodynamics (see Figures 8A and 8B) and model performance.

[0148] The prediction ratios demonstrate relatively low prediction rates for no / trivial MR compared to mild MR. For example, at a precision of 0.6 mild MR is labelled positive in ~9% of cases whereas no / trivial MR is labelled positive in ~1% of cases despite being 2.1 times more prevalent. The colour of each box in Figure 7 illustrates the percentage of positive predictions relative to the total number of cases. Mild MR positive predictions make up the bulk of false positives. Severe MR has a higher positive prediction ratio than moderate MR.

[0149] Another method to understand the performance and behaviour of the model is to analyse the haemodynamics across the spectrum of severity classes (0,1 , 2, 3, 4 refer to no, trivial, mild, moderate, severe MR) as compared to haemodynamics with respect to the classifier outputs. In Figures 8A and 8B, the Mean (and Standard Deviation) is shown across the severity classes for a number of relevant echocardiographic variables. Intensity of shading represents the direction of deviation of the mean value away from the population mean. In Figure 8A values without a border refer to a state of being more normal, values with a border refer to a state of being more abnormal. In Figure 8B values with a border refer to a state ofbeing more normal, values without a border refer to a state of being more abnormal . Also shown is a summary statistic known as the Partial Eta Squared (qA2), which measures the proportion of variance explained by a given variable. There is a strong gradient across the severity classes for several variables including LA_Volume_lndex and LV_Mitral_E_to_E_Prime_Ratio_Hierarchy, with a partial eta squared of 0.268 and 0.159 respectively. Ideally, these gradients should also be apparent between the negative and positive predictions of the model.

[0150] Analysis of the haemodynamics of each severity class demonstrates that the boundary between trivial and mild MR is much more distinct than that between mild and moderate MR. For example, the LA_Volume_lndex jumps from 33.2 + / - 16.5 ml / m2 for trivial MR to 62.6 + / - 34.7 ml / m2 for mild MR, but only increases to 72.5 + / - 43.2 ml / m2 for moderate MR. A similar trend is present for LV_Mitral_E_to_E_Prime_Ratio_Hierarchy and LV_Septal_E_Prime_Velocity among other variables.

[0151] The labels used for model training combine no / trivial / mild MR into class ‘0’ and moderate / severe MR into class T. Due to the indistinct boundary between mild and moderate MR, the haemodynamic gradient between these classes is weaker than the gradients across all severity classes. In Table 10, the Eta-squared for important variables is compared for the following classes 1 ) all severity classes (as shown in Figures 8A and 8B), 2) grouped severity: no / trivial / mild MR vs moderate / severe MR, 3) model outputs (negative vs positive prediction).

[0152] The Eta-squared (Eta_Sq)for the grouped severity classes is significantly lower than across all severity classes. The Eta_Sq across model outputs (negative vs positive predictions) are of similar magnitude to those across all severity classes, with somewhat higher eta squared for LV_Mitral_E_to_E_Prime_Ratio_Hierarchy, Mitral_E_Point_Velocity and LV Ejection Fraction Hierarchy and slightly lower eta squared for LA_Volume_lndex.Table 10: Eta squared correlation coefficient for real label and model outputs.Study 1 - Conclusion

[0154] NLP-derived physician-graded severity was deemed to be a suitable ground truth label for MR due to inaccuracies and low fill-rates of MR quantitation variables in the available very large real-world database. MDN imputation of physical echo measurements was used to fill in missing data to be used in the downstream MR model. The dataset was filtered to exclude ambiguous MR labels and the presence of Mitral Valve Replacement / Repair, before being divided into test and train datasets. A binary-moderate+ classifier was decided as the labelling scheme due to its high prevalence in the literature and close correspondence with the clinical and commercial needs of the product. A publicly available package AutoPyTorch was used to train an ensemble model to predict moderate+ MR. Model performance was considered unacceptable for further development as a Mitral Regurgitation classifier. Although the false negative rate for ‘no’ or ‘trivial’ MR was typically very low, overall performance was hampered by the indistinct haemodynamic boundary between mild and moderate MR, combined with the low prevalence of moderate or greater MR.

[0155] Preliminary clinical analysis of the outputs of the Al model within the secure NEDA environment suggested that although the prediction correctly identified patients with increased mortality outcomes, it did not stratify them based on MR severity. There was, however, preliminary evidence suggesting these patients were stratified based on a haemodynamic profile of left ventricular dysfunction. This discovery led to the subsequent model development pathway summarised in subsequent Studies. From hereon, the model discussed above will be referred to as the LVD Al (Left Ventricular Dysfunction Al) model.Study 2

[0156] In a particular example of the above methods, the LVD Al model was applied to a test set of patients that the Al had never previously seen (Cohort 317). The studies were then filtered for no evidence of right and / or left ventricular pacing based on NLP extracted data and the results were imported into the NEDA database independently from Echo IQ, so that mortality outcomes could be analysed.

[0157] The method of the current study is depicted in Figures 9A and 9B. The original total NEDA dataset 301 , containing 1077,145 individual echo studies, was split 30:70 into a training 303 and test 305 database for building an imputation framework as above. The 758,283 studies on 505,688 patients 305 who did not participate in the imputation training database were then subjected to interrogation of the accompanying text and cardiologist comments contained in the echocardiogram report. By the use of Natural Language Processing Al (NLP), any comment regarding mitral regurgitation (MR) was captured. Patients with no valid NLP MR comments became the validation cohort 901 , except for patients where >1 study formed part of the training 315 or test 317 database (the studies from these patients were excluded from analysis, and shown in Figure 9B as “ineligible studies” 903).

[0158] Patients with >1 valid NLP MR comment were randomly split into training 311 (75%, n=380,163 studies on 261 ,285 patients) or test 905 (25%, 128,236 studies on 87,173 patients) cohorts. The training database 311 was used to train the models described in Study 1 described above, and will not be discussed further in respect of this Study 2 since no data in the training dataset was analysed. The Test cohort 905 (87,173 patients) was filtered to a single echo per patient (last echo only) and the absence of an NLP extraction comment about a mitral valve replacement (MVR). Further, if any reference to right and / or left ventricular pacing was identified using NLP extracted data, the patient was also excluded from analysis. The validation cohort 901 underwent similar filtering to a single echo per patient (last echo only) and no MVR, and no evidence of right and / or left ventricular pacing based on NLP extracted data. The final test database 910 comprising 81 ,693 patients was the primary database used for analysis purposes. The validation database 907 is not used in this analysis.

[0159] The heart failure probability data for each patient in the Test and Validation cohorts shown in Figures 9A and 9B were merged via a direct matchup process (using PatientID and StudylD, with matches confirmed by random backward checks) with the main NEDA database.

[0160] For this Study 2, the last reported echo (i.e. most recent echo available) was included as described in Figure 9A, from individuals investigated from 1 / 1 / 2000 to 21 / 05 / 2019 with data linkage to the National Death Index data provided by the Australian Institute for Health and Welfare. If someone died, the primary cause of death was categorized according to ICD-10AM coding, with those in the range of I00-I99 categorized a cardiovascular-related death. All fatal events (including cardiovascular-related events) were identified during a median 1 ,541 (interquartile range [IQR] 820, 2,629 days) follow-up.

[0161] Probability data for the heart failure models were presented as a continuous probability (between 0 and 1 ), and as quartiles, quintiles and deciles (representing equal numbers of patients in each group).

[0162] No formal calculations of study power were performed given our ability to analyse >150,000 case-fatalities during 2.5 million person-years follow-up. Discrete variables were summarized by frequencies and percentages (with 95% confidence intervals [Cl] where appropriate). Continuous variables were summarized by standard measures of central tendency and dispersion (Mean, Standard Deviation; Median, IQR). The relevant applicable guidelines were applied, specifically the Diastolic Function Guidelines and the Heart Failure Guidelines. The results for each guideline group, and for the heart failure models (quintiles and deciles), were presented in tables, including indeterminate groups.

[0163] Long-term outcomes for each Heart Failure Model probability group were displayed using Kaplan-Meier survival curves. Using 5-year actuarial data (a minimum potential follow-up of 5 years for all patients, with censoring of data at 1825 days, and deaths recorded only during the initial 1825 days of follow-up), a series of Cox-Proportional Hazard Models (entry model) were then used to derive hazard ratios (HR) for all-cause and cardiovascular-related mortality (with censored events) for each Heart Failure Model group. The Cox Models comprised a two basic model adjusting for age and sex used for each of the Heart Failure Model groups, and then a fully adjusted model (a combination of age, sex, AF or other atrial arrhythmia, ventricular systolic and diastolic function parameters, and TR velocity) to derive adjusted HR with 95% CL All analyses were performed with SPSS v29.0 and statistical significance accepted at a 2-sided p-value of <0.05.

[0164] Standard heart failure guidelines were applied according to EF category, with preserved EF (LVEF >50%), mildly impaired EF (LVEF 40-49.9%), and impaired EF (LVEF <40%). Further classification of the HFpEF group was then undertaken according to filling pressure, and classified as normal filling pressure, indeterminate filling pressure, or increased filling pressure (i.e. as having HFpEF). In addition, patients were classified according to the 2016 Diastolic Function guidelines, with either preserved EF (LVEF >50%), or reduced EF (<50%), and the subcategories described by the Guidelines. The LVD Al probability output was presented as a continuous variable (from 0 to 1), indicating the Al’s raw output, and also into equal deciles, equal quintiles, and equal quartiles.

[0165] For ROC curves, the probability output was plotted as a continuous output against the categorical output of all-cause death. Analysis of the heart failure probability outputs included echocardiographic tables, and crosstabs against each heart failure and diastolicdysfunction group. It also included survival curves of Kaplan -Meier all-cause long-term survival, and 5-year all-cause and cardiovascular-related mortality after adjustment for age and sex. Hazard ratios were calculated from the Cox hazard analysis.

[0166] After exclusions, 81 ,693 patients were included for final analysis in the test database 910, comprising 41 ,888 males (51 .3%), of whom 64,816 individuals could have some form of heart failure guidelines applied. The LVEF was impaired (<40%) in 4,373 individuals (5.4% of the test cohort), mildly impaired (40-49.9%) in 3,627 (4.4%) and preserved (>50%) in 56,816 (69.5%).

[0167] Patients meeting guideline definitions for heart failure had the typical echocardiographic characteristics, with the HFpEF having preserved LVEF, LA dilatation, diastolic dysfunction and pulmonary hypertension, and HFmrEF and HFrEF showing similar characteristics apart from a lower LVEF. Diastolic function guidelines demonstrated normal diastolic function in 31 ,532 (55.5%) individuals, with 4,397 (7.7%) showing abnormal diastolic function, and 20,887 individuals (36.8%, mean age 64.2±17.8 years) with indeterminate diastolic function. As expected, cardiac function appeared normal in those with normal diastolic function and had characteristics of heart failure with preserved ejection fraction (HFpEF, accompanied by older age and higher BP) in the diastolic dysfunction group. The indeterminate diastolic function pressure group showed characteristics in between the other two groups. The impaired LVEF diastolic function was available in 8,000 patients. Of these, 1 ,110 (13.9%) had guideline evidence of normal left atrial (LA) pressure (LAP), 392 (4.9%) had increased LA pressure and the majority, comprising 5,777 individuals (mean age 69.6±15.5 years), had indeterminate LA pressure. Most individuals in all 3 groups had abnormal cardiac function with the least abnormal in the Grade 1 diastolic dysfunction group. The indeterminate LA pressure group showed characteristics most similar to the Grade 2 (increased LA pressure) group.

[0168] The LV dysfunction Al probability output (containing measurement abnormalities typically identified in heart failure) was available in all 81 ,693 patients, a substantial increase compared with the proportion of indeterminate outputs when traditional heart failure guidelines or diastolic function guidelines were applied. Patients were youngest in the first decile with a steep age gradient across each decile, showing a mean age of 47.1 ± 14.7 years in the first decile through to 77.4 ± 12.2 years in the 10th decile. This age gradient was accompanied by small increases in the aortic velocities and a small decrease in the body surface area, rising blood pressure, and marked increases in cardiac changes usually present in heart failure. Specifically, there was a substantial rise in the indexed LA volume, a fall in the LVEF in the upper deciles, a rising LV mass across deciles, accompanied by a corresponding rise in theE:e’ ratio, a fall in the septal e’ velocity, and a rise in the pulmonary artery systolic pressure. Similar characteristics were identified when the LVD Al was examined according to probability output quintiles. These findings indicate that the Al can be applied to a larger proportion of echocardiograms than current heart failure and diastolic function guidelines, and offer a more nuanced approach to identify increasing levels of left ventricular dysfunction. The Al identifies the same phenotypic abnormalities as the heart failure and diastolic function guidelines, but with a simple probability output that has widespread applicability.

[0169] When compared with the heart failure guidelines, patients with HFrEF tended to have a LVD Al high probability output, with 3,350 patients (76.6%) in the highest quintile. The distribution was similar but less marked in the HFmrEF groups (45.6%) with a broader distribution in the preserved EF groups.

[0170] It is important to note that normal individuals without heart failure have a normal EF, and therefore the term “Preserved EF” does not indicate heart failure is present. A small subgroup of patients with “Preserved EF” have HFpEF, which is identified by the presence of specific phenotypic abnormalities despite the presence of a normal EF. Indeed, among those with an LVEF >50%, 80% of patients with normal filling pressures were in the first two quintiles. Conversely, 80% of patients with increased filling pressure were in the upper three quintiles. Similarly, among those with a preserved LVEF (only some of whom had HFpEF), there was a steep age gradient across the probability output quintiles, and a worsening phenotype of increasing blood pressure, diastolic dysfunction and signs of increased left ventricular filling pressure. As the diastolic function worsened, there were greater numbers of individuals in the higher quintiles of the LVD Al probability output. This observation was consistent across normal and impaired EF diastolic function guideline groups. Importantly, when diastolic function was indeterminate, the distribution of LVD Al continued to identify individuals with the echocardiographic heart failure phenotype.

[0171] Mortality was increased in traditionally classified heart failure groups. Patients showing no echocardiographic evidence of heart failure had the best survival. The group with a preserved EF and normal filling pressure is the least likely to have clinical heart failure and has the best long-term survival. Each of the heart failure groups (preserved EF with increased filling pressure, mildly reduced EF and reduced EF) have an adverse survival with the HFrEF having the worst survival, comparable with the HFmrEF groups. Similar mortality statistics were observed with the diastolic function guidelines, where the survival was significantly worse for patients with indeterminate or abnormal diastolic function . Also, the corresponding survival in impaired LVEF categories was impaired for all groups, with echocardiographic signs ofincreased filling pressure having a slightly more adverse prognosis, although importantly, indeterminate filling pressure determination was also associated with adverse survival.

[0172] ROC analysis demonstrated that the LVD Al model yielded a good ability of the Al to predict mortality, with area under the ROC curve for all-cause long-term mortality at 0.771 (95% Cl 0.767-0.775, p<0.001 ). The mortality associated with each increasing decile beyond the first two (overlapping) deciles progressively increased as the probability output increased (see Figure 10 which shows the long-term cardiovascular-related mortality for each decile output of the LVD Al, adjusted for age and sex. HR = Hazard Ratio, with the first decile as the reference group. 95% confidence intervals are presented with the hazard ratios). This mortality association remained for long-term all-cause mortality as well as 5-year actuarial mortality. To account for the capability of the Al to identify at-risk individuals beyond the “traditional” heart failure echocardiographic variables, the Al was then subjected to more comprehensive adjustment for age, sex, AV peak velocity, body surface area (BSA), indexed LA volume, LVEF, and the indexed left ventricular mass. The results of this fully adjusted model demonstrated that the LVD Al continued to show a significant mortality association even with these adjustments. Further adjustment was made to include the tricuspid regurgitation velocity, and despite a weakening of the mortality association between the LVD Al probability output and mortality, higher quintiles continued to show a significantly increased mortality. The fully adjusted HR = 1 .81 (95% Cl 1 .42-2.31 , p<0.001 ) and HR = 2.10 (1 .59-2.76, p<0.001 ) for the upper two quintiles, compared with the lowest risk group.

[0173] The LVD Al was then examined for its ability to transcend the echocardiographic clinical practice guidelines for heart failure. Although the heart failure guidelines predicted mortality, the LVD Al remained independently associated with mortality.Study 2 - Conclusions

[0174] The Echo IQ LVD Al model predicts the heart failure phenotype via a continuous probability output scale. The characteristics identified by the model are typical for the echocardiographic characteristics of heart failure. These include heart failure with preserved LVEF (HFpEF), where a higher probability output is associated with a rising left ventricular mass, E:e’ ratio, a lower septal e’ velocity, a larger left atrium, and increasing pulmonary artery pressures. The model also appropriately identified patients with mildly reduced (40-49.9%) or reduced (<40%) LVEF.

[0175] Importantly, the model could be applied to every patient (even in the setting of missing data), although the minimum variable set required to maintain accuracy of the model is not yet determined.

[0176] The LVD Al model has a strong independent mortality association, including raw unadjusted long-term Kaplan-Meier mortality association, in 5-year actuarial models adjusted for age and sex, and in fully adjusted models including age, sex, and a comprehensive set of echocardiographic variables known to change in the setting of heart failure. Finally, the LVD Al model continues to be associated with mortality after adjustment for the existing Heart Failure guidelines.

[0177] The LVD Al developed herein has enormous potential for patient management. Potential applications for use of the LVD Al include:1 . Finding all patients at-risk across the spectrum of heart failure who would benefit from timely clinical review and titration of medical therapy,2. Finding more patients with the HFpEF syndrome who will benefit from clinical review and introduction of disease-modifying therapy,3. Categorising which patients may benefit most from introduction and / or titration of medical therapy,4. Assessing response of patients to medical therapy using the LVD Al system;5. Use of the LVD Al systems to triage high-risk patients for earlier treatment in resource-limited areas.Study 3

[0178] As discussed above, the Al model is an AutoPyTorch machine learning model that takes an input of 49 different, filled variables and runs to produce either a classification prediction or probability output.

[0179] To test the efficacy of the LVD Al model described herein, explainability tests were run against a predefined test set of studies. The original list of studies was derived from the full NEDA v2.0 dataset and then filtered, removing patients with an MVR in the process.

[0180] Once the designated studies for the test set were established, consistent with the data set in Study 1 described above, the results file for Cohort 317, was used to ensure that the test set being used to analyse the model’s explainability was the same as the studies that were tested on during model development. This stage of the data preprocessing also included a dimensional reduction of the test set to the same variables used in the LVD Al model.1 -dimensional analysis

[0181] The variable analysis was conducted with partial dependence scores. These scores are used to indicate relative weights of certain variables with regards to making a prediction of the target variable. Each input variable in the model is varied based on a set number of samples (in our case 20 per variable) and then the respective probability output is calculated. Depending on how much the probability output changes, different variables will be ranked as more or less important to the model’s ‘decision making’ process.

[0182] Individual Conditional Expectation (ICE) plots are a statistical visualization tool used in the context of machine learning models to gain deeper insights into the relationship between predictors and predictions. Unlike traditional Partial Dependence Plots (PDPs) which are the ‘average’ scores obtained in the ICE plots, ICE plots provide a more detailed view by showing the relationship for each individual data point rather than just an average, making them particularly useful for detecting interactions and variations in the model’s predictions. They also help in understanding how models behave when predictors are outside the training data range and can be used to visually assess additivity in the data generating process, enhancing the interpretability of complex models. For more information regarding these statistical techniques see Goldstein etal., Peaking Inside the Black Box: Visualising Statistical Learning with Plots of Individual Conditional Expectation., incorporated herein by reference.

[0183] Since all 49 input variables are tested, the input variable pipeline is developed to support parallelisation of input variables and data. Each variable does not have to wait for the previous one to complete, thus substantially decreasing the runtime of the pipeline. Furthermore, the dataset is split into two sections to optimise computational resources for each variable and then the results are rejoined at the end of the calculations. Upon completion of the runs for each feature name, all results are collated in a dictionary with the key being the variable name and the value being the output of the partial dependence function provided by the scikit-learn python package.1 - Dimensional Analysis Results

[0184] Figure 11 shows the partial dependence vs Raw variable value for a subset of model parameters with the global mean output probability output marked on each plot.

[0185] An average-value PDF plot maps the average output probabilities associated with a given value of a given variable. Plotting the static partial dependence results reveals a significant amount about variable strength within the model, as stronger variables will tend to have a larger absolute gradient when increasing or decreasing the value of the variable.Depending on how the variable influences the model, manipulation of stronger variables will denote a larger absolute change in the probability output whereas the opposite is true for weaker variables.

[0186] Variables such as “Age_At_Echo”, “LA VolumeJndex” and “LV Ejection Fraction Hierarchy” all have a large absolute gradient as depicted in Figure 11 . Thus, indicating that the probability output of an individual study will change depending on the values provided for these variables.

[0187] The variables exhibiting a very small rate of change, such as ‘LVOT_AV_Vel_Ratio’ (see Figure 11 ) did not appear to have significant influence on the model over the given value range as the probability output did not change as the variable value was altered . This suggests that the probability output of a study is highly independent of the input value of these variables and thus can be deemed ‘weaker’ in the context of model explainability.

[0188] Derivative Variable Plots are derived from the gradients of the partial dependence. These plots are extremely useful as they elucidate which variables have the largest gradients within the cohort and therefore which variables influence the probability output in the greatest capacity. Figure 12 shows the result of Absolute gradient of the partial dependence against sampled variable value for each variable in the model feature set. The top 10 contributing values appear in colour in Figure 12, with the remaining variables depicted as grey coloured lines without an associated label.

[0189] Variables with larger absolute gradients such as ‘Age_At_Echo’ and ‘LA_Volume_lndex’ appear with the largest derivative in Figure 12, consistent with the importance of these values in predicting LV dysfunction. The overall gradient plot also illustrates regions where there may be anomalous reactions to changing the values of certain variables. “AV_Peak_Velocity” and “LVOT_AV_VTI_Ratio” are both marked on the graph, however they seem to behave differently to the other 8 variables. These variables are different as they have a lower impact on the model (align with the average probability output) yet at certain distances from the mean (-0.35, -1 .1 respectively) have a sharp spike upwards. This behaviour is classified as erroneous, due to external disease processes (aortic stenosis), and unlikely to reflect the contribution of these variables impacts on overall model performance.

[0190] In the course of the analysis, it was also observed that certain variables, such as ‘TR Peak Velocity,’ exhibited unusual behaviour. As illustrated in Figure 12 ‘TR Peak Velocity’ is characterized by a central spike amongst a relatively low gradient in the dataset. While this behaviour was intriguing, it was determined to be of relatively lowsignificance to the overall performance of the model. This peculiar behaviour appears to align with an anticipated response threshold of 235 (for ‘TR Peak Velocity’), demonstrating the model’s ability to accurately capture and encapsulate previously undefined patterns within the dataset. Figure 12 also highlights a noticeable change in probability output based on the measured value of Right_Atrial_Volume_lndex_Hierarchy. In terms of defining explainability, the TR Peak Velocity, Right_Atrial_Volume_lndex, Right_Atrial_Pressure_Hierarchy and the Right_Ventricular_Systolic_Pressure are all interdependent. The high importance of the two right atrial measures in model performance may reflect a state of pulmonary hypertension in most instances, with the TR peak velocity only independently contributing in a subset of individuals.

[0191] Figure 13 describes the averaged gradient across the normalised range of values for each variable as shown in Figure 12. As such, this method provides a framework for possible important variables but does not encapsulate the step-by-step changes in variable so further analysis is needed to determine the order of importance for variables.

[0192] From this we can also conclude that the variables most subject to change are variables such as ‘LA_Volume_lndex’ and ‘Age_At_Echo’ whereas ‘Diastolic_BP’ and ‘PI_Peak_Velocity’ have a much smaller effect on the probability output. This table aligns with the conclusions outlined in Study 2, above, as each of the top five variables with the largest average gradients, are important to heart failure guidelines. The model also tends to favour the use of ‘Mitral_E_Point_Velocity’ over ‘LV_Septal_E_Prime_Velocity’ and ‘Mitral_A_Point_Velocity’, which suggests that the inherent correlation between the variables and their respective ratios is being encapsulated by the shared variable ‘Mitral_E_Point_Velocity’. Moreover, these results support the conclusion that the model does not predict other disease states such as Aortic Stenosis as important variables such as ‘AV Peak Velocity’ and ‘AVAVTIcalculated’ are classified as having an extremely low average gradient, and therefore do not have a major impact on the resultant probability output no matter the value input.

[0193] Centred ICE Plots of the most important variables as shown in Figure 14, are used to visualise the deviation of probability output per study, per variable. Yellow plots 1401 indicates the average probability output, red plots 1403 and green plots 1405 represent the 10 studies whose output increased or decreased the most respectively. All other studies are shown in grey. The centring process starts by selecting the minimum value for each of the target variables and then calculating a new curve for each individual study by finding the difference between the original curve and the curve at the minimum value. This processcentres all individual curves, thus emphasising the induced variance of changing a variable’s value.

[0194] As depicted in Figure 14, all variables demonstrate a cone-like behaviour when creating centred ICE plots. The graphs depict the probability output for every study given a certain value and as such describes the overall behaviour of a variable as it changes per study. Strong variables such as “Age_At_Echo” and “LA_Volume_lndex” clearly demonstrate the strong linear increases as the value of the variable increases. Stronger variables will also have high clustering of the study gradients as a change in value will have a significant impact on the probability output.

[0195] Graphs that imitate the behaviour depicted in ‘Diastolic_BP’ however, show an irregular change in probability output with low gradient clustering. This suggests that there is not a gradual change in the probability output given a change in the variable value (suggesting a ‘weaker’ variable), therefore the spikes in probability output changes are due to another threshold variables or the combination of the depicted variable with other variables.

[0196] Upon further analysis of a previously outlined variable ‘TR Peak Velocity’, the massive spike depicted in Figure 12 is also apparent at a similar threshold of 235. As such, with expert advice this variable can be identified as one of the more important variables that may influence the probability output if it is provided in the input data.

[0197] As observed in Figure 14, there are certain studies whereby changing the value of a specific variable leads to a dramatic change in the probability output. To illustrate what populations these studies belong to, it is useful to map them on a normal ICE plot which demonstrates the change in output of an individual study over the variable value range. Figure 15 thus shows Normal ICE plots of the outlier gradients determined by the centred ICE plots of Figure 14.

[0198] From the plots described in Figure 15 strong variables exhibit a distinct behaviour compared to their weaker counterparts. Plots 1501 and Plots 1502 represent the 10 studies whose output respectively increased or decreased the most. Studies with the largest and most clustered gradient changes over the value range are centred around the mean probability output. This implies that studies that are not at either probability output extreme are more susceptible to change due to the influence on an important variable. This is exhibited in the ‘LV Ejection Fraction Hierachy’ plot within Figure 15 where the largest gradient studies vary around the mean probability output (based on ‘LV Ejection Fraction Hiearchy’) are at the extremes. Furthermore, this behaviour suggests that for the studies not influenced by certain‘strong’ variables, there are other variables in that study that dictate the probability output and therefore changing the one depicted variable will not do anything to the resultant probability output.

[0199] Variables that do not follow this pattern exhibit significantly more ‘random’ behaviour. If the plot of variable value range against probability output does not demonstrate any clustering or linear increase / decrease in probability output per study, it can be concluded that the variable is not impactful on the probability output. ‘Diastolic_BP’ illustrates this phenomenon as studies with the greatest increase in probability output are widely spread across the probability output spectrum. Thus, variables that show behaviour such as ‘Diastolic_BP’ in Figure 15 can be deemed as ‘weaker’ when it comes to deriving feature importance.2 - Dimensional analysis

[0200] Further analysis was conducted on a select sub-group of predefined important variables. These variables were selected based on the conclusions from the 1 -D variable analysis as well as clinical guidance on the plots and tables produced. The variables selected for further analysis include the following:LA_Volume_lndex Mitral_E_Point_Velocity GenderfinalTR Peak Velocity Mitral_E_to_A_Ratio LV_Diastolic_Volume_SIMLV_Mitral_E_to_MV_E LV_Mass_2D_ASE_lndex RV_Diastolic_Basal_Prime_Septal_Ratio DiameterLV_Ejection_Fraction_LV_Septal_E_Prime_ Hierarchy AV Peak VelocityVelocityAge_At_Echo

[0201] The two-dimensional analysis pipeline has been strategically developed to bolster data processing and feature evaluation by means of parallelization . This pipeline also facilitates a systematic exploration of all possible combinations of 13 variables, enabling a thorough assessment of their interdependencies. Furthermore, the outcomes of these extensive analyses are recorded and organized within a data dictionary, serving as a structured repository for comprehensive documentation. The data dictionary is then securely stored within an Amazon S3 bucket readily available for further analysis.2 - Dimensional Analysis Results

[0202] Based on the 1 - Dimensional analysis results above, further investigation was conducted into the subset of 13 variables identified as important to the Al model. The variables were selected based on their overall average gradient, as well as their gradient behaviour, theregularity and clustering of their normal ICE plots as well as the behaviour of the centred -variable ICE plots. Variables with larger degrees of clustering and ‘regular’ behaviour were deemed as important and needed further insight into their interaction patterns. The overall goal of the 2-D analysis was to elucidate further the important variables as they would seemingly ‘dominate’ the other variables when it came to demonstrating their predictive ability on the output probabilities.

[0203] Contour Plots (Figures 16A, B and C) outline the distinct relationship between the select important variables. The plots are interpretable by the colour gradient of the plots, the direction of the colour gradient, and the individual values provided for each variable. Each individual plot is coloured by the average probability output given a cross section of two specific variable values. If the colour gradient remains around the purple range without much variation, such as what is observed in Figure 16A (‘LV_Septal_E_Prime_Velocity’ against ‘LV_Diastolic_Volume_SIM’), it can be concluded that no matter what the value for each of these variables, the model’s resulting probability output will not necessarily change significantly. This suggests that variables with this colour pattern are not as significant to the model as other variables. The analysis between these two variables can be extended to all correlated variables too. As outlined in the “Feature Selection” section, above, strongly correlated variables are collapsed into a single feature, which implies that the inferences made about Figure 16A can be extended to its correlated measures such as LV_Diastolic_Dimension_PLAX too.

[0204] On the other hand, variable with stark contrasts in colour gradients, such as shown in Figure 16B, ‘Age_At_Echo’ and ‘Mitral_E_Point_Velocity’, indicate a massive relative change in output probabilities given different values of the variable. Therefore, these two variables can be inferred as important to how the model creates a probability output.

[0205] The directionality of the colour gradient is also vital in understanding feature importance within 2-dimensional analysis. Colour gradients that are either vertical or horizontal (such as Figure 16C, the plot between ‘Mitral_E_Point_Velocity’ and ‘RV_Diastolic_Basal_Diameter’) suggest that one variable dominates the other when it comes to being a predictor of the target probability output. This is concluded due to the high variability in the probability output given the change in one variable whereas close to no relative change due to the other variable. This is consistent with the clinical understanding of heart failure, whereby the right ventricular dimensions are not expected to vary substantially within the expected range of left ventricular filling characteristics.

[0206] Proportional and inversely proportional relationships between variables are also elucidated via the colour gradient such as the relationship between ‘LV Ejection Fraction Hierarchy’ and ‘LV_Mitral_E_to_MV_E_Prime_Septal_Ratio’, and‘Mitral_E_Point_Velocity’ and ‘LA VolumeJndex’. This behaviour suggests that both variables are having an impact on the resultant output probabilities, something in different directions.

[0207] Overall, the preliminary contour plots provide valuable insight into the strength of certain variables within the model. Visually, it is possible to conclude that ‘Age_At_Echo’, ‘LA VolumeJndex’, ‘LV Ejection Fraction Hierarchy’, ‘Mitral_E_Point_Velocity’ are all very important to the model and provide a platform for the argument that they are minimum required variables for providing a sufficient model-based prediction.

[0208] Proportional and inversely proportional relationships between variables are also elucidated via the colour gradient such as the relationship between ‘LV Ejection Fraction Hierarchy’ and ‘LV_Mitral_E_to_MV_E_Prime_Septal_Ratio’, and ‘Mitral_E_Point_Velocity’ and ‘LA VolumeJndex’. This behaviour suggests that both variables are having an impact on the resultant output probabilities, something in different directions.

[0209] Overall, the preliminary contour plots provide valuable insight into the strength of certain variables within the model. Visually, it is possible to conclude that ‘Age_At_Echo’, ‘LA VolumeJndex’, ‘LV Ejection Fraction Hierarchy’, ‘Mitral_E_Point_Velocity’ are all very important to the model and provide a platform for the argument that they are minimum required variables for providing a sufficient model-based prediction.

[0210] Colour co-ordinating the normal ICE plots by variable can provide insights into the interactions between any two given variables. The resulting plots are relevant to a 2-dimensional analysis as they highlight studies that have a high true value of the target variable. Since the colours are derived from the true values defined in the test set, a variable is more ‘important’ to the model’s predictive ability if the colouring scheme shows a clear vertical colour gradient. Weaker target variables will show a more randomly dispersed colour range as high / low true values are not necessarily correlated with a high or low probability output. This plotting method was used in conjunction with all of the other methods outlined in this document to select the ‘T op’ variables for use as the Candidate Minimal Input Lists, discussed below.

[0211] Despite what was seen in the analysis conducted in the Variable gradient analysis, above, ‘LA VolumeJndex’ does not appear to have a large variation in values in the original test dataset. Granted there are a few studies with high values of LA VolumeJndex, majority of the colour distribution is centred around 1 standard deviation from the mean value. Despite this imbalance, there is still a vertical colour gradient within the plots with a bit of intermixing . This indicates that the LA VolumeJndex is still a strong predictor of the probability output as higher values of this variable are more correlated with higher probability outputs.

[0212] ‘LV Ejection Fraction Hierarchy’ acts differently to most of the other important variables. When analysing the coloured ICE plots it is apparent that LVEF has a strongly clustered vertical colour gradient, yet the gradient increases at lower values of the variable. As such it is possible to conclude that lower values of LVEF lead to a stronger predictor of the probability output.

[0213] ‘Diastolic_BP’ is an example of a weaker variable within the model training dataset. There is not a lot of colour dispersion within the graphs indicating that the values of Diastolic_BP are almost independent of the resulting probability output. Furthermore, due to the lack of clustering, it is possible to conclude that Diastolic_BP cannot be used as a predictor variable for the probability output either.Variable Analysis - Conclusions

[0214] From the Variable dimensional analysis discussed above, it can be concluded that the model parameters are not all equal in importance for running the model, there is a clear hierarchy in variable influence on the probability output. Variable behaviour cannot be solely described by partial dependence and ICE plots, they are also explained with clinical guidance.

[0215] The most important variables for the LVD Al model disclosed herein are thus found to include, but are not limited to:Age_At_EchoGenderfinalMitral_E_Point_VelocityLV Ejection Fraction HierarchyLA_Volume_lndex

[0216] Variables that have little impact on model performance include but are not limited to: Diastolic_BPRV_Diastolic_Basal_DiameterIVC_Diameter_ExpirationL V S t ro ke_ V o I u m e_S I MSystolic_BPImplementation

[0217] The methods of training and operating an Al assisted echocardiography system as disclosed herein may be implemented using a computer system 1800, such as the example computer system shown in Figure 17 with which embodiments described herein may be implemented wherein the systems and methods may be implemented as software, such as oneor more application programs executable within the computing device 1700. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part (and the corresponding code modules) performs the described methods, and a second part (and the corresponding code modules) manage a user interface between the first part and the user. The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 1700 from the computer readable medium, and then executed by the computer system 1700. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer system 1700 preferably effects an advantageous apparatus for Al assisted echocardiography.

[0218] In the example of Figure 17, example computer system 1700 and instructions for implementing the disclosed technologies in hardware, software, or a combination of hardware and software, are represented schematically, for example as boxes and circles, at the same level of detail that is commonly used by persons of ordinary skill in the art to which this disclosure pertains for communicating about computer architecture and computer systems implementations.

[0219] The example computing device 1700 can include, but is not limited to, one or more central processing units (CPUs) 1701 comprising one or more processors 1702, a system memory 1703, and a system bus 1704 that couples various system components including the system memory 1703 to the processing unit 1701 . The system bus 1704 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The computing device 1700 also typically includes computer readable media, which can include any available media that can be accessed by computing device 1700 and includes both volatile and non-volatile media and removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by the computing device 1000. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0220] The system memory 1703 includes computer storage media in the form of volatile and / or non-volatile memory such as read only memory (ROM) 1705 and random-access memory (RAM) 1706. A basic input / output system 1707 (BIOS), containing the basic routines that help to transfer information between elements within computing device 1700, such as during start-up, is typically stored in ROM 1705. RAM 1706- typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by processing unit 1701. By way of example, and not limitation, Figure 17 illustrates an operating system 1708, other program modules 1709, and program data 1710.

[0221] The computer readable instructions stored in memory 1703, ROM 1705, RAM 1706 or HDD storage 1711 may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and / or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP / IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. The instructions may implement a web server, web application server or web client. The instructions may be organized as a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or no SQL, an object store, a graph database, a flat file system or other data storage.

[0222] The computing device 1700 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, Figure 17 illustrates a hard disk drive 1711 that reads from or writes to non-removable, non-volatile magnetic media. Other removable / non-removable, volatile / non-volatile computer storage media that can be used with the example computing device include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 1711 is typically connected to the system bus 1704 through a non-removable memory interface such as interface 1712.

[0223] The drives and their associated computer storage media discussed above and illustrated in Figure 17, provide storage of computer readable instructions, data structures, program modules and other data for the computing device 1700. In Figure 17, for example, hard disk drive 1711 is illustrated as storing an operating system 1713, other program modules 1714, and program data 1715. Note that these components can either be the same as or different from operating system 1708, other program modules 1709 and program data 1710. Operating system 1713, other program modules 1714 and program data 1715 are given different numbers hereto illustrate that, at a minimum, they are different copies.

[0224] The computing device also includes one or more input / output (I / O) interfaces 1730 connected to the system bus 1704 including an audio-video interface that couples to output devices including one or more of a video display 1734 and loudspeakers 1735. Input / output interface(s) 1730 also couple(s) to one or more input devices including, for example a mouse 1731 , keyboard 1732 or touch sensitive device 1733 such as for example a smartphone or tablet device. In the embodiments disclosed herein, input interface 1730 may also comprise an echocardiography / ultrasound handpiece and computing device 1700 may comprise or be integrated with an echo / ultrasound workstation.

[0225] Of relevance to the descriptions below, the computing device 1700 may operate in a networked environment using logical connections to one or more remote computers. For simplicity of illustration, the computing device 1700 is shown in Figure 17 to be connected to a network 1720 that is not limited to any particular network or networking protocols, but which may include, for example Ethernet, Bluetooth or IEEE 802.X wireless protocols. The logical connection depicted in Figure 17 is a general network connection 1721 that can be a local area network (LAN), a wide area network (WAN) or other network, for example, the internet. The computing device 1700 is connected to the general network connection 1721 through a network interface or adapter 1722 which is, in turn, connected to the system bus 1704. In a networked environment, program modules depicted relative to the computing device 1700, or portions or peripherals thereof, may be stored in the memory of one or more other computing devices that are communicatively coupled to the computing device 1700 through the general network connection 1721 . It will be appreciated that the network connections shown are example and other means of establishing a communications link between computing devices may be used.Protocol Minimum Inputs

[0226] The smallest set of inputs to the LVD Al model where acceptable performance is maintained is significant in order to maximise the number of studies that will be accepted by the LVD Al model in a commercial application setting. The general methodology used is to simulatethe effect of missing data by removing the real measurement data in specific columns and analysing the resulting change in final output probabilities.

[0227] Once a list of columns is selected for testing, the raw data from NEDA v2.0 (Cohort 301 ) will be modified to remove all other variables. This reduced dataset will then be run through the two stages of the LVD-AI model: MDN imputation of missing data and the Al classification model as discussed above to produce a probability output of Heart Failure for each study in the test set (Cohort 317), which can then be compared to the original probability output outputs studied in the Studies discussed above.

[0228] Given a candidate set of minimally required inputs, we also calculate what percentage of echoes meet these requirements, known as the fill rate. All subsequent analysis needs to be understood in the context of the trade-off between fill rate and performance. Acceptable levels for both the performance and fill rate were not prescribed in advance, instead they were considered conditional on the final results.

[0229] The first step was to establish the validity of our column dropping and data generation methodology (baseline testing), then generate a series of datasets corresponding to candidate lists of variables, the choice of which was be based on previous analysis into the LVD Al model and feedback from clinicians. The resulting datasets were then analysed further.Baseline testing

[0230] Baseline testing was performed including:• Environment testing: Ensure the reproducibility of results on the Sydney server and AWS environments;• Seed testing: Test the effect of changing the imputation sampling method;• Column dropping method: Test the effect of column dropping methodology: o Keep only base column ; and o Keep correlated columns.Dataset Generation

[0231] A number of column lists were studied. Due to the highly manual and time-intensive process involved in mortality analysis, this technique was only applied to candidate lists already determined to be viable through automated analysis methods, starting with a couple of candidate lists based on qualitative analysis to test the column-dropping methodology and understand general trends in the change in probability output.

[0232] We then ran systematic column dropping tests, where the input variables are ordered based on feature importance and dropped one-by-one. Modifications to this list were needed based on clinical context and fill rate considerations. Results at this stage are highly dependent on the order in which columns are dropped.

[0233] Once a subset of highly important variables was decided upon, every possible combination of these variables was tested to ensure that every potential candidate list was studied. Based on these results, a set of viable final candidate lists was studied in-depth.

[0234] Relevant lists are articulated below. Note that Age_At_Echo and Genderfinal are never dropped from any list given that they have 100% fill rates in NEDA.• Initial candidate lists: o Data Science: as given in Variable Analysis - Conclusions. o Clinical recommendation.• Systematic column dropping: o Drop columns one-by-one based on a quantitative measure of feature importance.■ List provided in Candidate Minimal Input Lists. o Reorder list based on clinical advice.• Power set of top variables: o Take top 8 variables based on clinical advice. o Check every possible combination of top variables.• Clinically recommended candidate lists: o Consider fill rates and clinical context (e.g. Point Of Care echoes). o Several candidate lists chosen for deeper analysis.Inclusion / Exclusion criteria

[0235] Starting with the NEDA V2.0 data including Echo + Calculated values for patients aged 18 years and older, in the period 29 / 5 / 1985 - 26 / 6 / 2019, there are 1 ,077,145 echo-studies from 631 ,824 patients (Cohort 301). This was then filtered to a Model Testing Set (317) as discussed in Study 1 , discussed above, to include 84,946 patients, with one relevant echo -study from each patient.

[0236] Each candidate list was analysed with respect to how well it performs as a Heart Failure diagnostic tool. The new output probabilities were analysed in their raw from, and then stratified based on the quintile boundaries used in Study 2 above to facilitate comparison of the results.• Analyse movement of raw output probabilities and quintile distributions.Individual study analysis. o Look at the studies that change in output probabilities the most and find patterns.• Analyse haemodynamic distribution of real values across the quintile distributions.• Analyse relationship with guideline-defined Heart Failure.• Fill rate analysis. o Stratify analysis based on whether studies meet the minimum requirements.

[0237] The NEDA population was studied using the same methodologies described in Study 2, above in the same setting and study design, along with the study flow chart (Figure 3) and the study data including linkage with mortality. Following is the specific methodology of this minimum inputs sub-study.

[0238] No formal calculations of study power were performed given our ability to analyse >150,000 case-fatalities during 2.5 million person-years follow-up. Discrete variables were summarized by frequencies and percentages (with 95% confidence intervals [Cl] where appropriate). Continuous variables were summarized by standard measures of central tendency and dispersion (Mean, Standard Deviation; Median, IQR).

[0239] Long-term outcomes for each minimum inputs LVD Al Model probability output group were displayed using Kaplan-Meier survival curves. Using 5-year actuarial data (a minimum potential follow-up of 5 years for all patients, with censoring of data at 1825 days, and deaths recorded only during the initial 1825 days of follow-up), a series of Cox-Proportional Hazard Models (entry model) were then used to derive hazard ratios (HR) for all-cause and cardiovascular-related mortality (with censored events) for each LVD Al Model group. The Cox Models included adjustment for age and sex to derive adjusted HR with 95% CL All analyses were performed with SPSS v29.0 and statistical significance accepted at a 2-sided p-value of <0.05.Candidate Minimal Input Lists

[0240] An initial list of variables proposed in the order of perceived relevance to clinical heart failure analysis includes:LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, Mitral_E_Point_Velocity, LV_Mass_2D_ASE_lndex, TR Peak Velocity, LV_Mitral_E_to_MV_E_Prime_Septal_Ratio, Body_Surface_Area, LVOT Peak Velocity, LVOT AV VTI Ratio, Mitral_E_to_A_Ratio, Right_Atrial_Volume_lndex_Hierarchy, Right_Atrial_Pressure_HierarchyPulmonary_Vein_S_D_Ratio, Ascending_Aorta_Diameter, RVOT Peak Velocity, Mitral_A_Point_Velocity, LV_Diastolic_Volume_SIM, MV Peak Velocity, RVOT Peak Gradient, LA_Length_4C, MV Deceleration Time, PV gPeak Velocity, RVSP_Combined, Aortic_Arch_Diameter, IVS_Diastolic_Thickness, AVAVTIcalculated, AV Peak Velocity, RVOT_Velocity_Time_lntegral, LVOT_Stroke_Volume_lndex, Pulmonary_Vein_Systolic_Velocity, LV_Septal_E_Prime_Velocity, Pulmonary_Vein_Diastolic_Velocity, LVOT_AV_Vel_Ratio, Aortic_Root_Diameter, LV_Stroke_Volume_2D_Teich, Heart_Rate, MV_Velocity_Time_lntegral, RV_Diastolic_Basal_Diameter, LVOT Diameter, Pulmonary Vein A Velocity, Mitral_A_Duration, IVC_Diameter_Expiration, Systolic_BP, LV_Stroke_Volume_SIM, PI_Peak_Velocity, Diastolic_BP.

[0241] For the basis of further clinical analysis, the following variable lists are proposed:Top 13:LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, Mitral_E_Point_Velocity, LV_Mass_2D_ASE_lndex, TR Peak Velocity, LV_Mitral_E_to_MV_E_Prime_Septal_Ratio, Body_Surface_Area, LVOT Peak Velocity, LVOT AV VTI Ratio, Mitral_E_to_A_Ratio, Right_Atrial_Volume_lndex_Hierarchy, Right_Atrial_Pressure_Hierarchy.Top 8:LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, Mitral_E_Point_Velocity, LV_Mass_2D_ASE_lndex, TR Peak Velocity, LV_Mitral_E_to_MV_E_Prime_Septal_Ratio, Body_Surface_Area.Top 3:Body_Surface_Area, LV Ejection Fraction Hierarchy, Mitral_E_Point_Velocity.

[0242] Depending on the variables available in the echo-study report data, a set of clinical candidate variable lists are derived, including:Top 5 with LV Mass:LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, Mitral_E_Point_Velocity, LV_Mass_2D_ASE_lndexTop 5 with TR Peak Velocity:LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, Mitral_E_Point_Velocity, TR Peak VelocityTop 4 with no Doppler measurements (intended for Point Of Care Ultrasound - POCUS): LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, L V_Mass_2 D_AS E J ndexAn additional list was suggested by the Data Science team as a bridge between these lists and the Top 3 list.Top 4 with Mitral E Point Velocity:LA_Volume_lndex, LV_Systolic_Volume_SIM, LV Ejection Fraction Hierarchy, Mitral_E_Point_VelocityData Science Conclusions

[0243] Baseline tests confirmed that the computational environment used for testing did not affect the results, and that using correlated columns to impute a base column produced acceptable results. The randomness due to seeding was measured and the use of a fixed seed was implemented to minimize this variability.

[0244] Around 300 candidate lists of minimum required variables were tested in several phases to provide an exhaustive search of plausible candidates.

[0245] Generally output probabilities showed regression to the mean behaviour as the number of provided variables decreased. There was an accompanying general upward movement between quintiles.

[0246] Individual studies which showed the largest change in probability output were analysed in depth and several reasons were found for their high variability, including very low fill rates, extreme values in their real data and / or conflicting markers of disease.

[0247] The haemodynamics of the population stratified by quintile were analysed. It was shown that strong gradients remained when many unimportant columns were dropped. The strength of the correlation was reduced when the source variable was dropped for columns like LA_Volume_lndex and Mitral_E_Point_Velocity, whereas other top columns maintained their correlations by relying on the remaining information given to the algorithm.

[0248] The quintile movement of the population was stratified by guidelines-defined Heart Failure and Diastolic Dysfunction. It was shown that most of the upward quintile movement came from patients with preserved EF and normal or indeterminate filling pressures. Patients with high-risk guidelines classifications like HFrEF, HFmrEF with increased filling pressure, and Impaired EF with Grade 2 / 3 Diastolic Dysfunction became even more strongly weighed toward the upper quintiles and did not move down into lower quintiles as columns were dropped.

[0249] The population was stratified by whether they met the minimum requirements or not, and it was shown that those who met the requirements largely maintained the strength of theirhaemodynamic correlations across the quintiles more than those who did not meet the requirements, owing to the effects of missing information on this second group.Clinical Team Conclusions

[0250] The probability output of the Al model provides a meaningful output that predicts the echocardiographic characteristics typically found in both systolic and (predominantly) diastolic heart failure. These characteristics remained when the Al was exposed to a restricted set of echocardiographic variables that were clinically most likely to be abnormal in the setting of LV dysfunction.

[0251] The model performance was less optimal as the number of echo variables was progressively decreased, although surprisingly good performance was maintained for the higher probability output groups.

[0252] The performance of the model was inadequate where only 3 variables were provided, although the 2-D only, 4-variable model (with a view to a POCUS application) performed surprisingly well considering the minimum inputs provided. This simple model performed only slightly worse than the Top 5 (with LVMi) and Top 5 (with TR velocity) variable list. Finally, the 4- variable model (with a view to POCUS application) with the addition of the mitral E wave velocity performed similarly to the Top 5 (with LVMI).

[0253] In standard clinical echocardiography, the tricuspid regurgitation (TR) velocity is only measurable 50% of the time, and there are multiple clinical reasons why the septal e’ velocity may not be measured or recorded (e.g., in the setting of prior cardiac surgery, a pacemaker, or a left bundle branch block). In order to maximise the application of the Al to standard clinical echocardiography, it is preferable to exclude these from the minimum required variables list.

[0254] Therefore, several groups of minimal variables were identified that when provided, maintain the performance of the model, for standard clinical echocardiography. These viable minimum required variable groups are:• Top 5 variables (with LVMi): Age, sex, body weight, body height, LA volume index, LV systolic dimension (and by calculation, the LV systolic volume), LV ejection fraction (by any method), mitral inflow E wave, and LV mass index (thereby requiring the LV diastolic dimension, LV septal thickness, and the LV posterior wall thickness).• Top 4 variables (with LVMi): Age, sex, body weight, body height, LA volume index, LV systolic dimension (and by calculation, the LV systolic volume), LV ejection fraction (by any method), LV mass index (thereby requiring the LV diastolic dimension, LV septal thickness, and the LV posterior wall thickness)• POCUS Set A: Age, sex, body weight, body height, LV ejection fraction (by any method), LV mass index (thereby requiring the LV diastolic dimension, LV septal thickness, and the LV posterior wall thickness)• POCUS Set B: Age, sex, body weight, body height, LV ejection fraction (by any method), LV mass index (thereby requiring the LV diastolic dimension, LV septal thickness, and the LV posterior wall thickness), LA volume index• POCUS Set C: Age, sex, body weight, body height, LV ejection fraction (by any method), LV mass index (thereby requiring the LV diastolic dimension, LV septal thickness, and the LV posterior wall thickness), Mitral E Wave velocityInterpretationBus

[0255] In the context of this document, the term “bus” and its derivatives, while being described in a preferred embodiment as being a communication bus subsystem for interconnecting various devices including by way of parallel connectivity such as Industry Standard Architecture (ISA), conventional Peripheral Component Interconnect (PCI) and the like or serial connectivity such as PCI Express (PCIe), Serial Advanced Technology Attachment (Serial ATA) and the like, should be construed broadly herein as any system for communicating data.In Accordance With

[0256] As described herein, ‘in accordance with’ may also mean ‘as a function of’ and is not necessarily limited to the integers specified in relation thereto.Composite Items

[0257] As described herein, ‘a computer implemented method’ should not necessarily be inferred as being performed by a single computing device such that the steps of the method may be performed by more than one cooperating computing devices.

[0258] Similarly objects as used herein such as ‘web server’, ‘server’, ‘client computing device’, ‘computer readable medium’ and the like should not necessarily be construed as being a single object, and may be implemented as a two or more objects in cooperation, such as, for example, a web server being construed as two or more web servers in a server farm cooperating to achieve a desired goal or a computer readable medium being distributed in a composite manner, such as program code being provided on a compact disk activatable by a license key downloadable from a computer network.Database

[0259] In the context of this document, the term “database” and its derivatives may be used to describe a single database, a set of databases, a system of databases or the like. The system of databases may comprise a set of databases wherein the set of databases may be stored on a single implementation or span across multiple implementations. The term “database” is also not limited to refer to a certain database format rather may refer to any database format. For example, database formats may include MySQL, MySQLi , XML or the like.Wireless

[0260] The invention may be embodied using devices conforming to other network standards and for other applications, including, for example other WLAN standards and other wireless standards. Applications that can be accommodated include IEEE 802.11 wireless LANs and links, and wireless Ethernet.

[0261] In the context of this document, the term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a non-solid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not. In the context of this document, the term “wired” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a solid medium. The term does not imply that the associated devices are coupled by electrically conductive wires.Processes

[0262] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analysing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.Processor

[0263] In a similar manner, the term “processor” may refer to any device or portion of a device that processes electronic data, e.g., from registers and / or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and / or memory. A “computer”or a “computing device” or a “computing machine” or a “computing platform” may include one or more processors.

[0264] The methodologies described herein are, in one embodiment, performable by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included. Thus, one example is a typical processing system that includes one or more processors. The processing system further may include a memory subsystem including main RAM and / or a static RAM, and / or ROM.Computer-Readable Medium

[0265] Furthermore, a computer-readable carrier medium may form, or be included in a computer program product. A computer program product can be stored on a computer usable carrier medium, the computer program product comprising a computer readable program means for causing a processor to perform a method as described herein.Networked or Multiple Processors

[0266] In alternative embodiments, the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

[0267] Note that while some diagram(s) only show(s) a single processor and a single memory that carries the computer-readable code, those in the art will understand that many of the components described above are included, but not explicitly shown or described in order not to obscure the inventive aspect. For example, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.Additional Embodiments

[0268] Thus, one embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program thatare for execution on one or more processors. Thus, as will be appreciated by those skilled in the art, embodiments of the present invention may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium. The computer-readable carrier medium carries computer readable code including a set of instructions that when executed on one or more processors cause a processor or processors to implement a method. Accordingly, aspects of the present invention may take the form of a method, an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.Implementation

[0269] It will be understood that the steps of methods discussed are performed in one embodiment by an appropriate processor (or processors) of a processing (i.e. computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the invention is not limited to any particular implementation or programming technique and that the invention may be implemented using any appropriate techniques for implementing the functionality described herein. The invention is not limited to any particular programming language or operating system.Means For Carrying out a Method or Function

[0270] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor or a processor device, computer system, or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.Connected

[0271] Similarly, it is to be noticed that the term connected, when used in the claims, should not be interpreted as being limitative to direct connections only. Thus, the scope of the expression a device A connected to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. “Connected” may mean that two or more elements are either in direct physical or electricalcontact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.Embodiments

[0272] Reference throughout this specification to “one embodiment”, “an embodiment”, “one arrangement” or “an arrangement” means that a particular feature, structure or characteristic described in connection with the embodiment / arrangement is included in at least one embodiment / arrangement of the present invention. Thus, appearances of the phrases “in one embodiment / arrangement” or “in an embodiment / arrangement” in various places throughout this specification are not necessarily all referring to the same embodiment / arrangement, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments / arrangements.

[0273] Similarly, it should be appreciated that in the above description of example embodiments / arrangements of the invention, various features of the invention are sometimes grouped together in a single embodiment / arrangement, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment / arrangement. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment / arrangement of this invention.

[0274] Furthermore, while some embodiments / arrangements described herein include some but not other features included in other embodiments / arrangements, combinations of features of different embodiments / arrangements are meant to be within the scope of the invention, and form different embodiments / arrangements, as would be understood by those in the art. For example, in the following claims, any of the claimed embodiments / arrangements can be used in any combination.Specific Details

[0275] In the description provided herein, numerous specific details are set forth . However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail in order not to obscure an understanding of this description.Terminology

[0276] In describing the preferred embodiment of the invention illustrated in the drawings, specific terminology will be resorted to for the sake of clarity. However, the invention is not intended to be limited to the specific terms so selected, and it is to be understood that each specific term includes all technical equivalents which operate in a similar manner to accomplish a similar technical purpose. Terms such as “forward”, “rearward”, “radially”, “peripherally”, “upwardly”, “downwardly”, and the like are used as words of convenience to provide reference points and are not to be construed as limiting terms.Different Instances of Objects

[0277] As used herein, unless otherwise specified the use of the ordinal adjectives “first”, “second”, “third”, etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.Comprising and Including

[0278] In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” are used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.

[0279] Any one of the terms: “including” or “which includes” or “that includes” as used herein is also an open term that also means “including at least” the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.Scope of Invention

[0280] Thus, while there has been described what are believed to be the preferred arrangements of the invention, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications as fall within the scope of the invention . Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present invention.

[0281] Although the invention has been described with reference to specific examples, it will be appreciated by those skilled in the art that the invention may be embodied in many other forms.Industrial Applicability

[0282] It is apparent from the above, that the arrangements described are applicable to the imaging device industries, specifically for methods and systems for distributing digital results directly on or via imaging devices.

[0283] It will be appreciated that the methods and systems described / illustrated above at least substantially provide systems and methods for Al-assisted echocardiography.

[0284] The Al-assisted echocardiography systems and methods described herein, and / or shown in the drawings, are presented by way of example only and are not limiting as to the scope of the invention. Unless otherwise specifically stated, individual aspects and components of the systems and methods may be modified, or may have been substituted therefore known equivalents, or as yet unknown substitutes such as may be developed in the future, or such as may be found to be acceptable substitutes in the future. The systems and methods may also be modified for a variety of applications while remaining within the scope and spirit of the claimed invention, since the range of potential applications is great, and since it is intended that the present systems and methods be adaptable to many such variations.

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:1 . A method for processing a sparsely populated data source comprising:(a) retrieving data from a sparsely populated data source to form a base dataset comprising a plurality of patient echocardiography reports, the data source comprising a plurality of patient records, each patient record not requiring a full set of populated data fields corresponding to a medical measurement;(b) dividing the base dataset into two portions:(i). a first portion comprising a training dataset being a defined percentage,X%, of the base dataset; and(ii). a second portion comprising a holdout dataset being a defined percentage (100% - X%) of the base dataset;(c) analysing the training data set to jointly model variable relationships using a non-linear function approximation algorithm applied iteratively to the records of the training dataset to obtain a trained imputation model and measurement prediction protocols for populating unpopulated fields in the training data set;(d) imputing the predicted measurement values in the records of the holdout dataset;(e) applying a primary filter to the imputed holdout dataset to remove patient echocardiography reports comprising an absence of a first indication of the presence of, absence of, or severity of a predetermined disease state;(f) further dividing the imputed holdout dataset into two portions:(i). a first portion comprising an interim disease training dataset being a defined percentage, Y%, of the test / validation dataset; and(ii). a second portion comprising an interim disease test dataset being a defined percentage (100% - Y%) of the base dataset;(g) applying a secondary filter to each portion above to filter the interim datasets to remove patient echocardiography reports comprising a second indication of prior treatment for a predetermined disease state to obtain a disease training dataset and a disease test dataset;(h) applying a tertiary filter to each of the disease training dataset and a disease test dataset to filter the datasets to select a valid echocardiography report for each patient, wherein validity refers to the presence of the first indicator and the absence of the second indicator;(i) analysing the disease training dataset on the basis of predefined disease conditions in known patient records of the disease training dataset to form a classification model adapted to associate patient data to a probability output of a disease condition in patient records of the disease training data set;(j) validating the model comprising analysing the disease test dataset using the classification model, wherein the records of the disease test dataset comprise data associated with patient data, and determining a validation error comprising a probability of correctly predicting a patient associated with the disease condition in the records of the disease test dataset; and(k) repeating Steps (i) to (j) to minimise the validation error and computing a probability output of a probable disease state for each patient record in the disease test dataset.

2. The method as claimed in Claim 1 , further comprising recalibrating the probability output to correspond to the true probability of the disease state.

3. The method as claimed in either Claim 1 or Claim 2, further comprising defining a series of threshold values to divide the probability output or probability into bins that map to classes of disease severity e.g. low risk, medium risk, high risk, thereby to generate a predicted class that is indicative of degrees of LV Dysfunction that may be associated with a further disease state.

4. The method as claimed in any one of the preceding claims, wherein analysing the training data set is performed using a using a machine learning system.

5. The method as claimed in Claim 1 comprising: a first stage comprising an imputation model used to fill in missing values in the echocardiographic data; and a second stage comprising a classification model used to predict the disease state.

6. The method as claimed in Claim 5, wherein the imputation model comprises a Mixture Density Network.

7. The method as claimed in either Claim 5 or Claim 6, wherein the classification model comprises an automated machine learning algorithm configured to: train a large number of candidate models; and build an ensemble classifier by choosing a weighted subset of the candidate models to optimize performance on a given classification metric.

8. The method as claimed in Claim 7, wherein the classification model further comprises generating an F1 -score measuring a harmonic mean of precision and recall.

9. The method as claimed in Claim 8, wherein the candidate models are selected from a list of standard machine techniques including:Categorical Boost Classifier;Light Gradient Boosting Model;Extra Trees Classifier;Random Forest Classifier;K-Nearest Neighbours Classifier; andSupport Vector Classifier.

10. The method as claimed in Claim 9, wherein the neural network pipelines comprise one or more scaling techniques selected from the group including:Normalization;Minimum Maximum Scaling;Power Transformation;Quantile Transformation;Robust Scaling; andStandard Scaling.11 . The method as claimed in Claim 9, wherein the neural network pipelines comprise scaling techniques selected from the group including:Fast Independent Component Analysis;Kernel Principal Component Analysis;Random Kitchen Sinks;Nystroem Kernel Approximation;Polynomial Features;Truncated Singular Value Decomposition;Extra Trees Preprocessor Classification;Feature Agglomeration;Random Trees Embedding;Select Percentile Classification; andLib Linear SVC Preprocessing.

12. The method as claimed in any one of the preceding claims, wherein the disease state is mitral regurgitation.

13. The method as claimed in any one of the preceding claims, wherein the further disease state is clinical heart failure.

14. A method for generating a training set for training a model to predict mitral regurgitation from echocardiograph data, comprising the steps of: retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports; analysing the echocardiograph data to determine unpopulated data fields;populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model; filtering the patient records to remove patient records with no available diagnosis of mitral regurgitation severity; filtering the patient records to remove patient records from patients with more than one patient record and to select a valid echocardiography report for each patient, wherein validity refers to the presence of a first indicator and the absence of a second indicator; and generating a training set for training a machine learning or artificial intelligence system, the training set based on predefined disease conditions in the patient records.

15. A method as claimed in Claim 14, wherein the first indicator comprises the presence of, confirmed absence of, or severity of mitral regurgitation.

16. A method as claimed in Claim 14, wherein the second indicator comprises the presence of prior mitral valve replacement or repair.

17. A method of predicting heart failure from echocardiograph data, comprising the steps of: retrieving echocardiograph measurement data from a plurality of patient records comprising echocardiography reports; analysing the echocardiograph data to determine unpopulated data fields; populating the unpopulated data fields with imputed echocardiograph data determined by a machine learning model; calculating a probability output from a trained model; analysing echocardiograph measurement data of individual patient records from the echocardiograph data to determine a prediction of the presence of a disease state in the patient on the basis of the calculated probability output; and associating the presence of the disease state to a prediction of heart failure in the patient.

18. A method as claimed in Claim 17, wherein the disease state is left ventricular dysfunction.

19. A method as claimed in Claim 17, wherein a predetermined threshold can be applied to determine the clinical disease state and or likelihood of developing the clinical disease state of Heart Failure.

20. A method as claimed in Claim 18, wherein a predetermined threshold can be applied to determine the clinical disease state and or likelihood of developing the clinical disease state of left ventricular dysfunction.

Images