Deep learning-enabled electrocardiogram prediction of high coronary artery calcium score (CAC) based on computed tomography

Abstract

Receiving voltage-time data of a plurality of leads of an electrocardiograph of a subject, generating a feature vector from the voltage-time data, providing the feature vector to a pre-trained learning system, and receiving, from the pre-trained learning system, a display of the level of coronary artery calcium in the subject. A method, system, and computer program product for detecting and evaluating coronary artery calcium (CAC) (e.g., CAC scoring) are provided herein.

Description

Technical Field

[0001] Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63 / 342,275, filed May 16, 2022, which is hereby incorporated by reference in its entirety.

[0002] Embodiments of the present disclosure relate to methods for the detection and evaluation of coronary artery calcium and to the treatment of a subject identified with respect to cardiovascular risk. Computed tomography-based coronary artery calcium (CAC) scoring is recommended in adults with unknown cardiovascular risk to inform prevention strategies and statin prescribing. CAC has significant limitations as it exposes individuals to radiation, is costly, not readily available everywhere, and requires scoring by a specialized radiologist. Accordingly, there has been a long-standing unmet need for improved methods for detecting and evaluating CAC.

Summary of the Invention

[0003] Embodiments of the present disclosure provide methods and computer program products for the detection of coronary artery calcium and the prediction of CAC scores from a 12-lead electrocardiogram (ECG-AI).

[0004] In some aspects of the invention, disclosed herein is a method comprising receiving voltage-time data of a subject, wherein the voltage-time data includes voltage data of a plurality of leads of an electrocardiograph, generating a feature vector from the voltage-time data, providing the feature vector to a pre-trained learning system, and receiving, from the pre-trained learning system, a display of the level of CAC in the subject.

[0005] Aspects of the invention disclosed herein include an electrocardiograph having a plurality of leads and a computing node having a computer-readable storage medium embodied with program instructions, the program instructions being executable by a processor of the computing node, whereby the processor receives voltage-time data of a subject from an echocardiograph, the voltage-time data including voltage data of a plurality of leads, generates a feature vector from the voltage-time data, provides the feature vector to a pre-trained learning system, and receives a display of the level of CAC in the subject from the pre-trained learning system, and a system including the computing node.

[0006] In certain aspects of the invention, there is disclosed a computer program product for the evaluation of CAC, having a computer-readable storage medium embodied with program instructions, the program instructions being executable by a processor, whereby the processor receives voltage-time data of a subject from an echocardiograph, the voltage-time data including voltage data of a plurality of leads, generates a feature vector from the voltage-time data, provides the feature vector to a pre-trained learning system, and receives a display of the level of CAC in the subject from the pre-trained learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

DETAILED DESCRIPTION OF THE INVENTION

[0008] Approximately half of all deaths due to cardiovascular disease are caused by coronary artery disease (CAD). Patients who have experienced a non - lethal acute myocardial infarction or sudden death may not have had previous symptoms. Therefore, the identification of individuals, including asymptomatic individuals at higher risk of experiencing future cardiovascular events, is important for the implementation of preventive strategies. Characterization of coronary artery calcification has been shown to be equivalent to the total coronary atherosclerotic burden and the risk of cardiovascular events. However, the acquisition and analysis of CAC data are expensive, require highly specialized equipment and trained technicians, are not easily accessible, and are performed by computed tomography, which exposes individuals to radiation. Therefore, the invention provided herein is at least partially based on a deep learning (DL) algorithm designed and developed to predict CAC scores from a 12 - lead electrocardiogram (ECG - AI).

[0009] This specification discloses the development of an artificial intelligence (AI)-based tool for detecting coronary artery calcium (CAC) from a standard 12-lead electrocardiogram (ECG). To test an AI-driven model for early detection of CAC, a novel AI electrocardiogram (ECG) network was developed.

[0010] Convolutional neural networks provide a comprehensive approach for analyzing and interpreting the vast amount of data generated by a single ECG. The algorithm was developed using 12-lead ECG data from 43,210 consecutive patients who received clinically indicated CAC and ECGs within one year from 1997 to 2020. In some embodiments, the present invention may be implemented using single-lead and 6-lead ECG subsets. Additionally, smartphone-compatible electrodes enable point-of-care diagnosis with single-lead and 6-lead options, so that a compatible network can be developed and tested.

[0011] All models used voltage-time information from 12-lead ECGs as input. The modeling techniques investigated included convolutional neural networks with different structures, such as using all 12 leads as a single input.

[0012] Accordingly, in some aspects of the present invention, this specification discloses a method including receiving voltage-time data of a subject, wherein the voltage-time data includes voltage data of a plurality of leads of an electrocardiograph; generating a feature vector from the voltage-time data, wherein the feature vector includes a time series of values indicating amplitudes of the voltage-time data regarding the plurality of leads; providing the feature vector to a pre-trained learning system; and receiving, from the pre-trained learning system, a display of the level of CAC in the subject. Generating the feature vector may include generating a spectrogram based on the voltage data of the plurality of leads. In some embodiments, generating the feature vector includes grouping the voltage data of the plurality of leads into a plurality of subsets.

[0013] In some embodiments, such a method further includes receiving demographic information of a subject, and generating the feature vector includes adding the demographic information to the feature vector. In some such embodiments, the method further includes receiving genomic information of the subject. Generating the feature vector may include adding the genomic information to the feature vector. Without being bound by any particular methodology or theory, the genomic data may be from a patient with an increased risk of cardiovascular disease, for example, from a biological sample derived from a family history or genetic marker and / or protein marker. In some such embodiments, the learning system comprises a convolutional neural network. Such a convolutional neural network may include at least one residual connection.

[0014] In some embodiments, voltage-time data of a subject is received from an electrocardiograph. In a further embodiment, voltage-time data of the subject is received from an electronic medical record.

[0015] In some embodiments, the method further includes providing the display to an electronic health record system for storage in a health record associated with the subject. In some embodiments, the method further includes providing the display to a computing node for display to a user.

[0016] In some embodiments of the methods disclosed herein, the feature vector includes a matrix having a plurality of rows and a plurality of columns, the plurality of rows corresponding to a time dimension, and the plurality of columns corresponding to a spatial dimension. In some such embodiments, each of the plurality of rows corresponds to one of a plurality of leads, and each of the plurality of columns corresponds to a timestamp. In some embodiments, the time dimension has a resolution of 500 Hz. The convolutional neural network disclosed herein may comprise at least nine convolutional blocks and two fully connected blocks.

[0017] In some embodiments, the time period of the voltage-time data describes a plurality of cardiac cycles of the subject.

[0018] In some embodiments, the method further includes presenting a display of the level of CAC in the subject to a healthcare provider associated with the subject.

[0019] In some embodiments, the voltage-time data describes a single-lead ECG of the subject. In some embodiments, the single-lead ECG corresponds to one of the leads of a 12-lead ECG. In some embodiments, one of the leads of the 12-lead ECG to which the single-lead corresponds is Lead 1, Lead 2, Lead 3, AvF, AvL, AvR, or V1-V6.

[0020] In some embodiments, the voltage-time data is limited to describing a single cardiac cycle of the voltage-time data from the subject's ECG.

[0021] In some embodiments, the voltage-time data is limited to describing a single average cardiac cycle of the voltage-time data from the subject's ECG. In some embodiments, the voltage-time data is limited to describing a single central cardiac cycle of the voltage-time data from the subject's ECG. In some embodiments, the voltage-time data describes a single cardiac cycle and the voltage-time data is multi-dimensional and extracted from a vector cardiogram. In some embodiments, the voltage-time data describes normal sinus rhythm in the subject. In some embodiments, the feature vector is based on the characteristics of the normal sinus rhythm of the subject.

[0022] In some embodiments, the voltage-time data extends over a time interval of 30 seconds or less. In some embodiments, the voltage-time data extends over a time interval of 10 minutes or less.

[0023] In some embodiments, the subject is a mammal and the mammal is a human.

[0024] In some embodiments, the voltage-time data of the subject is based on fewer than 12 leads.

[0025] In some embodiments, the display of the level of CAC in the subject indicates at least one possible treatment plan. In some aspects, at least one possible monitoring or treatment plan includes administering treatment. In some embodiments, at least one possible treatment plan includes collecting a second series of voltage-time data.

[0026] In some embodiments, the method further includes obtaining data that describes a non-ECG profile for a mammal, generating one or more second neural network inputs that represent the non-ECG profile for the subject, and processing the first neural network input with the one or more second neural network inputs in a neural network to generate a feature vector.

[0027] In some embodiments, the display of the level of CAC in the subject indicates at least a threshold likelihood that the subject will experience CAC. In some embodiments, the method further includes determining a treatment to reduce the level of CAC in the subject in response to the display of at least the threshold likelihood that the subject will experience CAC.

[0028] In some embodiments, the voltage-time data from the subject is recorded over a first time interval, and the method further includes obtaining a second neural network input, where the second neural network input represents second voltage-time data of the subject recorded over a second time interval, and the first time interval and the second time interval are separated by a third time interval, and processing the first neural network input with the second neural network input in a neural network to generate a feature vector for the subject. In some embodiments, the third time interval is at least 1 minute, 1 hour, 1 day, 1 week, or 1 month. In some embodiments, the neural network further processes a third neural network input indicating the length of the third time interval between the first time interval and the second time interval when the first voltage-time data and the second voltage-time data were respectively recorded, along with the first neural network input and the second neural network input.

[0029] In some embodiments, a pre-trained learning system is trained by receiving a training set of voltage-time data from a plurality of CAC patients. In some aspects, the training set of voltage-time data is from one of a retrospective cohort subset or a prospective cohort subset.

[0030] In some embodiments, the voltage-time data includes analog data characterizing an ECG signal. In some embodiments, the voltage-time data includes numerical values specifying the amplitude of the ECG signal. In some embodiments, the voltage-time data includes image data characterizing the ECG signal. In some embodiments, the image data characterizing the ECG signal is in a pixel format including TIFF, PNG, or PDF file types.

[0031] Referring now to FIG. 1, a system for detecting or otherwise predicting the level of CAC according to an embodiment of the present disclosure is shown. As outlined above, in various embodiments, patient information including electrocardiogram (ECG) data is provided to a learning system to determine the level of CAC. Accordingly, aspects of the invention disclosed herein include an electrocardiograph having a plurality of leads and a computing node comprising a computer-readable storage medium embodied with program instructions, the program instructions being executable by a processor of the computing node, whereby the processor receives voltage-time data of a subject from an echocardiograph, the voltage-time data including voltage data of a plurality of leads, generates a feature vector from the voltage-time data, provides the feature vector to a pre-trained learning system, and receives a display of the level of CAC in the subject from the pre-trained learning system. The system also includes a computing node that executes a method including generating a spectrogram based on the voltage data of the plurality of leads. In some embodiments, generating the feature vector includes grouping the voltage data of the plurality of leads into a plurality of subsets.

[0032] In some embodiments, such a system further includes receiving demographic information of the subject, and generating the feature vector includes adding the demographic information to the feature vector. In some such embodiments, the system further includes receiving genomic information of the subject. Generating the feature vector may include adding the genomic information to the feature vector. Without being bound by any particular methodology or theory, the genomic data may be derived from a biological sample from a patient with an increased risk of cardiovascular disease, such as a family history or genetic and / or protein markers. In some such embodiments, the learning system comprises a convolutional neural network. Such a convolutional neural network may include at least one residual connection.

[0033] In some embodiments, voltage-time data of a subject is received from an electrocardiograph. In further embodiments, voltage-time data of the subject is received from an electronic medical record.

[0034] In some embodiments, the system further includes providing the display to an electronic health record system for storage in a health record associated with the subject. In some embodiments, the system further includes providing the display to a computing node for display to a user.

[0035] In some embodiments of the systems disclosed herein, the feature vector includes a matrix having a plurality of rows and a plurality of columns, the plurality of rows corresponding to a time dimension, and the plurality of columns corresponding to a spatial dimension. In some such embodiments, each of the plurality of columns corresponds to one of a plurality of leads, and each of the plurality of columns corresponds to a timestamp. In some embodiments, the time dimension has a resolution of 500 Hz. In some embodiments, the convolutional neural network comprises at least nine convolutional blocks and two fully connected blocks.

[0036] Patient data may be received from an electronic health record (EHR) 101. An electronic health record (EHR) or electronic medical record (EMR) may refer to a systematic collection of health information of patients and populations electronically stored in digital form. These records can be shared between different medical settings. The records can be shared through network-connected enterprise-scale information systems or other information networks and exchanges. The EHR may include various data including demographics, medical history, drugs and allergies, immunization status, test results, radiological images, vital signs, personal statistics such as age and weight, and billing information. The EHR system may be designed to store data and capture the patient's condition over time. In this way, there is no need to track the patient's previous paper medical records.

[0037] ECG data may be received directly from an electrocardiogram recording device 102. In an exemplary 12-lead ECG, 10 electrodes are placed on the surfaces of a patient's extremities and chest. The overall magnitude of the heart's electrical potential is then measured from 12 different angles (leads) and recorded over a fixed period (usually 10 seconds). In this way, the overall magnitude and direction of the heart's electrical depolarization are captured at each instant throughout the cardiac cycle.

[0038] An additional data store 103 may contain additional patient information as described herein. Suitable data stores include databases, flat files, and other structures known in the art.

[0039] It can be understood that the ECG data may be stored in the EHR for later retrieval. It can also be understood that the ECG data may be cached rather than being directly delivered to a learning system for further processing.

[0040] The learning system 104 receives patient information from one or more of the EHR 101, ECG 102, and additional data store 103. As described above, in some embodiments, the learning system comprises a convolutional neural network. In various embodiments, the input to the convolutional neural network includes voltage-time information of the ECG, which, in some embodiments, is paired with additional patient information such as demographic or genetic information.

[0041] The learning system 104 may be pre-trained using appropriate population data as exemplified to generate a display of the level of CAC. In some embodiments, the display is binary. In some embodiments, the display is a probability value indicating the likelihood of the level of CAC given the input patient data.

[0042] In some embodiments, the learning system 104 provides a display of the level of CAC for storage as part of the EHR. In this way, computer-aided diagnosis that can be referenced by a clinician is provided. In some embodiments, the learning system 104 provides a display of the level of CAC to the remote client 105. For example, the remote client may be a health app, a cloud service, or another consumer of diagnostic data. In some embodiments, the learning system 104 is incorporated into an ECG machine to provide immediate feedback to the user during testing.

[0043] In some embodiments, a feature vector is provided to the learning system. Based on the input features, the learning system generates one or more outputs. In some embodiments, the output of the learning system is a feature vector. The feature vector can include a time series of values indicating the amplitudes of voltage-time data for multiple leads.

[0044] In some embodiments, the learning system comprises an SVM. In other embodiments, the learning system comprises an artificial neural network. In some embodiments, the learning system is pre-trained using training data. In some embodiments, the training data is retrospective data. In some embodiments, the retrospective data is stored in a data store. In some embodiments, the learning system may be further trained by manual curation of previously generated outputs.

[0045] In some embodiments, the learning system is a trained classifier. In some embodiments, the trained classifier is a random decision forest. However, it can be understood that various other classifiers, including neural networks such as linear classifiers, support vector machines (SVMs), or recurrent neural networks (RNNs), are suitable for use according to the present disclosure.

[0046] Suitable artificial neural networks include, but are not limited to, feedforward neural networks, radial basis function networks, self-organizing maps, learning vector quantization, recurrent neural networks, Hopfield networks, Boltzmann machines, echo state networks, long short-term memory, bidirectional recurrent neural networks, hierarchical recurrent neural networks, probabilistic neural networks, modular neural networks, associative neural networks, deep neural networks, deep belief networks, convolutional neural networks, convolutional deep belief networks, large-capacity memory search neural networks, deep Boltzmann machines, deep stacking networks, tensor deep stacking networks, spike and slab restricted Boltzmann machines, composite hierarchical deep models, deep coding networks, multi-layer kernel machines, or deep Q networks.

[0047] In machine learning, a convolutional neural network (CNN) is a type of feedforward artificial neural network applicable to the analysis of visual images and other natural signals. A CNN consists of an input layer, an output layer, and multiple hidden layers. The hidden layers of a CNN typically consist of convolutional layers, pooling layers, fully connected layers, and normalization layers. The convolutional layer applies a convolution operation to the input and passes the result to the next layer. Convolution emulates the response of individual neurons to stimuli. Each convolutional neuron processes data only for its receptive field.

[0048] The convolution operation enables a reduction in free parameters compared to a fully connected feedforward network. In particular, by tiling a given kernel, it becomes possible to learn a fixed number of parameters regardless of the image size. This also reduces the memory footprint of a given network.

[0049] The parameters of the convolutional layer consist of a set of learnable filters (or kernels), which have small receptive fields but extend across the entire depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the inner product between the entries of the filter and the input, and generating a 2D activation map for that filter. As a result, the network learns filters that activate when detecting some specific kind of feature at some spatial location within the input.

[0050] In an exemplary convolution, the kernel contains multiple weights w1...w9. The sizes provided here are merely exemplary, and it can be understood that any kernel dimension can be used as described herein. The kernel is applied to each tile of the input (e.g., an image). The result for each tile is an element of the feature map. It can be understood that multiple kernels may be applied to the same input to generate multiple feature maps.

[0051] By stacking the feature maps of all the kernels, the full output volume of the convolutional layer is formed. Thus, all the entries within the output volume can also be interpreted as looking at small regions within the input and being the output of neurons that share neurons and parameters within the same feature map.

[0052] Convolutional neural networks may be implemented on various hardware including hardware CNN accelerators and GPUs.

[0053] Referring now to FIG. 2, a flowchart is provided that illustrates a method for detecting or otherwise predicting the level of CAC according to an embodiment of the present disclosure. At 201, voltage-time data of a subject is received. The voltage-time data includes voltage data of a plurality of leads of an electrocardiograph. At 202, a feature vector is generated from the voltage-time data. The feature vector includes a time series of values indicating the amplitudes of the voltage-time data for the plurality of leads. At 203, the feature vector is provided to a pre-trained learning system. At 204, an indication of the presence or absence of CAC in the subject is received from the pre-trained learning system.

[0054] Referring now to FIG. 3, a schematic diagram of an example of a computing node is shown. Computing node 10 is merely an example of a suitable computing node and is not intended to suggest any limitation as to the use or functionality of the embodiments described herein. Nevertheless, computing node 10 can implement and / or execute any of the above functions.

[0055] Computing node 10 has a computer system / server 12 that operates in a number of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices.

[0056] The computer system / server 12 can be described in the general context of computer system-executable instructions, such as program modules executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer system / server 12 may be implemented in a distributed cloud computing environment where tasks are executed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may be located on both local and remote computer system storage media including memory storage devices.

[0057] As shown in FIG. 3, the computer system / server 12 within the computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processor 16.

[0058] The bus 18 represents one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Extended ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0059] Computer system / server 12 typically includes various computer system-readable media. Such media may be any available media accessible by computer system / server 12, including both volatile and non-volatile media, removable and non-removable media.

[0060] System memory 28 can include computer system-readable media in the form of volatile memory such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown, typically referred to as a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from and writing to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media can be provided. In such cases, each can be connected to bus 18 by one or more data media interfaces. As further shown and described below, memory 28 may include at least one program product having a set of program modules (e.g., at least one of the program modules) configured to execute the functions of embodiments of the present disclosure.

[0061] The program / utilities 40 having a set of program modules 42 (at least one of the program modules 42) may be stored in, by way of example and not limitation, the memory 28, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules 42 generally execute the functions and / or methodologies of the embodiments described herein.

[0062] Also, the computer system / server 12 may communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, one or more devices that enable a user to interact with the computer system / server 12, and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), and such communication may be through an input / output (I / O) interface 22. Further, the computer system / server 12 may communicate with one or more networks such as a local area network (LAN), a general-purpose wide area network (WAN), and / or a public network (e.g., the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. Although not shown, it should be understood that other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0063] The present disclosure may be embodied as a system, method, and / or computer program product. For example, in some aspects or embodiments of the present invention, there is provided herein a computer program product for detecting and / or predicting the level of CAC, comprising a computer-readable storage medium embodied with program instructions executable by a processor, whereby the processor is configured to receive voltage-time data of a subject from an echocardiogram, the voltage-time data including voltage data of a plurality of leads; generate a feature vector from the voltage-time data; provide the feature vector to a pre-trained learning system; and receive a display of the level of CAC in the subject from the pre-trained learning system. Generating the feature vector may include generating a spectrogram based on the voltage data of the plurality of leads. In some embodiments, generating the feature vector includes grouping the voltage data of the plurality of leads into a plurality of subsets.

[0064] In some embodiments, such a computer program product further includes receiving demographic information of the subject, and generating the feature vector includes adding the demographic information to the feature vector. In some such embodiments, the computer program further includes receiving genomic information of the subject. Generating the feature vector may include adding the genomic information to the feature vector. Without being bound by any particular methodology or theory, the genomic data may be derived from a biological sample of a patient with an increased risk of cardiovascular disease, such as a family history or genetic and / or protein markers. In some such embodiments, the computer program product comprises a convolutional neural network. Such a convolutional neural network may include at least one residual connection.

[0065] In some embodiments, voltage-time data of a subject is received from an electrocardiograph. In further embodiments, voltage-time data of the subject is received from an electronic medical record.

[0066] In some embodiments, the computer program product further includes providing a display to an electronic health record system for storage in a health record associated with the subject. In some embodiments, the computer program product further includes providing a display to a computing node for display to a user.

[0067] In some embodiments of the computer program product disclosed herein, the feature vector may include a matrix having a plurality of rows and a plurality of columns, the plurality of rows corresponding to a time dimension and the plurality of columns corresponding to a spatial dimension. In some such embodiments, each of the plurality of columns corresponds to one of a plurality of leads and each of the plurality of columns corresponds to a timestamp. In some embodiments, the time dimension has a resolution of 500 Hz. In some embodiments, the convolutional neural network comprises at least nine convolutional blocks and two fully connected blocks.

[0068] The computer program product provided herein may include a computer-readable storage medium (or a plurality of computer-readable storage media) having computer-readable program instructions for causing a processor to execute aspects of the present disclosure.

[0069] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes, hereinafter, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be construed to be a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted via a wire.

[0070] The computer-readable program instructions described herein can be downloaded to each computing / processing device from a computer-readable storage medium or via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, or via an external computer or an external storage device. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0071] The computer-readable program instructions for carrying out the operations of this disclosure may be source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit for carrying out aspects of this disclosure.

[0072] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0073] These computer-readable program instructions may be provided to the processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that includes instructions for causing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium comprises an article including instructions for implementing the functions / operations specified in one or more blocks of the flowchart and / or block diagram.

[0074] Alternatively, the computer-readable program instructions may be loaded onto a computer, other programmable apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowchart and / or block diagram.

[0075] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of one or more executable instructions for implementing the specified logical function. In some alternative embodiments, the functions recited in the blocks may be performed out of the order presented in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the related functionality. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0076] The description of the various embodiments of the present disclosure is presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are chosen to best explain the principles of the embodiments, the practical application, or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0077] Example Example 1: Method A historical cohort of 43,210 consecutive patients who had clinically demonstrated CAC and ECG within 1 year between 1997 and 2020 was evaluated. Major modifiable cardiovascular (CV) risk factors were collected as part of preventive cardiology or general medical evaluation visits. The oldest recorded CAC was used. Patients taking statins at the time of CAC with paced rhythm or a history of myocardial infarction or missing variables were excluded. Convolutional neural networks based on ECG-AI output + / - CV risk factors were trained and validated to predict high CAC scores (≥300) for 60% and 20% random samples of each dataset respectively. Model performance metrics using the threshold that yielded 90% sensitivity were evaluated in an independent test set using the remaining 8,642 (20%) observations.

[0078] Definition and size of the training and validation cohorts: The cohort consisted of 43,210 consecutive patients who had clinically demonstrated computed tomography-based coronary artery calcium (CAC) scans for cardiovascular risk stratification between 1997 and 2020 and clinically demonstrated electrocardiograms (ECGs) within 1 year. Convolutional neural networks based on ECG-AI output + / - CV risk factors were trained and validated to predict high CAC scores (≥300) for 60% and 20% random samples of each dataset respectively. The association of the ECG-AI algorithm output was also validated to predict mortality in this cohort.

[0079] Overview of algorithm architecture and design: · CAC: · The coronary artery calcium burden was quantified by CAC scoring of non-contrast ECG-gated cardiac CT images. · Sequential 3-mm thick images were obtained using either an electron beam or multislice CT scanner. · A natural language processing (NLP) algorithm was used to extract the CAC score and percentile from the radiologist's narrative report. · The NLP output achieved a predefined threshold of 95% accuracy. ·ECG: ·For training purposes, each ECG was converted into a 12×5000 matrix, where the first dimension represents the spatial leads and the second time series represents 10 seconds at 500 Hz. ·A convolutional neural network (CNN)-based model was developed using the Keras framework with TensorFlow (Google; Mountain View, CA, USA) implemented in Python. ·This framework has previously been successfully used for a validated model created to screen for left ventricular dysfunction and age and gender estimated from standard 12-lead ECGs. ·The CNN was trained to generate binary classification models based on different levels of CAC (i.e., >0, ≥100, ≥300) regardless of the presence or absence of tabular data (i.e., clinical characteristics). ·This CNN consists of a stack of convolutional layers, max-pooling layers, and batch normalization layers interspersed with ReLU activation functions and is completed with a fully connected layer. ·The model was trained for 30 epochs with a learning rate of 0.001 and a batch size of 32. For models that also included tabular data, the same hyperparameters and CNN architecture were used. ·The ECG features extracted from the last CNN layer were then concatenated with the tabular variables before the activation function of the final layer. ·The model was created using (i) ECG data only, (ii) ECG data combined with age and gender, and (iii) ECG data, age, gender, and cardiovascular risk factors.

[0080] Example 2: Results Performance of the algorithm (AUC, sensitivity, specificity, PPV / NPV, etc.): ·The area under the receiver operating characteristic curve (ROC), sensitivity, specificity, and accuracy of the ECG-AI age and gender algorithm for CAC score ≥300 were 0.83, 0.90%, 0.56%, and 0.60%, respectively. · The ECG-AI + age and gender algorithm showed equivalent performance (Delong's p-value > 0.05) when compared to an algorithm that included ECG-AI, age, gender, and major modifiable CV risk factors. · The performance of the model was consistent across subgroups including those without CV risk factors. · The ECG-AI algorithm outputs individuals identified with almost twice the mortality risk over the follow-up period. · This association · is independent of age, gender, conventional cardiovascular risk factors, current cardiovascular risk prediction algorithms, and importantly, CT-based CAC score + / - conventional risk factors. · The output additive value, i.e., individuals identified as having a low or high mortality risk within the latest cardiovascular risk prediction algorithm and CT-based CAC score categories.

[0081] The ECG-AI + age and gender algorithm showed equivalent performance (see Table 1) (Delong's p-value > 0.05) when compared to an algorithm that included ECG-AI, age, gender, and major modifiable CV risk factors. The performance of the model was consistent across subgroups including those without CV risk factors (see Figure 4). · Age 55 ± 9.8 · Gender, 31% female · 1,857 (20%) participants had CAC ≥ 300.

Table 1

[0082] Deep learning-enabled ECG algorithms can help predict high CAC scores even without additional information on modifiable CV risk factors. Therefore, the algorithms can be used to improve the selection of subjects more likely to have high CAC.

[0083] All publications (including patents, patent applications, and sequence accession numbers mentioned in this specification) are hereby incorporated by reference in their entirety, as if each individual publication were specifically and individually indicated to be incorporated by reference. In case of conflict, the present application, including any definitions herein, will control.

[0084] Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments of the invention described herein. Such equivalents are intended to be encompassed by the following claims.

Claims

1. Receiving voltage-time data of a subject, wherein the voltage-time data includes voltage data of a plurality of leads of an electrocardiograph; Generating a feature vector from the voltage-time data, wherein the feature vector includes a time series of values indicating amplitudes of an ECG for the plurality of leads; Providing the feature vector to a pre-trained learning system; Receiving, from the pre-trained learning system, a display of a level of coronary artery calcium (CAC) in the subject; A method comprising the above steps.

2. The method according to claim 1, wherein generating the feature vector includes generating a spectrogram based on the voltage data of the plurality of leads.

3. The method according to claim 1, wherein generating the feature vector includes grouping the voltage data of the plurality of leads into a plurality of subsets.

4. The method further includes receiving demographic information of the subject, and generating the feature vector includes adding the demographic information to the feature vector. The method according to claim 1.

5. The method further includes receiving genomic information of the subject, and generating the feature vector includes adding the genomic information to the feature vector. The method according to claim 1.

6. The method according to claim 1, wherein the learning system comprises a convolutional neural network.

7. The method according to claim 6, wherein the convolutional neural network includes at least one residual connection.

8. The method according to claim 1, wherein the voltage-time data of the subject is received from an electrocardiograph.

9. The method according to claim 1, wherein the voltage-time data of the subject is received from an electronic medical record.

10. Further comprising providing the display to an electronic health record system for storage in a health record associated with the subject. The method according to claim 1.

11. Further comprising providing the display to a computing node for display to a user. The method according to claim 1.

12. The method according to claim 1, wherein the feature vector includes a matrix having a plurality of rows and a plurality of columns, the plurality of rows corresponding to a time dimension, and the plurality of columns corresponding to a spatial dimension.

13. The method according to claim 12, wherein each of the plurality of rows corresponds to one of the plurality of leads, and each of the plurality of columns corresponds to a timestamp.

14. The method according to claim 12, wherein the time dimension has a resolution of 500 Hz.

15. The method according to claim 6, wherein the convolutional neural network comprises at least nine convolutional blocks and two fully connected blocks.

16. An electrocardiograph comprising a plurality of leads, A computing node comprising a computer-readable storage medium embodied with program instructions, the program instructions being executable by a processor of the computing node, whereby the processor Receiving voltage-time data of a subject from an echocardiograph, the voltage-time data including voltage data of the plurality of leads; Generating a feature vector from the voltage-time data, the feature vector including a time series of values indicating the amplitude of the ECG for the plurality of leads; Providing the feature vector to a pre-trained learning system; Receiving, from the pre-trained learning system, a display of the level of CAC in the subject; A computing node that executes a method including: A system comprising.

17. The system according to claim 16, wherein generating the feature vector includes generating a spectrogram based on the voltage data of the plurality of leads.

18. The system according to claim 16, wherein generating the feature vector includes grouping the voltage data of the plurality of leads into a plurality of subsets.

19. Further comprising receiving demographic information of the subject, and generating the feature vector includes adding the demographic information to the feature vector. The system according to claim 16.

20. Further comprising receiving genomic information of the subject, and generating the feature vector includes adding the genomic information to the feature vector. The system according to claim 16.

21. The system according to claim 16, wherein the learning system comprises a convolutional neural network.

22. The system according to claim 21, wherein the convolutional neural network includes at least one residual connection. **Claim 23** The system according to claim 16, wherein the voltage-time data of the subject is received from an electrocardiograph. **Claim 24** The system according to claim 16, wherein the voltage-time data of the subject is received from an electronic medical record. **Claim 25** The system according to claim 16, further comprising providing the display to an electronic health record system for storage in a health record associated with the subject. **Claim 26** The system according to claim 16, further comprising providing the display to a computing node for display to a user. **Claim 27** The system according to claim 16, wherein the feature vector includes a matrix having a plurality of rows and a plurality of columns, the plurality of rows corresponding to a time dimension, and the plurality of columns corresponding to a spatial dimension. **Claim 28** The system according to claim 27, wherein each of the plurality of columns corresponds to one of the plurality of leads, and each of the plurality of columns corresponds to a timestamp. **Claim 29** The system according to claim 27, wherein the time dimension has a resolution of 500 Hz. **Claim 30** The system according to claim 21, wherein the convolutional neural network comprises at least nine convolutional blocks and two fully connected blocks. **Claim 31** A computer program product for detecting a level of CAC, comprising a computer-readable storage medium embodied with program instructions executable by a processor, whereby the processor receives voltage-time data of a subject from an echocardiograph, the voltage-time data including voltage data of the plurality of leads, generates a feature vector from the voltage-time data, the feature vector including a time series of values indicating amplitudes of an ECG with respect to the plurality of leads, provides the feature vector to a pre-trained learning system, receives a display of a level of CAC in the subject from the pre-trained learning system, and executes a method including the above steps. **Claim 32** The computer program product according to claim 31, wherein generating the feature vector includes generating a spectrogram based on the voltage data of the plurality of leads.

33. The computer program product according to claim 31, wherein generating the feature vector includes grouping the voltage data of the plurality of leads into a plurality of subsets.

34. The computer program product according to claim 31, further comprising receiving demographic information of the subject, and generating the feature vector includes adding the demographic information to the feature vector. The computer program product according to claim 31.

35. The computer program product according to claim 31, further comprising receiving genomic information of the subject, and generating the feature vector includes adding the genomic information to the feature vector. The computer program product according to claim 31.

36. The computer program product according to claim 31, wherein the learning system comprises a convolutional neural network.

37. The computer program product according to claim 36, wherein the convolutional neural network includes at least one residual connection.

38. The computer program product according to claim 31, wherein the voltage-time data of the subject is received from an electrocardiograph.

39. The computer program product according to claim 31, wherein the voltage-time data of the subject is received from an electronic medical record.

40. The computer program product according to claim 31, further comprising providing the display to an electronic health record system for storage in a health record associated with the subject. The computer program product according to claim 31.

41. The computer program product according to claim 31, further comprising providing the display to a computing node for display to a user. The computer program product according to claim 31.

42. The computer program product according to claim 31, wherein the feature vector includes a matrix having a plurality of rows and a plurality of columns, the plurality of rows corresponding to a time dimension, and the plurality of columns corresponding to a spatial dimension.

43. The computer program product according to claim 41, wherein each of the plurality of columns corresponds to one of the plurality of leads, and each of the plurality of columns corresponds to a timestamp.

44. The computer program product according to claim 42, wherein the time dimension has a resolution of 500 Hz.

45. The convolutional neural network comprises at least nine convolutional blocks and two fully-connected blocks, the computer program product according to claim 36.

46. The time period of the voltage-time data describes a plurality of cardiac cycles of the subject, the method according to claim 1.

47. The method according to claim 1, further comprising presenting the display of the level of the CAC in the subject to a healthcare provider associated with the subject.

48. The voltage-time data describes a single-lead ECG of the subject, the method according to claim 1.

49. The method according to claim 48, wherein the single-lead ECG corresponds to one of the leads of a 12-lead ECG.

50. One of the leads of the 12-lead ECG to which the single-lead ECG corresponds is Lead 1, Lead 2, Lead 3, AvF, AvL, AvR, or V1-V6, the method according to claim 49.

51. The voltage-time data is limited to describing a single cardiac cycle of the voltage-time data from the ECG of the subject, the method according to claim 1.

52. The voltage-time data is limited to describing a single average cardiac cycle of the voltage-time data from the ECG of the subject, the method according to claim 1.

53. The voltage-time data is limited to describing a single central cardiac cycle of the voltage-time data from the ECG of the subject, the method according to claim 1.

54. The voltage-time data describes a single cardiac cycle, and the voltage-time data is multi-dimensional and extracted from a vector cardiogram, the method according to claim 1.

55. The voltage-time data describes normal sinus rhythm in the subject, the method according to claim 1.

56. The feature vector is based on the characteristics of normal sinus rhythm of the subject, the method according to claim 1.

57. The voltage-time data extends over a time interval of 30 seconds or less, the method according to claim 1.

58. The voltage-time data extends over a time interval of 10 minutes or less, the method according to claim 1.

59. The subject is a mammal, the method according to claim 1.

60. The mammal is a human, the method according to claim 59.

61. The voltage-time data of the subject is based on less than 12 leads, the method according to claim 1.

62. The method according to claim 1, wherein the display of the level of the CAC in the subject indicates at least one possible treatment plan.

63. The method according to claim 62, wherein the at least one possible monitoring or treatment plan includes administering a treatment.

64. The method according to claim 62, wherein the at least one possible treatment plan includes collecting a second series of voltage-time data.

65. Obtaining data describing a non-ECG profile for a mammal, Generating one or more second neural network inputs representing the non-ECG profile for the subject, Processing the first neural network input with the one or more second neural network inputs in the neural network to generate the feature vector, The method according to claim 1, further comprising.

66. The method according to claim 1, wherein the display of the level of the CAC in the subject indicates at least a threshold likelihood that the subject experiences CAC.

67. The method according to claim 66, further comprising determining a treatment for reducing the level of CAC in the subject in response to the display of at least a threshold likelihood that the subject experiences CAC.

68. The voltage-time data from the subject is recorded over a first time interval, and the method is Obtaining a second neural network input, wherein the second neural network input represents second voltage-time data of the subject recorded over a second time interval, and the first time interval and the second time interval are separated by a third time interval, Processing the first neural network input with the second neural network input in the neural network to generate the feature vector in the subject, The method according to claim 1, further comprising.

69. The method according to claim 68, wherein the third time interval is at least one minute, one hour, one day, one week, or one month.

70. The method according to claim 69, wherein the neural network further processes a third neural network input indicating the length of a third time interval between the first time interval and the second time interval when the first voltage-time data and the second voltage-time data are respectively recorded, together with the first neural network input and the second neural network input.

71. The method according to claim 1, wherein the pre-trained learning system is trained by receiving a training set of voltage-time data from a plurality of CAC patients.

72. The method according to claim 71, wherein the training set of the voltage-time data is from one of a retrospective cohort subset or a prospective cohort subset.

73. The method according to claim 1, wherein the voltage-time data includes analog data characterizing an ECG signal.

74. The method according to claim 1, wherein the voltage-time data includes numerical values specifying the amplitude of an ECG signal.

75. The method according to claim 1, wherein the voltage-time data includes image data characterizing an ECG signal.

76. The method according to claim 75, wherein the image data characterizing the ECG signal is in a pixel format including a TIFF, PNG, or PDF file type.

Images