ECG retrieval and interpretation based on dual ECG and text embedding models

Abstract

An embodiment of the present disclosure provides a system and method for ECG search based on a dual ECG and text embedding model. A text machine learning (ML) model may be trained to generate a text embedding based on a received textual representation of an ECG diagnosis. The text ML model may be used to train an ECG encoding ML model to generate an ECG embedding based on received ECG lead data. A database may be populated with a plurality of ECG embeddings, each of the plurality of ECG embeddings being generated based on ECG lead data of a previously diagnosed ECG. In response to receiving a query ECG, the ECG ML model may generate a query embedding, and a similarity score between the query embedding and each of the plurality of ECG embeddings may be determined. The top-K results may be stored and displayed / visualized based on the similarity scores.

Description

[Technical field]

[0001] cross reference This application claims the benefit of U.S. Patent Application No. 17 / 585,358, filed January 26, 2022, and entitled “ECG SEARCH AND INTERPRETATION BASED ON A DUAL ECG AND TEXT EMBEDDING MODEL,” the entire contents of which are incorporated herein by reference.

[0002] Aspects of the present disclosure relate to electrocardiogram (ECG) interpretation, and in particular, to searching and classifying ECGs to aid in ECG interpretation. [Background technology]

[0003] Cardiovascular disease is the leading cause of death worldwide. In 2008, 30% of all deaths on earth could be attributed to cardiovascular disease. It is estimated that by 2030, over 23 million people will die from cardiovascular disease each year. Cardiovascular disease is equally common among all residents of first and third world countries, affecting people regardless of socio-economic status.

[0004] Arrhythmia is a cardiac condition in which the electrical activity of the heart is irregular, or faster (tachycardia) or slower (bradycardia) than normal. While many arrhythmias are not life-threatening, some can cause cardiac arrest and even cardiac death. In fact, cardiac arrhythmias are one of the most common causes of death on the way to the hospital. Atrial fibrillation (A-fib) is the most common cardiac arrhythmia. In A-fib, electrical conduction through the ventricles is irregular and disrupted. While A-fib may not cause any symptoms, it is often associated with palpitations, shortness of breath, fainting, chest pain, or congestive heart failure, and also increases the risk of stroke. A-fib is usually diagnosed by taking an electrocardiogram (ECG) of the subject. To treat A-fib, patients may receive medication to slow down the heartbeat or change the heart rhythm. Patients may also take anticoagulants to prevent strokes, or even undergo surgical interventions, including cardiac ablation, to treat A-fib. In another example, ECG may provide decision support for Acute Coronary Syndromes (ACS) by interpreting various rhythm and morphological conditions, including Myocardial Infarction (MI) and ischemia.

[0005] Often, patients with A-fib (or other types of arrhythmias) are monitored for extended periods of time to manage the disease. For example, patients may be given a Holter monitor or other ambulatory electrocardiography device to continuously monitor the electrical activity of the cardiovascular system, for example, for at least 24 hours. Such monitoring can be important, among other things, to detect conditions such as acute coronary syndrome (ACS).

[0006] The American Heart Association and the European Society of Cardiology recommend that a 12-lead ECG be collected as soon as possible for patients with possible ACS when symptoms are present. Prehospital ECGs have been found to significantly reduce time to treatment and have been shown to have better survival rates. Time to first ECG is so important that it is a quality and performance metric monitored by several regulatory agencies. According to the 2015 National Health Statistics, over 7 million people visited emergency departments (EDs) in the United States (US) with a chief complaint of chest pain or symptoms related to ACS. In the US, ED visits are increasing at a rate of 3.2% per year, and outside the US, ED visits are increasing 3%-7% per year. The most accurate and specific method of ECG interpretation for ACS is to compare the current ECG to the patient's previously recorded ECG to see if there are any significant changes in the ST-T segment and QRS complex. Summary of the Invention [Means for solving the problem]

[0007] The described embodiments and their advantages may best be understood by referring to the following description taken in conjunction with the accompanying drawings, which are not intended to limit changes in form and detail that may be made to the described embodiments by those skilled in the art without departing from the spirit and scope of the described embodiments. [Brief description of the drawings]

[0008] [Figure 1A] FIG. 1 is a block diagram illustrating an example system, according to some embodiments of the present disclosure. [Figure 1B] FIG. 1 illustrates a single dipole heart model with a 12-lead set represented in a 6-axis system, according to some embodiments of the present disclosure. [Diagram 2]FIG. 1 is a block diagram illustrating a cloud services system according to some embodiments of the present disclosure. [Diagram 3] FIG. 2 is a block diagram illustrating training of a text encoder in accordance with certain aspects of the present disclosure. [Figure 4] FIG. 2 is a block diagram illustrating training of an ECG encoder in accordance with some embodiments of the present disclosure. [Diagram 5] FIG. 2 is a block diagram illustrating a joint embedding space, in accordance with some embodiments of the present disclosure. [Figure 6A] 11 is a block diagram illustrating the performance of an ECG search according to some embodiments of the present disclosure. [Figure 6B] FIG. 13 illustrates a timeline visualization of ECG search results according to some embodiments of the present disclosure. [Figure 6C] FIG. 13 illustrates a timeline visualization of ECG search results according to some embodiments of the present disclosure. [Figure 7] FIG. 2 is a block diagram illustrating training of a classifier according to some embodiments of the present disclosure. [Figure 8A] 1 is a flow diagram of a method for performing an ECG search, according to some embodiments of the present disclosure. [Figure 8B] 1 is a flow diagram of a method for performing ECG classification, according to some embodiments of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an example computing device that may perform one or more of the operations described herein, in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Computer-generated ECG interpretations have been used for many years, and many of the systems that generate them operate based on input from experts and a set of predefined criteria. Recently, the use of deep learning models to generate ECG interpretations has been explored, but has not been widely applied to real medical devices and systems. One of the main reasons is that machine learning models, such as DNN models, are generally “black boxes” in that they provide an interpretation of the ECG but do not show why a certain result was reached. For comprehensive multi-lead ECG interpretation, there are many classes, e.g., rhythm and morphology interpretations, that usually require some explanation or justification for the final interpretation. This is different from the simple types of detection, e.g., performed by smart watches and other wearable devices for AFIB and sinus rhythm detection. Utilizing machine learning model-based ECG interpretations while adding explicit justification for the results of the interpretation is a very important task to further expand the utilization of machine learning ECG interpretation models to various clinical applications.

[0010] The present disclosure addresses these and other deficiencies by providing a system and method for ECG searching based on a dual ECG and text embedding model. A processing device may train a text machine learning (ML) model to generate a text embedding based on a received textual representation of an ECG diagnosis. The processing device may use the text ML model to train an ECG encoding ML model to generate an ECG embedding based on the received ECG lead data, where ECG embeddings generated from similar ECG lead data are close to each other in vector space. The processing device may populate a database with a plurality of ECG embeddings, where each of the plurality of ECG embeddings is generated based on ECG lead data of a previously diagnosed ECG. In response to receiving a query ECG, the processing device may use the ECG ML model to generate a query embedding and may determine a similarity score between the query embedding and each of the plurality of ECG embeddings. The processing device may sort the ECG embeddings in descending order based on the similarity score and may display / visualize (or transmit to the local computing device 120 for display / visualization) the top K results.

[0011] FIG. 1A illustrates a system 100 in which aspects of the disclosure may be implemented. The system 100 may be specified for use by a first user, for example, by the first user's physician. Alternatively, the system 100 may be used without input from a physician or other third party. The system 100 may include a local computing device 120 of the first user. The local computing device 120 may be loaded with a user interface, dashboard, or other subsystems of the cardiology management system 100. For example, the local computing device 120 may be loaded with a mobile software application (shown as 101A in FIG. 1) for interfacing with the system 100. The mobile software application 101A may be configured to interface with one or more biometric sensors (e.g., ECG monitor 110) and may include software and a user interface for managing biometric data collected by the local computing device 120 from the one or more biometric sensors. Local computing device 120 may include any suitable computing device, such as, for example, a tablet computer, a smartphone, a server computer, a desktop computer, a laptop computer, or a body-worn computing device (e.g., a smart watch or other wearable). In some embodiments, local computing device 120 may include a single computing device or may include multiple interconnected computing devices (e.g., multiple servers arranged in a cluster).

[0012] The local computing device 120 may be coupled to one or more biosensors. For example, the local computing device 120 may be coupled to an ECG monitor 110, which may include a set of electrodes for recording ECG (electrocardiogram) data of the first user's heart (also referred to herein as "taking an ECG"). The ECG data may be recorded or taken using a set of electrodes placed at multiple locations on the first user's skin. The electrical signals recorded between electrode pairs may be referred to as leads, and FIG. 1B shows a 12-lead set including leads I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, and V6, all represented on a 6-axis system. Various numbers of leads may be used to record ECG data, and different numbers and combinations of electrodes may be used to form the various leads. Exemplary numbers of leads used to take an ECG are 1, 2, 6, and 12 leads. For example, the ECG monitor 110 may be a device that includes ten electrodes (six on the user's chest and one on each of the user's arms and legs) that may provide a 12-lead ECG. The electrode placed on the right arm may be referred to as the RA. The electrode placed on the left arm may be referred to as the LA. The RA and LA electrodes may be placed in the same location on the left and right arms, e.g., near the wrist. The leg electrodes may be referred to as the RL on the right leg and the LL on the left foot. The RL and LL electrodes may be placed in the same location on the left and right legs, e.g., near the ankle.

[0013] In some embodiments, the ECG monitor 110 may include a handheld ECG monitor (such as, for example, the KardiaMobile® or KardiaMobile® 6L devices by AliveCor® Inc.) that includes fewer electrodes (e.g., two or three electrodes). In these embodiments, the electrodes may be used to measure a subset of the leads shown in FIG. 2, such as Lead I (e.g., voltage between the left arm and the right arm) simultaneously with Lead II (e.g., voltage between the left leg and the right arm), and Lead I simultaneously with another one of the chest leads, such as Lead V2 or V5. It should be noted that other combinations of leads are possible. If desired, additional leads may be algorithmically derived (e.g., by the ECG monitor 110 itself or the local computing device 120) from the determined subset of leads. For example, an amplified limb lead may be determined from values ​​measured by the LA, RA, LL, and RL electrodes. The augmented vector right (aVR) may be equal to RA-(LA+LL) / 2 or -(I+II) / 2. The extension vector left (aVL) may be equal to LA-(RA+LL) / 2 or I-II / 2. The extension vector down (aVF) may be equal to LL-(RA+LA) / 2 or -II-I / 2. In some embodiments, the ECG monitor 110 itself or the local computing device 120 may utilize a machine learning (ML) model to derive the full 12-lead set from a subset of the measured leads. In some embodiments, the ECG monitor 110 may be in the form of a smartphone or a wearable device such as a smart watch. In some embodiments, the ECG monitor 110 may be a handheld sensor coupled to the local computing device 120 with an intermediate protective case / adapter.

[0014] The ECG data recorded by the ECG monitor 110 may include, for example, the electrical activity of the first user's heart. A typical heartbeat may include several variations in electrical potentials, which may be categorized into waves and complexes, including P waves, QRS complexes, T waves, and sometimes U waves, as known in the art. The shape and duration of the P waves may be related to the size of the user's atria (e.g., indicative of atrial enlargement) and may be a first source of heartbeat characteristics specific to the user.

[0015] The duration, amplitude, and morphology of each of the Q, R, and S waves may vary from person to person, and in particular may vary significantly in users with cardiac disease or abnormalities. For example, a Q wave greater than 1 / 3 the height of an R wave or greater than 40 ms in duration may indicate a myocardial infarction and provide a unique characteristic of the user's heart. Similarly, other healthy ratios of Q and R waves may be used to distinguish the heartbeats of different users.

[0016] The ECG monitor 110 may be used by a first user to measure ECG data and transmit the measured ECG data to a local computing device 120 using a suitable wired or wireless connection (e.g., a Wi-Fi connection, a Bluetooth® connection, a near field communication (NFC) connection, an ultrasonic signal transmission connection, etc.).

[0017] The ECG data may be recorded continuously by the user at regular intervals. For example, the intervals may be once a day, once a week, once a month, or some other predetermined interval. The ECG data may be recorded at the same or different times and under similar or different circumstances as described herein. The ECG data may also be recorded at the same or different times of the interval (e.g., the ECG data may be captured asynchronously). Alternatively or additionally, the ECG data may be recorded on demand by the user at various discrete times, such as when the user feels chest pain or experiences other unusual or abnormal sensations, or upon instruction to do so, for example, from the user's physician. In another embodiment, the ECG data may be recorded continuously over a period of time (e.g., by a Holter monitor or by some other wearable device).

[0018] Each ECG data record may be time-stamped and annotated with additional data by the user or health care provider to describe user characteristics. For example, the local computing device 120 (e.g., its mobile app 101A) may include a user interface for data entry that allows the user to enter the user's user characteristics, including, for example, a user ID. The local computing device 120 may add the user characteristics to the ECG data and transmit the ECG data to the cloud service system 140.

[0019] The ECG data may be transmitted by the local computing device 120 to the cloud service system 140 for storage and analysis. The transmission may be at regular intervals, such as real-time, hourly, daily, weekly, and / or any interval in between, or may be on-demand. The local computing device 120 and the cloud service system 140 may be coupled to each other (e.g., operatively coupled, communicatively coupled, and may communicate data / messages with each other) via a network 130. The network 130 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or a combination thereof. In one embodiment, the network 130 may include a wired or wireless infrastructure, which may be provided by one or more wireless communication systems, such as Wi-Fi hotspots connected to the network 130, and / or a wireless carrier system, which may be implemented using various data processing equipment, communication towers (e.g., cell towers), and the like. The network 130 may carry communications (eg, data, messages, packets, frames, etc.) between the local computing device 120 and the cloud service system 140.

[0020] Machine learning (ML) models are suitable for continuous monitoring of one or more criteria to identify anomalies or trends, large or small, in input data compared to the training examples used to train the model. The ML models described herein may be trained with ECG data from a population of users and / or other training examples that serve the design needs of the model. Machine learning models that may be used with the embodiments described herein include, by way of example and not limitation, Bayesian, Markovian, Gaussian processes, clustering algorithms, generative models, kernel and neural network algorithms. Some embodiments utilize machine learning models based on trained neural networks (e.g., trained recurrent neural networks (RNN) or trained convolutional neural networks (CNN)).

[0021] For example, the ML model may include a trained CNN ML model that puts input data (e.g., ECG data) into a convolutional layer (also called a hidden layer) and applies a set of trained weights or filters to the input data of each of the convolutional layers. The output of the first convolutional layer is an activation map, which is the input to the second convolutional layer, to which trained weights or filters (not shown) are applied, and the output of the subsequent convolutional layers is an activation map that represents increasingly complex features of the input data to the first layer. After each convolutional layer, a nonlinear layer (not shown) is applied to introduce nonlinearity to the problem, which may include activation functions such as tanh, sigmoid, or ReLU. In some cases, a pooling layer (not shown) may be applied after the nonlinear layer, also called a downsampling layer, which basically takes a filter and a stride of the same length, applies it to the input, and outputs the maximum number for each sub-region that the filter convolves. Other options for pooling are average pooling and L2 regularization pooling. The pooling layer reduces the spatial dimension of the input volume, reducing computational cost and controlling overfitting. The final layer of the network is the fully connected layer, which takes the output of the last convolutional layer and outputs an n-dimensional output vector representing the quantity to be predicted, which may yield the predicted output. The trained weights may be different in each of the convolutional layers.

[0022] To achieve real-world prediction / detection, the neural network needs to be trained with known data inputs or training examples, resulting in a trained CNN. To train the CNN, many different training examples (e.g., ECG data from a user) are input into the model. Those skilled in the art of neural networks will fully appreciate that the above description provides a somewhat simplified neural network to provide some context for the present discussion, and that the application of any neural network alone or in combination with other neural networks or other entirely different machine learning models is equally applicable and within the scope of some embodiments described herein.

[0023] FIG. 2 illustrates a cloud service system 140 according to some embodiments of the present disclosure. As illustrated in FIG. 2, the cloud service system 140 may be a computing device including hardware such as a processing device 140B (e.g., a processor, a central processing unit (CPU)), a memory 140A (e.g., a random access memory (RAM)), a storage device (e.g., a hard disk drive (HDD), a solid state drive (SSD), etc.), and other hardware devices (e.g., a sound card, a video card, etc.). In some embodiments, the memory 140A may be a persistent storage device capable of storing data. The persistent storage device may be a local storage unit or a remote storage unit. The persistent storage device may be a magnetic storage device, an optical storage device, a solid state storage device, an electronic storage device (main memory), or a similar storage device. The persistent storage device may also be a monolithic / single device or a distributed set of devices. The memory 140A may be configured for long-term storage of data and may retain data between power on / off cycles of the cloud service system 140. The memory 140A may store ECG data accumulated over time for a user as well as a number of other users. The accumulated ECG data over time for a particular user may form a time series health record for that particular user. Cloud services system 140 may include any suitable type of computing device or machine having a programmable processor, including, for example, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, etc. In some embodiments, cloud services system 140 may include a single computing device or may include multiple interconnected computing devices (e.g., multiple servers arranged in a cloud storage cluster).

[0024] Memory 140A may further include an ECG encoder training module 141 and an ECG search module 143, each of which may be executed by processing device 140B to perform some of the functions described herein. Processing device 140B may execute ECG encoder training module 141 to train an ECG encoder for use with ECG search module 143, as described in more detail herein. Memory 140A may further include training data 150, which may include textual representations of a plurality of ECG diagnoses for use in training text encoder 145, as described in more detail herein. Memory 140A may further include training data 155, which may include ECG recordings (i.e., raw lead data) and textual representations of corresponding diagnoses for each of a plurality of ECGs. ECG recordings, as used herein, may refer to raw lead data of the ECG.

[0025] Upon executing the ECG encoder training module 141, the processing device 140B may train the text encoder 145 (shown in FIG. 3) in a semi-supervised manner based on the training data 150 to learn a representation function (referred to herein as a text embedding) that transforms the textual representation of the ECG diagnosis into a vector in an embedding space. The training data 150 may include a textual representation of each of a plurality of ECG diagnoses, as described in further detail herein. The text encoder 145 may be any suitable ML model capable of extracting a learnable, transformable representation from a sequence, such as, for example, the bidirectional encoder representations from transformers (BERT) sequence-to-sequence model. The embedding space is a relatively low-dimensional space that contains learned continuous vector representations of discrete variables into which higher-dimensional vectors can be transformed. Semi-supervised (or self-supervised learning) involves training an ML model using a limited set of labels, or no labels at all, to infer patterns from the data on which the ML model is trained. In some cases, semi-supervised learning may include the use of proxy tasks, as described in more detail herein.

[0026] Embeddings make it easier to do machine learning with large inputs, such as sparse vectors representing words, and can be learned and reused across models. The textual representation of each ECG diagnosis in the training data 150 may be an ordered sequence of diagnosis codes representing the diagnosis generated in the ECG. Each diagnosis code in the sequence may represent the observed condition (e.g., code 22 = normal sinus rhythm), a grammatical modifier (e.g., code 179 = and), and an adverb / adverbial phrase (e.g., code 211 = sometimes). For example, the diagnosis code sequence [19, 221, 1766] is translated to "sinus rhythm with premature ventricular complexes." Although embodiments of the present disclosure are described using an ordered sequence of diagnosis codes representing ECG diagnoses as the textual representation of ECG diagnoses for illustrative purposes, the embodiments are not so limited and may be implemented using any suitable textual representation of an ECG diagnosis.

[0027] FIG. 3 illustrates a process of training the text encoder 145. The text encoder 145 may receive as input a first sequence of diagnostic codes from the training data 150 corresponding to ECG diagnoses. The first sequence of diagnostic codes may be [19, 221, 1766, 1687, 0, 0, 0...], representing "sinus rhythm premature supraventricular complexes in a series otherwise normal ECG". The processing device 140B may remove extraneous information from the sequence of diagnostic codes, such as free text portions and dates. The number of diagnostic codes / sequence lengths of the input and output sequences are set to 16 as shown in FIG. 3 and may be selected arbitrarily. As can be seen, the first sequence of diagnostic codes may be padded with zeros to fill the 16 code length (may be [19, 221, 1766, 1687, 0, 0, 0...]). In some embodiments, the number of diagnostic codes in the input and output sequences may be selected to obtain ideal performance.

[0028] The text encoder 145 may learn to encode a sequence of diagnostic codes into text embeddings (vectors in the embedding space) by training on a masked prediction task. Thus, for a first sequence of diagnostic codes, the processing device 140B randomly removes the diagnostic code from the sequence and encodes it as <mask>In the example of FIG. 3, the diagnostic code representing "supraventricular extra complex" is masked, and the text encoder 145 is trained (e.g., via a classifier layer of the text encoder 145) to predict the masked diagnostic code by outputting a sequence of probability distributions for possible diagnostic codes that may fit the masked token. Since most masked prediction examples have multiple possible completions, it is reasonable to expect that given sufficient sequences of diagnostic codes from the training data 150, the text encoder 145 (e.g., via a conditional dependency layer) will learn a representation function that captures the conditional dependencies between the different diagnostic codes and encodes the similarity of the ECG diagnoses involved. Continuing with the example of FIG. 3, the text encoder 145 may come to understand that while two or more diagnostic codes could potentially appear after "sinus rhythm," the phrase "sequence of" indicates a high probability of a potential VTAC, and thus the mask is most likely to be filled with "ventricular extra complex." The text encoder 145 may assign other phrases, such as "sinus complexes," other possibilities, and lower probabilities.

[0029] Note that what the text encoder 145 is actually learning is the probability distribution of different diagnostic codes that may fit the masked tokens, which ultimately informs how the sequence of diagnostic codes is understood / interpreted. More specifically, the representation function of the text encoder 145 may map the sequence of diagnostic codes to text embeddings (vectors), and the classifier layer of the text encoder 145 may map the text embeddings to probability distributions of tokens. The classifier layer may be trained to predict the masked diagnostic code from the embedding of the representation function at the position of the masked diagnostic code. Since the diagnostic code at that position is masked, the representation function must generate this embedding from the context (i.e., by using the unmasked diagnostic code of the sequence). Contexts that produce similar distributions are likely to have similar embeddings. For example, there may be two training examples, “normal sinus rhythm, normal ECG” and “sinus rhythm, normal ECG”, where the second diagnostic code of each is masked (“normal sinus rhythm, <mask>" and "Sinus rhythm, <mask>"). The text encoder 145 is likely to learn that "normal sinus rhythm" and "sinus rhythm" are similar (and will produce similar context embeddings) since "normal ECG" is the target prediction in both cases.

[0030] Once training of the text encoder 145 is complete, the text encoder 145 may receive a sequence of diagnosis codes and output a sequence of vectors (consecutive real numbers) that capture all the diagnosis information a doctor or healthcare professional needs, such that similar diagnoses are near each other in the embedding space.

[0031] ECG search may be implemented by training ECG encoder 147 to learn a representation function (called an "ECG embedding") that transforms ECG recordings (e.g., 10-second, 12-lead) into vectors in an embedding space. ECG encoder 147 has the same properties as text encoder 145 in that ECGs with similar diagnoses will be pushed into the same region of the embedding space, and ECGs with different diagnoses will be pushed into different regions. Thus, processing device 140B may train ECG encoder 147 using the joint embedding space between ECG recordings and the textual representations of the corresponding diagnoses. To do this, processing device 140B may use the representation function learned by text encoder 145 to supervise the training of ECG encoder 147. However, processing device 140B may utilize a flexible form of supervision that simply uses the text embeddings as a starting point for learning the joint embeddings. The processing device 140B may train the ECG encoder 147 using training data 155, which may include ECG recordings (i.e., raw lead data) and textual representations of each corresponding diagnosis for multiple ECGs (i.e., sequences of diagnosis codes).

[0032] FIG. 4 illustrates training of ECG encoder 147. ECG encoder 147 may receive an ECG recording (from training data 155) that includes raw ECG lead data and may utilize multiple layers, with each layer downsampling the raw ECG lead data and adding more information / channels. A channel is a dimension that represents the features of the input at a point in time. A typical convolutional network converts the input to a time dimension by downsampling, possibly increasing the number of channels. For example, the input (raw ECG lead data) may have dimensions of 3000 (time steps)×1 (channels), and the first layer of the network may output raw ECG lead data with dimensions of 1500×32, the second layer of the network may output raw ECG lead data with dimensions of 750×64, and the third layer of the network may output raw ECG lead data with dimensions of 375×128. As one goes deeper into the network, the raw ECG lead data will have more channels and fewer time steps. The network may convert the local information into global information. Finally, the time dimension may be reduced to one, and each output may represent global information about the raw ECG lead data.

[0033] The ECG encoder 147 may include a convolutional network, a lead combiner, and a convolutional residual network (not shown). The convolutional network may downsample and extract features from each lead independently (performing the same operation on each lead). The lead combiner may integrate and mix information from all of the leads. The convolutional residual network may perform additional processing and downsample using a technique sometimes called information bottlenecking, where information is passed through a smaller space, thereby forcing the ECG encoder 147 to learn how to represent that information more efficiently and discard information that is extraneous or irrelevant. In this way, the processing device 140B may train the ECG encoder 147 to learn how to represent the raw lead data of each ECG in the training data 155 more efficiently and discard unnecessary information (as ECG recordings often have a significant amount of redundant information). In some embodiments, during training, the processing device 140B may randomly zero out individual leads with a probability of 10% to make the ECG encoder 147 more robust to the effects of bad lead contact and / or lost or corrupted lead data. Entire leads are dropped out to encourage the ECG encoder 147 to learn lead-independent features rather than strongly correlating its output to one "best" lead or a subset of "best" leads. The processing device 140B may train the ECG embedding projection layer 405, which may include a learnable linear transformation that transforms the output sequence of the ECG encoder 147 (ECG embedding) into the unified embedding space 410. The ECG embedding projection layer 405 may include a fully connected layer (not shown) that outputs vectors of length (the size of the unified embedding space 410) 256. The ECG embedding projection layer 405 may also divide the output vector by its Euclidean normal (i.e., L2-normalize the output vector) so that the output vector is always a unit-length vector.

[0034] As shown in FIG. 4, the shape of the output of the text encoder 145 is a [16 x 128] array representing 16 embedding vectors, each of length 128. The processing device 140B may train the text embedding projection layer 415 to convert the output sequence (text embedding) of the text encoder 145 to the unified embedding space 410. The text embedding projection layer 415 may include a fully connected layer (not shown) that requires a one-dimensional input, and therefore the text embedding projection layer 415 may flatten the output of the text encoder 145 to make it one-dimensional. The fully connected layer may convert the output sequence (text embedding) of the text encoder 145 to the unified embedding space 410 by outputting a vector of length 256 (the size of the unified embedding space 410). The text embedding projection layer 405 may also L2-normalize the output vector so that the output vector is always a unit-length vector. For every ECG in training data 155, text encoder 145 may take as input the corresponding sequence of diagnostic codes and generate a text embedding (as described above), and ECG encoder 147 may take as input the corresponding raw lead data and generate an ECG embedding (as described above). Processing device 140B may project the text embedding and ECG embedding (both L2 normalized to unit length) into the joint embedding space 410 using their respective fully connected layers (ECG embedding projection layer 405 and text embedding projection layer 415).

[0035] The joint embedding space 410 is where the processing device 140B (running the ECG encoder training module 141) may apply a loss function to train the ECG encoder 147 so that the ECG encoder 147 can learn to match ECG embeddings with corresponding text embeddings. FIG. 5 shows a matrix that represents the joint embedding space 410. For each training ECG, the text encoder 145 may encode a sequence of diagnosis codes representing the diagnosis of the training ECG into a text embedding, and the ECG encoder 147 may encode the raw lead data of the training ECG into an ECG embedding. As shown in FIG. 5, the text embeddings of each training ECG are T1 through T2 across rows 505. N and the ECG embeddings of each training ECG are represented as l1 to l1 in column 510. N The processing device 140B may utilize a loss function to train the ECG encoder 147 to maximize the similarity between matching ECGs and text embeddings and minimize the similarity between dissimilar ECGs and text embeddings, thereby simultaneously training the ECG encoder 147 and the text embedding projection layer. More specifically, for each training ECG, the processing device 140B may employ the dot product of the corresponding ECG embedding and text embedding. The training objective is to make all diagnosis entries (corresponding to the dot product between the matching text and ECG embeddings) as close to 1 as possible. Training the ECG encoder 147 as described above may enable the ECG encoder 147 to map pairs of similar ECGs to embeddings with high similarity and pairs of dissimilar ECGs to embeddings with low similarity.

[0036] Upon receiving a query ECG from a user (e.g., via the local computing device 120 as described herein), the processing device 140B may execute the ECG search module 143 to utilize the trained ECG encoder 147 to perform an ECG search. FIG. 6 illustrates the process of performing an ECG search. The processing device 140B may prepare a searchable database 605 of ECG embeddings for each ECG in the ECG database 160 (including textual representations of multiple previously recorded ECGs and their diagnoses) by using the ECG encoder 147 to create an ECG embedding for each ECG in the ECG database 160. Note that the ECG database 160 may include previous ECGs of a particular patient for analysis purposes as described herein with respect to FIGS. 6B and 6C. As shown in FIG. 6A, the database 605 may include a file name, a diagnosis, and an ECG embedding for each of the ECGs in the ECG database 160. To search the database 605 for ECGs similar to the query ECG, the query ECG is encoded by the ECG encoder 147 to create a query embedding.

[0037] The processing device 140B (running the ECG search module 143) may calculate a similarity score between the query embedding and each ECG embedding in the database 605. The ECG embedding vectors may have 256 components and be normalized to unit length, and pairs of vectors may be compared using vector dot products as a metric. In this way, the processing device 140B may use the dot product between the query embedding and the ECG embedding as a similarity score and may calculate a similarity score for the query embedding and each ECG embedding. In some embodiments, the similarity scores may be calculated quickly and in parallel using a distributed query engine such as Presto or Spark. The processing device 140B may sort the records in descending order based on the similarity score and may display / visualize the top K results (or send them to the local computing device 120 for display / visualization).

[0038] The use of dual embedded models to perform enhanced ECG searches may be used in a variety of ways. In one example, embodiments of the present disclosure may be used to find ECGs that resemble a selected ECG (e.g., to determine if a particular patient has previously had an ECG like the selected ECG). In another example, embodiments of the present disclosure may be used to identify trends, changes, and / or seasonality in a particular patient's cardiac health (e.g., to determine if the ECG is normal for a particular patient or if there has been a change in the patient's ECG that requires further analysis). In line with these examples, in some embodiments, the processing device 140B may generate and display a timeline view of a patient's ECG history, which may enable a user (e.g., a doctor or nurse) to quickly identify an ECG of interest. The user may select one ECG or a pair displayed below the timeline, either as a single ECG or two ECGs side-by-side for comparison. 6B and 6C show different "modes" of the timeline view, where each circle icon may represent an ECG recording, the X-axis may represent time, and the Y-axis may represent ECG embeddings (including those based on ECGs in the searchable database 605 and query embeddings based on a query ECG) projected in one dimension. The two modes shown in Figures 6B and 6C differ in how the Y-axis is defined, as described in more detail herein.

[0039] A timeline view of a patient's ECG recordings may be used for serial comparisons, with the first step being to determine whether a significant change has occurred in the rhythm and / or morphology of the patient's ECG recordings. A threshold for significant change may be established from the correlation of the dual embedded variables. If the correlation is higher than the threshold, there is no significant change between the current ECG and the reference ECG, and therefore the interpretation status remains unchanged. If the correlation is lower than the threshold, this may indicate that some significant change has occurred. Further analysis of the ECG parameters and embedded variables may define what type of change has occurred, such as ST-T changes in the case of ACS, or QRS duration changes in the case of bundle branch block.

[0040] 6B shows the timeline view in "ECG-ECG" mode, in which the Y-axis may represent the similarity score between each ECG in the searchable database 605 and the query (reference) ECG (i.e., the similarity score between their respective embeddings). The most recent (i.e., query) ECG may be the default reference ECG, but the user may designate a new reference ECG by interacting with (e.g., double-clicking on) the appropriate icon of the ECG they wish to designate as the reference ECG. All other ECGs on the timeline (other than the reference ECG) may be vertically arranged according to their similarity to the reference ECG.

[0041] FIG. 6C shows the timeline view in “ECG-TEXT” mode, in which the Y-axis may represent the similarity score between the textual representation of the diagnosis and each ECG in the searchable database 605 (i.e., the similarity score between their respective text / ECG embeddings). The user may select or type a diagnosis (e.g., “Atrial Fibrillation” shown in the example of FIG. 6C) in the provided data entry area 630, and each ECG in the ECG database 160 may be projected on the Y-axis based on their similarity to the given diagnosis. The text embedding for “Normal Sinus Rhythm” may be mapped to 0 on the Y-axis by default. This allows the user to immediately spot potentially abnormal ECGs as well as understand trends at a glance. As can be seen in FIG. 6C, the height of each ECG in the searchable database 605 on the Y-axis may correspond to the dot product of the ECG embedding of that ECG and the (similarity score between) the text embedding of the entered diagnosis.

[0042] There may be situations where it is desirable to be able to focus on a particular number of conditions when performing an ECG search. For example, a data scientist may wish to search a database of ECGs for records that may have a diagnosis. In another example, a physician may wish to search a patient's ECG history, which is particularly relevant in the context of a mobile / home ECG user who often has many unlabeled ECGs. However, because the ECG encoder 147 is trained based on matching text embeddings without reference to a specific classification goal, as explained above, execution of the ECG search module 143 may yield results that are more general (and not focused on a particular condition). Thus, in some embodiments, the processing device 140B may execute the classification module 142 (instead of the ECG search module 143) to further train the classifier 149 to classify ECG search results that meet the particular conditions of the classification that the user is attempting, as shown in FIG. 7. FIG. 7 illustrates a process of training the classifier 149 to classify ECG search results that focus on the particular conditions of the classification that the user is attempting. Processing device 140B (running classification module 142) may use the output of ECG encoder 147 as input for classifier 149 to train it. Classifier 149 may be a simple ML model. For example, in some embodiments, classifier 149 may simply perform linear regression based on input data from ECG encoder 147. Classifier 149 may be ideal in situations where it can be trained on a smaller sample set of high quality data (i.e., well-labeled data). Indeed, such an embodiment may enable processing device 140B to use pre-trained ECG encoder 147 as a backbone for transfer learning. More specifically, ECG encoder may pre-process a small sample of ECG data to generate ECG embeddings. A small-scale classifier may thus be trained on (embedding, label) pairs.This requires less training data and uses fewer parameters than training directly on the (ECG, label) pairs.

[0043] 8A is a flow diagram of a method 800 for performing an ECG search, according to some embodiments of the present disclosure. Method 800 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, processor, processing device, central processing unit (CPU), system on chip (SoC), etc.), software (e.g., instructions running / executed on a processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments, method 800 may be performed by a computing device (e.g., cloud service system 140 shown in FIG. 2).

[0044] 2 and with further reference to FIG. 3 , method 800 begins at block 805 with execution of ECG encoder training module 141, where processing device 140B may train text encoder 145 (shown in FIG. 3 ) in a semi-supervised manner based on training data 150 to learn representation functions (referred to herein as text embeddings) that transform textual representations of ECG diagnoses into vectors in an embedding space. Training data 150 may include textual representations of each of a plurality of ECG diagnoses, as described in further detail herein. Text encoder 145 may be any suitable ML model capable of extracting learnable, transformable representations from sequences, such as, for example, the Bidirectional Encoder Representation by Transformers (BERT) sequence-to-sequence model.

[0045] FIG. 3 illustrates a process of training the text encoder 145. The text encoder 145 may receive as input a first sequence of diagnostic codes from the training data 150 corresponding to ECG diagnoses. The first sequence of diagnostic codes may be [19, 221, 1766, 1687, 0, 0, 0...], representing "sinus rhythm, series of supraventricular extra complexes, otherwise normal ECG". The processing device 140B may remove extraneous information from the sequence of diagnostic codes, such as free text portions and dates. The number of diagnostic codes / sequence lengths of the input and output sequences are set to 16 as shown in FIG. 3 and may be selected arbitrarily. As can be seen, the first sequence of diagnostic codes may be padded with zeros to fill the 16 code length (may be [19, 221, 1766, 1687, 0, 0, 0...]). In some embodiments, the number of diagnostic codes in the input and output sequences may be selected to obtain ideal performance.

[0046] The text encoder 145 may learn to encode a sequence of diagnostic codes into vectors in the embedding space (text embedding) by training on a masked prediction task. Thus, for a first sequence of diagnostic codes, the processing device 140B randomly removes the diagnostic code from the sequence and encodes it as <mask>3, the diagnostic code representing "supraventricular extra complex" is masked, and the text encoder 145 is trained (e.g., via a classifier layer of the text encoder 145) to predict the masked diagnostic code by outputting a sequence of probability distributions over possible diagnostic codes that can fit into the masked space. Since most masked prediction examples have multiple possible completions, it is reasonable to expect that given sufficient sequences of diagnostic codes from the training data 150, the text encoder 145 (e.g., via a conditional dependency layer) will learn a representation function that captures the conditional dependencies between the different diagnostic codes and encodes the similarity of the ECG diagnoses involved. Continuing with the example of FIG. 3, the text encoder 145 may come to understand that while two or more diagnostic codes could potentially appear after "sinus rhythm," the phrase "sequence of" indicates a high probability of a potential VTAC, and thus the mask is most likely to be filled with "ventricular extra complex." <mask>Other words that could fit into the token are assigned lower probabilities. Note that what the text encoder 145 is actually learning is the probability distribution of different codes that can fit into the masked slots that ultimately informs how the sequence of diagnostic codes is understood / interpreted.

[0047] Once training of the text encoder 145 is complete, the text encoder 145 may receive a sequence of diagnosis codes and output a sequence of vectors (consecutive real numbers) that capture all the diagnosis information a doctor or healthcare professional needs, such that similar diagnoses are near each other in the embedding space.

[0048] In block 810, processing device 140B may train ECG encoder 147 to learn a representation function (called an "ECG embedding") that transforms ECG recordings (e.g., 10-second, 12-lead) into vectors in the embedding space. ECG encoder 147 has the same properties as text encoder 145 in that ECGs with similar diagnoses will be pushed into the same region of the embedding space, and ECGs with different diagnoses will be pushed into different regions. Thus, processing device 140B may train ECG encoder 147 using the joint embedding space between ECG recordings and the text representations of the corresponding diagnoses. To do this, processing device 140B may use the representation function learned by text encoder 145 to supervise the training of ECG encoder 147. However, processing device 140B may utilize a flexible form of supervision that simply uses the text embeddings as a starting point for learning the joint embeddings. The processing device 140B may train the ECG encoder 147 using training data 155, which may include ECG recordings (i.e., raw lead data) and textual representations of each corresponding diagnosis for multiple ECGs (i.e., sequences of diagnosis codes).

[0049] FIG. 4 illustrates training of ECG encoder 147. ECG encoder 147 may receive ECG records (from training data 155) that contain raw ECG lead data and may utilize multiple layers, with each layer downsampling the raw ECG lead data and adding more information / channels. This concept is sometimes referred to as an information bottleneck, where information is passed through a smaller space, thereby forcing ECG encoder 147 to learn how to represent that information more efficiently and discard information that is extraneous or irrelevant. In this manner, processing device 140B may train ECG encoder 147 to learn how to represent the raw lead data of each ECG in training data 155 more efficiently and discard unnecessary information (as ECG records often have a significant amount of redundant information). For every ECG in training data 155, text encoder 145 may take as input the corresponding sequence of diagnostic codes and generate a text embedding, and ECG encoder 147 takes as input the corresponding raw lead data and generates an ECG embedding. Processing device 140B may project the text embeddings and the ECG embeddings (both L2 normalized to unit length) into a joint embedding space 410 using a fully connected layer.

[0050] The joint embedding space 410 is where the processing device 140B (running the ECG encoder training module 141) may apply a loss function to train the ECG encoder 147 so that the ECG encoder 147 can learn to match ECG embeddings with corresponding text embeddings. FIG. 5 shows a matrix that represents the joint embedding space 410. For each training ECG, the text encoder 145 may encode a sequence of diagnosis codes representing the diagnosis of the training ECG into a text embedding, and the ECG encoder 147 may encode the raw lead data of the training ECG into an ECG embedding. As shown in FIG. 5, the text embeddings of each training ECG are T1 through T2 across rows 505. N and the ECG embeddings of each training ECG are represented as l1 to l1 in column 510. N The processing device 140B may utilize a loss function to train the ECG encoder 147 to maximize the similarity between matching ECGs and text embeddings and minimize the similarity between dissimilar ECGs and text embeddings, thereby simultaneously training the ECG encoder 147 and the text embedding projection layer. More specifically, for each training ECG, the processing device 140B may employ the dot product of the corresponding ECG embedding and text embedding. The training objective is to make all diagnosis entries (corresponding to the dot product between the matching text and ECG embeddings) as close to 1 as possible. Training the ECG encoder 147 as described above may enable the ECG encoder 147 to map pairs of similar ECGs to embeddings with high similarity and pairs of dissimilar ECGs to embeddings with low similarity.

[0051] In block 815, the processing device 140B may prepare a searchable database 605 of ECG embeddings for each ECG in the ECG database 160 (including a textual representation of multiple previously recorded ECGs and their diagnoses) by using the ECG encoder 147 to create an ECG embedding for each ECG in the ECG database 160. As shown in FIG. 6, the database 605 may include a file name, a diagnosis, and an ECG embedding for each ECG in the ECG database 160. In block 820, upon receiving a query ECG, the query ECG is encoded by the ECG encoder 147 to create a query embedding. In block 825, the processing device 140B (running the ECG search module 143) may calculate a similarity score between the query embedding and each ECG embedding in the database 605. The ECG embedding vectors have 256 components and are normalized to unit length, and pairs of vectors may be compared using vector dot products as a metric. In this manner, processing device 140B may use the dot product between the query embedding and the ECG embedding as a similarity score and may calculate a similarity score for the query embedding and each ECG embedding. In some embodiments, the similarity scores may be calculated quickly and in parallel using a distributed query engine such as Presto or Spark. Processing device 140B may sort the ECG embeddings in descending order based on the similarity score and may display / visualize the top K results (or return them to local computing device 120 for display / visualization).

[0052] 8B is a flow diagram of a method 850 of performing ECG classification according to some embodiments of the present disclosure. The method 850 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, processor, processing device, central processing unit (CPU), system on chip (SoC), etc.), software (e.g., instructions running / executed on a processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments, the method 850 may be performed by a computing device (e.g., cloud service system 140 shown in FIG. 3).

[0053] There may be situations where it is desirable to be able to focus on a specific number of conditions when performing an ECG search. However, because the ECG encoder 147 is trained based on matching text embeddings without reference to a specific classification goal, as described above, execution of the ECG search module 143 may result in more general (and not focused on specific conditions) results. The method 850 may begin with blocks 855 and 860, which are similar to blocks 805 and 810 described above with respect to FIG. 8A. In block 865, the processing device 140B may execute the classification module 142 (instead of the ECG search module 143) to further train the classifier 149 to classify ECG search results that meet the specific conditions of the classification that the user is attempting, as shown in FIG. 7. FIG. 7 illustrates a process of training the classifier 149 to classify ECG search results that focus on the specific conditions of the classification that the user is attempting. Processing device 140B (running classification module 142) may use the output of ECG encoder 147 as input for classifier 149 to train it. Classifier 149 may be a simple ML model. For example, in some embodiments, classifier 149 may simply perform linear regression based on the input data from ECG encoder 147. Classifier 149 may be ideal in situations where it can be trained on a smaller sample set of high quality data (i.e., well-labeled data).

[0054] FIG. 9 shows a diagrammatic representation of a machine in the example form of a computer system 900, in which a set of instructions are for causing the machine to perform any one or more of the methods described herein for performing an ECG search.

[0055] In alternative embodiments, the machine may be connected (networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch or bridge, a hub, an access point, a network access control device, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine. Furthermore, although merely one machine is described, the term "machine" shall also be considered to include any collection of machines that individually or together execute a set (or sets) of instructions to perform any one or more of the methods described herein. In one embodiment, the computer system 900 may represent a server.

[0056] The exemplary computer system 900 includes a processing device 902, a main memory 904 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM)), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 918, which communicate with each other via a bus 930. Any of the signals provided via the various buses described herein may be time multiplexed with other signals and provided via one or more common buses. In addition, the interconnections between circuit components or blocks may be depicted as buses or single signal lines. Each of the buses may alternatively be one or more single signal lines, and each of the single signal lines may alternatively be a bus.

[0057] Computing device 900 may further include a network interface device 908 that may communicate with a network 920. Computing device 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), and an acoustic signal generating device 916 (e.g., a speaker). In one embodiment, the video display unit 910, the alphanumeric input device 912, and the cursor control device 914 may be combined into a single component or device (e.g., an LCD touch screen).

[0058] The processing device 902 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processing device 902 may also be one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The processing device 902 is configured to execute ECG search instructions 925 to perform the operations and steps described herein.

[0059] The data storage device 915 may include a machine-readable storage medium 928 on which one or more sets of ECG retrieval instructions 925 (e.g., software) embodying any one or more of the methods of functionality described herein are stored. The ECG retrieval instructions 925 may also reside, completely or at least partially, within the main memory 904 or within the processing device 902 during execution thereof by the computer system 900, with the main memory 904 and the processing device 902 also constituting machine-readable storage media. The ECG retrieval instructions 925 may further be transmitted or received over the network 920 via the network interface device 908.

[0060] Although the machine-readable storage medium 928 is shown to be a single medium in the exemplary embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media (e.g., centralized or distributed databases, or associated caches and servers) that store one or more sets of instructions. A machine-readable medium includes any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). A machine-readable medium may include, but is not limited to, magnetic storage media (e.g., floppy diskettes), optical storage media (e.g., CD-ROMs), magneto-optical storage media, read-only memory (ROM), random access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or another type of medium suitable for storing electronic instructions.

[0061] The above description shows numerous specific details, such as examples of specific systems, components, methods, etc., to enable a thorough understanding of some embodiments of the present disclosure. However, it will be apparent to those skilled in the art that at least some embodiments of the present disclosure can be practiced without these specific details. In other cases, well-known components or methods are not described in detail or are presented in simple block diagram form to avoid unnecessarily obscuring the present disclosure. Thus, the specific details shown are examples only. Certain embodiments may differ from these illustrative details and still be considered to be within the scope of the present disclosure.

[0062] Additionally, some embodiments may be practiced in a distributed computing environment in which a machine-readable medium is stored on and / or executed by two or more computer systems. Additionally, information communicated between computer systems may be pulled or pushed across a communications medium connecting the computer systems.

[0063] Embodiments of the claimed subject matter include, but are not limited to, various operations described herein. These operations may be performed by hardware components, software, firmware, or any combination thereof.

[0064] Although the operations of the methods herein are shown and described in a particular order, the order of operations of each method may be changed such that some operations may be performed in reverse order or some operations may be performed, at least in part, simultaneously with other operations. In alternative embodiments, separate operation instructions or sub-operations may be intermittent or alternating.

[0065] The above description of illustrated implementations of the invention, including those described in the Abstract, are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Specific implementations, and examples, of the invention have been described herein for purposes of illustration, and those skilled in the art will recognize that various equivalent modifications are possible within the scope of the invention. The words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other aspects or designs. Instead, use of the words "example" or "exemplary" is intended to illustrate a concept. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any of the natural inclusive permutations. That is, if X includes A, or X includes B, or X includes both A and B, then "X includes A or B" is satisfied under any of the foregoing cases. In addition, the articles "a" and "an" used in this application and the appended claims should generally be construed to mean "one or more" unless otherwise specified or unless the singular is intended from the context. Moreover, the use of the terms "embodiment" or "one embodiment" or "implementation" or "one implementation" throughout does not refer to the same embodiment or implementation unless so stated. Furthermore, the terms "first", "second", "third", "fourth", etc., used herein are meant as labels to distinguish between different elements and may not necessarily have a sequential meaning according to their numerical designation.

[0066] It will be appreciated that variations of the above-disclosed and other features and functions, or alternatives thereof, may be combined into any other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations, or improvements therein may subsequently be made by those skilled in the art, which are also intended to be encompassed by the following claims. The claims may encompass hardware, software, or combinations thereof. [Explanation of symbols]

[0067] 100 Systems 101 Mobile Software Applications 110 ECG Monitor 120 Local Computing Devices 130 Network 140 Cloud Service System 140A Memory 140B Processing Device 141 ECG Encoder Training Module 142 Classification Module 143 ECG Search Module 145 Text Encoder 147 ECG Encoder 149 Classifier 150 training data 155 training data 160 ECG database 405 ECG Embedded Projection Layer 410 Integrated Embedded Space 415 Text-embedded projection layer 605 ECG database 902 Processing Device 925 ECG retrieval command 904 Main memory 906 Static Memory 908 Network Interface Device 920 Network 910 Video Display 912 Alphanumeric Input Device 930 Bus 914 Cursor Control Device 916 Signal Generating Device 918 Data Storage Devices 928 Machine-readable storage medium< / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. training a text machine learning (ML) model to generate text embeddings based on the received text representation of the ECG diagnosis; using the text ML model to train an ECG encoding ML model to generate ECG embeddings based on the received ECG lead data such that ECG embeddings generated from similar ECG lead data are close to each other in vector space; populating a database with a plurality of ECG embeddings, each of the plurality of ECG embeddings being generated based on ECG lead data of a previously diagnosed ECG; in response to receiving a query ECG, generating a query embedding using the ECG model; determining a similarity score between the query embedding and each of the plurality of ECG embeddings; A method comprising:

2. ranking each of the plurality of ECG embeddings based on a respective similarity score between the ECG embedding and the query embedding; displaying in descending order the number of ECG embeddings having the highest similarity scores; The method of claim 1, further comprising:

3. training the ECG ML model using the text ML model, generating, by the text ML model, training text embeddings based on a text representation of a diagnosis for each of a plurality of training ECGs to create a plurality of training text embeddings; generating, with the ECG ML model, a training ECG embedding based on induction data for each of the plurality of training ECGs to create a plurality of training text embeddings; projecting the plurality of training ECG embeddings and the plurality of training text embeddings into a joint embedding space; utilizing a loss function to train the ECG ML model to match each training ECG embedding with a corresponding training text embedding; 2. The method of claim 1, comprising:

4. Utilizing the loss function to train the ECG ML model, 4. The method of claim 3, comprising: for each of the plurality of training ECG embeddings, determining a dot product of the training ECG embedding with each of the training text embeddings, wherein the ECG ML model will make the dot product of a training ECG embedding and a training text embedding of the training ECG as close to 1 as possible.

5. The method of claim 1 , wherein the textual representation of the ECG diagnosis comprises a set of diagnosis codes.

6. Training the text ML model comprises: For each textual representation of a number of training ECG diagnoses, randomly removing diagnostic codes from the corresponding set of diagnostic codes of the training ECG diagnoses and replacing the removed diagnostic codes with mask tokens; training the text ML model to predict the deleted diagnostic code based on the remaining diagnostic codes of the set of diagnostic codes; 6. The method of claim 5, comprising:

7. The method of claim 1 , wherein the text ML model is a Bidirectional Encoder Representation with Transformers (BERT) model.

8. an electrocardiogram (ECG) monitor for recording ECG data of a user; A cloud service system, comprising: training a text machine learning (ML) model to generate text embeddings based on the received text representation of the ECG diagnosis; training an ECG encoding ML model using the text ML model to generate ECG embeddings based on the received ECG lead data such that ECG embeddings generated from similar ECG lead data are close to each other in vector space; populating a database with a plurality of ECG embeddings, each of the plurality of ECG embeddings being generated based on ECG lead data of a previously diagnosed ECG; in response to receiving a query ECG, generating a query embedding using the ECG model; determining a similarity score between the query embedding and each of the plurality of ECG embeddings; A cloud service system for carrying out A system comprising:

9. The cloud service system, ranking each of the plurality of ECG embeddings based on a respective similarity score between the ECG embedding and the query embedding; Displaying in descending order the number of ECG embeddings with the highest similarity scores; The system of claim 8, further comprising:

10. To train the ECG ML model using the text ML model, the cloud service system: generating, by the text ML model, training text embeddings based on a text representation of a diagnosis for each of a plurality of training ECGs to create a plurality of training text embeddings; generating a training ECG embedding based on induction data for each of the plurality of training ECGs to create a plurality of training text embeddings with the ECG ML model; projecting the plurality of training ECG embeddings and the plurality of training text embeddings into a joint embedding space; using a loss function to train the ECG ML model to match each training ECG embedding with a corresponding training text embedding; The system of claim 8,

11. To utilize the loss function to train the ECG ML model, the cloud service system:

11. The system of claim 10, wherein for each of the plurality of training ECG embeddings, a dot product of the training ECG embedding with each of the training text embeddings is determined, and the ECG ML model is configured to make the dot product of a training ECG embedding and a training text embedding of the training ECG as close to 1 as possible.

12. The system of claim 8 , wherein the textual representation of the ECG diagnosis includes a set of diagnosis codes.

13. To train the text ML model, the cloud service system: For each textual representation of a number of training ECG diagnoses, randomly removing diagnostic codes from a corresponding set of diagnostic codes of the training ECG diagnoses and replacing the removed diagnostic codes with mask tokens; training the text ML model to predict the deleted diagnostic code based on the remaining diagnostic codes of the set of diagnostic codes; The system of claim 12,

14. 10. The system of claim 8, wherein the text-ML model is a Bidirectional Encoder Representation with Transformers (BERT) model.

15. A non-transitory computer readable medium having instructions stored thereon, the instructions, when executed by a processing device, causing the processing device to: training a text machine learning (ML) model to generate text embeddings based on the received text representation of the ECG diagnosis; training an ECG encoding ML model using the text ML model to generate ECG embeddings based on the received ECG lead data such that ECG embeddings generated from similar ECG lead data are close to each other in vector space; populating a database with a plurality of ECG embeddings, each of the plurality of ECG embeddings being generated based on ECG lead data of a previously diagnosed ECG; in response to receiving a query ECG, generating a query embedding using the ECG model; determining a similarity score between the query embedding and each of the plurality of ECG embeddings; A non-transitory computer-readable medium for causing

16. the processing device comprising: ranking each of the plurality of ECG embeddings based on a respective similarity score between the ECG embedding and the query embedding; Displaying in descending order the number of ECG embeddings with the highest similarity scores; 16. The non-transitory computer-readable medium of claim 15, further comprising:

17. To train the ECG ML model using the text ML model, the processing device: generating, by the text ML model, training text embeddings based on a text representation of a diagnosis for each of a plurality of training ECGs to create a plurality of training text embeddings; generating a training ECG embedding based on induction data for each of the plurality of training ECGs to create a plurality of training text embeddings with the ECG ML model; projecting the plurality of training ECG embeddings and the plurality of training text embeddings into a joint embedding space; using a loss function to train the ECG ML model to match each training ECG embedding with a corresponding training text embedding; 16. The non-transitory computer-readable medium of claim 15,

18. To utilize the loss function to train the ECG ML model, the processing device:

20. The non-transitory computer-readable medium of claim 17, wherein for each of the plurality of training ECG embeddings, a dot product of the training ECG embedding with each of the training text embeddings is determined, and the ECG ML model causes the dot product of a training ECG embedding and a training text embedding of the training ECG to be as close to 1 as possible.

19. 16. The non-transitory computer-readable medium of claim 15, wherein the textual representation of the ECG diagnosis includes a set of diagnosis codes.

20. To train the text ML model, the processing device: For each textual representation of a number of training ECG diagnoses, randomly removing diagnostic codes from a corresponding set of diagnostic codes of the training ECG diagnoses and replacing the removed diagnostic codes with mask tokens; training the text ML model to predict the deleted diagnostic code based on the remaining diagnostic codes of the set of diagnostic codes; 20. The non-transitory computer-readable medium of claim 19,

Images