Continuous monitoring of user health using mobile devices

Abstract

Disclosed herein are devices, systems, methods, and platforms for continuously monitoring a user's health status, e.g., cardiac health status. The present disclosure describes systems, methods, devices, software, and platforms for continuously monitoring a user's health indicator data (e.g., without limitation, PPG signal, heart rate, or blood pressure) from a user device in combination with corresponding (temporal) data related to factors ("other factors") that may affect the health indicator to determine whether the user has normal health when judged by or compared to either (i) a group of individuals affected by similar other factors, or (ii) the user himself / herself affected by similar other factors.

Description

[Technical field]

[0001] The present invention relates to continuous monitoring of a user's health using mobile devices. [Background technology]

[0002] Indicators of an individual's physiological health ("health indicators")--for example, but not limited to, heart rate, heart rate variability, blood pressure, and ECG (electrocardiogram) are some examples--can be measured or calculated at discrete times from data collected to measure the health indicator. In many cases, the value of a health indicator at a particular time, or changes over time, provide information about the state of an individual's health. For example, a low or high heart rate or blood pressure, or an ECG clearly indicating myocardial ischemia, may indicate the need for immediate intervention. However, a reading, a series of readings, or changes in the readings of these indicators over time may provide information that is not recognized by the user or even by medical personnel when attention is needed.

[0003] For example, arrhythmias may occur continuously or intermittently. Continuous arrhythmias may be most reliably diagnosed from an individual's electrocardiogram. Since continuous arrhythmias are always present, ECG analysis may be applied at any time to diagnose the arrhythmia. ECG may also be used to diagnose intermittent arrhythmias. However, since intermittent arrhythmias may be asymptomatic and / or are intermittent by definition, diagnosis presents the challenge of applying diagnostic techniques when an individual is experiencing the arrhythmia. Thus, the actual diagnosis of intermittent arrhythmias is notoriously difficult. This particular difficulty is compounded with asymptomatic arrhythmias, which account for nearly 40% of arrhythmias in the United States. Boriani G. and Pettorelli D., "Atrial Fibrillation Burden and Atrial Fibrillation type: Clinical Significance and Impact on the Risk of Stroke and Decision Making for Long-term Anticoagulation," Vascul Pharmacol., 83:26-35 (August 2016), p. 26.

[0004] Sensor and mobile electronics technologies exist that allow health indicators to be monitored and recorded frequently or continuously. However, the capabilities of these sensor platforms often exceed the ability of traditional medical science to interpret the data they generate. The physiological meaning of health indicator parameters, such as heart rate, is often only clear in a specific medical context; for example, heart rate is traditionally evaluated as a single scalar value out of context from other data / information that may affect the health indicator. A resting heart rate within the range of 60-100 beats per minute (BPM) may be considered normal. A user may typically measure resting heart rate manually once or twice a day.

[0005] Mobile sensor platforms (e.g., mobile blood pressure cuffs, mobile heart rate monitors, mobile ECG devices) may be capable of continuously monitoring health indicators (e.g., heart rate), for example generating measurements every second or every five seconds while simultaneously acquiring other data about the user, such as, without limitation, activity level, body position, and environmental parameters such as temperature, pressure, position, etc. Over a 24-hour period, this can result in thousands of independent health indicator measurements. There is relatively little data or medical consensus on what a "normal" series of thousands of measurements looks like, as opposed to one or two measurements per day. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 9,420,956 [Patent Document 2] U.S. Patent No. 9,572,499 [Patent Document 3] U.S. Patent No. 9,351,654 [Patent Document 4] U.S. Patent No. 9,247,911 [Patent Document 5] U.S. Patent No. 9,254,095 [Patent Document 6] U.S. Patent No. 8,509,882 [Patent Document 7] US Patent Application Publication No. 2015 / 0018660 [Patent Document 8] US Patent Application Publication No. 2015 / 0297134 [Patent Document 9] US Patent Application Publication No. 2015 / 0320328 [Non-patent literature]

[0007] [Non-Patent Document 1] Boriani G. and Pettorelli D., "Atrial Fibrillation Burden and Atrial Fibrillation type: Clinical Significance and Impact on the Risk of Stroke and Decision Making for Long-term Anticoagulation," Vascul Pharmacol., 83:26-35 (August 2016), p. 26 Summary of the Invention [Problem to be solved by the invention]

[0008] Devices currently used to continuously measure a user / patient's health indicators range from bulky, invasive and cumbersome devices to simple wearable or handheld mobile devices. Currently, these devices do not provide the ability to effectively utilize the data to continuously monitor a person's health. It is left to the user or a medical professional to evaluate the health indicators taking into account other factors that may affect these health indicators to determine the user's health status. [Means for solving the problem]

[0009] Certain features described herein are set forth with particularity in the appended claims. The features and advantages of the disclosed embodiments will be better understood by reference to the following detailed description that sets forth illustrative embodiments, in which the principles described herein are utilized, and the accompanying drawings, in which: [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1 illustrates a convolutional neural network that may be used by some embodiments described herein. [Figure 1B]FIG. 1 illustrates a convolutional neural network that may be used by some embodiments described herein. [Figure 2A] FIG. 1 illustrates a recurrent neural network that may be used by some embodiments described herein. [Figure 2B] FIG. 1 illustrates a recurrent neural network that may be used by some embodiments described herein. [Diagram 3] FIG. 1 illustrates another recurrent neural network that may be used with some embodiments described herein. [Figure 4A] 1 is a hypothetical data plot to demonstrate the application of some embodiments described herein. [Figure 4B] 1 is a hypothetical data plot to demonstrate the application of some embodiments described herein. [Figure 4C] 1 is a hypothetical data plot to demonstrate the application of some embodiments described herein. [Figure 5A] FIG. 1 illustrates another recurrent neural network according to some embodiments described herein and hypothetical plots used to explain some of these embodiments. [Figure 5B] FIG. 1 illustrates another recurrent neural network according to some embodiments described herein and hypothetical plots used to explain some of these embodiments. [Figure 5C] FIG. 1 illustrates another recurrent neural network according to some embodiments described herein and hypothetical plots used to explain some of these embodiments. [Figure 5D] FIG. 1 illustrates another recurrent neural network according to some embodiments described herein and hypothetical plots used to explain some of these embodiments. [Figure 5E]FIG. 1 illustrates another recurrent neural network according to some embodiments described herein and hypothetical plots used to explain some of these embodiments. [Figure 6] FIG. 1 illustrates an unrolled recurrent neural network according to some embodiments described herein. [Figure 7A] FIG. 1 illustrates a system and device according to some embodiments described herein. [Figure 7B] FIG. 1 illustrates a system and device according to some embodiments described herein. [Figure 8] FIG. 1 illustrates a method according to some embodiments described herein. [Figure 9A] FIG. 1 shows a hypothetical plot of heart rate versus time to demonstrate one or more embodiments and methods according to certain embodiments described herein. [Figure 9B] FIG. 1 shows a hypothetical plot of heart rate versus time to demonstrate one or more embodiments and methods according to certain embodiments described herein. [Figure 10] FIG. 1 illustrates a method according to some embodiments described herein. [Figure 11] 1 is a hypothetical data plot to demonstrate the application of some embodiments described herein. [Figure 12] FIG. 1 illustrates a system and device according to some embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Large amounts of data, the complexity of interactions between health indicators and other factors, and limited clinical guidance may limit the effectiveness of monitoring systems that attempt to detect anomalies in continuous and / or ambulatory sensor data through specific rules based on traditional medical practice.Embodiments described herein include devices, systems, methods, and platforms that utilize predictive machine learning models and can detect anomalies in an unsupervised manner from time series of health indicator data alone or in combination with other factor (as defined herein) data.

[0012] Atrial fibrillation (AF or Afib) is present in 1-2% of the general population, and the presence of AF increases the risk of morbidity and adverse outcomes, such as stroke and heart failure. Boriani G. and Pettorelli D., "Atrial Fibrillation Burden and Atrial Fibrillation type: Clinical Significance and Impact on the Risk of Stroke and Decision Making for Long-term Anticoagulation," Vascul Pharmacol., 83:26-35 (August 2016), p. 26. Many people with AFib, some of which are estimated to be as high as 40% of AF patients, may be asymptomatic, and these asymptomatic patients have similar risk factors for stroke and heart failure as symptomatic patients. Id. However, symptomatic patients can take active measures, such as taking blood thinners or other medications, to reduce the risk of adverse outcomes. With the use of implantable electrical devices (CIEDs), it is possible to detect asymptomatic AF (so-called subclinical AF or SAF) and the duration for which a patient has AF. Id. From this information, the amount of time these patients spend in AF, i.e., AF burden, can be determined. Id. AF burdens greater than 5-6 minutes, and especially greater than 1 hour, are associated with significantly increased risk of stroke and other adverse health outcomes. Id. Thus, if the AF burden could be measured in asymptomatic patients, early interventional treatments could be undertaken, reducing the risk of adverse health outcomes associated with AF. Id. Detecting SAF is difficult and typically requires some form of continuous monitoring. Currently, continuous monitoring for AF requires bulky, sometimes invasive, expensive devices, and such monitoring requires a high level of oversight and scrutiny by medical professionals.

[0013] For example, many devices continuously acquire data to measure or calculate health index data, including but not limited to FitBit®, Apple Watch®, Polar®, smartphones, tablets, among others, in the class of wearable and / or mobile devices. Other devices include permanent or semi-permanent devices (e.g., Holter) that are attached to or placed on the user / patient, and other devices may include larger devices in hospitals that may be mobile by being on carts. However, little is done with this measurement data other than periodic observations on a display or establishing simple data thresholds. Observations of the data, even by trained medical professionals, can frequently appear normal, with one major exception being when the user has an easily identifiable acute symptom. It is tremendously difficult and practically impossible for medical professionals to continuously monitor health indexes and find outliers and / or trends in the data that may indicate something more significant.

[0014] As used herein, a platform comprises one or more customized software applications (or "applications") configured to interact with each other, either locally or through a distributed network, including the cloud and the Internet. The applications of the platform described herein are configured to collect and analyze user data and may include one or more software models. In some embodiments of the platform, the platform may comprise one or more hardware components (e.g., one or more sensing devices, or microprocessors). In some embodiments, the platform is configured to operate together with one or more devices and / or one or more systems. That is, the devices described herein are, in some embodiments, configured to execute the platform's applications using an embedded processor, and in some embodiments, the platform is utilized by a system comprising one or more computing devices that interact with or execute one or more applications of the platform.

[0015] This disclosure describes systems, methods, devices, software, and platforms for continuously monitoring a user's data (e.g., but not limited to, PPG signal, heart rate, or blood pressure) related to one or more health indicators from a user device in combination with corresponding (in time) data related to factors (herein referred to as "other factors") that may affect the health indicators to determine whether the user has normal health when judged or compared, for example, but not limited to, either (i) a group of individuals affected by similar other factors, or (ii) the user himself / herself affected by similar other factors. In some embodiments, the measured health indicator data is input into a trained machine learning model that, alone or in combination with other factor data, determines the probability that the user's measured health indicator is considered to be within a healthy range and notifies the user of this if it is not. A user that is not within the healthy range may increase the probability that the user may be experiencing a health event that warrants high fidelity information to confirm a diagnosis, such as an arrhythmia that the user may be symptomatic or asymptomatic. The notification may take the form of, for example, requesting the user to obtain an ECG. Other high fidelity measurements may be required, blood pressure, pulse oximetry to name two, ECG being but one example. The high fidelity measurement, in this embodiment the ECG, may be evaluated by an algorithm and / or a medical professional to provide an indication or diagnosis (collectively referred to herein as "diagnosis", recognizing that only a physician may provide a diagnosis). In the ECG example, the diagnosis may be AFib or any number of other well-known medical conditions diagnosed utilizing the ECG.

[0016] In further embodiments, the diagnosis is used to label a low-fidelity data series (e.g., heart rate or PPG), which may include other factor data series. This low-fidelity data series labeled with the high-fidelity diagnosis is used to train a high-fidelity machine learning model. In these further embodiments, the training of the high-fidelity machine learning model may be trained by unsupervised learning or may be updated with new training examples from time to time. In some embodiments, the user's measured low-fidelity health indicator data series, and optionally the corresponding (in time) data series of other factors, are input into a trained high-fidelity machine learning model, which determines the probability and / or prediction that the user is experiencing or has experienced the diagnosed medical condition for which the high-fidelity machine learning model was trained. This probability may include the probability of when the event will begin and when it will end. Some embodiments may, for example, calculate the user's atrial fibrillation (AF) burden, or the length of time the user will experience AF over time. Previously, AF burden could only be determined using cumbersome and expensive Holter or implanted continuous ECG monitoring devices. Thus, some embodiments described herein can continuously monitor a user's health status and notify the user of changes in health status by continuously monitoring health indicator data (e.g., but not limited to, PPG data, blood pressure data, and heart rate data) obtained from a device worn by the user, either alone or in combination with corresponding data for other factors. As used herein, "other factors" includes anything that may affect a health indicator and / or affect data representing a health indicator (e.g., PPG data). These other factors may include various factors such as, but not limited to, temperature, altitude, activity level, weight, sex, diet, standing, sitting, lying, lying, weather, and BMI, to name a few.In some embodiments, a mathematical or empirical model, rather than a machine learning model, may be used to determine when to notify a user to obtain a high-fidelity measurement result, which may then be analyzed and used to train a high-fidelity machine learning model as described herein.

[0017] Some embodiments described herein may detect outliers for a user in an unsupervised manner by receiving a primary time series of health indicator data, and optionally receiving one or more secondary time series of other factor data that correspond in time with the primary time series of health indicator data, where the secondary series may originate from a sensor or an external data source (e.g., via a network connection, computer API, etc.); providing the primary and secondary time series to a pre-processor that may perform operations such as filtering, caching, averaging, time alignment, buffering, upsampling, and downsampling on the data; providing the time series of data to a machine learning model that is trained and / or configured to utilize values ​​of the primary and secondary time series to predict a next value of the primary series at a future time; comparing the predicted primary time series value generated by the machine learning model at a particular time t with the measured value of the primary time series at time t; and alerting or prompting the user to take action if the difference between the predicted future time series and the measured time series exceeds a threshold or criterion.

[0018] Thus, some embodiments described herein detect when the observed behavior of a primary series of physiological data over time and / or in response to an observed secondary series of data differs from what would be expected given the training examples used to train the model. When the training examples are collected from normal individuals or from data that has already been classified as normal for a particular user, the system can act as an anomaly detector. If the data is simply obtained from a particular user without other classification, the system can act as a change detector and detect changes in the health indicator data that the primary series is measuring relative to the time the training data was captured.

[0019] Described herein are software platforms, systems, devices, and methods for generating trained machine learning models and using the trained machine learning models to predict or determine the probability when a user's measured health indicator data (primary series) under the influence of other factors (secondary series) is outside the normal limits for a healthy population under the influence of similar other factors (i.e., global model) or outside the normal limits for that particular user under the influence of similar other factors (i.e., personalized model), and a notification of such is sent to the user. In some embodiments, the user may be prompted to obtain additional measured high-fidelity data that can be used to generate a different trained high-fidelity machine learning model that has the ability to label the already obtained low-fidelity user health indicator data and predict or diagnose anomalies or events using only the low-fidelity health indicator data, where such anomalies are typically only identified or diagnosed using high-fidelity data.

[0020] Some embodiments described herein may include inputting a user's health indicator data and, optionally, inputting corresponding (in time) data of other factors into a trained machine learning model, which predicts the user's health indicator data or a probability distribution of health indicator data at future time steps. The prediction in some embodiments is compared to the user's measured health indicator data at the predicted time step, and if the absolute value of the difference exceeds a threshold, the user is notified that the health indicator data is outside of normal range. This notification may, in some embodiments, include a diagnosis or instruction to do something, such as, but not limited to, to obtain additional measurements or contact a medical professional. In some embodiments, the health indicator data and corresponding (in time) data of the other factors from a healthy population of people are used to train the machine learning model. It will be understood that the other factors in the training example used to train the machine learning model may not be population averages, but rather, the data for each of the other factors corresponds in time to the collection of health indicator data for the individuals in the training example.

[0021] Some embodiments are described as receiving discrete data points in time, predicting discrete data points at future times from the input, and then determining whether the loss between the discrete measured input at the future time and the predicted value at the future time exceeds a threshold. Those skilled in the art will readily appreciate that the input data and output predictions can take forms other than discrete data points or scalars. For example, but not by way of limitation, health indicator data series (also referred to herein as primary series) and other data series (also referred to herein as secondary series) can be divided into segments in time. Those skilled in the art will recognize that the manner in which the data is segmented is a matter of design choice and can take many different forms.

[0022] In some embodiments, the health indicator data series (also referred to herein as the primary series) and other data series (also referred to herein as the secondary series) are partitioned into two segments: past, which represents all data before a certain time t, and future, which represents all data at or after time t. In these embodiments, the health indicator data series for the past time segment and all other data series for that past time segment are input into a machine learning model configured to predict the most likely future segment (or distribution of likely future segments) of health indicator data. Alternatively, in these embodiments, the health indicator data series for the past time segment, all other data series for that past time segment, and other data series from the future segment are input into a machine learning model configured to predict the most likely future segment (or distribution of likely future segments) of health indicator data. The predicted future segment of health indicator data is compared with the user's measured health indicator data in the future segment to determine the loss and whether the loss exceeds a threshold, and if so, some action is taken. This action may include, for example, but is not limited to, notifying the user to obtain additional data (e.g., ECG or blood pressure), notifying the user to contact a medical professional, or automatically triggering the acquisition of additional data. Automatic acquisition of additional data may include, for example, but is not limited to, ECG acquisition via a sensor (wired or wireless) operably coupled to the user-worn computing device, or blood pressure via a mobile cuff worn around the user's wrist or other suitable body part and coupled to the user-worn computing device. A segment of data may include a single data point, multiple data points over a period of time, an average, median, or mode, which may include an exact average, of these data points over the period of time. In some embodiments, the segments may overlap in time.

[0023] These embodiments detect when the observed behavior or measurement of a health indicator series of data over time as affected by the corresponding (time-wise) other factor series of the data differs from that expected from training examples collected under similar other factors. If the training examples are collected from healthy individuals under similar other factors, or from data previously classified as healthy for a particular user under similar other factors, these embodiments act as anomaly detectors from a healthy population or a particular user, respectively. If the training examples are simply obtained from a particular user without other classification, these embodiments act as change detectors, detecting changes in the health indicator at the time of measurement with respect to the time the training examples were collected for the particular user.

[0024] In some embodiments described herein, machine learning is used to continuously monitor the health indicators of a person under the influence of one or more other factors, and evaluate whether the person is healthy in terms of a population classified as healthy under the influence of similar other factors. Those skilled in the art will readily understand that many different machine learning algorithms or models (including but not limited to Bayesian, Markovian, Gaussian process, clustering algorithms, generative models, kernel and neural network algorithms) can be used without going beyond the scope described herein. As will be understood by those skilled in the art, a typical neural network may employ, for example, but not limited to, one or more layers of nonlinear activation functions to predict the output for a received input, and may include one or more hidden layers in addition to input and output layers. The output of each hidden layer in some of these networks is used as the input to the next layer in the network. Examples of neural networks include, for example, but not limited to, generative neural networks, convolutional neural networks, and recurrent neural networks.

[0025] In some embodiments of the health monitoring system, an individual's heart rate and activity data are monitored as low fidelity data (e.g., heart rate or PPG data) to detect medical conditions (e.g., AFib) that are typically detected using high fidelity data (e.g., ECG data). For example, the individual's heart rate may be provided by a sensor continuously or at discrete intervals (e.g., every 5 seconds). The heart rate may be determined based on PPG, pulse oximetry, or other sensors. In some embodiments, activity data may be generated as the number of steps taken, the amount of movement sensed, or other data points indicative of activity levels. The low fidelity (e.g., heart rate) data and activity data may then be input into a machine learning system to determine a prediction of a high fidelity outcome. For example, the machine learning system may use the low fidelity data to predict arrhythmia or other indications indicative of the user's cardiac health. In some embodiments, the machine learning system may use input of a segment of data input to determine a prediction. For example, an hour's worth of activity level data and heart rate data may be input into the machine learning system. The system may then use this data to generate a prediction of a medical condition such as atrial fibrillation. Various embodiments of the invention are described in further detail below.

[0026] Referring to FIG. 1A, a trained convolutional neural network (CNN) 100 (an example of a feedforward network) puts input data 102 (e.g., a photo of a boat) into convolutional layers (also called hidden layers) 103 and applies a set of trained weights or filters 104 to the input data 106 in each of the convolutional layers 103. The output of the first convolutional layer is an activation map (not shown), which is the input to a second convolutional layer where trained weights or filters (not shown) are applied, and the output of the subsequent convolutional layers is an activation map that represents increasingly complex features of the input data to the first layer. After each convolutional layer, a nonlinear layer (not shown) is applied to introduce nonlinearity to the problem, which may include tanh, sigmoid, or ReLU. In some cases, a pooling layer (not shown) may be applied after the nonlinear layer, also called a downsampling layer, which basically takes the same length filter and stride and applies it to the input and outputs the maximum number in all the subregions that the filter convolves around. Other options for pooling are average pooling and L2 norm pooling. The pooling layer reduces the spatial dimension of the input volume to reduce computational cost and control overfitting. The final layer of the network is a fully connected layer, which receives the output of the last convolutional layer and outputs an n-dimensional output vector representing the quantity to be predicted, e.g., the probability of image classification: car 20%, boat 75%, bus 5%, bicycle 0%, i.e., resulting in a predicted output 106(O *), which is, for example, likely to be an image of a boat. The output may be a scalar-valued data point predicted by the network, for example, a stock price. The trained weights 104 may be different for each of the convolutional layers 103, as described in more detail below. To achieve this real-world prediction / detection (e.g., it's a boat), the neural network needs to be trained on known data inputs or training examples, resulting in the trained CNN 100. To train the CNN 100, many different training examples (e.g., many photos of boats) are input to the model. Those skilled in the art of neural networks will appreciate that the above description provides some context for the description of the present invention. It will be fully understood that this presents a somewhat simplified view of CNNs that result in the above-mentioned neural networks, and that application of CNNs alone or in combination with other neural networks is equally applicable and within the scope of some embodiments described herein.

[0027] FIG. 1B illustrates training a CNN 108. In FIG. 1B, the convolutional layer 103 is divided into separate hidden convolutional layers 105, 105′ to the convolutional layer 105. n-1 , with the final nth layer being a fully connected layer. It will be appreciated that the final layer may be multiple fully connected layers. Training examples 111 are input to a convolutional layer 103, which is then fed to a non-linear activation function (not shown) and weights 110, 110' through 110. n are applied sequentially to the training examples 111, and the output of the hidden layer is input to the next layer, and so on to the final n-th fully connected layer 105. nproduces an output 114. The output or prediction 114 is compared to the training examples 111 (e.g., a photo of a boat), resulting in a difference 116 between the output or prediction 114 and the training examples 111. If the difference or loss 116 is less than some preset loss (e.g., the output or prediction 114 predicted that the object was a boat), the CNN has converged and is considered trained. If the CNN has not converged, backpropagation is used to change the weights 110 and 110' to 110 according to how close the prediction is to the known input. n is updated. Those skilled in the art will appreciate that methods other than backpropagation may be used to adjust the weights. A second training example (e.g., a different photo of a boat) is input, and the process is repeated again with the updated weights, then updated again, and so on, until the nth training example (e.g., the nth photo of the nth boat) is input. This is repeated over and over with the same n training examples until the convolutional neural network (CNN) is trained or converges to the correct output for the known input. After the CNN 108 is trained, the weights 110, 110′ to 110 nare fixed and used in the trained CNN 100, which are the weights 104 as shown in FIG. 1A. As described, there are different weights for each of the convolutional layers 103 and the fully connected layers. The trained CNN 100 or model is then sent image data, which determines or predicts that it has been trained to predict / identify (e.g., a boat) as described above. Any trained model, CNN, RNN, etc., may be further trained, i.e., weights may be allowed to be modified, using additional training examples or predicted data output by the model that is then used as training examples. Machine learning models may be trained "offline", e.g., trained once on a computing platform separate from the platform that uses / executes the trained model, and then transferred to that platform. Alternatively, the embodiments described herein may periodically or continuously update the machine learning model based on newly acquired training data. This updated training can be done on a separate computing platform that sends the updated trained model to the platform that uses / executes the retrained model over a network connection, or the training / retraining / update process can be done on the platform itself as new data is acquired. Those skilled in the art will understand that CNNs can be applied to fixed-array data (e.g., photos, characters, words, etc.) or time series of data. For example, ordered health indicator data and other factor data can be modeled using CNNs. In some embodiments, feed-forward, CNNs are used with skip connections and Gaussian mixture model outputs to determine probability distributions for predicted health indicators, e.g., heart rate, PPG, or arrhythmia.

[0028] In some embodiments, other types and configurations of neural networks can be utilized. The number of convolutional layers can be increased or decreased, as can the number of fully connected layers. In general, the optimal number and ratio of convolutional layers to fully connected layers can be set empirically by determining which configuration performs best on a given dataset. It is also possible to reduce the number of convolutional layers to zero, leaving a fully connected network. The number of convolutional filters and the width of each filter can also be increased or decreased.

[0029] The output of the neural network can be a single scalar value corresponding to an accurate prediction for the primary time series, or it can be a logistic regression, with each category corresponding to a particular range or class of primary time series values, and one of ordinary skill in the art will readily appreciate that there are any number of other outputs.

[0030] The use of Gaussian mixture model outputs in some embodiments is intended to constrain the network to learning well-shaped probability distributions and improve generalization to limited training data. The use of multiple elements in some embodiments of Gaussian mixture models is intended to enable the model to learn multi-modal probability distributions. Machine learning models that combine or aggregate the results of different neural networks can also be used, and the results can be combined.

[0031] Another approach to modeling ordered data is machine learning models that have memory or states that can be updated from previous predictions to apply to subsequent predictions. In particular, some embodiments described herein utilize recurrent neural networks. Referring to the example of FIG. 2A, a diagram of a trained recurrent neural network (RNN) 200 is shown. The trained RNN 200 has updatable states (S) 202 and trained weights (W) 204. Input data 206 is input to the states 202 where the weights (W) 204 are applied, and predictions 206 (P* ) is output. In contrast to a linear neural network (e.g., CNN 100), state 202 is updated based on the input data, thereby sequentially acting as a memory from the previous state for the next prediction with the next data. By updating the state, the RNN has a circular or looping characteristic. For ease of illustration, FIG. 2B shows an unrolled trained RNN 200 and its applicability to ordered data. When unrolled, the RNN appears similar to a CNN, but in the unrolled RNN, each of the seemingly similar layers appears as a single layer with updated states and the same weights are applied in each iteration of the loop. Those skilled in the art will appreciate that a single layer may itself have sublayers, but for ease of illustration, a single layer is shown here. The input data (I t ) 208 is the state (S t ) 210, and the cell (C t ) 212, the trained weights 204 are applied. t The output of 212 is the prediction at time step t+1.

[0032]

number

[0033] 214 and updated state S t+1 216. Similarly, C t+1 In 220, I t+1 218 is S t+1 216, the same trained weights 204 are applied, and C t+1 The output of 220 is

[0034]

number

[0035] 222. As pointed out above, S t+1 is S t It is updated from St+1 is the S from the previous time step t For example, but not limited to, this memory may include previous health indicator data or previous other factor data from one or more previous time steps. This process continues for n steps, I t+n 224 is S t+n 226 and the same weight 204 is applied. t+n The output of

[0036]

number

[0037] In particular, these states are updated from previous time steps, giving the RNN the benefit of memory from previous states. This property makes RNNs an alternative choice for making predictions on ordered data for some embodiments. However, as mentioned above, there are other machine learning techniques that are suitable for performing such predictions on ordered data, including CNNs.

[0038] RNNs, like CNNs, can take a data string as input and output a predicted data string. A simple way to illustrate this aspect of using RNNs is to use the example of natural language prediction. Take the phrase The sky is blue. This word string (i.e., data) has a context. Thus, as the state is updated, the data string is updated from one iteration to the next, giving the context of predicting blue. As just described, RNNs have a memory component to aid in making predictions on ordered data. However, the memory in the updated state of the RNN may be limited in terms of how far it can look back, which is similar to short-term memory. When predicting ordered data where a longer lookback, similar to long-term memory, is desired, a fine-tuning to the RNN just described may be used to achieve this. A sentence where the word to be predicted is not obvious from the words immediately preceding or surrounding it is also a simple example to illustrate, i.e., Mary speaks fluent French. That French is a correct prediction is not clear from the words immediately preceding it, only that some language is a correct prediction, but which language? The correct prediction can be in the context of words separated by gaps larger than a single word string. Long short-term memory (LSTM) networks are a special kind of RNN that can learn these long-term(er) dependencies.

[0039] As mentioned above, RNNs have a relatively simple repetitive structure, for example they may have a single layer with a nonlinear activation function (e.g., tanh or sigmoid). LSTMs have a similar concatenated structure, but (for example) have four neural network layers instead of one. These additional neural network layers give LSTMs the ability to remove or add information to the state (S) by using structures called cell gates. See Id. FIG. 3 shows a cell 300 for an LSTM RNN. Line 302 represents the cell state (S) and can be thought of as an information highway, where it is relatively easy for information to flow along the cell state unchanged. See Id. Cell gates 304, 306, and 308 determine how much information is allowed to pass through the state or travel along the information highway. Cell gate 304 first selects the cell state S. t This is the so-called forget gate layer. See Id. Next, cell gates 306 and 306' determine what information is added to the cell state, and cell gates 308 and 308' predict what will be output from the cell state.

[0040]

number

[0041] The information highway or cell state is determined as the now updated cell state S for use in the next cell. t+1 LSTMs allow RNNs to have more persistent or long-term memory. LSTMs give RNN-based machine learning models an additional advantage in that they take into account context in output predictions that are farther away from the input data in space or time, depending on how the data is ordered, compared to simpler RNN structures.

[0042] In some embodiments, by utilizing an RNN, the primary and secondary time series may not be presented as vectors to the RNN at each time step. Instead, the RNN may be provided only with the current values ​​of the primary and secondary time series along with future values ​​or aggregation functions of the secondary time series within the prediction interval. In this manner, the RNN uses a persistent state vector to hold information about previous values ​​for use in making predictions.

[0043] Machine learning is well suited for continuous monitoring of one or more criteria to identify large and small, anomalies or trends in input data compared to the training examples used to train the model. Thus, in some embodiments described herein, the user's health index data and optionally other factor data are input into a trained machine learning model, which predicts what the healthy person's health index data will look like at the next time step and compares the prediction with the user's measured health index data at future time steps. If the absolute value of the difference (e.g., loss, described below) exceeds a threshold, the user is notified that the health index data is not within normal or healthy range. The threshold is a number set by the designer, and in some embodiments, may be changed by the user, allowing the user to adjust the notification sensitivity. The machine learning model of these embodiments may be trained on the health index data alone or in combination with corresponding (in time) other factor data from a population of healthy people, or on other training examples to meet the design requirements for the model.

[0044] Data from health indicators, like heart rate data, is ordered data, more specifically, time ordered data. Heart rate can be measured in many different ways, for example, but not limited to, by measuring electrical signals from a chest cuff or derived from a PPG signal. Some embodiments receive a derived heart rate from a device, with each data point (e.g., heart rate) generated at approximately equal intervals (e.g., 5s). However, in some cases and in other embodiments, the derived heart rate is not provided at approximately equal time steps, for example, because the data required for the derivation is unreliable (e.g., PPG signals are unreliable due to the device being moved or light pollution). The same is true for obtaining secondary series of data from motion sensors or other sensors used to collect other factor data.

[0045] The raw signal / data (ECG, electrical signals from a chest girdle, or PPG signals) itself is a time series of data that may be used in accordance with some embodiments. For simplicity and without limitation, this description uses PPG to refer to data representative of a health indicator. Those skilled in the art will readily appreciate that any form of data, raw data, waveforms or numbers derived from the raw data or waveforms for a health indicator may be used in accordance with some embodiments described herein.

[0046] Machine learning models that may be used with the embodiments described herein include, for example, but are not limited to, Bayesian, Markovian, Gaussian processes, clustering algorithms, generative models, kernels, and neural network algorithms. Some embodiments utilize machine learning models based on trained neural networks, other embodiments utilize recurrent neural networks, and additional embodiments use LSTM RNNs. For simplicity, but without limitation, recurrent neural networks are used to illustrate some embodiments of the description herein.

[0047] 4A-4C show hypothetical plots of PPG (FIG. 4A), steps taken (FIG. 4B), and temperature (FIG. 4C) versus time. PPG is an example of health index data, and steps, activity level, and temperature are exemplary other factor data for other factors that may affect the health index data. As one skilled in the art will appreciate, the other data may be obtained from any of many known sources, including, but not limited to, accelerometer data, GPS data, weight scale, user input, and the like, and may include, but is not limited to, temperature, activity (running, walking, sitting, cycling, falling, climbing stairs, steps, etc.), BMI, weight, height, age, and the like. A first dotted line drawn vertically across all three plots represents the time t at which user data is obtained for input into a trained machine learning model (described below). The hash plot line in FIG. 4A represents predicted or likely output data 402, and the solid line 404 in FIG. 4A represents measured data. FIG. 4B is a hypothetical plot of the number of steps taken by a user at various times, and FIG. 4C is a hypothetical plot of temperature at various times.

[0048] 5A-5B show a schematic diagram for a trained recurrent neural network 500 that receives the input data shown in FIG. 4A-4C, namely, PPG (P), steps (R), and temperature (T). Again, it is emphasized that these input data (P, R, and T) are only examples of health index data and other factor data. It will also be understood that data for multiple health indexes may be input and predicted, and more or less than two other factor data may be used, the selection depending on what the model is designed for. Those skilled in the art will further understand that the other factor data is collected in a time-corresponding manner to the collection or measurement of the health index data. In some cases, for example, weight, the other factor data remains relatively constant over a certain period of time.

[0049] FIG. 5A shows the trained neural network 500 as a loop. P, T, and R are input to the states 502 of the RNN 500, weights W are applied, and the RNN 500 outputs a predicted PPG 504 (P * In step 506, the difference PP * (ΔP * ) is calculated, and in step 508, |ΔP * It is determined whether | is greater than a threshold. If yes, in step 510, the user is notified / warned that the health indicator is outside the limits / thresholds predicted as normal or predicted for a healthy person. The warning / notification / detection can be, for example, but not limited to, a suggestion to see / get a doctor, a simple notification like haptic feedback, a request to take additional measurements like an ECG, or a simple note without a recommendation, or any combination of these. |ΔP * If | is less than or equal to the threshold, step 512 does nothing. In both steps 510 and 512, the process is repeated with new user data at the next time step. In this embodiment, the state may be updated following the output of the predicted data, and the predicted data may be used in updating the state.

[0050] In another embodiment, not shown, a primary series of heart rate data (derived from a PPG signal) and a secondary series of other factor data are provided to a trained machine learning model, which may be an RNN or CNN, other machine learning model, or a combination of models. In this embodiment, the machine learning model receives as input at a reference time t: A. A vector of length 300 (V H ), B.V. H At least one vector of length 300 containing recent other factor data, e.g., step counts, at the approximate time of each sample in O ), C. Entry V at index iDT (i) is health index sample V H (i) timestamp and V H A vector of length 300 containing the time difference between the (i-1) timestamp (V TD ), D. A scalar prediction interval other factor rate O that represents the average other factor rate (e.g., step rate) measured over the time interval from t to t+τ, which may be, for example, but is not limited to, 2.5 minutes, a prediction interval into the future. rate (for example, but not limited to, a step rate).

[0051] The output of this embodiment may be, for example, a probability distribution characterizing the predicted heart rate measured over the time period from t to t+τ. In some embodiments, the machine learning model is trained with training examples that include a continuous time series of health indicator data and other factor data series. In another embodiment, the notification system assigns a timestamp to each predicted health indicator (e.g., heart rate) distribution for t+τ / 2, thereby centering the predicted distribution within the predicted interval (τ). The notification logic in this embodiment then performs a time-domain analysis to determine if all samples are of length W in this example. L = 2*(τ) or within a 5 minute sliding window (W) and calculate the three parameters. 1. All health indicator series data within the time window

[0052]

number

[0053] The average value. 2. Health indicators

[0054]

number

[0055] The average of all model predictions for which the prediction timestamp falls within the time window. 3. Time Window

[0056]

number

[0057] The root mean square median of each predicted health indicator distribution in 4. In one embodiment, Ψ is the threshold

[0058]

number

[0059] or

[0060]

number

[0061] A notification is generated if

[0062] In this embodiment, an alert is generated when the measured health index is more than a multiple of a standard deviation away from the mean of the predicted health index values ​​within a particular window W. The window W may be applied in a sliding fashion across the sequence of measured and predicted health index values, with each window overlapping in time with the previous window by a designer-specified number of minutes, e.g., 0.5 minutes.

[0063] The notification may take any number of different forms: for example, but not limited to, it may notify the user to take an ECG and / or blood pressure, it may instruct a computing system (e.g., a wearable, etc.) to automatically take an ECG or blood pressure (for example), it may notify the user to see a doctor, or it may simply inform the user that their health indicator data is not normal.

[0064] In this embodiment, V as an input to the model DT The selection of the model is V H The present embodiment is intended to allow the use of information contained in the variable intervals between health indicator data in the vector V, which may be the result of algorithmically deriving the health indicator data from raw data that is less than consistent. For example, heart rate samples are generated by the Apple Watch algorithm only when it has reliable enough raw PPG data to output a reliable heart rate value, resulting in irregular time gaps between heart rate samples. Similarly, in this embodiment, other factor data (V O ) to handle different and irregular sample rates between the primary series (health indicators) and the secondary series (other factors). In this embodiment, the secondary series is remapped or interpolated onto the same time points as the primary time series.

[0065] Furthermore, in some embodiments, the composition of data from the secondary time series presented as input to the machine learning model from a future prediction time interval (e.g., after t) may be modified. In some embodiments, a single scalar value containing the average other factor data rate over the prediction interval may be modified with multiple scalar values, e.g., one for each secondary time series. Or, a vector of values ​​may be used over the prediction interval. In addition, the prediction interval may itself be adjusted. For example, a shorter prediction interval may result in faster response to changes and improved detection of events with short(er) fundamental timescales, but may also be more sensitive to interference from noise sources, such as motion artifacts.

[0066] Similarly, the output predictions of the machine learning model itself need not be scalar. For example, in some embodiments, a time series of predictions may be generated for multiple times t in the time interval between t and t+τ, and the alerting logic may compare each of these predictions to the predicted values ​​in the same time interval.

[0067] In this prior embodiment, the machine learning model itself may include, for example, a seven-layer feedforward neural network. The first three layers may be convolutional layers that include 32 kernels, each with a kernel width of 24 and a stride of 2. The first layer receives as input the array V H , V O , and V TD The final four layers are fully connected layers and all but the last may utilize a hyperbolic tangent activation function. The output of the third layer may be flattened into one array for input to the first fully connected layer. The final layer outputs 30 values ​​that parameterize a Gaussian mixture model with 10 mixtures (mean, variance, and weights for each mixture). The network uses skip connections between the first and third fully connected layers, and the output of layer 6 is summed with the output of layer 4 to generate the input to layer 7. Standard batch normalization may be used in all layers except the last layer, with a decay rate of 0.97. The use of skip connections and batch normalization can improve the ability to propagate gradients through the network.

[0068] The choice of machine learning model can affect the performance of a system. Machine learning model configuration can be divided into two types of considerations. First, the internal architecture of the model, which refers to the choice of model type (convolutional neural network, recurrent neural network, generalized nonlinear regression such as random forests) as well as the parameters that characterize the model's implementation (typically the number of parameters and / or the number of layers, the number of decision trees, etc.). Second, the external architecture of the model - the arrangement of data fed into the model and the specific parameters of the problem the model is being asked to solve. The external architecture can be characterized in part by the dimensionality and type of data fed into the model as input, the time range that the data spans, and the pre- or post-processing performed on the data.

[0069] In general, the choice of external architecture is a balance between increasing the number of parameters and amount of information supplied as input that can increase the predictive ability of the machine learning model, along with the available storage and computational power to train and evaluate larger models, and the availability of a sufficient amount of data to prevent overfitting.

[0070] Many variations of the external architecture of the model described in some embodiments are possible. The number of input vectors, as well as the absolute length (number of elements) and the time span of coverage can be modified. It is not necessary for each input vector to be the same length or to cover the same time span. The data need not be sampled evenly in time, for example, but not limited to, a 6-hour history of heart rate data may be taken, where data less than 1 hour before t is sampled at a rate of 1 Hz, data more than 1 hour before t and less than 2 hours before t is sampled at a rate of 0.5 Hz, and data older than 2 hours is sampled at a rate of 0.1 Hz, where t is the reference time.

[0071] FIG. 5B shows the trained RNN 500 being unrolled. t , R t , and T t ) is the state at time t (S t ) 514 and the trained weights 516 are applied. t )518 output is the prediction at time t+1

[0072]

number

[0073] 520 and updated state S t+1 522. Similarly, C t+1 In 524, the input data (P t+1 , R t+1 , and T t+1 )513' is S t+1522, the trained weights 516 are applied, and C t+1 The output of 524 is

[0074]

number

[0075] 523. As pointed out above, S t+1 is S t This is the result of updating S t+1 is the cell in the previous time step (C t )518 operation from S t This process continues for n steps, and the input data (P n , R n , and T n )513" is S n 530 and the trained weights 516 are applied. Cell C t The output is the predicted 532

[0076]

number

[0077] In particular, the trained RNN applies the same weights throughout, but what is important is that the state is updated from the previous time step, giving the RNN the benefit of memory from the previous time step. Those skilled in the art will appreciate that the temporal order of inputting the dependent health indicator data may vary and still produce favorable results. For example, t-1 ) and the measured health indicator data from the current time step (e.g., R t and T t ) other factor data from the current time step (S t ) and the model can enter the state at the current time step

[0078]

number

[0079] The health index is predicted at and compared with the health index data measured at the current time step to determine whether the user's health index is normal or within a healthy range, as described above.

[0080] FIG. 5C shows another embodiment of a trained RNN to determine whether ordered data of a user's health indicators, PPG in our example, is within a band or threshold for healthy subjects. The input data in this embodiment is a linear combination

[0081]

number

[0082] where:

[0083]

number

[0084] is the predicted health index value at time t, and P t is the measured health index at time t. In this embodiment, α ranges from 0 to 1 nonlinearly as a function of loss (L), where loss and α are described in more detail below. Of present interest is the data P measured when α is close to zero. t is input to the network, and the predicted data when α is close to 1

[0085]

number

[0086] is input to the network and prediction is made at the next time step. t ) may be optionally entered.

[0087] I t and O t is the state S t , which in some embodiments represents the predicted health index data at time step t+1.

[0088]

number

[0089] Probability distribution of (β)

[0090]

number

[0091] where β (P*) is the predicted health index (P * ) In some embodiments, the probability distribution function is the predicted health index value at t+1

[0092]

number

[0093] As will be appreciated by those skilled in the art, the sampling is performed to select β (P*) may be sampled using different methods depending on the network designer's goals, which may include taking the mean, maximum, or random sampling of a probability distribution. Using the measured data at time t+1, β t+1 By evaluating t+1 would have predicted for the measured data.

[0094] To illustrate this concept, Figure 5D shows a hypothetical probability distribution for a range of hypothetical health indicator data at time t+1. The function may be sampled, for example, with a maximum probability of 0.95, so that the predicted health indicator at time t+1 is

[0095]

number

[0096] Determine the probability distribution (β t+1 ) also includes measured or actual health indicator data.

[0097]

number

[0098] to determine the probability that the model would have predicted if the actual data had been fed into the model. In this example,

[0099]

number

[0100] is 0.85.

[0101] The loss may be defined to aid in determining whether to notify the user that their health status is not within the normal range as predicted by the trained machine learning model. The loss is selected to model how close the predicted data is to the actual or measured data. Those skilled in the art will appreciate that there are many ways to define the loss. In other embodiments described herein, for example, the absolute value of the difference between the predicted data and the actual data (|ΔP * In some embodiments, the loss (L) is L=-ln[β (P) ] as well, where,

[0102]

number

[0103] L is a measure of how close the predicted data is to the measured or actual data. β (P) L is in the range of 0 to 1, with 1 meaning the predicted value and the measured value being the same. Thus, a low loss indicates that the predicted value is probably the same or close to the measured value. In this context, this means that the measured data looks like it comes from a healthy / normal individual. In some embodiments, a threshold for L is set, e.g., L>5, and the user is notified that the health index data is outside of a range that is considered healthy. In other embodiments, the loss may be averaged over a period of time and the average compared to the threshold. In some embodiments, the threshold itself may be a function of a statistical calculation of the predicted value or may be the average of the predicted value. In some embodiments, the following formula may be used to notify the user that the health index is not within the healthy range:

[0104]

number

[0105] <P range > is determined by averaging health indicator data measured over a certain time range,

[0106]

number

[0107] is determined by averaging the predicted health indicator data over the same time range,

[0108]

number

[0109] is the median of the series of standard deviations derived from the network over the same time range,

[0110]

number

[0111] teeth,

[0112]

number

[0113] is a function of the standard deviation evaluated at , and can act as a threshold.

[0114] Averaging methods that may be used include, for example and without limitation, the mean, arithmetic mean, median, and mode. In some embodiments, outliers are removed so as not to skew the calculated numerical values.

[0115] Input data for the embodiment shown in FIG. 5C

[0116]

number

[0117] Referring again to α t is defined as a function of L and ranges from 0 to 1. For example, α(L) may be a linear function, or a nonlinear function, or may be linear over one range of L and nonlinear over another range of L. In one example, as shown in FIG. 5E, the function α(L) is linear for L between 0 and 3, quadratic for L between 3 and 13, and unity for L greater than 13. For this embodiment, when L is between 0 and 3 (i.e., when the predicted health index data and the measured health index data are approximately equal), the input data I t+1When α-1 approaches zero, the measured data P t+1 When L is large, for example, greater than 13, α(L) is 1, which is the input data

[0118]

number

[0119] Let α(L) be the predicted health index at time t+1. As L is between 1 and 13, α(L) varies as a quadratic function, and the relative contribution of the predicted and measured health index data to the input data also varies. The linear combination of the predicted and measured health index data weighted by α(L) allows, in this embodiment, to weight the input data between the predicted and measured data at any particular time step. In all these examples, the input data is weighted according to other contributing data (O t ) This is just one example of self-sampling, where some combination of predicted and measured data is used as input to the trained network. Those skilled in the art will appreciate that many others may be used.

[0120] The machine learning model in an embodiment uses a trained machine learning model. In some embodiments, the machine learning model uses a recurrent neural network, which requires a trained RNN. For example, without limitation, FIG. 6 shows an unrolled RNN that demonstrates training an RNN in accordance with some embodiments. Cell 602 has an initial state S0 604 and a weight matrix W 606. Step rate data R0, temperature data T0, and initial PPG data P0 at time step zero are input into state S0, and weights W are applied to produce a predicted PPG at the first time step.

[0121]

number

[0122] is output from cell 602, and using the PPG (P1) obtained at time step 1,

[0123]

number

[0124] Cell 602 also outputs the updated state 608 (S1) at time step 1, which goes into cell 610. The step rate data R1, temperature data T1, and PPG data P1 at time step 1 are input into S1 and weights 606 W are applied to calculate the predicted PPG at time step 2.

[0125]

number

[0126] is output from cell 610, and using the PPG (P2) obtained at time step 2,

[0127]

number

[0128] Cell 610 also outputs the updated state 612 (S2) at time step 2, which goes into cell 614. The step rate data R3, temperature data T3, and PPG data (P3) at time step 3 are input into S2 and weights 606 W are applied to calculate the predicted PPG at time step 3.

[0129]

number

[0130] is output from cell 614, and using the PPG (P3) obtained at time step 3,

[0131]

number

[0132] is calculated. This means that the state 616 at time step n is output,

[0133]

number

[0134] This continues until ΔP is calculated. * ' is used in backpropagation to adjust the weight matrix, which is similar to training a convolutional neural network. However, unlike a convolutional network, the same weight matrix is ​​applied in the recurrent neural network at each iteration, which is only modified by backpropagation during training. Many training examples including health index data and corresponding other factor data are input to the RNN 600 many times until convergence. As previously described, LSTM RNNs may be used in some embodiments where the state of such a network provides longer-term contextual analysis of the input data, and the network may make better predictions when it learns (more) long-term correlations. As further mentioned, those skilled in the art will readily appreciate that other machine learning models also fall within the scope of the embodiments described herein, and may include, for example, but not limited to, CNNs or other feedforward networks.

[0135] FIG. 7A shows a system 700 that predicts whether a user's measured health indicator is within the normal range for an indicator of a healthy person under similar other factors, or outside the normal threshold. The system 700 has a machine learning model 702 and a health detector 704. An embodiment for the machine learning model 702 includes, for example (but not limited to), a trained machine learning model, a trained RNN, a CNN, or other feed-forward network. The trained RNN, other network, or combination of networks can be trained on training examples from a population of healthy people from whom the health indicator data and corresponding (in time) other factor data were collected. Alternatively, the trained RNN, other network, or combination of networks can be trained on training examples from a specific user, thereby making it a personalized trained machine learning model. Those skilled in the art will understand that the training examples from different populations can be selected depending on the use or design for the trained network and the system in general. Those skilled in the art will also readily understand that the health indicator data in this and other embodiments can be one or more health indicators. For example, but not limited to, one or more of PPG data, heart rate data, blood pressure data, body temperature data, blood oxygen level data, and the like, can be used to train the model and predict the health of the user. The health detector 704 uses the machine learning model 702 and predictions 708 from the input data 710 to determine whether the loss or other metrics determined by analyzing the predicted output from the measured data exceed a threshold considered normal and therefore unhealthy. The system 700 then outputs a notification or status of the user's health. The notification can take many forms as described herein. The input generator 706 continuously acquires data from the user wearing or in contact with the sensor (not shown) with a sensor, the data representing one or more health indicators of the user. Corresponding (in time) other factor data can be collected by another sensor or acquired through other means as described herein or as would be readily apparent to one of ordinary skill in the art.

[0136] The input generator 706 may also collect data to determine / calculate other factor data. The input generator may include, for example, but not limited to, a smartwatch, a wearable or mobile device (e.g., an Apple Watch® or FitBit® smartphone, a tablet, or a laptop computer), a combination of a smartwatch and a mobile device, a surgically implanted device capable of transmitting data to a mobile device or other portable computing device, or a device on a cart in a medical facility. Preferably, the user input generator 706 has a sensor (e.g., a PPG sensor, an electrode sensor) for measuring data related to one or more health indicators. The smartwatch, tablet, mobile phone, or laptop computer of some embodiments may be equipped with a sensor or the sensor may be placed at a remote location (surgically implanted, in contact with the body separate from the mobile device, or some separate device), in all these cases the mobile device communicates with the sensor to collect the health indicator data. In some embodiments, the system 700 may be provided on a mobile device alone, in combination with other mobile devices, or in combination with other computing systems via communication over a network through which these devices communicate. For example, but not by way of limitation, the system 700 may be a smart watch or wearable, with the machine learning model 702 and health detector 704 located on the device, e.g., in the memory of the watch or firmware on the watch. The watch may have a user input generator 706 and may communicate with other computing devices (e.g., mobile phones, tablets, laptop computers, or desktop computers) via direct communication, wireless communication (e.g., WiFi, sound, Bluetooth, etc.), or over a network (e.g., the Internet, an intranet, an extranet, etc.), or a combination thereof, with the trained machine learning model 702 and health detector 704 located on the other computing devices. Those skilled in the art will appreciate that the present invention may be practiced with the present invention in a variety of different configurations. For example, it will be appreciated that any number of configurations of system 700 may be utilized without departing from the scope of the embodiments described herein.

[0137] Referring to FIG. 7B, a smartwatch 712 according to one embodiment is illustrated. The smartwatch 712 includes a watch 714 that contains all circuitry and a microprocessor (not shown) known to those skilled in the art. The watch 714 also includes a display 716 on which the user's health indicator data 718, in this example, heart rate data, may be displayed. Also displayed on the display 716 may be a predicted health indicator band 720 for a normal or healthy population. In FIG. 7B, the user's measured heart rate data does not exceed the predicted healthy band, so in this particular example, no notification is provided. The watch 714 may also include a watch band 722 and a high fidelity sensor 724, for example, an ECG sensor. Alternatively, the watch band 722 may be an inflatable cuff for measuring blood pressure. A low fidelity sensor 726 (shown in shading) is provided on the back of the watch 714 to collect user health indicator data, such as PPG data, which may be used to derive heart rate data or other data, such as blood pressure. Alternatively, as will be appreciated by those skilled in the art, in some embodiments a fitness band such as a FitBit or Polar may be used, which has similar processing capabilities and other factor measuring devices (e.g., ppg and accelerometers).

[0138] FIG. 8 illustrates one embodiment of a method 800 for continuously monitoring a user's health status. In step 802, user input data is received, which may include data for one or more health indicators (i.e., a primary series of data) and corresponding (in time) data for other factors (i.e., a secondary series of data). In step 804, the user data is input to a trained machine learning model, which may include a trained RNN, CNN, other feed-forward network, or other neural network known to those skilled in the art, as described herein. In some embodiments, the health indicator input data may be one or a combination, e.g., a linear combination, of predicted health indicator data and measured health indicator data, as described in some embodiments herein. In step 806, data for one or more predicted health indicators at a time step is output, which may include, for example, but not limited to, a single predicted value, a probability distribution as a function of the predicted value. At step 808, a loss is determined based on the predicted health index, for example, but not limited to, the loss may be a simple difference between the predicted and measured health index, or some other appropriately selected loss function (e.g., the negative logarithm of the probability distribution evaluated at the value for the measured health index). At step 810, it is determined whether the loss exceeds a threshold considered normal or unhealthy, which may be, for example, but not limited to, a simple numerical value chosen by a designer, or a more complex function of some parameter related to the prediction. If greater than this threshold, at step 812, the user is notified that the user's health index has exceeded the threshold considered normal or healthy. The notification may take many forms, as described herein. In some embodiments, this information may be visualized to the user. For example, but not limited to, information such as a graph showing the distribution of (i) the measured health index data (e.g., heart rate) and other factor data (e.g., step count) as a function of time, and (ii) the predicted health index data (e.g., predicted heart rate values) generated by the machine learning model may be displayed on a user interface. In this way, a user can visually compare the measured data points to the predicted data points and determine by visual inspection, for example, whether a heart rate falls within the range predicted by the machine learning model.

[0139] Some embodiments described herein refer to using a threshold value to determine whether or not to notify the user. In one or more of these embodiments, the user may change the threshold value to adjust or tune the system or method to more accurately match their personal health knowledge. For example, if the physiological indicator used is blood pressure and the user has a higher blood pressure, the embodiment may frequently warn / notify the user that the user's health indicator is outside of the normal or healthy range from a model trained on a healthy population. Thus, some embodiments allow the user to increase the threshold value to be notified less frequently that the user's health indicator data has exceeded what is considered normal or healthy.

[0140] In some embodiments, raw data is preferably used for the health indicators. If the raw data is processed to derive a particular measurement, e.g., heart rate, this derived data can be used according to the embodiments. In some situations, the provider of the health monitoring device does not control the raw data, but rather receives processed data in the form of a calculated health indicator, e.g., heart rate or blood pressure. As one skilled in the art will appreciate, the form of data used to train the machine learning model should match the form of data collected from the user and input to the trained model, otherwise the predictions may prove incorrect. For example, the Apple Watch provides heart rate measurement data at unequal time steps and does not provide raw PPG data. In this example, the user is wearing an Apple Watch that outputs heart rate data according to Apple's PPG processing algorithm that uses heart rate data at unequal time steps. The model is trained on this data. With Apple deciding to change the algorithm for providing heart rate data, the model trained on data from the previous algorithm may become obsolete and cannot be used with data input from the new algorithm. To account for this potential problem, in some embodiments, when collecting data to train the model, irregularly spaced data (such as heart rate, blood pressure data, or ECG data) are resampled onto a regularly spaced grid and samples from the regularly spaced grid. If Apple, or other suppliers of data, change their algorithms, the model only needs to be retrained on newly collected training examples, but the model does not need to be rebuilt to match the algorithm changes.

[0141] In a further embodiment, the trained machine learning model may be trained on the user's data, resulting in a personalized trained machine learning model. This personalized trained machine learning model can be used in place of or in combination with the machine learning model trained on a healthy population as described herein. When used alone, the user's data is input into the personalized trained machine learning model, which outputs a prediction of the individual's health indicators that are normal for that user at the next time step, which is then compared with the actual / measured data from the next time step in a manner consistent with the embodiments described herein to determine whether the user's health indicators differ by a certain threshold from those predicted to be normal for that user. In addition, this personalized machine learning model can be used in combination with a machine learning model trained on training examples from a healthy population, thereby generating predictions and associated notifications related to both what is predicted to be normal for that individual user and what is predicted to be normal for the healthy population.

[0142] FIG. 9A shows a method 900 according to another embodiment, and FIG. 9B shows a hypothetical plot 902 of heart rate (for example, but not limited to) as a function of time for illustration. In step 904 (FIG. 9A), the user heart rate data (or other health indicator data) and, optionally, corresponding (in time) other factor data are received and input into a personalized trained machine learning model. In some embodiments, the personalized trained model is trained on the user's individual health indicator data and, optionally, corresponding (in time) other data described herein. Thus, in step 906, the personalized trained machine learning model predicts normal heart rate data for that individual user under the conditions of other factors, and in step 908, identifies deviations or anomalies in the user's health indicator data compared to what is predicted to be normal for that particular user. In some embodiments, the user's health indicator data is received from a wearable device worn by the user (e.g., Apple Watch, smart watch, FitBit®, etc.) or from another mobile device (e.g., tablet, computer, etc.) in communication with a sensor worn by the user (e.g., Polar® strap, PPG sensor, etc.), as described throughout the description herein.

[0143] The loss may be defined to aid in determining whether to notify the user in step 908 that the user's measured data is abnormal compared to what is predicted to be normal for that particular user. The loss is selected to model how close the prediction is to the actual or measured data. Those skilled in the art will appreciate that there are many ways to define the loss. In other embodiments described herein that are equally applicable here, for example, the absolute value of the difference between the predicted value and the absolute value |ΔP * | is a form of loss. In some embodiments, the loss (L) is L=-ln[β (P) ] as well, where,

[0144]

number

[0145] L is generally a measure of how close the predicted data is to the measured data. In this example, we use the probability distribution β (P) L ranges from 0 to 1, with 1 meaning the predicted data and the measured data are the same. Thus, low loss, in some embodiments, indicates that the predicted data is probably the same or close to the measured data. In some embodiments, a threshold for L is set, e.g., L>5, and the user is notified that there is an abnormal condition that deviates from what is predicted for that particular user. This notification can take many forms, as described elsewhere herein. As also described elsewhere herein, other embodiments may average the loss over a period of time and compare the average to a threshold. In some embodiments, the threshold itself may be a function of a statistical calculation of the predicted data or an average of the predicted data, as described in more detail elsewhere herein. Loss is described in more detail elsewhere herein and will not be described further here for the sake of brevity. Those skilled in the art will also appreciate that the input and predicted data may be scalar values, or segments of data, over a period of time. For example, and not by way of limitation, a system designer may be interested in a 5 minute data segment, inputting all data prior to time t and all other data at t+5 minutes, predicting health index data at t+5 minutes, and determining the loss between the measured health index data for the t+5 minute segment and the predicted health index data for the t+5 minute segment.

[0146] At step 908, it is determined whether an abnormality exists. As will be explained, this may be determined if the loss exceeds a threshold. As already explained, the threshold is set based on the purpose of the system being designed, at the designer's choice. In some embodiments, the threshold may be modified by the user, but preferably not in this embodiment. If an abnormality does not exist, the process is repeated at step 904. If an abnormality exists, at step 910, the user is notified or alerted to obtain a high fidelity measurement, such as, but not limited to, an ECG or blood pressure measurement. At step 912, the high fidelity data is analyzed by an algorithm, a medical professional, or both, and described as normal or abnormal, and if not normal, some diagnosis may be made, such as AFib, tachycardia, bradycardia, atrial flutter, or hyper / hypotension, depending on the high fidelity measurement obtained. For clarity, it is noted that the notification to record high fidelity data is equally applicable and possible in other embodiments, and in certain embodiments using the general model described above. The high fidelity measurements may in some embodiments be obtained directly by the user using a mobile monitoring system, such as an ECG or blood pressure measurement system, which in some embodiments may be associated with a wearable device. Alternatively, the notification step 910 triggers automatic acquisition of the high fidelity measurements. For example, the wearable device may communicate with a sensor (either hard-wired or via wireless communication) to obtain ECG data, or with a blood pressure cuff system (e.g., a wearable wristband or armband cuff) to automatically obtain blood pressure measurements, or with an implanted device such as a pacemaker or ECG electrodes. Systems for remotely obtaining ECGs are provided, for example, by AliveCor, Inc., such systems include (but are not limited to) one or more sensors that contact the user at two or more locations, the sensors collect electrical cardiac data that is transmitted, either wired or wirelessly, to a mobile computing device, and an app provides the data to an algorithm, a medical professional, or both.The wearable generates an ECG strip from the data, which can be analyzed by a mobile computing device. Alternatively, the sensor may be a blood pressure monitor, and the blood pressure data is transmitted to the mobile computing device, either wired or wirelessly. The wearable itself may be a blood pressure system with the ability to measure health indicator data, and optionally a cuff with an ECG sensor similar to that described above. The mobile computing device may be, for example, but not limited to, a computer tablet (e.g., iPad®), a smartphone (e.g., iPhone®), a wearable (e.g., Apple Watch®), or a device in a medical facility (which may be attached to a cart). The mobile computing device may be, in some embodiments, a computer in communication with a laptop computer or some other mobile device. Those skilled in the art will understand that a wearable or a smart watch may also be considered a mobile computing device with respect to the functionality provided in the context of the embodiments described herein. In the case of a wearable, the sensor may be attached to a band on the wearable and the sensor may transmit data wirelessly or via wires to a computing device / wearable, or the band may be a blood pressure monitoring cuff, or both as previously described. In the case of a mobile phone, the sensor may be a pad attached to the phone or located remotely from the phone, where the pad senses electrical cardiac signals and communicates the data wirelessly or via wires to the wearable or other mobile computing device.More detailed descriptions of some of these systems are provided in one or more of U.S. Patent Nos. 9,420,956, 9,572,499, 9,351,654, 9,247,911, 9,254,095, and 8,509,882, as well as one or more of U.S. Patent Application Publication Nos. 2015 / 0018660, 2015 / 0297134, and 2015 / 0320328, all of which are incorporated herein in their entirety for all purposes. As previously described, in step 912, the high fidelity data is analyzed to provide an explanation or diagnosis.

[0147] In step 914, the diagnosis or classification of the high fidelity measurement is received by a computing system, which in some embodiments may be a mobile or wearable computing system used to collect the user's heart rate data (or other health indicator data), and in step 916, the low fidelity health indicator data series (heart rate data in this example) is labeled with a diagnosis. In step 918, the labeled user's low fidelity data series is used to train a high fidelity machine learning model, and optionally, other factor data series are also provided to train the model. The trained high fidelity machine learning model, in some embodiments, has the ability to receive the measured low fidelity health indicator data series (e.g., heart rate data or PPG data) and optionally other factor data, and provide a probability or predict, or diagnose, or detect when the user is experiencing an event that is typically diagnosed or detected using high fidelity data. The trained high fidelity machine learning model is able to do this because it has already been trained on the user's health diagnosis data (and optionally other factor data) labeled with a diagnosis in the high fidelity data. Thus, the trained model has the ability to predict when a user has an event (e.g., Afib, high blood pressure, etc.) associated with one or more of the labels based solely on the measured low fidelity health indicator input data series, e.g., heart rate or ppg data (and optionally other factor data). As one skilled in the art will appreciate, the training of the high fidelity model may be performed on the user's mobile device, away from the user's mobile device, a combination of the two, or in a distributed network. For example, but not limited to, the user's health indicator data may be stored in a cloud system, and the data may be labeled in the cloud using the diagnosis from step 914. One skilled in the art will readily appreciate that there are any number of ways and manners to store, label, and access this information.Alternatively, a globally trained high-fidelity model can be used, which is trained on labeled training examples from a population of people experiencing those conditions that are typically diagnosed or detected with high-fidelity measurements. These global training examples provide low-fidelity data series (e.g., heart rate) that are labeled with the condition diagnosed using the high-fidelity measurements (e.g., Afib called from an ECG by a medical professional or algorithm).

[0148] 9B, plot 902 shows a summary of heart rate plotted as a function of time. Deviations 920 from the user's normal heart rate data occurred at times t1, t2, t3, t4, t5, t6, t7, t8. Normal, as mentioned above, means that the expected data for this particular user was within a threshold range of the measured data, and the deviations are outside the threshold. Upon deviation from normal, some embodiments prompt the user to obtain a more definitive or higher fidelity reading, for example, an ECG reading, identified as, but not limited to, ECG1, ECG2, ECG3, ECG4, ECG5, ECG6, ECG7, ECG8. As mentioned above, the high fidelity reading can be obtained automatically or the user can obtain it, and it can be something other than an ECG, for example, blood pressure. The high fidelity reading is analyzed by an algorithm, a medical professional, or both to identify the high fidelity data as normal / abnormal and further identify / diagnose the abnormality, for example, but not limited to, AFib. This information is used to label health indicator data (e.g., heart rate or PPG data) at the point of anomaly 920 in the user's ordered data.

[0149] The distinction between high-fidelity data and low-fidelity data is that high-fidelity data or measurements are typically used to make decisions, detections, or diagnoses, whereas low-fidelity data cannot be readily used for such. For example, ECG scans can be used to identify, detect, or diagnose arrhythmias, whereas heart rate or PPG data typically cannot. As will be appreciated by those skilled in the art, the discussion herein relating to machine learning algorithms (e.g., Bayesian, Markov, Gaussian process, clustering algorithms, generative models, kernels, and neural network algorithms) applies equally to all embodiments described herein.

[0150] In some situations, a user may be asymptomatic even though a problem may exist, and even if symptomatic, it may be impractical to obtain the high fidelity measurements necessary to make a diagnosis or detection. For example, but not limited to, arrhythmias, particularly AF, may not be present, and even when symptoms do occur, it is very difficult to record an ECG at that time, and it is extremely difficult to continuously monitor the user without expensive, bulky, and sometimes invasive monitoring devices. As described elsewhere herein, it is important to understand when a user experiences AF, since AF may be a causative factor in at least other serious medical conditions, among others, stroke. Similarly, as described elsewhere, AF burden may have similar importance. Some embodiments enable continuous monitoring of arrhythmias (e.g., AF) or other serious medical conditions using only continuous monitoring of low fidelity health indicator data such as heart rate or ppg, along with optional other contributing data.

[0151] FIG. 10 shows a method 1000 according to some embodiments of the health monitoring system and method. In step 1002, measured or actual user low fidelity health indicator data (e.g., heart rate or PPG data from a sensor on a wearable) is received, and optionally corresponding (in time) other factor data that may affect the health indicator data described herein. As described elsewhere herein, the low fidelity health indicator data may be measured by a mobile computing device, such as a smart watch, other wearable, or computer tablet. In step 1004, the user's low fidelity health indicator data (and optionally other factor data) are input into a trained high fidelity machine learning model, which in step 1006 outputs a predicted identification or diagnosis for the user based on the measured low fidelity health indicator data (and optionally corresponding (in time) other factor data). In step 1008, it is asked whether the identification or diagnosis is normal, and if yes, the process starts again. If the identification or diagnosis is not normal, the user is notified of the problem or detection at step 1010. Optionally, the system, method, or platform may be set up to notify any combination of the user, family, friends, medical professionals, emergency services 911, or the like. Who of these people is notified may depend on the identification, detection, or diagnosis. If the identification, detection, or diagnosis is life-threatening, certain people may be contacted or notified, and if the diagnosis is not life-threatening, they may not be notified. In addition, in some embodiments, the measured health indicator data series is input into a trained high-fidelity machine learning model to calculate the length of time the user is experiencing an abnormal event (e.g., the difference between the start and end of a predicted abnormal event), thereby allowing a deeper understanding of the abnormal burden on the user. In particular, AF burden may be crucial to understand in preventing strokes and other serious medical conditions.Thus, some embodiments enable continuous monitoring of abnormal events by mobile, wearable, or other portable computing devices that are only capable of acquiring low-fidelity health factor data and, optionally, other factor data.

[0152] FIG. 11 illustrates example data 1100 analyzed based on low fidelity data to generate a high fidelity output prediction or detection, according to some embodiments described herein. Although described with reference to detection of atrial fibrillation, similar data may be generated for additional prediction of a high fidelity diagnosis based on low fidelity measurements. A first chart 1110 illustrates heart rate calculations over time for a user. The heart rate may be determined based on PPG data or other heart rate sensors. A second chart 1120 illustrates activity data for the user for the same time period. For example, the activity data may be determined based on step counts or other measurements of the user's movement. A third chart 1130 illustrates classifiers output from a machine learning model and horizontal thresholds for when notifications are generated. The machine learning model may generate a prediction based on input of low fidelity measurements. For example, the data in the first chart 1110 and the second chart 1120 may be analyzed by a machine learning system as further described above. The analysis result of the machine learning system may be provided as an atrial fibrillation probability, as shown in chart 1130. When the probability exceeds a threshold, shown in this case as exceeding a confidence level of 0.6, the health monitoring system may trigger a notification or other alert to the user, a physician, or others associated with the user.

[0153] In some embodiments, the data in the charts 1110 and 1120 may be provided as continuous measurements to the machine learning system. For example, heart rate and activity level may be generated as measurements every 5 seconds to provide accurate measurements. A segment of time with multiple measurements may then be input to the machine learning model. For example, previous hours of data may be used as input to the machine learning model. In some embodiments, a shorter or longer period of time may be provided rather than one hour. As shown in FIG. 11, the output chart 1130 provides an indication of the period of time that the user is experiencing an abnormal health event. For example, the period when a prediction is made with a particular confidence level may be used by the health monitoring system to determine atrial fibrillation. This value may be used to determine the atrial fibrillation burden on the user during the measurement time period.

[0154] In some embodiments, the machine learning model for generating the predicted output in the chart 1130 may be trained based on the labeled user data. For example, the labeled user data may be provided based on high fidelity data (such as ECG readings) taken during a time period when low fidelity data (e.g., PPG, heart rate) and other data (e.g., activity level or steps) are also available. In some embodiments, the machine learning model is designed to determine whether there is a high probability that there was atrial fibrillation during a preceding time period. For example, the machine learning model may take an hour's worth of low fidelity data as input and provide a probability that there was an event. Thus, the training data may include several hours' worth of recorded data for a population of individuals. The data may be a health event labeling time when a medical condition was diagnosed based on the high fidelity data. Thus, if there was a health event labeling time based on the high fidelity data, the machine learning model may determine that any hour's worth of window of low fidelity data with that event input to the untrained machine learning model should provide a prediction of the health event. The untrained machine learning model may then be updated based on a comparison of the prediction to its label. After iterating over multiple iterations and determining that the machine learning model has converged, it may be used by the health monitoring system to monitor the user for atrial fibrillation based on the low-fidelity data. In various embodiments, medical conditions other than atrial fibrillation may be detected using the low-fidelity data.

[0155] FIG. 12 illustrates a diagrammatic representation of a machine in the exemplary form of a computer system 1200 on which a set of instructions may be executed to cause the machine to perform one or more of the methodologies described herein. In other embodiments, the machine may be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch or bridge, a hub, an access point, a network access control device, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify actions to be performed by the machine. Moreover, although only a single machine is illustrated, the term "machine" or "machine" shall be construed to include a machine or a collection of machines that individually or in conjunction execute a set of instructions (or multiple sets of instructions) to perform one or more of the methodologies described herein. In one embodiment, computer system 1200 may represent a server, mobile computing device, wearable, or the like configured to perform the health monitoring described herein.

[0156] The exemplary computer system 1200 comprises a processing device 1202, a main memory 1204 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM)), a static memory 1206 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 1218, which communicate with each other via a bus 1230. Any of the signals provided on the various buses described herein may be time-shared with other signals and provided on one or more common buses. In addition, the interconnections between circuit components or blocks may be depicted as buses or as single signal lines. Each of these buses may alternatively be one or more single signal lines, and each of the single signal lines may alternatively be a bus.

[0157] The processing device 1202 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor implementing other instruction sets or a processor implementing a combination of instruction sets. The processing device 1202 may also be one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 1202 is configured to execute the processing logic 1226, which may be one example of the health monitor 1250 and related systems for performing the operations and steps described herein.

[0158] The data storage device 1218 may comprise a machine-readable storage medium 1228 having stored thereon one or more sets of instructions 1222 (e.g., software) embodying any one or more of the methods of functionality described herein, including instructions that cause the processing device 1202 to perform the health monitor 1250 and related processes described herein. The instructions 1222 may also reside, completely or at least partially, within the main memory 1204 or the processing device 1202 during execution by the computer system 1200, with the main memory 1204 and the processing device 1202 also constituting machine-readable storage media. The instructions 1222 may further be transmitted or received over the network 1220 via the network interface device 1208.

[0159] The machine-readable storage medium 1228 may also be used to store instructions for performing methods for monitoring user health as described herein. Although the machine-readable storage medium 1228 is shown in the exemplary embodiment as being a single medium, the term "machine-readable storage medium" should be interpreted as including a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that store one or more sets of instructions. A machine-readable medium comprises a mechanism for storing information in a form (e.g., software, processing application) that can be read by a machine (e.g., a computer). A machine-readable medium may include, but is not limited to, a magnetic storage medium (e.g., floppy diskette), an optical storage medium (e.g., CD-ROM), a magneto-optical storage medium, a read-only memory (ROM), a random access memory (RAM), an erasable programmable memory (e.g., EPROM and EEPROM), a flash memory, or another type of medium suitable for storing electronic instructions.

[0160] The foregoing description sets forth numerous specific details, such as examples of specific systems, components, methods, and the like, to provide a thorough understanding of some embodiments of the present disclosure. However, it will be apparent to one of ordinary skill in the art that at least some embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known components or methods have not been described in detail or have been presented in simple block diagram form in order not to unnecessarily obscure the present disclosure. Thus, the specific details set forth are merely illustrative. It is contemplated that certain embodiments may vary from these illustrative details and still be within the scope of the present disclosure.

[0161] Additionally, some embodiments may be practiced in a distributed computing environment in which the machine-readable medium is stored on multiple computer systems and / or executed by multiple computers. Additionally, information transferred between computer systems may be either pulled or pushed over the communications medium connecting the computer systems.

[0162] Embodiments of the claimed subject matter include, but are not limited to, various operations described herein, which may be performed by hardware components, software, firmware, or any combination thereof.

[0163] Although the operations of the methods herein are illustrated and described in a particular order, the order of operations of each method may be changed such that some operations may be performed in reverse order or some operations may be performed at least in part concurrently with other operations. In alternative embodiments, the orders or sub-operations of different operations may be intermittent or alternating.

[0164] The above description of the illustrated implementations of the invention, including the contents described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Specific implementations of, and examples for, the invention are described herein for illustrative purposes, although various equivalent modifications are possible within the scope of the invention, as those skilled in the art will recognize. The words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" should not necessarily be construed as being preferred or superior to other aspects or designs. Rather, the use of the words "example" or "exemplary" is intended to illustrate a concept. The word "or" or "or" as used in this application is intended to mean inclusive "or" or "or" rather than exclusive "or" or "or". That is, unless otherwise stated or clear from the context, "X includes A or B" is intended to mean a natural inclusive permutation. That is, if X contains A, or X contains B, or X contains both A and B, then "X contains A or B" is satisfied under said cases. In addition, the articles "a" and "an" used in this application and the appended claims in English should generally be construed to mean "one or more" unless otherwise noted or unless the singular form is clear from the context. Furthermore, the use of the words "embodiment" or "one embodiment" or "implementation" or "one implementation" throughout is not intended to refer to the same embodiment or implementation unless described as such. Furthermore, the words "first", "second", "third", "fourth", etc., used herein are intended to be labels to distinguish different elements and may not necessarily have an ordinal meaning according to their numerical designations.

[0165] It will be understood that variations or other forms of the above-disclosed and other features and functions may be combined into other different systems or applications. Various presently unforeseen or unanticipated other forms, modifications, alterations, improvements thereon may be made by those skilled in the art that are also intended to be encompassed by the following claims. The claims may encompass hardware, software, or combinations thereof. In addition to the embodiments described above, the present disclosure includes, without limitation, the following exemplary implementations.

[0166] Some example implementations provide a method for monitoring a user's cardiac health, which may include receiving measured health index data and other factor data of a user at a first time, inputting, by a processing device, the health index data and other factor data into a machine learning model, where the machine learning model generates predicted health index data at a next time step, receiving the user's data at the next time step, determining, by the processing device, a loss at the next time step, where the loss is a measure between the predicted health index data at the next time step and the user's measured health index data at the next time step, determining that the loss exceeds a threshold, and outputting a notification to the user in response to determining that the loss exceeds the threshold.

[0167] In some exemplary implementations of the method of any exemplary implementation, the trained machine learning model is a trained generative neural network. In some exemplary implementations of the method of any exemplary implementation, the trained machine learning model is a feedforward network. In some exemplary implementations of the method of any exemplary implementation, the trained machine learning model is an RNN. In some exemplary implementations of the method of any exemplary implementation, the trained machine learning model is a CNN.

[0168] In some example implementations of the method of any example implementation, the trained machine learning model is trained on training examples from one or more of a healthy population, a cardiac disease population, and a user.

[0169] In some exemplary implementations of the method of any exemplary implementation, the loss at the next time step is the absolute value of the difference between the predicted health index data at the next time step and the user's measured health index at the next time step.

[0170] In some example implementations of the method of any example implementation, the predicted health index data is a probability distribution, and the predicted health index data at the next time step is sampled from this probability distribution.

[0171] In some exemplary implementations of the method of any exemplary implementation, the predicted health index data at the next time step is sampled according to a sampling technique selected from the group consisting of maximum probability predicted health index data and random sampling of health index data predicted from a probability distribution.

[0172] In some exemplary implementations of the method of any exemplary implementation, the predicted health index data is a probability distribution (β), and the loss is determined based on the negative logarithm of the probability distribution at the next time step evaluated on the user's measured health index at the next time step. In some exemplary implementations of the method of any exemplary implementation, the method further includes self-sampling the probability distribution.

[0173] In some exemplary implementations of the method of any exemplary implementation, the method further includes averaging predicted health index data over the period of the time steps, averaging measured health index data of the user over the period of the time steps, and determining a loss based on an absolute value of a difference between the predicted health index data and the measured health index data.

[0174] In some exemplary implementations of the method of any exemplary implementation, the measured health indicator data includes PPG data. In some exemplary implementations of the method of any exemplary implementation, the measured health indicator data includes heart rate data.

[0175] In some example implementations of the method of any example implementation, the method further includes resampling the irregularly spaced heart rate data onto a regularly spaced grid, where the heart rate data is sampled from the regularly spaced grid.

[0176] In some example implementations of the method of any example implementation, the measured health indicator data is one or more health indicator data selected from the group consisting of PPG data, heart rate data, pulse oximeter data, ECG data, and blood pressure data.

[0177] Some example limitations provide an apparatus that includes a mobile computing device having a processing device, a display, a health indicator data sensor, and a memory having instructions stored thereon that, when executed by the processing device, cause the processing device to receive measured health indicator data from the health indicator data sensor at a time and other factor data at a first time, input the health indicator data and the other factor data into a trained machine learning model, the trained machine learning model generating predicted health indicator data at a next time step, receiving the measured health indicator data and the other factor data at the next time step, determining a loss at the next time step, the loss being a measure between the predicted health indicator data at the next time step and the measured health indicator data at the next time step, and outputting a notification if the loss at the next time step exceeds a threshold.

[0178] In some exemplary implementations of any exemplary device, the trained machine learning model includes a trained generative neural network. In some exemplary implementations of any exemplary device, the trained machine learning model includes a feedforward network. In some exemplary implementations of any exemplary device, the trained machine learning model is an RNN. In some exemplary implementations of the method of any exemplary implementation, the trained machine learning model is a CNN.

[0179] In some example implementations of any of the example devices, the trained machine learning model is trained on training examples from one of the groups consisting of a healthy population, a cardiac disease population, and a user.

[0180] In some exemplary implementations of any exemplary device, the predicted health index data is a point prediction of the user's health index at the next time step, and the loss is the absolute value of the difference between the predicted health index data and the measured health index data at the next time step.

[0181] In some example implementations of any of the example devices, the predicted health index data is sampled from a probability distribution generated from a machine learning model.

[0182] In some example implementations of any of the example devices, the predicted health index data is sampled according to a sampling technique selected from the group consisting of maximum probability and random sampling from a probability distribution.

[0183] In some example implementations of any example device, the predicted health index data is a probability distribution (β) and the loss is determined based on the negative logarithm of β evaluated on the user's measured health index at the next time step.

[0184] In some exemplary implementations of any of the exemplary apparatus, the processing device further defines a function α ranging from 0 to 1, where It contains a linear combination of the user's measured and predicted health index data as a function of α.

[0185] In some example implementations of any of the example apparatus, the processing device further performs self-sampling of the probability distribution.

[0186] In some exemplary implementations of any exemplary apparatus, the processing device further averages predicted health index data sampled from the probability distribution over a period of time steps using an averaging method, averages measured health index data of the user over the period of time steps using the averaging method, and defines the loss as the absolute value of the averaged predicted health index data and the measured health index data.

[0187] In some example implementations of any of the example devices, the averaging method includes one or more methods selected from the group consisting of calculating the mean, calculating the arithmetic mean, calculating the median, and calculating the mode.

[0188] In some exemplary implementations of any exemplary device, the measured health indicator data includes PPG data from a PPG signal. In some exemplary implementations of any exemplary device, the measured health indicator data is heart rate data. In some exemplary implementations of any exemplary device, the heart rate data is collected by resampling irregularly spaced heart rate data onto a regularly spaced grid, and the heart rate data is sampled from the regularly spaced grid. In some exemplary implementations of any exemplary device, the measured health indicator data is one or more health indicator data selected from the group consisting of PPG data, heart rate data, pulse oximeter data, ECG data, and blood pressure data.

[0189] In some example implementations of any of the example apparatus, the mobile device is selected from the group consisting of a smart watch, a fitness band, a computer tablet, and a laptop computer.

[0190] In some example implementations of any example apparatus, the mobile device further comprises a user high-fidelity sensor, and the notification requests the user to obtain high-fidelity measurement data, and the processing device further receives an analysis of the high-fidelity measurement data, labels the user's measured health indicator data with the analysis to generate labeled user health indicator data, and trains a personalized trained high-fidelity machine learning model using the labeled user health indicator data as a training example.

[0191] In some exemplary implementations of any exemplary device, the trained machine learning model is stored on a memory. In some exemplary implementations of any exemplary device, the trained machine learning model is stored on a remote memory, the remote memory is separate from the computing device, and the mobile computing device is a wearable computing device. In some exemplary implementations of any exemplary device, the personalized trained high-fidelity machine learning model is stored on a memory. In some exemplary implementations of any exemplary device, the personalized trained high-fidelity machine learning model is stored on a remote memory, the remote memory is separate from the computing device, and the mobile computing device is a wearable computing device.

[0192] In some example implementations of any of the example apparatus, the processing device is further configured to predict that the user is experiencing atrial fibrillation and to determine the user's atrial fibrillation burden.

[0193] Some example implementations provide a method for monitoring a user's cardiac health, which may include receiving measured low-fidelity health indicator data and other factor data of a user at a first time, inputting data including the user health indicator data and other factor data at the first time into a personalized trained high-fidelity machine learning model, where the personalized trained high-fidelity machine learning model makes a prediction of whether the user's health indicator data is abnormal, and if the prediction is abnormal, sending a notification that the user's health is abnormal.

[0194] In some example implementations of the method of any example implementation, a personalized trained high-fidelity machine learning model is trained on measured low-fidelity user health indicator data that has been labeled with analysis of the high-fidelity measurement data.

[0195] In some example implementations of the method of any example implementation, the analysis of the high fidelity measurement data is based on user-specific high fidelity measurement data.

[0196] In some example implementations of the method of any example implementation, the personalized high-fidelity machine learning model outputs a probability distribution, and predictions are sampled from that probability distribution.

[0197] In some example implementations of the method of any example implementation, the predictions are sampled according to a sampling technique selected from the group consisting of: maximum probability prediction and random sampling of predictions from a probability distribution.

[0198] In some example implementations of the method of any example implementation, an averaged prediction is determined by averaging the predictions over a period of time steps using an averaging method, and the averaged prediction is used to determine whether the user's health indicator data is normal or abnormal.

[0199] In some example implementations of the methods of any example implementation, the averaging method includes one or more methods selected from the group consisting of calculating the mean, calculating the arithmetic mean, calculating the median, and calculating the mode.

[0200] In some exemplary implementations of the method of any exemplary implementation, the personalized trained high fidelity machine learning model is stored in a memory of the user's wearable device. In some exemplary implementations of the method of any exemplary implementation, the measured health index data and other factor data are time segments of data over a period of time.

[0201] In some example implementations of the method of any example implementation, the personalized trained high-fidelity machine learning model is stored in a remote memory, the remote memory being located at a location away from the user's wearable computing device.

[0202] In some example implementations, the health monitoring apparatus may include a mobile computing device having a microprocessor, a display, a user health indicator data sensor, and a memory having instructions stored therein which, when executed by the microprocessor, cause the processing device to receive low-fidelity health indicator data and other factor data measured at a first time, the measured health indicator data acquired by the user health indicator data sensor, input data including the health indicator data and other factor data at the first time into a trained high-fidelity machine learning model, the trained high-fidelity machine learning model making a prediction as to whether the user's health indicator data is normal or abnormal, and in response to the prediction being abnormal, sending at least a notification to the user indicating that the user's health is abnormal.

[0203] In some exemplary implementations of the health monitoring device of any exemplary implementation, the trained high-fidelity machine learning model is a trained high-fidelity generative neural network. In some exemplary implementations of the health monitoring device of any exemplary implementation, the trained high-fidelity machine learning model is a trained recurrent neural network (RNN). In some exemplary implementations of the health monitoring device of any exemplary implementation, the trained high-fidelity machine learning model is a trained feedforward neural network. In some exemplary implementations of the health monitoring device of any exemplary implementation, the trained high-fidelity machine learning model is a CNN.

[0204] In some exemplary implementations of the health monitoring device of any exemplary implementation, a trained high-fidelity machine learning model is trained on measured user health indicator data that is labeled based on user-specific high-fidelity measurement data.

[0205] In some exemplary implementations of the health monitoring device of any exemplary implementation, the trained high-fidelity machine learning model is trained on low-fidelity health indicator data labeled based on high-fidelity measurement data, and the low-fidelity health indicator data and the high-fidelity measurement data are data from a subject population.

[0206] In some example implementations of the health monitoring device of any example implementation, the high-fidelity machine learning model outputs a probability distribution and predictions are sampled from that probability distribution.

[0207] In some exemplary implementations of the health monitoring device of any exemplary implementation, the predictions are sampled according to a sampling technique selected from the group consisting of maximum probability prediction and random sampling of predictions from a probability distribution.

[0208] In some example implementations of the health monitoring device of any example implementation, an averaged prediction is determined by averaging the predictions over a period of time steps using an averaging method, and the averaged prediction is used to determine whether the user's health index data is normal or abnormal.

[0209] In some example implementations of the health monitoring device of any example implementation, the measured health index data and other factor data are time segments of data over a period of time.

[0210] In some example implementations of the health monitoring device of any example implementation, the averaging method includes one or more methods selected from the group consisting of calculating the mean, calculating the arithmetic mean, calculating the median, and calculating the mode.

[0211] In some exemplary implementations of the health monitoring device of any exemplary implementation, the personalized trained high-fidelity machine learning model is stored in a memory. In some exemplary implementations of the health monitoring device of any exemplary implementation, the personalized trained high-fidelity machine learning model is stored in a remote memory, the remote memory being located at a location remote from the wearable computing device. In some exemplary implementations of the health monitoring device of any exemplary implementation, the mobile device is selected from the group consisting of a smart watch, a fitness band, a computer tablet, and a laptop computer. [Explanation of symbols]

[0212] 4 layers 6 layers 7 layers 100 pre-trained Convolutional Neural Networks (CNNs) 102 Input Data 103 Convolutional layer (also called hidden layer) 104 Pre-trained weights or filters 105, 105' hidden convolutional layer 105 n-1 Convolutional Layer 106 Input Data 108CNN 110, 110' to 110 n Weight 111 Training Example 114 Output or Prediction 116 Difference 200 pre-trained recurrent neural networks (RNNs) 202 Updateable state (S) 204 Trained Weights (W) 206 Input Data 206 Predictions 208 Input Data (I t ) 210 State at time t (S t ) 212 Cell (C t ) 214 Prediction at time step t+1 216 Updated Status S t+1 218 I t+1 220 C t+1 224 I t+n 226S t+n 300 cells 304, 306, 306', 308, 308' Cell Gate 402 Output Data 404 Solid Line 500 pre-trained recurrent neural networks 502 Status 504 PPG 513 Input Data 513' Input data (P t+1 , R t+1 , and T t+1 ) 514 State at time t (S t ) 516 Trained Weights 518 cells (Ct) 520 Prediction at time t+1 522 Condition S t+1 524C t+1 530S n 532 Predictions 602 Cells 604 Initial state S0 606 Weighting matrix W 608 Updated state at time step 1 610 Cell 612 Updated state at time step 2 614 Cells 700 System 702 Machine Learning Models 704 Health Detector 706 Input Generator 708 Predictions 710 Input Data 712 Smart Watch 714 Watch 716 Display 718 User health index data 720 Predicted Health Index Bands 722 Watch Band 724 High Fidelity Sensor 726 Low Fidelity Sensors 920 Deviation 1000 ways 1110 First Chart 1120 Second Chart 1130 Third Chart 1200 Computer Systems 1202 Processing device 1204 Main memory 1206 Static Memory 1218 Data Storage Devices 1222 instruction set 1226 Processing Logic 1228 Machine-readable storage medium 1230 Bus 1250 Health Monitor

Claims

1. 1. A method for monitoring a user's cardiac health, performed by a processing device, comprising: receiving measured low fidelity health indicator data and other factor data of a user at a first time; inputting the low fidelity health indicator data and other factor data into a personalized trained machine learning model, the personalized trained machine learning model generating predicted normal low fidelity health indicator data for a next time step; receiving low fidelity health indicator data for the user at the next time step; determining a loss at the next time step, the loss being a measure between the normal predicted low fidelity health indicator data at the next time step and the user's measured low fidelity health indicator data at the next time step; determining that the loss exceeds a threshold; responsive to determining that the loss exceeds a threshold, outputting a notification to the user to obtain high fidelity measurement data; Including, acquiring high fidelity measurement data based on the notification, and analyzing the acquired high fidelity measurement data; labeling the measured low-fidelity health indicator data of the user at the next time step based on the analysis of the acquired high-fidelity measurement data; training a personalized trained machine learning model based on the labeled measured low-fidelity health indicator data of the user; The method further comprising:

2. 2. The method of claim 1, wherein the personalized trained machine learning model comprises a trained generative neural network, a feedforward network, a recurrent neural network, or a convolutional neural network.

3. 2. The method of claim 1, wherein the personalized trained machine learning model is trained with training examples from one or more of a healthy population, a cardiac disease population, and the user.

4. The method of claim 1, wherein the loss at the next time step is an absolute value of the difference between the predicted normal low fidelity health indicator data at the next time step and the user's measured low fidelity health indicator data at the next time step.

5. The method of claim 1 , wherein the predicted normal low fidelity health indicator data is a probability distribution, and the predicted normal low fidelity health indicator data at the next time step is sampled from the probability distribution.

6. averaging the predicted normal low fidelity health indicator data over a period of a preceding time step; averaging the measured low fidelity health indicator data of the user over the period of a preceding time step; The method of claim 1 , further comprising determining the loss based on an absolute value of a difference between the predicted normal low fidelity health index data and the measured low fidelity health index data.

7. The method of claim 1 , wherein the measured low fidelity health indicator data includes pulse plethysmography data, heart rate data, or heart rate variability.

8. A processing device; a low fidelity health indicator data sensor operably coupled to the processing device; a memory, which when executed by the processing device, causes the processing device to: receiving measured low fidelity health indicator data from the low fidelity health indicator data sensor at a time and other factor data at a first time; inputting the low fidelity health indicator data and other factor data into a personalized trained machine learning model, the personalized trained machine learning model generating low fidelity health indicator data predicted to be normal at a next time step; receiving low fidelity health indicator data and other factor data measured at the next time step; determining a loss at the next time step, the loss being a measure between the predicted normal low fidelity health index data at the next time step and the measured low fidelity health index data at the next time step; outputting a notification to acquire high fidelity measurement data if the loss at the next time step exceeds a threshold. a memory having instructions stored therein; The instructions cause the processing device to: acquiring high fidelity measurement data based on the notification, and analyzing the acquired high fidelity measurement data; labeling the measured low-fidelity health indicator data at the next time step based on an analysis result of the acquired high-fidelity measurement data; training a personalized trained machine learning model based on the labeled low-fidelity health indicator data; The apparatus further comprises:

9. 10. The apparatus of claim 8, wherein the personalized trained machine learning model comprises a trained generative neural network, a feedforward network, a recurrent neural network, or a convolutional neural network.

10. 10. The apparatus of claim 8, wherein the personalized trained machine learning model is trained with training examples from one of the groups consisting of a healthy population, a cardiac disease population, and a user.

11. 10. The apparatus of claim 8, wherein the predicted normal low-fidelity health indicator data is sampled from a probability distribution generated from the personalized trained machine learning model.

12. The processing device comprises: predicting that the user is experiencing atrial fibrillation; 10. The apparatus of claim 8, further configured to determine the atrial fibrillation burden of the user.

Images