Continuously monitor user health using mobile devices

Abstract

Disclosed herein are devices, systems, methods, and platforms for continuously monitoring a user's health status (e.g., heart health status). The present invention describes systems, methods, devices, software, and platforms for continuously monitoring a user's health indicator data (e.g., but not limited to, PPG signals, heart rate, or blood pressure, etc.) from a user device in combination with (temporally) corresponding data related to factors that may affect the health indicator ("other factors") to determine whether the user has normal health by, for example, but not limited to, determining or comparing with, for example, but not limited to, the following: i) a group of individuals affected by similar other factors; or ii) the user himself affected by similar other factors.

Description

Technical Field

[0001] The present application relates to continuous monitoring of user health using a mobile device. Background Art

[0002] Indicators of an individual's physiological health ("health indicators") (such as, but not limited to, heart rate, heart rate variability, blood pressure, and ECG (electrocardiogram)) can be measured or calculated at any discrete time point or points based on data collected to measure the health indicators. In many cases, the value of a health indicator at a particular time, or the change over time, provides information about the individual's health status. For example, a low or high heart rate or blood pressure, or an ECG that clearly demonstrates myocardial ischemia, may indicate the need for immediate intervention. However, it should be noted that the readings, a series of readings, or changes in readings over time of these indicators may provide information that cannot be recognized by the user or even a health professional.

[0003] Arrhythmias, for example, may occur continuously or may occur intermittently. Arrhythmias that occur continuously can be most clearly diagnosed by an individual's electrocardiogram. Because continuous arrhythmias are always present, ECG analysis can be applied at any time to diagnose arrhythmias. The ECG can also be used to diagnose intermittent arrhythmias. However, because intermittent arrhythmias may be asymptomatic and / or intermittent by definition, diagnosis presents the challenge of applying diagnostic techniques while an individual is experiencing an arrhythmia. Therefore, the actual diagnosis of intermittent arrhythmias is very difficult. This particular difficulty is coupled with the fact that asymptomatic arrhythmias account for nearly 40% of arrhythmias in the United States. Boriani G. and Pettorelli D., Atrial Fibrillation Burden and Atrial Fibrillation type: Clinical Significance and Impact on the Risk of Stroke and Decision Making for Long-term Anticoagulation, Vascul Pharmacol. ,83:26-35(August 2016),page 26.

[0004] There are sensors and mobile electronic technologies that allow frequent or continuous monitoring and recording of health indicators. However, the capabilities of these sensor platforms often exceed the ability of traditional medical science to interpret the data generated by the sensors. The physiological meaning of health indicator parameters such as heart rate is often well defined only in a specific medical context: for example, heart rate is traditionally evaluated as a single scalar value based on other data / information that may affect health indicators out of context. A resting heart rate in the range of 60 to 100 beats per minute (BPM) can be considered normal. Users generally may manually measure their resting heart rate once or twice a day.

[0005] A mobile sensor platform (e.g., a mobile blood pressure cuff; a mobile heart rate monitor; or a mobile ECG device) may be capable of continuously monitoring a health indicator (e.g., heart rate), for example being capable of generating a measurement every second or every 5 seconds, while also acquiring other data about the user, such as, but not limited to, activity level, body position, and environmental parameters such as air temperature, air pressure, location, etc. Over a 24 hour period, this may result in thousands of separate health indicator measurements. There is relatively little data or medical consensus on what a "normal" sequence of thousands of measurements looks like, as opposed to once or twice a day.

[0006] Current devices for continuously measuring health indicators of users / patients range from bulky, invasive and inconvenient devices to simple wearable or handheld mobile devices. Currently, these devices do not provide the ability to effectively utilize data to continuously monitor personal health. Users or health professionals are relied upon to evaluate these health indicators based on other factors that may affect the health indicators to determine the health status of the user. Summary of the invention

[0007] According to one aspect of the present application, a method for monitoring a user's heart health includes: receiving measured health indicator data and other factor data of the user at a first time; inputting the health indicator data and other factor data into a machine learning model through a processing device, wherein the machine learning model generates predicted health indicator data at a next time step; receiving user data at the next time step; determining a loss at the next time step through the processing device, wherein the loss is a measure between the predicted health indicator data at the next time step and the measured health indicator data of the user at the next time step; judging that the loss exceeds a threshold; and outputting a notification to the user in response to judging that the loss exceeds the threshold.

[0008] According to another aspect of the present application, a device includes: a processing device; a health indicator data sensor, which is operably coupled to the processing device; and a memory, on which instructions are stored that, when executed by the processing device, cause the processing device to perform the following operations: receiving measured health indicator data from the health indicator data sensor at a certain time and other factor data at a first time; inputting the health indicator data and other factor data into a trained machine learning model, and wherein the trained machine learning model generates predicted health indicator data at the next time step; receiving the measured health indicator data and other factor data at the next time step; determining the loss at the next time step, wherein the loss is a measure between the predicted health indicator data at the next time step and the measured health indicator data at the next time step; and outputting a notification if the loss at the next time step exceeds a threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Certain features described herein are particularly set forth in the appended claims. A better understanding of the features and advantages of the disclosed embodiments will be obtained by reference to the following detailed description and accompanying drawings which set forth exemplary embodiments utilizing the principles described herein, in which:

[0010] Figure 1 A-1B depict convolutional neural networks that may be used according to some embodiments as described herein;

[0011] Figure 2 A-2B depict recurrent neural networks that may be used according to some embodiments as described herein;

[0012] Figure 3 depicts an alternative recurrent neural network that may be used in accordance with some embodiments as described herein;

[0013] Figures 4A to 4C depicts a hypothetical data plot used to demonstrate the application of some embodiments as described herein;

[0014] Figures 5A to 5E depicts an alternative recurrent neural network according to some embodiments as described herein and a hypothetical plot used to describe some of these embodiments;

[0015] Figure 6 depicts an unfolded recurrent neural network according to some embodiments as described herein;

[0016] Figures 7A-7B depicts systems and apparatus according to some embodiments as described herein;

[0017] Figure 8 depicts methods according to some embodiments as described herein;

[0018] Figures 9A to 9B depicts a hypothetical plot of heart rate versus time according to methods as described herein and to illustrate one or more embodiments;

[0019] Fig.10 depicts methods according to some embodiments as described herein;

[0020] Fig.11 depicts a hypothetical data plot used to demonstrate the application of some embodiments as described herein; and

[0021] Fig.12 Systems and apparatus according to some embodiments as described herein are depicted. DETAILED DESCRIPTION

[0022] The volume of large data, the complexity of the interactions between health indicators and other factors, and limited clinical guidance may limit the effectiveness of any monitoring system that attempts to detect anomalies in continuous and / or flowing sensor data through specific rules based on traditional medical practice. Embodiments described herein include devices, systems, methods, and platforms that can detect anomalies in a time series in an unsupervised manner using predictive machine learning models based on health indicator data alone or in combination with other factors (as defined herein) data.

[0023] Atrial fibrillation (AF or AFib) occurs in 1-2% of the general population, and the presence of AF increases the risk of morbidity and adverse outcomes such as stroke and heart failure. Boriani G. and Pettorelli D., Atrial Fibrillation Burden and Atrial Fibrillation type: Clinical Significance and Impact on the Risk of Stroke and Decision Making for Long-term Anticoagulation, Vascul Pharmacol., 83:26-35 (August 2016), p. 26. In many people (estimated to be up to 40% of AF patients), AFib may be asymptomatic, and these asymptomatic patients have similar risk profiles for stroke and heart failure as symptomatic patients. Id. However, symptomatic patients can take active measures (such as taking blood thinners or other medications) to reduce the risk of negative outcomes. Asymptomatic AF (so-called silent AF or SAF) and the duration that a patient is in AF can be detected using an implantable electrical device (CIED). Id. Based on this information, the time these patients spend in AF or AF burden can be determined. Id. AF burden greater than 5-6 minutes, and particularly greater than 1 hour, is associated with a significantly increased risk of stroke and other negative health outcomes. Id. Therefore, the ability to measure AF burden in asymptomatic patients can enable earlier interventional treatment and can reduce the risk of negative health outcomes associated with AF. Id. Detection of SAF is challenging and generally requires some form of continuous monitoring. Currently, continuous monitoring of AF requires bulky, sometimes invasive and expensive equipment, with such monitoring requiring a high level of medical professional oversight and review.

[0024] Many devices continuously acquire data to provide measurements or calculations of health indicators, such as but not limited to Smartphones, tablet computers, and the like fall into the category of wearable devices and / or mobile devices. Other devices include permanent or semi-permanent devices on or in the user / patient (e.g., Holter monitors), while others may include larger devices that are mobile within a hospital on a cart. However, little processing is done on the measurement data beyond periodic observation of the measurement data on a display or establishment of simple data thresholds. Observation of the data (even by trained medical professionals) may often appear normal, with one major exception being situations where the user has easily identifiable acute symptoms. It is difficult and nearly impossible for medical professionals to continuously monitor health indicators and observe data anomalies and / or trends that may indicate a more serious condition.

[0025] As used herein, a platform includes one or more custom software applications (or "applications") that are configured to interact with each other locally or through a distributed network including the cloud and the Internet. The applications of the platform as described herein are configured to collect and analyze user data and may include one or more software models. In some embodiments of the platform, the platform includes one or more hardware components (e.g., one or more sensing devices or microprocessors). In some embodiments, the platform is configured to operate with one or more devices and / or one or more systems. That is, in some embodiments, the devices as described herein are configured to use a built-in processor to run the applications of the platform, and in some embodiments, the platform is utilized by a system including one or more computing devices that interact with or run one or more applications of the platform.

[0026] The present invention describes systems, methods, devices, software and platforms for continuously monitoring user data (e.g., but not limited to, PPG signals, heart rate, or blood pressure, etc.) from a user device in combination with (temporally) corresponding data related to factors that may affect the health indicators (referred to herein as "other factors") to determine whether the user has normal health by, for example, but not limited to, determining or comparing with, for example, but not limited to, the following: i) a group of individuals affected by similar other factors; or ii) the user himself affected by similar other factors. In some embodiments, the measured health indicator data is input into a trained machine learning model alone or in combination with other factor data, wherein the machine learning model determines the probability that the user's measured health indicator is considered to be within a healthy range, and notifies the user of this situation if the user's measured health indicator is considered to be not within a healthy range. Users who are not within a healthy range may increase the likelihood that the user may be experiencing a health event (such as arrhythmia that may be symptomatic or asymptomatic) that requires high-fidelity information to confirm the diagnosis. The notification may take the form of, for example, requesting the user to obtain an ECG. Other high-fidelity measurements (blood pressure, pulse oximeter, etc.) may be requested, and ECG is just one example. The high fidelity measurement (ECG in this example) can be evaluated by algorithms and / or medical professionals for notification or diagnosis (collectively referred to herein as "diagnosis," recognizing that only a physician can make a diagnosis). In the ECG example, the diagnosis could be AFib or any other number of well-known conditions that are diagnosed using an ECG.

[0027] In further embodiments, the diagnosis is used to label a low-fidelity data sequence (e.g., heart rate or PPG), which may include other factor data sequences. This low-fidelity data sequence labeled with a high-fidelity diagnosis is used to train a high-fidelity machine learning model. In these further embodiments, the training of the high-fidelity machine learning model can be trained by unsupervised learning, or can be updated from time to time with new training examples. In some embodiments, the user's measured low-fidelity health indicator data sequence and optionally the corresponding (temporal) data sequence of other factors are input into the trained high-fidelity machine learning model to determine the probability and / or prediction that the user is experiencing or has experienced a diagnostic condition, wherein the high-fidelity machine learning model is trained for the diagnostic condition. Such probabilities may include probabilities of when an event begins and when an event ends. For example, some embodiments may calculate a user's atrial fibrillation (AF) burden, or the amount of time a user experiences AF over time. Previously, AF burden could only be determined using cumbersome and expensive dynamic electrocardiograms or implantable continuous ECG monitoring devices. Therefore, some embodiments described herein can continuously monitor the health status of the user and notify the user of changes in health status by continuously monitoring health indicator data (such as but not limited to PPG data, blood pressure data, and heart rate data, etc.) obtained from a device worn by the user, either alone or in combination with corresponding data of other factors. "Other factors" as used herein include any factors that may affect health indicators and / or may affect data representing health indicators (e.g., PPG data). These other factors may include various factors such as but not limited to: air temperature, altitude, exercise level, weight, gender, diet, standing, sitting, falling, lying down, weather, and BMI, etc. In some embodiments, a mathematical or empirical model that is not a machine learning model can be used to determine when to notify the user to obtain a high-fidelity measurement, which can then be analyzed and used to train a high-fidelity machine training model as described herein.

[0028] Some embodiments described herein may detect anomalies in a user in an unsupervised manner by: receiving a primary time series of health indicator data; optionally receiving a secondary time series of one or more other factor data corresponding in time to the primary time series of health indicator data, which secondary series may be from sensors or from external data sources (e.g., via a network connection, a computer API, etc.); providing the primary time series and the secondary time series to a preprocessor, which may perform operations such as filtering, caching, averaging, time alignment, buffering, upsampling, and downsampling on the data; providing the time series of data to a machine learning model, which is trained and / or configured to use the values ​​of the primary time series and the secondary time series to predict the next value of the primary time series at a future time; comparing the predicted primary time series value generated by the machine learning model at a specific time t with the measured value of the primary time series at time t; and alerting or prompting the user to take action if the difference between the predicted future time series and the measured time series exceeds a threshold or standard.

[0029] Thus, some embodiments described herein detect when the behavior of a primary sequence of observed physiological data differs from the behavior that would be expected given the training examples used to train the model, relative to the passage of time and / or in response to a secondary sequence of observed data. In cases where the training examples are collected from normal individuals or from data previously classified as normal for a particular user, the system can be used as an anomaly detector. If the data is simply acquired from a particular user without any other classification, the system can be used as a change detector to detect changes in the health indicator data being measured by the primary sequence relative to the time the training data was captured.

[0030] Described herein are software platforms, systems, apparatus, and methods for generating a trained machine learning model and using the model to predict or determine the probability that measured health indicator data (primary sequence) of a user affected by other factors (secondary sequence) is outside the normal boundaries of a healthy population affected by similar other factors (i.e., a global model), or outside the normal boundaries of the specific user affected by similar other factors (i.e., a personalized model), wherein such notification is provided to the user. In some embodiments, the user may be prompted to obtain additional measured high-fidelity data that may be used to label previously acquired low-fidelity user health indicator data to generate a different trained high-fidelity machine learning model having the ability to predict or diagnose anomalies or events using only low-fidelity health indicator data, where such anomalies are typically identified or diagnosed using only high-fidelity data.

[0031] Some embodiments described herein may include inputting a user's health indicator data, and optionally inputting corresponding data (in time) of other factors into a trained machine learning model, wherein the trained machine learning model predicts the user's health indicator data or the probability distribution of health indicator data at a future time step. In some embodiments, the predicted and predicted measured health indicator data of the user at the time step are compared, wherein if the absolute value of the difference exceeds a threshold, the user is notified that his or her health indicator data is outside the normal range. In some embodiments, the notification may include an instruction to diagnose or do something, such as but not limited to obtaining additional measurements or contacting a health professional. In some embodiments, the machine learning model is trained using health indicator data from a healthy population and corresponding data (in time) of other factors. It should be understood that the other factors in the training examples used to train the machine learning model may not be the average of the population, but rather the data of each of the other factors corresponds in time to the set of health indicator data of the individuals in the training examples.

[0032] Some embodiments are described as receiving discrete data points in time, predicting discrete data points at future times based on inputs, and then determining whether the loss between the discrete measured inputs at future times and the predicted values ​​at future times exceeds a threshold. Those skilled in the art will readily appreciate that the input data and output predictions can take forms other than discrete data points or scalars. For example, but not limited to, health indicator data sequences (also referred to herein as primary sequences) and other data sequences (also referred to herein as secondary sequences) can be divided into time segments. Those skilled in the art will recognize that the manner in which the data is segmented is a matter of design choice and can take many different forms.

[0033] Some embodiments divide the health indicator data sequence (also referred to herein as the main sequence) and other data sequences (also referred to herein as the secondary sequence) into two segments: the past, representing all data before a specific time t; and the future, representing all data at or after time t. These embodiments input the health indicator data sequence of the past time segment and all other data sequences of the past time segment into a machine learning model, which is configured to predict the most likely future segment of the health indicator data (or the distribution of possible future segments). Optionally, these embodiments input the health indicator data sequence of the past time segment, all other data sequences of the past time segment, and other data sequences of the future segment into a machine learning model, which is configured to predict the most likely future segment of the health indicator data (or the distribution of possible future segments). The predicted future segment of the health indicator data is compared with the measured health indicator data of the user at the future segment to determine the loss and whether the loss exceeds a threshold, in which case some actions are taken. The action may include, for example, but not limited to: notifying the user to obtain additional data (e.g., ECG or blood pressure); notifying the user to contact a health professional; or automatically triggering the acquisition of additional data. Automatically acquiring additional data may include, for example, but not limited to, ECG acquisition via a sensor operably coupled (wired or wirelessly) to a computing device worn by a user, or blood pressure via a mobile cuff around the user's wrist or other appropriate body part and coupled to a computing device worn by the user. The data segments may include a single data point, a number of data points over a period of time, an average of these data points over the period of time, where the average may include a true mean, median, or mode. In some embodiments, the segments may overlap in time.

[0034] These embodiments detect when the behavior or measurement of a health indicator data sequence observed relative to the passage of time, as influenced (temporally) by the corresponding other factor data sequence, differs from the behavior or measurement expected based on training examples collected under similar other factors. If the training examples are collected from healthy individuals under similar other factors, or from data previously classified as healthy for a specific user under similar other factors, these embodiments act as anomaly detectors from a healthy population or a specific user, respectively. If the training examples are simply obtained from a specific user without any other classification, these embodiments act as change detectors to detect changes in the health indicators of the specific user when measured relative to the time when the training examples were collected.

[0035] Some embodiments described herein utilize machine learning to continuously monitor a person's health indicators under the influence of one or more other factors, and assess whether the person is healthy according to being classified as a healthy crowd under the influence of similar other factors. As will be readily understood by those skilled in the art, without exceeding the scope described herein, a variety of different machine learning algorithms or models (including but not limited to Bayes, Markov, Gausian processes, clustering algorithms, generative models, kernels and neural network algorithms) can be used. As understood by those skilled in the art, a typical neural network uses one or more layers of, for example, but not limited to, nonlinear activation functions to predict the output of receiving input, and in addition to the input layer and the output layer, one or more hidden layers may also be included. The output of each hidden layer of some of these networks is used as the input of the next layer in the network. Examples of neural networks include, for example, but not limited to, generative neutral networks, convolutional neural networks, and recursive neural networks.

[0036] Some embodiments of the health monitoring system monitor the heart rate and activity data of an individual as low-fidelity data (e.g., heart rate or PPG data), and detect conditions (e.g., AFib) that are usually detected using high-fidelity data (e.g., ECG data). For example, the heart rate of an individual may be provided by a sensor continuously or at discrete intervals (such as every five seconds). The heart rate may be determined based on a PPG, a pulse oximeter, or other sensor. In some embodiments, the activity data may be generated as the number of steps taken, the amount of movement sensed, or other data points indicating the activity level. Then, the low-fidelity (e.g., heart rate) data and the activity data may be input into a machine learning system to determine a prediction of a high-fidelity result. For example, a machine learning system may use low-fidelity data to predict arrhythmias or other indications of a user's heart health. In some embodiments, a machine learning system may use input of a fragment of data input to determine a prediction. For example, one hour of activity level data and heart rate data may be input into a machine learning system. Then, the system may use these data to generate predictions of conditions such as atrial fibrillation. Various embodiments of the present invention are discussed in more detail below.

[0037] refer to Figure 1A, a trained convolutional neural network (CNN) 100 (an example of a feed-forward network) takes input data 102 (e.g., a picture of a boat) into a convolutional layer (also called a hidden layer) 103, and a series of trained weights or filters 104 are applied to the input data 106 in each convolutional layer 103. The output of the first convolutional layer is an activation map (not shown), which is the input to the second convolutional layer to which the trained weights or filters (not shown) are applied, where the output of subsequent convolutional layers results in activation maps that represent increasingly complex features of the input data of the first layer. After each convolutional layer, a nonlinear layer (not shown) is applied to introduce nonlinearity, where the nonlinear layer may include tanh, sigmoid, or ReLU. In some cases, a pooling layer (not shown) (also called a downsampling layer) may be applied after the nonlinear layer, where the pooling layer essentially takes filters of the same length and stride, applies them to the input, and outputs the maximum number of filters per sub-region for which the convolution operation is performed. Other options for pooling are average pooling and L2 norm pooling. The pooling layer reduces the spatial dimension of the input volume, thereby reducing computational cost and controlling overfitting. The last layer of the network is a fully connected layer, which takes the output of the previous convolutional layer and outputs an n-dimensional output vector representing the quantity to be predicted (e.g., the probability of image classification: 20% car, 75% boat, 5% bus and 0% bicycle), that is, the predicted output 106 (O*) is obtained, for example, this may be a picture of a boat. The output can be a scalar value data point that the network is predicting, such as a stock price. As described more fully below, the trained weights 104 may be different for each convolutional layer 103. In order to achieve this real-world prediction / detection (e.g., it is a boat), it is necessary to train the neural network on known data inputs or training examples to obtain a trained CNN 100. In order to train the CNN 100, many different training examples (e.g., many pictures of boats) are input into the model. Those skilled in the art of neural networks will fully appreciate that the above description provides a somewhat simplified view of CNNs to provide some context for the current discussion, and will fully appreciate that the application of any CNN alone or in combination with other neural networks will be equally applicable and within the scope of some of the embodiments described herein.

[0038] Figure 1 B shows the training of CNN 108. Figure 1 In B, the convolutional layer 103 is shown as a single hidden convolutional layer 105, 105' until the convolutional layer 105 n-1 , and the last nth layer is a fully connected layer. It should be understood that the last layer can be more than one fully connected layer. The training example 111 is input to the convolutional layer 103, and the nonlinear activation function (not shown) and weights 110, 110' to 110 nThis is applied to training examples 111 successively, where the output of any hidden layer is the input to the next layer, and so on, until the final nth fully connected layer 105 n An output 114 is generated. The output or prediction 114 is compared to a training example 111 (e.g., a picture of a boat) and a difference 116 is obtained between the output or prediction 114 and the training example 111. If the difference or loss 116 is less than some preset loss (e.g., the output or prediction 114 predicts that the object is a boat), then the CNN has converged and is considered trained. If the CNN has not yet converged, the weights 110 and 110' to 110 are updated using a back-propagation technique based on how close the predictions are to the known inputs. n Those skilled in the art will appreciate that methods other than back propagation may be used to adjust the weights. A second training example (e.g., a different picture of a ship) is input and the process is repeated again with the updated weights, and the weights are updated again, and so on, until the nth training example (e.g., the nth picture of the nth ship) has been input. The process is repeated over and over again with the same n training examples until the convolutional neural network (CNN) is trained or converges on the correct output for the known input. Once the CNN 108 is trained, the weights 110, 110′ to 110′ are adjusted. n (i.e. Figure 1 The weights 104 depicted in A) are fixed and used for the trained CNN 100. As explained, there are different weights for each convolutional layer 103 and each fully connected layer. The trained CNN 100 or model is then fed image data to determine or predict what it is trained to predict / recognize (e.g., a ship), as described above. Any trained model, CNN, RNN, etc. can be further trained using additional training examples or prediction data output by the model and then used as training examples, i.e., weights can be allowed to be modified. The machine learning model can be trained "offline", for example, trained on a computing platform separate from the platform using / executing the trained model and then transmitted to the platform using / executing the trained model. Optionally, the embodiments described herein can periodically or continuously update the machine learning model based on newly acquired training data. This update training can be performed on a separate computing platform that transfers the updated trained model to the platform using / executing the retrained model through a network connection, or the training / retraining / update process can be performed on the platform itself that uses / executing the retrained model when new data is acquired. Those skilled in the art will understand that CNN is applicable to data in fixed arrays (e.g., pictures, characters, words, etc.) or time series of data. For example, CNN can be used to model serialized health indicator data and other factor data. Some embodiments utilize a feedforward CNN with skip connections and Gaussian mixture model outputs to determine the probability distribution of predicted health indicators (e.g., heart rate, PPG, or arrhythmia).

[0039] Some embodiments may utilize other types and configurations of neural networks. The number of convolutional layers and the number of fully connected layers may be increased or decreased. In general, the optimal number and ratio of convolutional layers relative to fully connected layers may be set experimentally by determining which configuration provides the best performance for a given data set. The number of convolutional layers may be reduced to zero, leaving a fully connected network. The number of convolutional filters and the width of each filter may also be increased or decreased.

[0040] The output of the neural network can be a single scalar value corresponding to an accurate forecast of the primary time series. Alternatively, the output of the neural network can be a logistic regression in which each category corresponds to a specific range or category of primary time series values, where these primary time series values ​​are any number of optional outputs that are readily understood by those skilled in the art.

[0041] In some embodiments, the use of Gaussian mixture model outputs is intended to constrain the network to learn well-formed probability distributions and improve generalization to limited training data. In some embodiments, the use of multiple elements in the Gaussian mixture model is intended to allow the model to learn multi-modal probability distributions. Machine learning models that combine or aggregate the results of different neural networks can also be used, where the results can be combined.

[0042] Machine learning models that have an updateable memory or state from previous predictions to apply to subsequent predictions are another approach for modeling sequential data. In particular, some embodiments described herein utilize recurrent neural networks. Figure 2 1, a diagram of a trained recurrent neural network (RNN) 200 is shown. The trained RNN 200 has updateable states (S) 202 and trained weights (W) 204. Input data 206 is input into the states 202 where the weights (W) 204 are applied, and a prediction 206 (P*) is output. In contrast to a linear neural network (e.g., CNN 100), the states 202 are updated based on the input data, thereby serving as a memory from the previous state to be used sequentially for the next prediction using the next data. Updating the states provides a cyclic or looping feature to the RNN. To better illustrate, Figure 2 B shows an unfolded trained RNN 200 and its applicability to sequential data. When unfolded, the RNN behaves similarly to a CNN, but in the unfolded RNN, each apparently similar layer behaves as a single layer with updated states, where the same weights are applied in each iteration of the loop. Those skilled in the art will appreciate that a single layer may itself have sublayers, but for clarity of explanation, a single layer is described here. t )208 input to the state at time t (S t)210, and the neuron (C t ) 212 and applying the trained weights 204. t The output of 212 is the prediction at time step t+1 and the updated state S t+1 216. Similarly, in C t+1 220, I t+1 218 input to S t+1 216, the same trained weights 204 are applied, and C t+1 The output of 220 is As mentioned above, by S t Update S t+1 , so S t+1 With S from the previous time step t For example, but not limited to, the memory may include previous health indicator data or previous other factor data from one or more previous time steps. The process continues for n steps, where I t+n 224 input to S t+n 226, and apply the same weights as 204. Neuron C t+n The output is the prediction In particular, the state is updated according to the previous time step, thereby providing the RNN with the benefit of memory from the previous state. For some embodiments, this property makes RNN a viable option for making predictions on serialized data. Nevertheless, and as mentioned above, there are other suitable machine learning techniques for making such predictions on serialized data, including CNNs.

[0043] Like CNN, RNN can process a data string as input and output a predicted data string. A simple way to explain this aspect of using RNN is to use an example of natural language prediction. Take the following phrase as an example: The sky is blue. The word string (i.e., data) has a context. Therefore, as the state is updated, the data string is updated from one iteration to the next, thereby providing a context for predicting blue. As just described, RNN has a memory component to assist in predicting serialized data. However, the memory in the updated state of the RNN may be limited in how far it can be traced back, similar to short-term memory. When it is desired to predict serialized data with a longer traceback (similar to long-term memory), this can be achieved using fine-tuning of the RNN just described. The sentence in which it is unclear what word to predict from the previous or surrounding words is again a simple example to explain the following: Mary speaks fluent French. It is not clear from the previous words that French is the correct prediction; it is only clear that a certain language is the correct prediction, but which language is the correct prediction? The correct prediction may lie in the context of words separated by larger intervals than a single string of words. Long Term Memory (LSTM) networks are a special type of RNN that are able to learn these (longer) long-term dependencies.

[0044] As mentioned above, RNNs have a relatively simple repetitive structure, for example, they include a single layer with a nonlinear activation function (e.g., tanh or sigmoid). Similarly, LSTMs have a chain-like structure, but (for example) have four neural network layers instead of one. These additional neural network layers provide LSTMs with the ability to remove or add information relative to the state (S) by using structures called neuron gates. Ibid. Figure 3 Neuron 300 for an LSTM RNN is shown. Line 302 represents the neuron state (S) and can be thought of as an information highway; it is relatively easy for information to flow along an unchanged neuron state. Ibid. Neuron gates 304, 306, and 308 determine how much information is allowed to pass through the state or along the information highway. Neuron gate 304 first decides what to pass from neuron state S. t Next, neuron gates 306 and 306' determine what information will be added to the neuron state, and neuron gates 308 and 308' determine what information will be output from the neuron state as a prediction. The information highway or neuron state is now the updated neuron state S t+1, for use in the next neuron. LSTM allows RNNs to have more persistent or (longer) long-term memory. Compared to simpler RNN structures, LSTM provides the following additional advantages to RNN-based machine learning models: Depending on how the data is serialized, the output prediction takes into account context that is separated from the input data for a longer period of time.

[0045] In some embodiments utilizing RNNs, the primary and secondary time series may not be provided to the RNN as vectors at each time step. Instead, only the current values ​​of the primary and secondary time series, and the future values ​​or aggregate functions of the secondary time series within the prediction interval are provided to the RNN. In this way, the RNN uses a persistent state vector to retain information about previous values ​​for use in making predictions.

[0046] Machine learning is very suitable for continuously monitoring one or more criteria to identify anomalies or trends of varying sizes in the input data compared to the training examples used to train the model. Therefore, some embodiments described herein input the user's health indicator data and optional other factor data into a trained machine learning model, which predicts what the health indicator data of a healthy person will look like at the next time step, and compares the prediction with the user's measured health indicator data at the future time step. If the absolute value of the difference (e.g., the loss described below) exceeds a threshold, the user is notified that his or her health indicator data is not within a normal or healthy range. The threshold is a number set by the designer, and in some embodiments can be changed by the user to allow the user to adjust the notification sensitivity. The machine learning models of these embodiments can be trained individually or in combination with (in time) corresponding other factor data by health indicator data from a healthy population, or the machine learning models of these embodiments can be trained by other training examples to meet the design requirements of the model.

[0047] Data from health indicators (e.g., heart rate data) is serialized data, more particularly time-serialized data. Heart rate, for example but not limited to, can be measured in a variety of different ways, for example, measuring an electrical signal from a chest strap or derived from a PPG signal. Some embodiments obtain a heart rate derived from a device, where each data point (e.g., heart rate) is generated at approximately equal intervals (e.g., 5 seconds). However, in some cases and in other embodiments, the derived heart rate is not provided in approximately equal time steps, for example because the data required for the derivation is unreliable (e.g., because the device moves or due to light pollution, the PPG signal is unreliable). The same is true for obtaining secondary data sequences from motion sensors or other sensors used to collect data on other factors.

[0048] The raw signal / data (electrical signal from ECG, chest strap or PPG signal) itself is a time series of data that can be used according to some embodiments. For the sake of clarity and not limitation, this specification uses PPG to refer to data representing health indicators. Those skilled in the art will readily understand that according to some embodiments described herein, health indicator data, raw data, waveforms, or digital forms derived from raw data or waveforms can be used.

[0049] Machine learning models that can be used with the embodiments described herein include, for example, but not limited to, Bayesian, Markov, Gausian processes, clustering algorithms, generative models, kernel and neural network algorithms. Some embodiments utilize machine learning models based on trained neural networks, other embodiments utilize recursive neural networks, and additional embodiments use LTSMRNN. For clarity and not limitation, some embodiments of this specification will be described using recursive neural networks.

[0050] Figures 4A to 4C PPG( Figure 4A ), the steps taken ( Figure 4B ) and temperature ( Figure 4C ) versus time. The PPG is an example of health indicator data, where steps, activity level, and air temperature are exemplary other factors data that may affect the health indicator data. Those skilled in the art will understand that other data can be obtained from any of many known sources including but not limited to accelerometer data, GPS data, weight scales, user input, etc., and may include but are not limited to air temperature, activity (running, walking, sitting, biking, falling, climbing stairs, stepping, etc.), BMI, weight, height, age, etc. The first dashed line extending vertically on all three plots represents the time t at which user data is obtained for input into the trained machine learning model (discussed below). Figure 4A The hash plotted lines in represent predicted or possible output data 402, and Figure 4A The solid line 404 in represents the measured data. Figure 4B is a hypothetical plot of the number of steps taken by a user over time, and Figure 4C is a hypothetical plot of temperature at various times.

[0051] Figures 5A-5B The trained recurrent neural network 500 receives Figures 4A to 4CSchematic diagram of the input data (i.e., PPG (P), steps (R), and air temperature (T)) depicted in . Once again, it is emphasized that these input data (P, R, and T) are merely examples of health indicator data and other factor data. It should also be understood that data for more than one health indicator can be input and predicted, and more or less than two other factor data can be used, where the selection depends on what the model is designed for. Those skilled in the art will also understand that the other factor data is collected to correspond in time with the collection or measurement of the health indicator data. In some cases (e.g., weight), the other factor data will remain relatively constant over a period of time.

[0052] Figure 5A The trained neural network 500 is depicted as a loop. P, T, and R are input into the state 502 of the RNN 500 to which the weight W is applied, and the RNN 500 outputs a predicted PPG 504 (P*). In step 506, the difference PP is calculated. * (ΔP * ), and at step 508, determine |ΔP * | is greater than a threshold. If so, step 510 notifies / warns the user that his / her health indicator is outside the boundary / threshold predicted as normal or predicted for a healthy person. The warning / notification / detection can be, for example, but not limited to, a suggestion to see a doctor / consult a doctor, a simple notification such as tactile feedback, a request to take additional measurements such as ECG, or a simple annotation without any suggestion, or any combination thereof. If |ΔP * | is less than or equal to the threshold, then step 512 does nothing. In both steps 510 and 512, the process is repeated at the next time step using new user data. In this embodiment, the state is updated after the predicted data is output, and the predicted data can be used when updating the state.

[0053] In another embodiment (not shown), the primary sequence of heart rate data (e.g., derived from the PPG signal) and the secondary sequence of other factor data are provided to a trained machine learning model, which can be an RNN, a CNN, other machine learning models, or a combination of these models. In this embodiment, the machine learning model is configured to receive the following at a reference time t as input:

[0054] A. A vector (V ) of length 300 of the last 300 samples of any health indicator data (e.g., heart rate in beats per minute) up to and including time t. H );

[0055] B. Contains V HAt least one vector (V ) of length 300 of the most recent other factor data (e.g., number of steps) at the approximate time of each sample in O );

[0056] C. A vector with a length of 300 (V TD ), where index i is the input V DT (i) Contains health indicator sample V H (i) Timestamp and V H The time difference between the timestamps of (i-1); and

[0057] D. Scalar prediction interval other factor rate O representing the average other factor rate (e.g., step rate) measured over the time period from t to t+τ rate (eg, but not limited to, step rate), where τ may be, for example, but not limited to, 2.5 minutes, and is the future prediction interval.

[0058] The output of this embodiment can be, for example, a probability distribution that characterizes the predicted heart rate measured over the time period from t to t+τ. In some embodiments, the machine learning model is trained using training examples that include a continuous time series of health indicator data and other factor data series. In an optional embodiment, the notification system assigns a timestamp t+τ / 2 to each predicted health indicator (e.g., heart rate) distribution, thereby concentrating the predicted distribution within the prediction interval (τ). In this embodiment, the notification logic then considers a time series of length W. L = 2*(τ) or in this case all samples within a sliding window (W) of 5 minutes and calculate three parameters:

[0059] 1. The average value of all serialized data of health indicators within the time window

[0060] 2. The average of all model predictions for health indicators whose timestamps fall within the time window Among them; and

[0061] 3. The median value of the root mean square of the distribution of each predicted health indicator within the time window in

[0062] 4. In one embodiment, if or (where ψ is a threshold), a notification is generated.

[0063] In this embodiment, a warning is generated when the measured health index within a particular window W is more than a certain number of standard deviations from the mean of the predicted health index value. The window W can be applied in a sliding manner in the sequence of measured health index values ​​and predicted health index values, where each window overlaps the previous window in time by a designer-specified fraction (e.g., 0.5 minutes).

[0064] The notification may take any number of different forms. For example, but not limited to, the user may be notified to obtain an ECG and / or blood pressure, the computing system (e.g., a wearable computing system, etc.) may be instructed to automatically obtain an ECG or blood pressure (for example), the user may be notified to see a doctor, or the user may simply be informed that health indicator data is not normal.

[0065] In this embodiment, V is used as the input of the model. DT The choice is intended to allow the model to take advantage of V H The information contained in the variable spacing between health indicator data in V, where the variable spacing may come from the algorithm that derives the health indicator data from less consistent raw data. For example, heart rate samples are generated by the Apple Watch algorithm only when there is sufficiently reliable raw PPG data to output reliable heart rate values, which results in irregular time gaps between heart rate samples. In a similar manner, this embodiment utilizes other factor data (V) with the same length as other vectors O ) to handle the different and irregular sampling rates between the primary sequence (health indicators) and the secondary sequence (other factors). In this embodiment, the secondary sequence is remapped or interpolated to the same time point as the primary time series.

[0066] In addition, in some embodiments, the configuration of the data in the sub-time series that exist as input to the machine learning model for future prediction time intervals (e.g., after t) can be modified. In some embodiments, a single scalar value containing the average other factor data rate within the prediction interval can be modified using multiple scalar values ​​(e.g., one scalar value per sub-time series). Alternatively, a vector of values ​​within the prediction interval can be used. In addition, the prediction interval itself can be adjusted. For example, a shorter prediction interval can provide a faster response to changes and improved detection of events with a (shorter) basic time metric, but may also be more sensitive to interference from noise sources (e.g., motion artifacts).

[0067] Similarly, the output predictions of the machine learning model itself need not be scalars. For example, some embodiments may generate a time series of predictions for multiple times t within the time interval between t and t+τ, and the alert logic may compare each of these predictions with a measured value within the same time interval.

[0068] In this previous embodiment, the machine learning model itself may include, for example, a 7-layer feedforward neural network. The first 3 layers may be convolutional layers containing 32 kernels, each with a kernel width of 24 and a stride of 2. The first layer may have as input an array V in three channels. H 、V O and V TD . The last 4 layers can be fully connected layers, and all fully connected layers except the last layer use the hyperbolic tangent activation function. The output of the third layer can be flattened into an array for input to the first fully connected layer. The last layer outputs 30 values, parameterizing the Gaussian mixture model into 10 mixtures (with three parameters for each mixture: mean, variance, and weight). The network uses skip connections between the first and third fully connected layers, so that the output of layer 6 is summed with the output of layer 4 to produce the input of layer 7. Standard batch normalization can be used on all layers except the last layer, with a decay of 0.97. Using skip connections and batch normalization can improve the ability to propagate gradients through the network.

[0069] The choice of machine learning model may affect the performance of the system. Machine learning model configuration can be divided into two types of considerations. The first is the internal architecture of the model, that is, the choice of model type (convolutional neural network, recursive neural network, generalized nonlinear regression of random forest, etc.), and the parameters that characterize the implementation of the model (generally the number of parameters, and / or the number of layers, the number of decision trees, etc.). The second is the external architecture of the model-the arrangement of the data being fed into the model and the specific parameters of the problem the model is asked to solve. The characteristics of the external architecture can be partly the dimensions and type of data provided as input to the model, the time range spanned by the data, and the pre-processing or post-processing of the data.

[0070] In general, the choice of external architecture is a balance between increasing the number of parameters, which can increase the predictive power of the machine learning model (with available storage and computing power to train and evaluate larger models), and increasing the amount of information provided as input, and the availability of sufficient amounts of data to prevent overfitting.

[0071] Many variations of the external architecture of the model discussed in some embodiments are possible. The number of input vectors as well as the absolute length (number of elements) and the time span covered can be modified. The input vectors need not be of the same length or cover the same time span. The data need not be sampled isochronously, for example but not limited to, a 6-hour history of heart rate data can be provided, wherein data less than one hour before t is sampled at a rate of 1 Hz, data more than one hour before t but less than two hours before t is sampled at a rate of 0.5 Hz, and data earlier than two hours is sampled at a rate of 0.1 Hz, where t is a reference time.

[0072] Figure 5B The unrolled trained RNN 500 is shown. t , R t and T t ) is input to the state at time t (S t ) 514, and apply the trained weights 516. Neuron (C t The output of 518 is the prediction at time t+1 and the updated state S t+1 522. Similarly, in C t+1 In 524, the input data (P t+1 , R t+1 and T t+1 )513' Input to S t+1 522, and applying the trained weights 516, and C t+1 The output of 524 is As mentioned above, by updating S t Get S t+1 , so S t+1 With S from the previous time step t The neurons (C t )518. The process continues for n steps, where the input data (P n , R n and T n )513" input to S n 530, and applying the trained weights 516. Neuron C t The output is the prediction In particular, the trained RNN always applies the same weights, but more importantly, updates the state based on the previous time step, thereby providing the RNN with the benefit of memory from the previous time step. Those skilled in the art will appreciate that the temporal order of inputting dependent health indicator data can be varied and still produce the desired results. For example, measured health indicator data from a previous time step (e.g., P t-1 ) and other factor data from the current time step (e.g., R t and T t ) can be input to the current time step (S t ), where the model predicts the health indicator at the current time step As mentioned above, the health indicator The measured health indicator data at the current time step is compared to determine whether the user's health indicator is normal or within a healthy range.

[0073] Figure 5CAn alternative embodiment of a trained RNN is shown to determine whether the user's health indicator serialized data (PPG in our example) is within the band or threshold of a healthy person. The input data in this embodiment is a linear combination in is the predicted health index value at time t, and P t is the measured health indicator at time t. In this embodiment, as a function of loss (L), the nonlinear range of α is 0 to 1, where loss and α are discussed in more detail below. Now it is worth noting that when α is close to 0, the measured data P t is input into the network, and when α is close to 1, the predicted data (P t * ) is input into the network to make a prediction at the next time step. Other factor data at time t can also be optionally input ( t ).

[0074] I t and O t It is state S t The input of which in some embodiments, the state S t Output the predicted health indicator data at time step t+1 The probability distribution of where β (P*) Is a predictive health indicator (P * ). In some embodiments, the probability distribution function is sampled to select the predicted health indicator value at t+1 As those skilled in the art will appreciate, different methods can be used to implement β according to the goals of the network designer. (P*) Sampling can be performed, where the method may include obtaining the mean, maximum, or random sampling of the probability distribution. Use the measurement data at time t+1 to evaluate β t+1 Provides status S t+1 The predicted probability for the measured data.

[0075] To illustrate this concept, Figure 5D The hypothetical probability distribution of the hypothetical health indicator data range at time t+1 is shown. For example, the function is sampled with a maximum probability of 0.95 to determine the predicted health indicator at time t+1 Also uses measured or actual health indicator data To evaluate the probability distribution (β t+1 ) and determines the probability that the model would predict if the actual data had been input into the model. In this example, It is 0.85.

[0076] A loss can be defined to help determine whether to notify a user that his or her health condition is not within the normal range predicted by the trained machine learning model. The loss is selected to model how close the predicted data is to the actual or measured data. Those skilled in the art will understand many ways to define a loss. In other embodiments described herein, for example, the absolute value of the difference between the predicted data and the actual data (|ΔP * |) is the loss. In some embodiments, the loss (L) may be L = -ln[β (P) ],in L is a measure of how close the predicted data are to the measured or actual data. (P) In the range of 0 to 1, where 1 means that the predicted value and the measured value are the same. Therefore, low loss means that the predicted value is likely to be the same as or close to the measured value; in this context, low loss means that the measured data looks like it comes from a healthy / normal person. In some embodiments, a threshold for L is set, such as L>5, where the user is notified that the health indicator data is outside the range considered healthy. Other embodiments may take the average of the losses over a period of time and compare the average to a threshold. In some embodiments, the threshold itself may be a statistical calculation of the predicted values ​​or a function of the average of the predicted values. In some embodiments, the following formula may be used to notify the user that the health indicator is not within a healthy range:

[0077]

[0078] <P range >Determined by averaging the measured health indicator data within a certain time range;

[0079] Determined by averaging the predicted health indicator data over the same time frame; is the median of the series of standard deviations obtained from the network over the same time frame; and

[0080] is is a function of the standard deviation of the evaluations at , and can be used as a threshold.

[0081] Averaging methods that can be used include, for example, but are not limited to, average, mean, median, and mode. In some embodiments, outliers are deleted so as not to skew the calculated numbers.

[0082] Return to reference Figure 5C The input data of the embodiment depicted in α tis defined as a function of L and is in the range of 0 to 1. For example, α(L) may be a linear function or a nonlinear function, or may be linear in a certain range of L and nonlinear in a separate range of L. In one example, Figure 5E As shown, the function α(L) is linear for L between 0 and 3, quadratic for L between 3 and 13, and 1 for L greater than 13. For this embodiment, when L is between 0 and 3 (i.e., when the predicted health indicator data and the measured health indicator data approximately match), as α-1 approaches zero, the input data I t+1 Approximately the measured data P t+1 When L is large (for example, greater than 13), α(L) is 1, which makes the input data is the time (the predicted health indicator at time t+1). When L is between 1 and 13, α(L) varies quadratically, and the relative contributions of the predicted health indicator data and the measured health indicator data to the input data also vary. In this embodiment, the linear combination of the predicted health indicator data and the measured health indicator data weighted by α(L) allows the input data at any particular time step to be weighted between the predicted data and the measured data. In all of these examples, the input data may also include other factor data (O t ). This is just one example of self-sampling, where some combination of predicted data and measured data is used as input to the trained network. Those skilled in the art will appreciate that other examples may be used.

[0083] The machine learning model in the embodiment uses a trained machine learning model. In some embodiments, the machine learning model uses a recursive neural network, which requires a trained RNN. As an example and not limitation, Figure 6 An unfolded RNN is depicted to demonstrate training the RNN in accordance with some embodiments. Neuron 602 has an initial state S0 604 and a weight matrix W 606. Step rate data R0, air temperature data T0, and initial PPG data P0 at time step 0 are input into state S0, weight W is applied, and the prediction at the first time step is output from neuron 602 And use the PPG(P1) obtained at time step 1 to calculate Neuron 602 also outputs an updated state 608 (S1) at time step 1, which enters neuron 610. The step rate data R1, air temperature data T1, and PPG data P1 at time step 1 are input into S1, weight 606W is applied, and the prediction at time step 2 is output from neuron 610. And use the PPG(P2) obtained at time step 2 to calculate Neuron 610 also outputs an updated state 612 (S2) at time step 2, which enters neuron 614. The step rate data R3, air temperature data T3, and PPG data (P3) at time step 3 are input into S2, weight 606W is applied, and the prediction at time step 3 is output from neuron 614. And use the PPG obtained at time step 3 (P3) to calculate This process continues until the state 616 at time step n is output and the Similar to the training of convolutional neural networks, ΔP is used in back propagation * ' to adjust the weight matrix. However, unlike convolutional networks, the same weight matrix in recursive neural networks is applied in each iteration; during training, the weight matrix is ​​only modified in backpropagation. Many training examples with health indicator data and corresponding other factor data are repeatedly input into the RNN 600 until it converges. As previously discussed, LTSM RNNs may be used in some embodiments, where the state of such networks provides a longer-term contextual analysis of the input data, thereby providing better predictions when the network is aware of (more) long-term correlations. As mentioned, and those skilled in the art will readily appreciate, other machine learning models will fall within the scope of the embodiments described herein, and may include, for example, but not limited to, CNNs or other feedforward networks.

[0084] Fig. 7AA system 700 is depicted for predicting whether a user's measured health indicator is within or outside a threshold that is normal for a healthy person under similar other factors. The system 700 has a machine learning model 702 and a health detector 704. For example (but not limited to), an embodiment of the machine learning model 702 includes a trained machine learning model, a trained RNN, a CNN, or other feedforward network. The trained RNN, other networks, or a combination of networks can be trained by training examples from a healthy population from which health indicator data and (temporally) corresponding other factor data are collected. Optionally, the trained RNN, other networks, or a combination of networks can be trained by training examples from a specific user to make it a personalized trained machine learning model. Those skilled in the art will understand that in general, training examples from different populations can be selected based on the use or design of the trained network and system. Those skilled in the art will also readily understand that the health indicator data in this embodiment and other embodiments can be one or more health indicators. For example, but not limited to, one or more of PPG data, heart rate data, blood pressure data, body temperature data, and blood oxygen concentration data can be used to train the model and predict the health of the user. The health detector 704 uses the prediction 708 from the machine learning model 702 and the input data 710 to determine whether the loss or other metrics determined by analyzing the predicted output with the measured data exceed a threshold that is considered normal and is therefore unhealthy. The system 700 then outputs a notification or the user's health status. The notification can take many forms as discussed herein. The input generator 706 uses a sensor (not shown) to continuously obtain data from a user wearing the sensor or in contact with the sensor, wherein the data represents one or more health indicators of the user. The corresponding other factor data (in time) can be collected by another sensor, or obtained by other means described herein or obvious to those skilled in the art.

[0085] The input generator 706 may also collect data to determine / calculate other factor data. For example, but not limited to, the input generator may include a smart watch, wearable or mobile device (e.g., Apple or The system 700 may be a combination of a smartwatch and a mobile device, a surgically implanted device with the ability to send data to a mobile device or other portable computing device, or a device on a cart in a medical care facility. Preferably, the user input generator 706 has a sensor (e.g., a PPG sensor, an electrode sensor, etc.) for measuring data related to one or more health indicators. The smartwatch, tablet, mobile phone, or laptop computer of some embodiments may carry the sensor, or the sensor may be placed remotely (surgically embedded, contacting the body away from the mobile device, or some separate device), wherein in all these cases, the mobile device communicates with the sensor to collect health indicator data. In some embodiments, the system 700 may be provided on a mobile device alone, in combination with other mobile devices, or in combination with other computing systems via communication over a network through which these devices can communicate. For example, but not limited to, the system 700 may be a smartwatch or wearable device with a machine learning model 702 and a health detector 704, wherein the machine learning model 702 and the health detector 704 are located on the device (e.g., the memory of the watch or the firmware on the watch). The watch may have a user input generator 706 and communicate with other computing devices (e.g., mobile phones, tablet computers, laptop computers, or desktop computers, etc.) via direct communication, wireless communication (e.g., WiFi, sound, Bluetooth, etc.), or through a network (e.g., the Internet, an intranet, an extranet, etc.), or a combination thereof, where the trained machine learning model 702 and the health detector 704 may be located on the other computing devices. Those skilled in the art will appreciate that any number of configurations of the system 700 may be utilized without exceeding the scope of the embodiments described herein.

[0086] refer to Figure 7B , depicting a smartwatch 712 according to an embodiment. Smartwatch 712 includes a watch 714, which contains all circuits and a microprocessor (not shown) known to those skilled in the art. Watch 714 also includes a display 716, where the user's health indicator data 718 (heart rate data in this example) can be displayed on the display 716. A predicted health indicator band 720 for a normal or healthy population can also be displayed on the display 716. Figure 7BIn the example, the user's measured heart rate data does not exceed the predicted health band, so in this particular example, no notification will be made. The watch 714 may also include a watch band 722 and a high-fidelity sensor 724 (e.g., an ECG sensor). Optionally, the watch band 722 may be an expandable cuff for measuring blood pressure. A low-fidelity sensor 726 (shown in shading) is provided on the back of the watch 714 to collect user health indicator data such as PPG data, which may be used to derive, for example, heart rate data or other data such as blood pressure. Alternatively, as will be appreciated by those skilled in the art, in some embodiments, a fitness bracelet (such as FitBit or Polar, etc.) may be used, wherein the fitness bracelet has similar processing capabilities and other factor measurement devices (e.g., ppg and accelerometers).

[0087] Figure 8An embodiment of a method 800 for continuously monitoring a user's health status is depicted. Step 802 receives user input data, which may include data of one or more health indicators (also referred to as a primary data sequence) and corresponding data (also referred to as a secondary data sequence) of other factors (also referred to as a secondary data sequence). Step 804 inputs the user data into a trained machine learning model, which may include a trained RNN, CNN, other feedforward networks as described herein, or other neural networks known to those skilled in the art. In some embodiments, the health indicator input data may be one or a combination of predicted health indicator data and measured health indicator data, such as a linear combination, as described in some embodiments herein. Step 806 outputs data of one or more predicted health indicators at a certain time step, wherein the output may include, for example, but not limited to, a single predicted value, a probability distribution as a function of the predicted value. Step 808 determines a loss based on the predicted health indicator, wherein, for example, but not limited to, the loss may be a simple difference between the predicted health indicator and the measured health indicator, or some other appropriately selected loss function (e.g., the negative logarithm of the probability distribution of the evaluation of the value of the measured health indicator). Step 810 determines whether the loss exceeds a threshold value that is considered normal or unhealthy, where the threshold value can be, for example, but not limited to, a simple number selected by the designer, or a more complex function of some parameters related to the prediction. If the loss is greater than the threshold, step 812 notifies the user that his or her health index exceeds the threshold value that is considered normal or healthy. As described herein, the notification can take many forms. In some embodiments, the information can be visible to the user. For example, but not limited to, the information can be displayed on a user interface, such as a graph showing the following (i) measured health index data (e.g., heart rate) and other factor data (e.g., number of steps) as a function of time, (ii) the distribution of predicted health index data (e.g., predicted heart rate value) generated by the machine learning model. In this way, the user can visually compare the measured data points with the predicted data points, and judge by visual observation, for example, whether his or her heart rate falls within the range expected by the machine learning model.

[0088] Some of the embodiments described herein have mentioned the use of thresholds to determine whether to notify the user. In one or more of these embodiments, the user can change the threshold to adjust or fine-tune the system or method to more closely match the user's personal health knowledge. For example, if the physiological indicator used is blood pressure and the user has high blood pressure, the embodiment may frequently warn / notify the user that his / her health indicator is outside the normal or healthy range based on a model trained with healthy people. Therefore, some embodiments allow the user to increase the threshold so that the user is not notified so frequently that his / her health indicator data exceeds the range considered normal or healthy.

[0089] Some embodiments preferably use raw data of health indicators. If the raw data is processed to derive a specific measurement (e.g., heart rate), the derived data can be used according to the embodiment. In some cases, the provider of the health monitoring device cannot control the raw data, instead, the received data is data processed in the form of a calculated health indicator (e.g., heart rate or blood pressure). As will be appreciated by those skilled in the art, the form of data used to train the machine learning model should match the form of data collected from the user and input into the trained model, otherwise the prediction may prove to be wrong. For example, Apple Watch provides heart rate measurement data at unequal time steps, but does not provide raw PPG data. In this example, the user wears an Apple Watch, which outputs heart rate data using heart rate data at unequal time steps according to Apple's PPG processing algorithm. The model is trained with this data. Apple decided to change its algorithm for providing heart rate data, which may make the model trained with data from the previous algorithm obsolete for using data input from the new algorithm. To address this potential problem, some embodiments resample irregularly spaced data (heart rate, blood pressure data, or ECG data, etc.) to a regularly spaced grid and sample according to the regularly spaced grid when collecting data to train the model. If Apple or other data providers change their algorithms, the model only needs to be retrained with the newly collected training examples, without having to reconstruct the model to account for the algorithmic changes.

[0090] In a further embodiment, the trained machine learning model can be trained by user data to obtain a personalized trained machine learning model. This trained personalized machine learning model can be used instead of the machine learning model trained by healthy people described here or in combination with it. If a personalized trained machine learning model itself is used, the user's data is input into the machine learning model, which will output a prediction of the individual health index in the next time step that is normal for the user, and then the prediction is compared with the actual / measured data from the next time step in a manner consistent with the embodiments described here to determine whether the user's health index differs from the health index predicted to be normal for the user by a certain threshold. In addition, this personalized machine learning model can be used in combination with a machine learning model trained by training examples from a healthy population to generate predictions and related notifications related to both the health index predicted to be normal for the individual user and the health index predicted to be normal for the healthy population.

[0091] Fig.9A A method 900 according to another embodiment is depicted, and Fig. 9B A hypothetical plot 902 of heart rate as a function of time is shown for purposes of explanation (for example but not limitation). Step 904 ( Fig.9A ) receives the user's heart rate data (or other health indicator data) and optional (temporally) corresponding other factor data, and inputs the data into a personalized trained machine learning model. In some embodiments, as described herein, the personalized trained model is trained using the user's individual health indicator data and optional (temporally) corresponding other data. Therefore, in step 906, the personalized trained machine learning model predicts the normal heart rate data of the individual user under the conditions of other factors, and step 908 compares the user's health indicator data with the health indicator data predicted to be normal for the particular user to identify abnormalities or anomalies in the user's health indicator data. As discussed in this specification, some embodiments receive information from a wearable device on the user (e.g., Apple Watch, smart watch, etc.), or from sensors on the user (e.g. Other mobile devices (e.g., tablet computers, computers, etc.) that communicate with the user's health indicator data.

[0092] A loss can be defined to help determine whether to notify the user in step 908 that the user's measured data is abnormal for the data predicted to be normal for this particular user. The loss is selected to model how close the prediction is to the actual or measured data. Those skilled in the art will understand many ways to define the loss. In other embodiments described herein and equally applicable, for example, the absolute value of the difference between the predicted value and the measured value |ΔP * | is the form of loss. In some embodiments, the loss (L) can be L = -ln[β (P) ],in L is usually a measure of how close the predicted data are to the measured data. (P)(in this example, a probability distribution) is in the range of 0 to 1, where 1 means that the predicted data and the measured data are the same. Therefore, in some embodiments, low loss indicates that the predicted data is likely to be the same or close to the measured data. In some embodiments, a threshold value of L is set, such as L>5, where the user is notified of the existence of an abnormal condition based on the prediction for a specific user. As described elsewhere in this document, this notification can take many forms. In addition, as described elsewhere in this document, other embodiments can obtain the average value of the loss within a certain time period and compare the average value with a threshold. In some embodiments, as described in more detail elsewhere in this document, the threshold itself can be a function of a statistical calculation of the predicted data or the average value of the predicted data. The loss has been described in detail elsewhere in this document, and for the sake of brevity, it will not be discussed further here. Those skilled in the art will understand that the input and predicted data can be scalar values, or data fragments within a certain time period. For example, but not limited to, the system designer may be interested in a 5-minute data fragment and input all data before time t and all other data for t+5 minutes, predict the health indicator data for t+5 minutes, and determine the loss between the measured health indicator data of the t+5 minute fragment and the predicted health indicator data of the t+5 minute fragment.

[0093] Step 908 determines whether there is an abnormality. As discussed, it can be determined whether the loss exceeds a threshold. As previously described, the threshold is set by the designer's choice and based on the purpose of the system being designed. In some embodiments, the threshold can be modified by the user, but preferably, it is not modified in this embodiment. If there is no abnormality, the process is repeated at step 904. If there is an abnormality, step 910 notifies or warns the user to obtain a high-fidelity measurement, such as but not limited to an ECG or blood pressure measurement. In step 912, the high-fidelity data is analyzed by an algorithm, a health professional, or both, and is described as normal or abnormal, and if abnormal, some diagnoses can be assigned based on the obtained high-fidelity measurements, such as AFib, tachycardia, bradycardia, atrial fibrillation, or high / low blood pressure. For clarity, it should be noted that the notification used to record high-fidelity data is equally applicable and possible in other embodiments and in the specific embodiments using the general model described above. In some embodiments, the high-fidelity measurement can be obtained directly by the user using a mobile monitoring system (such as an ECG or blood pressure system, etc.), which in some embodiments can be associated with a wearable device. Optionally, the notification step 910 enables automatic acquisition of high-fidelity measurements. For example, the wearable device can communicate with a sensor (either hardwired or via wireless communication) and obtain ECG data, or it can communicate with a blood pressure cuff system (e.g., a wristband or armband cuff of the wearable device) to automatically obtain blood pressure measurements, or it can communicate with an implanted device such as a pacemaker or ECG electrode. For example, AliveCor, Inc. provides a system for remotely obtaining an ECG, which includes (but is not limited to) one or more sensors in contact with a user in two or more locations, wherein the sensor collects ECG data that is sent to a mobile computing device by wire or wirelessly, wherein the app generates an ECG band based on the data, which can be analyzed by an algorithm, a medical professional, or both. Optionally, the sensor can be a blood pressure monitor, wherein the blood pressure data is sent to the mobile computing device by wire or wirelessly. The wearable device itself can be a blood pressure system with a cuff capable of measuring health indicator data and optionally an ECG sensor similar to the ECG sensor described above. The mobile computing device can be, for example, but not limited to, a tablet computer (e.g., iPad), a smartphone (e.g., ), a wearable device (e.g., an Apple Watch), or a device in a medical care facility (which may be mounted on a cart). In some embodiments, the mobile computing device may be a laptop or computer that communicates with some other mobile device. Those skilled in the art will appreciate that a wearable device or smartwatch would also be considered a mobile computing device in terms of the capabilities provided in the context of the embodiments described herein. In the case of a wearable device, the sensor may be placed on a cuff of the wearable device, where the sensor may send data to the computing device / wearable device wirelessly or by wire, or the cuff may also be a blood pressure monitoring cuff, or both as described above. In the case of a mobile phone, the sensor may be a pad attached to or remote from the phone, where the pad senses ECG signals and communicates the data to the wearable device or other mobile computing device wirelessly or by hardwire. A more detailed description of some of these systems is provided in one or more of U.S. Patent Nos. 9,420,956, 9,572,499, 9,351,654, 9,247,911, 9,254,095, and 8,509,882 and one or more of U.S. Patent Application Publication Nos. 2015 / 0018660, 2015 / 0297134, and 2015 / 0320328, all of which are incorporated herein for all purposes. Step 912 analyzes the high-fidelity data and provides a description or diagnosis as previously described.

[0094] In step 914, a diagnosis or classification of a high-fidelity measurement is received by a computing system, which in some embodiments may be a mobile or wearable computing system for collecting a user's heart rate data (or other health indicator data), and in step 916, a sequence of low-fidelity health indicator data (heart rate data in this example) is labeled with a diagnosis. In step 918, a high-fidelity machine learning model is trained using the labeled sequence of low-fidelity data of the user, and optionally other factor data sequences are also provided to train the model. In some embodiments, the trained high-fidelity machine learning model is capable of receiving a sequence of measured low-fidelity health indicator data (e.g., heart rate data or PPG data) and optional other factor data, and giving a probability that the user is experiencing an event that is typically diagnosed or detected using high-fidelity data, or predicting or diagnosing or detecting when the user is experiencing an event that is typically diagnosed or detected using high-fidelity data. The trained high-fidelity machine learning model is able to do this because it has been trained using the user's health indicator data (and optional other factor data) labeled with the diagnosis of the high-fidelity data. Thus, the trained model is able to predict when an event associated with one or more markers (e.g., Afib, hypertension, etc.) occurs to the user based solely on measuring low-fidelity health indicator input data sequences (e.g., heart rate or ppg data) (and optional other factor data). Those skilled in the art will appreciate that the training of the high-fidelity model can be performed on the user's mobile device, away from the user's mobile device, both, or in a distributed network. For example, but not limited to, the user's health indicator data can be stored in a cloud system, and the data can be marked in the cloud using the diagnosis from step 914. Those skilled in the art will readily understand any number of methods and means for storing, marking, and accessing this information. Alternatively, a globally trained high-fidelity model can be used, which will be trained by labeled training examples from people who are experiencing these conditions that are typically diagnosed or detected using high-fidelity measurements. These global training examples will provide low-fidelity data sequences (e.g., heart rate) labeled with conditions diagnosed using high-fidelity measurements (e.g., Afib called by a medical professional or algorithm based on an ECG).

[0095] Reference now Fig. 9B, plot 902 shows a schematic diagram of heart rate plotted as a function of time. Anomalies 920 relative to the user's normal heart rate data occur at times t1, t2, t3, t4, t5, t6, t7, 8. As described above, normal means that the predicted data for this particular user is within the threshold of the measured data, where the anomaly is outside the threshold. When abnormal relative to normal, some embodiments prompt the user to obtain a more explicit or high-fidelity reading, such as but not limited to an ECG reading identified as ECG1, ECG2, ECG3, ECG4, ECG5, ECG6, ECG7, ECG8. As described above, high-fidelity readings can be obtained automatically, the user can obtain high-fidelity readings, or the high-fidelity readings can be something other than ECG, such as blood pressure. The high-fidelity readings are analyzed by algorithms, health professionals, or both to identify the high-fidelity data as normal / abnormal, and further identify / diagnose abnormalities (such as but not limited to AFib). This information is used to mark health indicator data (eg, heart rate or PPG data) at abnormal points 920 in the user's serialized data.

[0096] The difference between high-fidelity and low-fidelity data is that high-fidelity data or measurements are typically used to make judgments, detections, or diagnoses, while low-fidelity data may not be easily used for such judgments, detections, or diagnoses. For example, an ECG scan can be used to identify, detect, or diagnose arrhythmias, while heart rate or PPG data generally do not provide such capabilities. As will be appreciated by those skilled in the art, the descriptions herein of machine learning algorithms (e.g., Bayesian, Markov, Gausian processes, clustering algorithms, generative models, kernel and neural network algorithms) are equally applicable to all embodiments described herein.

[0097] In some cases, despite these issues, the user may still be asymptomatic, and even if symptoms are present, it may be impractical to obtain the high fidelity measurements necessary to make a diagnosis or detection. For example, but not limited to, an arrhythmia, particularly AF, may not be present, and even if symptoms do exist, it may be very difficult to record an ECG at that moment, and it may be very difficult to continuously monitor the user without expensive, bulky, and sometimes invasive monitoring devices. As discussed elsewhere in this document, it is important to know when a user is experiencing AF because AF may be at least a cause of a stroke, in addition to other serious conditions. Similarly and as discussed elsewhere, AF burden may have similar inputs. Some embodiments allow for continuous monitoring of arrhythmias (e.g., AF) or other serious conditions using only continuous monitoring of low-fidelity health indicator data (such as heart rate or ppg) and optionally other factor data.

[0098] Fig.10Method 1000 according to some embodiments of health monitoring systems and methods is depicted. Step 1002 receives measured or actual low-fidelity health indicator data of a user (e.g., heart rate or PPG data from a sensor on a wearable device), and optionally receives (time-corresponding) other factor data that may affect the health indicator data described herein. As discussed elsewhere herein, the low-fidelity health indicator data can be measured by a mobile computing device such as a smart watch, other wearable device, or tablet computer. In step 1004, the low-fidelity health indicator data of the user (and optional other factor data) is input into a trained high-fidelity machine learning model, which outputs a predicted recognition or diagnosis for the user based on the measured low-fidelity health indicator data (and optional (time-corresponding) other factor data) in step 1006. Step 1008 asks whether the recognition or diagnosis is normal, and if so, restarts the process. If the recognition or diagnosis is abnormal, step 1010 notifies the user of the problem or detection. Optionally, the system, method, or platform can be set to notify any combination of the user, family, friends, medical care professionals, or emergency 911, etc. Which of these people is notified may depend on the identification, detection, or diagnosis. In the case of identification, detection, or diagnosis of life-threatening conditions, certain people may be contacted or notified, and these people may not be notified if the diagnosis is not life-threatening. In addition, in some embodiments, the measured health indicator data sequence is input into a trained high-fidelity machine learning model, and the amount of time the user is experiencing an abnormal event (e.g., the difference between the start and stop of the predicted abnormal event) is calculated, thereby allowing a better understanding of the user's abnormal load. In particular, understanding AF load can be very important in preventing strokes and other serious conditions. Therefore, some embodiments allow continuous monitoring of abnormal events using mobile computing devices, wearable computing devices, or other portable devices that can only obtain low-fidelity health factor data and optional other factor data.

[0099] Fig.11Depicted is example data 1100 for analyzing based on low-fidelity data to generate a high-fidelity output prediction or detection according to some embodiments described herein. Although described with reference to the detection of atrial fibrillation, similar data can be generated for additional predictions of high-fidelity diagnosis based on low-fidelity measurements. The first graph 1110 shows the calculation of the user's heart rate over time. The heart rate can be determined based on PPG data or other heart rate sensors. The second graph 1120 shows the user's activity data during the same time period. For example, the activity data can be determined based on the number of steps or other measurements of the user's movement. The third graph 1130 shows the classifier output from the machine learning model and the level threshold when the notification is generated. The machine learning model can generate predictions based on the input of low-fidelity measurements. For example, the data in the first graph 1110 and the second graph 1120 can be analyzed by a machine learning system as further described above. The results of the machine learning system analysis can be provided as the probability of atrial fibrillation shown in the graph 1130. When the probability exceeds a threshold (shown in this case as above 0.6 confidence), the health monitoring system may trigger a notification or other alert to the user, a physician, or other users associated with the user.

[0100] In some embodiments, the data in graphs 1110 and 1120 can be provided to a machine learning system as continuous measurements. For example, heart rate and activity level can be generated as measurements every 5 seconds for accurate measurements. Time segments with multiple measurements can then be input into a machine learning model. For example, data from the previous hour can be used as input to a machine learning model. In some embodiments, shorter or longer time periods can be provided instead of an hour. Fig.11 As shown, output graph 1130 provides an indication of time periods during which the user is experiencing abnormal health events. For example, the health monitoring system can use time periods predicted above a certain confidence level to determine atrial fibrillation. This value can then be used to determine the user's atrial fibrillation load during the measurement time period.

[0101] In some embodiments, the machine learning model used to generate the predicted output in the graph 1130 can be trained based on the marked user data. For example, the marked user data can be provided based on high-fidelity data (such as ECG readings, etc.) acquired during a time period in which low-fidelity data (e.g., PPG, heart rate, etc.) and other data (e.g., activity level or steps, etc.) are also available. In some embodiments, the machine learning model is designed to determine whether atrial fibrillation may exist during the previous time period. For example, the machine learning model can obtain one hour of low-fidelity data as input and provide the possibility of an event. Therefore, the training data may include multiple hours of recorded data for a population of individuals. In the case of diagnosing a condition based on high-fidelity data, the data can be a health event marking time. Therefore, if there is a health event marking time based on high-fidelity data, the machine learning model can determine that the low-fidelity data of any one hour window with the event input into the untrained machine learning model should provide a prediction of the health event. Then, the untrained machine learning model can be updated based on comparing the prediction with the marking. After repeating multiple iterations and determining that the machine learning model has converged, the health monitoring system can use the machine learning model to monitor the user's atrial fibrillation based on the low-fidelity data. In various embodiments, the low-fidelity data can be used to detect other conditions besides atrial fibrillation.

[0102] Fig.12 A schematic representation of a machine in an example form of a computer system 1200 is shown, wherein within the computer system 1200, an instruction set for causing the machine to perform any one or more methods discussed herein can be executed. In an optional embodiment, the machine can be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The machine can operate in the capacity of a server or client machine in a client-server network environment, or as a peer machine in a peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web device, a server, a network router, a switch or bridge, a hub, an access point, a network access control device, or any machine capable of executing an instruction set (sequential or otherwise) for specifying the action to be taken by the machine. In addition, although only a single machine is shown, the term "machine" should also be deemed to include any collection of machines that execute one (or more) instruction sets individually or collectively to perform any one or more methods discussed herein. In one embodiment, computer system 1200 may represent a server, a mobile computing device, a wearable device, or the like configured to perform health monitoring as described herein.

[0103] The exemplary computer system 1200 includes a processing device 1202, a main memory 1204 (e.g., a read-only memory (ROM), a flash memory, a dynamic random access memory (DRAM)), a static memory 1206 (e.g., a flash memory, a static random access memory (SRAM), etc.), and a data storage device 1218 that communicate with each other via a bus 1230. Any signal provided via the various buses described herein may be time multiplexed with other signals and provided via one or more common buses. In addition, the interconnections between circuit components or blocks may be shown as buses or single signal lines. Each bus may alternatively be one or more single signal lines, and each single signal line may alternatively be a bus.

[0104] The processing device 1202 represents one or more general-purpose processing devices, such as a microprocessor or a central processing unit. More specifically, the processing device can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processing device 1202 can also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. The processing device 1202 is configured to execute processing logic 1226, which can be an example of a health monitor 1250 and related systems for performing the operations and steps discussed herein.

[0105] The data storage device 1218 may include a machine-readable storage medium 1228 on which is stored one or more sets of instructions 1222 (e.g., software) embodying any one or more methods of functionality described herein, including instructions for causing the processing device 1202 to perform the health monitor 1250 and related processes described herein. The instructions 1222 may also reside, in whole or in part, within the main memory 1204 or within the processing device 1202 during execution of the instructions 1222 by the computer system 1200; the main memory 1204 and the processing device 1202 also constitute machine-readable storage media. The instructions 1222 may also be sent or received over the network 1220 via the network interface device 1208.

[0106] As described herein, the machine-readable storage medium 1228 may also be used to store instructions for performing a method for monitoring the health of a user. Although in an exemplary embodiment, the machine-readable storage medium 1228 is shown as a single medium, the term "machine-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache and server) storing one or more instruction sets. Machine-readable media include any mechanism for storing information in a machine (e.g., computer) readable form (e.g., software, processing applications). Machine-readable media may include, but are not limited to, magnetic storage media (e.g., floppy disks), optical storage media (e.g., CD-ROMs), magneto-optical storage media, read-only memory (ROM), random access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or other types of media suitable for storing electronic instructions.

[0107] The above description sets forth many specific details, such as examples of specific systems, components and methods, etc., to provide a good understanding of multiple embodiments of the present invention. However, it will be apparent to those skilled in the art that at least some embodiments of the present invention can be implemented without these specific details. In other cases, well-known components or methods are not described in detail or presented in a simple block diagram format to avoid unnecessarily obscuring the present invention. Therefore, the specific details set forth are merely exemplary. Specific embodiments may be different from these exemplary details, and still may be expected to be within the scope of the present invention.

[0108] In addition, some embodiments may be implemented in a distributed computing environment, where the machine-readable medium is stored on or executed by more than one computer system. In addition, information transmitted between computer systems may be pulled or pushed in the communication medium connecting the computer systems.

[0109] Embodiments of the claimed subject matter include, but are not limited to, the various operations described herein. These operations may be performed by hardware components, software, firmware, or a combination thereof.

[0110] Although the operation of the method herein is shown and described in a particular order, the order of the operation of each method can be changed so that some operations can be performed in reverse order, or some operations can be performed at least partially simultaneously with other operations. In another embodiment, the instructions or sub-operations of different operations can be adopted in an intermittent or alternating manner.

[0111] The above description of the exemplary implementation of the present invention (including the content described in the abstract) is not intended to be exhaustive or to limit the present invention to the exact form disclosed. As will be appreciated by those skilled in the art, although the specific implementation and examples of the present invention are described here for illustrative purposes, various equivalent modifications may also be within the scope of the present invention. The term "example" or "exemplary" used here means to be used as an example, instance or illustration. Any aspect or design described here as "example" or "exemplary" is not necessarily understood to be preferred or advantageous over other aspects or designs. On the contrary, the term "example" or "exemplary" is used to present concepts in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clearly visible from the context, "X includes A or B" is intended to mean any natural inclusive arrangement. That is, if X includes A, X includes B, or X includes both A and B, then in any of the above cases, "X includes A or B" is satisfied. In addition, the articles "a" and "an" used in the present application and the appended claims should generally be understood to mean "one or more", unless otherwise specified or it is clear from the context that a single form is intended. In addition, the use of the terms "embodiment" or "one embodiment" or "implementation" or "an implementation" throughout the specification is not intended to mean the same embodiment or implementation unless described as such. In addition, the terms "first", "second", "third", "fourth", etc. used herein refer to marks used to distinguish different elements and may not necessarily have ordinal meanings according to their numerical designations.

[0112] It should be understood that variations or alternatives to the above disclosed and other features and functions may be combined into other different systems or applications. Various substitutions, modifications, variations or improvements not currently foreseen or expected may be subsequently made by those skilled in the art, which are also intended to be covered by the following claims. The claims may cover embodiments in hardware, software or a combination thereof.

[0113] In addition to the above-mentioned embodiments, the present invention also includes but is not limited to the following exemplary implementations.

[0114] Some example implementations provide a method for monitoring a user's cardiac health. The method may include: receiving measured health indicator data and other factor data of a user at a first time; inputting the health indicator data and other factor data into a machine learning model through a processing device, wherein the machine learning model generates predicted health indicator data at a next time step; receiving the user's data at a next time step; determining the loss at the next time step through a processing device, wherein the loss is a measure between the predicted health indicator data at the next time step and the measured health indicator data of the user at the next time step; determining that the loss exceeds a threshold; and outputting a notification to the user in response to determining that the loss exceeds the threshold.

[0115] In some example implementations of the method of any example implementation, the trained machine learning model is a trained generative neural network. In some example implementations of the method of any example implementation, the trained machine learning model is a feedforward network. In some example implementations of the method of any example implementation, the trained machine learning model is an RNN. In some example implementations of the method of any example implementation, the trained machine learning model is a CNN.

[0116] In some example implementations of the method of any example implementation, the trained machine learning model is trained with training examples from one or more of: healthy people, people with heart disease, and users.

[0117] In some example implementations of the method of any example implementation, the loss at the next time step is an absolute value of a difference between the predicted health indicator data at the next time step and the measured health indicator of the user at the next time step.

[0118] In some example implementations of the method of any example implementation, the predicted health indicator data is a probability distribution, and wherein the predicted health indicator data at the next time step is sampled according to the probability distribution.

[0119] In some example implementations of the method of any example implementation, the predicted health indicator data at the next time step is sampled according to a sampling technique selected from the group consisting of: predicted health indicator data of maximum probability; and randomly sampling the predicted health indicator data according to a probability distribution.

[0120] In some example implementations of the method of any example implementation, the predicted health indicator data is a probability distribution (β), and wherein the loss is determined based on the negative logarithm of the probability distribution at the next time step evaluated using the measured health indicator of the user at the next time step. In some example implementations of the method of any example implementation, the method also includes self-sampling of the probability distribution.

[0121] In some example implementations of the method of any example implementation, the method also includes: averaging the predicted health indicator data over the time period of the time step; averaging the measured health indicator data of the user over the time period of the time step; and determining the loss based on the absolute value of the difference between the predicted health indicator data and the measured health indicator data.

[0122] In some example implementations of the method of any example implementation, the measured health indicator data includes PPG data. In some example implementations of the method of any example implementation, the measured health indicator data includes heart rate data.

[0123] In some example implementations of the method of any example implementation, the method further comprises resampling the irregularly spaced heart rate data onto a regularly spaced grid, wherein the heart rate data is sampled according to the regularly spaced grid.

[0124] In some example implementations of the method of any example implementation, the measured health indicator data is one or more health indicator data selected from the group consisting of: PPG data, heart rate data, pulse oximeter data, ECG data, and blood pressure data.

[0125] Some example limitations provide an apparatus comprising a mobile computing device, the mobile computing device comprising: a processing device; a display; a health indicator data sensor; and a memory storing instructions that, when executed by the processing device, cause the processing device to perform the following operations: receiving measured health indicator data from the health indicator data sensor at a first time and other factor data at the first time; inputting the health indicator data and other factor data into a trained machine learning model, and wherein the trained machine learning model generates predicted health indicator data at a next time step; receiving the measured health indicator data and other factor data at a next time step; determining a loss at a next time step, wherein the loss is a measure between the predicted health indicator data at the next time step and the measured health indicator data at the next time step; and outputting a notification if the loss at the next time step exceeds a threshold.

[0126] In some example implementations of any example device, the trained machine learning model includes a trained generative neural network. In some example implementations of any example device, the trained machine learning model includes a feed-forward network. In some example implementations of any example device, the trained machine learning model is an RNN. In some example implementations of the method of any example implementation, the trained machine learning model is a CNN.

[0127] In some example implementations of any example device, the trained machine learning model is trained with training examples from one of the group consisting of: healthy people, people with heart disease, and users.

[0128] In some example implementations of any example device, the predicted health indicator data is a point prediction of the user's health indicator at the next time step, and wherein the loss is an absolute value of a difference between the predicted health indicator data at the next time step and the measured health indicator data at the next time step.

[0129] In some example implementations of any of the example devices, the predicted health indicator data is sampled according to a probability distribution generated by a machine learning model.

[0130] In some example implementations of any of the example devices, the predicted health indicator data is sampled according to a sampling technique selected from the group consisting of: maximum probability; and random sampling according to a probability distribution.

[0131] In some example implementations of any of the example devices, the predicted health indicator data is a probability distribution (β), and wherein the loss is determined based on a negative logarithm of β evaluated using the measured health indicator of the user at the next time step.

[0132] In some example implementations of any of the example devices, the processing means is further configured to define a function α ranging from 0 to 1, where I t Includes a linear combination of the user's measured health indicator data and predicted health indicator numbers as a function of α.

[0133] In some example implementations of any of the example apparatus, the processing device is further configured to perform self-sampling of the probability distribution.

[0134] In some example implementations of any example device, the processing device is also used to: use an averaging method to average the predicted health indicator data sampled according to the probability distribution within the time period of the time step; use an averaging method to average the measured health indicator data of the user within the time period of the time step; define the loss as the absolute value of the difference between the averaged predicted health indicator data and the measured health indicator data.

[0135] In some example implementations of any of the example devices, the averaging method includes one or more methods selected from the group consisting of: calculating an average value, calculating an arithmetic mean, calculating a median, and calculating a mode.

[0136] In some example implementations of any example device, the measured health indicator data includes PPG data from a PPG signal. In some example implementations of any example device, the measured health indicator data is heart rate data. In some example implementations of any example device, the heart rate data is collected by resampling irregularly spaced heart rate data onto a regularly spaced grid and sampling the heart rate data according to the regularly spaced grid. In some example implementations of any example device, the measured health indicator data is one or more health indicator data selected from the group consisting of: PPG data, heart rate data, pulse oximeter data, ECG data, and blood pressure data.

[0137] In some example implementations of any of the example apparatus, the mobile device is selected from the group consisting of: a smart watch; a fitness band; a tablet computer; and a laptop computer.

[0138] In some example implementations of any example apparatus, the mobile device also includes a user high-fidelity sensor, wherein the notification requests the user to obtain high-fidelity measurement data, and wherein the processing device is further used to: receive an analysis of the high-fidelity measurement data; use the analysis to label the user's measured health indicator data to generate labeled user health indicator data; and use the labeled user health indicator data as a training example to train the trained personalized high-fidelity machine learning model.

[0139] In some example implementations of any example apparatus, the trained machine learning model is stored in memory. In some example implementations of any example apparatus, the trained machine learning model is stored in remote memory, wherein the remote memory is separate from the computing device, and wherein the mobile computing device is a wearable computing device. In some example implementations of any example apparatus, the trained personalized high-fidelity machine learning model is stored in memory. In some example implementations of any example apparatus, the trained personalized high-fidelity machine learning model is stored in remote memory, wherein the remote memory is separate from the computing device, and wherein the mobile computing device is a wearable computing device.

[0140] In some example implementations of any of the example devices, the processing device is further configured to predict that the user is experiencing atrial fibrillation and determine the user's atrial fibrillation burden.

[0141] Some example implementations provide a method for monitoring a user's cardiac health. The method may include: receiving low-fidelity user health indicator data and other factor data measured at a first time; inputting data including the user health indicator data and other factor data at the first time into a personalized trained high-fidelity machine learning model, wherein the personalized trained high-fidelity machine learning model predicts whether the user's health indicator data is abnormal; and sending a notification of the user's health abnormality if an abnormality is predicted.

[0142] In some example implementations of the method of any example implementation, the trained personalized high-fidelity machine learning model is trained by measuring low-fidelity user health indicator data labeled using analysis of high-fidelity measurement data.

[0143] In some example implementations of the method of any example implementation, the analysis of the high-fidelity measurement data is based on user-specific high-fidelity measurement data.

[0144] In some example implementations of the method of any example implementation, the personalized high-fidelity machine learning model outputs a probability distribution, wherein the predictions are sampled according to the probability distribution.

[0145] In some example implementations of the method of any example implementation, the predictions are sampled according to a sampling technique selected from the group consisting of: maximum probability predictions; and sampling predictions according to a probability distribution.

[0146] In some example implementations of the method of any example implementation, an average prediction is determined by averaging the predictions within a time period of time steps using an averaging method, and wherein the average prediction is used to determine whether the user's health indicator data is normal or abnormal.

[0147] In some example implementations of the method of any example implementation, the averaging method includes one or more methods selected from the group consisting of: calculating an average value, calculating an arithmetic mean, calculating a median, and calculating a mode.

[0148] In some example implementations of the method of any example implementation, the personalized high-fidelity trained machine learning model is stored in a memory of the user's wearable device. In some example implementations of the method of any example implementation, the measured health indicator data and other factor data are time segments of data within a certain time period.

[0149] In some example implementations of the method of any example implementation, the personalized high-fidelity trained machine learning model is stored in a remote memory, wherein the remote memory is located at a location remote from the user's wearable computing device.

[0150] In some example implementations, a health monitoring device may include a mobile computing device comprising: a microprocessor; a display; a user health indicator data sensor; and a memory storing instructions that, when executed by the microprocessor, cause the processing device to perform the following operations: receiving measured low-fidelity health indicator data and other factor data at a first time, wherein the measured health indicator data is obtained by the user health indicator data sensor; inputting data including the health indicator data and other factor data at the first time into a trained high-fidelity machine learning model, wherein the trained high-fidelity machine learning model predicts whether the user's health indicator data is normal or abnormal; and in response to the prediction being abnormal, sending a notification of the user's health abnormality to at least the user.

[0151] In some example implementations of the health monitoring device of any example implementation, the trained high-fidelity machine learning model is a trained high-fidelity generative neural network. In some example implementations of the health monitoring device of any example implementation, wherein the trained high-fidelity machine learning model is a trained recurrent neural network (RNN). In some example implementations of the health monitoring device of any example implementation, the trained high-fidelity machine learning model is a trained feedforward neural network. In some example implementations of the health monitoring device of any example implementation, the trained high-fidelity machine learning model is a CNN.

[0152] In some example implementations of the health monitoring device of any example implementation, the trained high-fidelity machine learning model is trained by measured user health indicator data labeled based on user-specific high-fidelity measurement data.

[0153] In some example implementations of the health monitoring device of any example implementation, a trained high-fidelity machine learning model is trained using low-fidelity health indicator data labeled based on high-fidelity measurement data, wherein the low-fidelity health indicator data and the high-fidelity measurement data are from a population of subjects.

[0154] In some example implementations of the health monitoring device of any example implementation, the high-fidelity machine learning model outputs a probability distribution, wherein the predictions are sampled according to the probability distribution.

[0155] In some example implementations of the health monitoring device of any example implementation, the predictions are sampled according to a sampling technique selected from the group consisting of: a maximum probability prediction; and randomly sampling the predictions according to a probability distribution.

[0156] In some example implementations of the health monitoring device of any example implementation, an average prediction is determined by averaging the predictions within a time period of time steps using an averaging method, and wherein the average prediction is used to determine whether the user's health indicator data is normal or abnormal.

[0157] In some example implementations of the health monitoring device of any example implementation, the measured health indicator data and other factor data are time segments of data within a time period.

[0158] In some example implementations of the health monitoring device of any example implementation, the averaging method includes one or more methods selected from the group consisting of: calculating an average value, calculating an arithmetic mean, calculating a median, and calculating a mode.

[0159] In some example implementations of the health monitoring device of any example implementation, the personalized high-fidelity trained machine learning model is stored in a memory. In some example implementations of the health monitoring device of any example implementation, the personalized high-fidelity trained machine learning model is stored in a remote memory, wherein the remote memory is located at a location remote from the wearable computing device. In some example implementations of the health monitoring device of any example implementation, the mobile device is selected from the group consisting of: a smart watch; a fitness band; a tablet computer; and a laptop computer.

Claims

1. A method for monitoring a user's heart health, the method comprising: Receiving low-fidelity measured health indicator data and other factor data of a user at a first time; Inputting the health indicator data and other factor data into a machine learning model through a processing device, wherein the machine learning model generates predicted health indicator data at a next time step based on the health indicator data and other factor data at the first time; receiving user data at the next time step; determining, by the processing device, a loss at the next time step, wherein the loss is a measure between the predicted health indicator data at the next time step and the user's low-fidelity measured health indicator data at the next time step; determining that the loss exceeds a threshold; and In response to determining that the loss exceeds a threshold, outputting a notification to a user, Among them, the machine learning model is a high-fidelity machine learning model trained using a low-fidelity data sequence after high-fidelity diagnostic labeling.

2. The method according to claim 1, wherein: Trained machine learning models include trained generative neural networks, feed-forward networks, recurrent neural networks, or convolutional neural networks.

3. The method according to claim 1, wherein: The trained machine learning model is trained on one or more of the following: healthy people and people with heart disease.

4. The method according to claim 1, wherein: The loss at the next time step is an absolute value of a difference between the predicted health indicator data at the next time step and the measured health indicator of the user at the next time step.

5. The method according to claim 1, wherein: The predicted health indicator data is a probability distribution, and wherein the predicted health indicator data at the next time step is sampled according to the probability distribution.

6. The method according to claim 1, wherein: The predicted health indicator data is a probability distribution, ie, β, and wherein the loss is determined based on the negative logarithm of the probability distribution at the next time step evaluated using the measured health indicator of the user at the next time step. The method of claim 6 , further comprising self-sampling of the probability distribution.

8. The method according to claim 1, further comprising: averaging the predicted health indicator data over the time period of the previous time steps; averaging the measured health indicator data of the user over the time period of the previous time step; as well as The loss is determined based on an absolute value difference between the predicted health indicator data and the measured health indicator data.

9. The method according to claim 1, wherein: The measured health indicator data includes pulse volume plethysmography data, heart rate data or heart rate variability.

10. A device for monitoring a user's heart health, comprising: Processing device; a health indicator data sensor operably coupled to the processing device; as well as A memory storing instructions that, when executed by the processing device, cause the processing device to perform the following operations: receiving low-fidelity measured health indicator data and other factor data from the health indicator data sensor at a first time; Inputting the health indicator data and other factor data into a trained machine learning model, and wherein the trained machine learning model generates predicted health indicator data at a next time step based on the health indicator data and other factor data at the first time; receiving measured health indicator data and other factor data at the next time step; determining a loss at the next time step, wherein the loss is a measure between the predicted health indicator data at the next time step and the low-fidelity measured health indicator data at the next time step; and outputting a notification if the loss at said next time step exceeds a threshold, Among them, the machine learning model is a high-fidelity machine learning model trained using a low-fidelity data sequence after high-fidelity diagnostic labeling.

11. The device according to claim 10, wherein: The trained machine learning model includes a trained generative neural network, a feedforward network, a recurrent neural network, or a convolutional neural network.

12. The device according to claim 10, wherein: The trained machine learning model is trained on one or more of the following: healthy people and people with heart disease.

13. The apparatus according to claim 10, wherein: The predicted health indicator data is sampled according to the probability distribution generated by the machine learning model.

14. The apparatus according to claim 10, wherein: The predicted health indicator data is a probability distribution, namely β, and wherein the loss is determined based on the negative logarithm of β evaluated using the measured health indicator of the user at the next time step.

15. The apparatus according to claim 10, wherein: The processing device is also used for: predicting that the user is experiencing atrial fibrillation; and Determine the user's atrial fibrillation burden.

Images